* [PATCH v4 01/30] x86/PoD: map one page at a time in p2m_pod_zero_check()
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-22 14:54 ` Roger Pau Monné
2026-10-19 19:42 ` [PATCH v4 02/30] x86/mm: refuse to promote an L4 while the domain has no vCPU George Dunlap
` (28 subsequent siblings)
29 siblings, 1 reply; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
p2m_pod_zero_check() maps every page of its batch that passes its
checks, up to POD_SWEEP_STRIDE (16) of them, and keeps them mapped
across the p2m lookups and updates which follow, each of which maps
page-table pages of its own. Its callers hold mappings too: the p2m
lookup which found the entry to populate still has its page-table page
mapped, and a PV mmu_update() the L1 page it is writing.
map_domain_page() cannot always provide that many at once. It uses the
mapcache on behalf of PV vCPUs only: in debug builds for every page, in
release builds for pages beyond the reach of the directmap in PV guests'
page-tables (5TiB, 3.5TiB with CONFIG_BIGMEM). The mapcache has
MAPCACHE_VCPU_ENTRIES (16) slots per vCPU of the domain, shared by its
vCPUs, and a mapping which finds no slot free or reclaimable hits the
BUG_ON() in map_domain_page(): the host crashes.
The sweep runs in the context of whichever vCPU populates a PoD entry
while the domain's PoD cache is empty. The guest's own vCPUs, being
HVM, map through the directmap; a PV vCPU gets there when its domain
maps or copies the guest's memory: a device model in dom0 or in a stub
domain, a backend doing grant copies. A PV domain with a single vCPU,
as a device-model stub domain or a dom0 booted with dom0_max_vcpus=1
is, has 16 slots, and a sweep over 16 candidate pages needs more.
The sweep has mapped its whole batch at once since 9ad9a1609fbe ("PoD
memory 5/9: emergency scan"), when x86-64 reached all memory through the
directmap. The per-vCPU budget came with the mapcache's return to
x86-64 in 4b28bf6ae90b ("x86: re-introduce map_domain_page() et al").
Map one page at a time instead, as p2m_pod_zero_check_superpage()
already does: map a page for the quick check and unmap it again, and
map it anew for the full check after the p2m update and the TLB flush.
The checks and their order are unchanged. Each page is mapped twice,
which costs little next to scanning it, and the error paths no longer
have mappings to undo.
Fixes: 4b28bf6ae90b ("x86: re-introduce map_domain_page() et al")
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- Track the pages to check in a bitmask walked with for_each_set_bit(),
not an array of bools (Andrew).
- Drop the new comment above the first loop; edit the old one's "and map
the pages" instead (Andrew).
NB this issue was raised in review and verified only by code
inspection; it hasn't been demonstrated.
---
xen/arch/x86/mm/p2m-pod.c | 72 +++++++++++++++++----------------------
1 file changed, 31 insertions(+), 41 deletions(-)
diff --git a/xen/arch/x86/mm/p2m-pod.c b/xen/arch/x86/mm/p2m-pod.c
index 4602c32cff..ed298fa7d2 100644
--- a/xen/arch/x86/mm/p2m-pod.c
+++ b/xen/arch/x86/mm/p2m-pod.c
@@ -901,9 +901,10 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
{
mfn_t mfns[POD_SWEEP_STRIDE];
p2m_type_t types[POD_SWEEP_STRIDE];
- unsigned long *map[POD_SWEEP_STRIDE];
+ unsigned int check = 0;
+ const unsigned long *map;
struct domain *d = p2m->domain;
- unsigned int i, j, max_ref = 1;
+ unsigned int j, max_ref = 1;
BUG_ON(count > POD_SWEEP_STRIDE);
@@ -911,8 +912,11 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
if ( paging_mode_shadow(d) )
max_ref++;
- /* First, get the gfn list, translate to mfns, and map the pages. */
- for ( i = 0; i < count; i++ )
+ /*
+ * First, get the gfn list, translate to mfns, and check for being a
+ * plausible candidate.
+ */
+ for ( unsigned int i = 0; i < count; i++ )
{
p2m_access_t a;
@@ -921,9 +925,8 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
/*
* If this is ram, and not a pagetable or a special page, and
- * probably not mapped elsewhere, map it; otherwise, skip.
+ * probably not mapped elsewhere, check it; otherwise, skip.
*/
- map[i] = NULL;
if ( p2m_is_ram(types[i]) )
{
const struct page_info *pg = mfn_to_page(mfns[i]);
@@ -932,7 +935,7 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
(pg->count_info & PGC_allocated) &&
!(pg->count_info & PGC_shadowed_pt) &&
((pg->count_info & PGC_count_mask) <= max_ref) )
- map[i] = map_domain_page(mfns[i]);
+ check |= 1U << i;
}
}
@@ -940,20 +943,23 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
* Then, go through and check for zeroed pages, removing write permission
* for those with zeroes.
*/
- for ( i = 0; i < count; i++ )
+ for_each_set_bit ( i, check )
{
- if ( !map[i] )
- continue;
-
/* Quick zero-check */
+ map = map_domain_page(mfns[i]);
for ( j = 0; j < 16; j++ )
- if ( *(map[i] + j) != 0 )
- goto skip;
+ if ( map[j] != 0 )
+ break;
+ unmap_domain_page(map);
/* Try to remove the page, restoring old mapping if it fails. */
- if ( p2m_set_entry(p2m, gfns[i], INVALID_MFN, PAGE_ORDER_4K,
+ if ( j < 16 ||
+ p2m_set_entry(p2m, gfns[i], INVALID_MFN, PAGE_ORDER_4K,
p2m_populate_on_demand, p2m->default_access) )
- goto skip;
+ {
+ check &= ~(1U << i);
+ continue;
+ }
/*
* See if the page was successfully unmapped. (Allow one refcount
@@ -961,6 +967,8 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
*/
if ( (mfn_to_page(mfns[i])->count_info & PGC_count_mask) > 1 )
{
+ check &= ~(1U << i);
+
/*
* If the previous p2m_set_entry call succeeded, this one shouldn't
* be able to fail. If it does, crashing the domain should be safe.
@@ -970,38 +978,27 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
{
ASSERT_UNREACHABLE();
domain_crash(d);
- goto out_unmap;
+ return;
}
-
- skip:
- unmap_domain_page(map[i]);
- map[i] = NULL;
-
- continue;
}
}
p2m_tlb_flush_sync(p2m);
/* Now check each page for real */
- for ( i = 0; i < count; i++ )
+ for_each_set_bit ( i, check )
{
- if ( !map[i] )
- continue;
-
- for ( j = 0; j < (PAGE_SIZE / sizeof(*map[i])); j++ )
- if ( *(map[i] + j) != 0 )
+ map = map_domain_page(mfns[i]);
+ for ( j = 0; j < (PAGE_SIZE / sizeof(*map)); j++ )
+ if ( map[j] != 0 )
break;
-
- unmap_domain_page(map[i]);
-
- map[i] = NULL;
+ unmap_domain_page(map);
/*
* See comment in p2m_pod_zero_check_superpage() re gnttab
* check timing.
*/
- if ( j < (PAGE_SIZE / sizeof(*map[i])) )
+ if ( j < (PAGE_SIZE / sizeof(*map)) )
{
/*
* If the previous p2m_set_entry call succeeded, this one shouldn't
@@ -1012,14 +1009,7 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
{
ASSERT_UNREACHABLE();
domain_crash(d);
- out_unmap:
- /*
- * Something went wrong, probably crashing the domain. Unmap
- * everything and return.
- */
- for ( i = 0; i < count; i++ )
- if ( map[i] )
- unmap_domain_page(map[i]);
+ return;
}
}
else
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* Re: [PATCH v4 01/30] x86/PoD: map one page at a time in p2m_pod_zero_check()
2026-10-19 19:42 ` [PATCH v4 01/30] x86/PoD: map one page at a time in p2m_pod_zero_check() George Dunlap
@ 2026-10-22 14:54 ` Roger Pau Monné
0 siblings, 0 replies; 39+ messages in thread
From: Roger Pau Monné @ 2026-10-22 14:54 UTC (permalink / raw)
To: George Dunlap; +Cc: xen-devel, Jan Beulich, Andrew Cooper, Teddy Astie
On Mon, Oct 19, 2026 at 08:42:32PM +0100, George Dunlap wrote:
> p2m_pod_zero_check() maps every page of its batch that passes its
> checks, up to POD_SWEEP_STRIDE (16) of them, and keeps them mapped
> across the p2m lookups and updates which follow, each of which maps
> page-table pages of its own. Its callers hold mappings too: the p2m
> lookup which found the entry to populate still has its page-table page
> mapped, and a PV mmu_update() the L1 page it is writing.
>
> map_domain_page() cannot always provide that many at once. It uses the
> mapcache on behalf of PV vCPUs only: in debug builds for every page, in
> release builds for pages beyond the reach of the directmap in PV guests'
> page-tables (5TiB, 3.5TiB with CONFIG_BIGMEM).
This is IMO too verbose. It doesn't add any value to the
justification here what's the memory limit at which the mapcache starts
to be used. I also don't think it's relevant to list what causes
mapcache to be used in general. It just distracts from the
justification of the change, by making the patch description unwieldy
long.
> The mapcache has
> MAPCACHE_VCPU_ENTRIES (16) slots per vCPU of the domain, shared by its
> vCPUs, and a mapping which finds no slot free or reclaimable hits the
> BUG_ON() in map_domain_page(): the host crashes.
>
> The sweep runs in the context of whichever vCPU populates a PoD entry
> while the domain's PoD cache is empty. The guest's own vCPUs, being
> HVM, map through the directmap; a PV vCPU gets there when its domain
> maps or copies the guest's memory: a device model in dom0 or in a stub
> domain, a backend doing grant copies. A PV domain with a single vCPU,
> as a device-model stub domain or a dom0 booted with dom0_max_vcpus=1
> is, has 16 slots, and a sweep over 16 candidate pages needs more.
>
> The sweep has mapped its whole batch at once since 9ad9a1609fbe ("PoD
> memory 5/9: emergency scan"), when x86-64 reached all memory through the
> directmap. The per-vCPU budget came with the mapcache's return to
> x86-64 in 4b28bf6ae90b ("x86: re-introduce map_domain_page() et al").
>
> Map one page at a time instead, as p2m_pod_zero_check_superpage()
> already does: map a page for the quick check and unmap it again, and
> map it anew for the full check after the p2m update and the TLB flush.
> The checks and their order are unchanged. Each page is mapped twice,
> which costs little next to scanning it, and the error paths no longer
> have mappings to undo.
>
> Fixes: 4b28bf6ae90b ("x86: re-introduce map_domain_page() et al")
> Assisted-by: Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
Acked-by: Roger Pau Monné <roger@xenproject.org>
I've got one suggestion below, but doesn't change the logic.
> ---
> Changes in v4:
> - Track the pages to check in a bitmask walked with for_each_set_bit(),
> not an array of bools (Andrew).
> - Drop the new comment above the first loop; edit the old one's "and map
> the pages" instead (Andrew).
>
> NB this issue was raised in review and verified only by code
> inspection; it hasn't been demonstrated.
> ---
> xen/arch/x86/mm/p2m-pod.c | 72 +++++++++++++++++----------------------
> 1 file changed, 31 insertions(+), 41 deletions(-)
>
> diff --git a/xen/arch/x86/mm/p2m-pod.c b/xen/arch/x86/mm/p2m-pod.c
> index 4602c32cff..ed298fa7d2 100644
> --- a/xen/arch/x86/mm/p2m-pod.c
> +++ b/xen/arch/x86/mm/p2m-pod.c
> @@ -901,9 +901,10 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
> {
> mfn_t mfns[POD_SWEEP_STRIDE];
> p2m_type_t types[POD_SWEEP_STRIDE];
> - unsigned long *map[POD_SWEEP_STRIDE];
> + unsigned int check = 0;
> + const unsigned long *map;
> struct domain *d = p2m->domain;
> - unsigned int i, j, max_ref = 1;
> + unsigned int j, max_ref = 1;
>
> BUG_ON(count > POD_SWEEP_STRIDE);
>
> @@ -911,8 +912,11 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
> if ( paging_mode_shadow(d) )
> max_ref++;
>
> - /* First, get the gfn list, translate to mfns, and map the pages. */
> - for ( i = 0; i < count; i++ )
> + /*
> + * First, get the gfn list, translate to mfns, and check for being a
> + * plausible candidate.
> + */
> + for ( unsigned int i = 0; i < count; i++ )
> {
> p2m_access_t a;
>
> @@ -921,9 +925,8 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
>
> /*
> * If this is ram, and not a pagetable or a special page, and
> - * probably not mapped elsewhere, map it; otherwise, skip.
> + * probably not mapped elsewhere, check it; otherwise, skip.
> */
> - map[i] = NULL;
> if ( p2m_is_ram(types[i]) )
> {
> const struct page_info *pg = mfn_to_page(mfns[i]);
> @@ -932,7 +935,7 @@ p2m_pod_zero_check(struct p2m_domain *p2m, const gfn_t *gfns, unsigned int count
> (pg->count_info & PGC_allocated) &&
> !(pg->count_info & PGC_shadowed_pt) &&
> ((pg->count_info & PGC_count_mask) <= max_ref) )
> - map[i] = map_domain_page(mfns[i]);
> + check |= 1U << i;
Not that I have a strong opinion, but I think it's clearer if you use
__{set,clear}_bit().
Thanks, Roger.
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v4 02/30] x86/mm: refuse to promote an L4 while the domain has no vCPU
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
2026-10-19 19:42 ` [PATCH v4 01/30] x86/PoD: map one page at a time in p2m_pod_zero_check() George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-23 7:05 ` Roger Pau Monné
2026-10-19 19:42 ` [PATCH v4 03/30] x86/mm: purge unneeded destroy_perdomain_mapping() George Dunlap
` (27 subsequent siblings)
29 siblings, 1 reply; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
init_xen_l4_slots() puts the domain's per-domain L3 in slot 260 of the
L4s it fills. Since fcf91ce9de66 ("x86/pv: don't populate the GDT/LDT
L3 slot at domain creation") a PV domain gets that L3 with its first
vCPU (pv_create_gdt_ldt_l1tab()). Only mapcache_domain_init() makes it
earlier, with the domain, and release builds skip that unless memory
hotplug is enabled or the host's memory reaches past the part of the
directmap PV guests can see. So in release builds on most hosts, an L4
promoted before XEN_DOMCTL_max_vcpus has created the domain's vCPUs gets
a slot 260 entry made from a NULL page (0x803e900000000063), pointing
far beyond the host's memory.
Debug builds happen to have the L3 that early. Refuse such an L4 in
every build, so that there is one rule: promote_l4_table() returns
-EINVAL while the domain has no vCPU 0, before taking any reference.
libxl is unaffected: it creates the vCPUs before it builds or restores a
domain.
There's no way to trigger the issue from the current toolstack; this
is just belt-and-braces.
Fixes: fcf91ce9de66 ("x86/pv: don't populate the GDT/LDT L3 slot at domain creation")
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- New in this version, split out of "x86/hvm: introduce per-vCPU L3
page-table": promote_l4_table() fails while the domain has no vCPU
---
xen/arch/x86/mm.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index e42cd33c0d..3f6f9f3bd4 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -1798,11 +1798,21 @@ static int promote_l4_table(struct page_info *page)
{
struct domain *d = page_get_owner(page);
mfn_t l4mfn = page_to_mfn(page);
- l4_pgentry_t *pl4e = map_domain_page(l4mfn);
+ l4_pgentry_t *pl4e;
unsigned int i;
int rc = 0;
unsigned int partial_flags = page->partial_flags;
+ /*
+ * init_xen_l4_slots() puts the domain's per-domain L3 in slot 260, and a
+ * PV domain may get that only with its first vCPU. The domain has no
+ * vCPUs before XEN_DOMCTL_max_vcpus.
+ */
+ if ( !domain_vcpu(d, 0) )
+ return -EINVAL;
+
+ pl4e = map_domain_page(l4mfn);
+
for ( i = page->nr_validated_ptes; i < L4_PAGETABLE_ENTRIES;
i++, partial_flags = 0 )
{
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* Re: [PATCH v4 02/30] x86/mm: refuse to promote an L4 while the domain has no vCPU
2026-10-19 19:42 ` [PATCH v4 02/30] x86/mm: refuse to promote an L4 while the domain has no vCPU George Dunlap
@ 2026-10-23 7:05 ` Roger Pau Monné
2026-10-26 10:28 ` George Dunlap
0 siblings, 1 reply; 39+ messages in thread
From: Roger Pau Monné @ 2026-10-23 7:05 UTC (permalink / raw)
To: George Dunlap; +Cc: xen-devel, Jan Beulich, Andrew Cooper, Teddy Astie
On Mon, Oct 19, 2026 at 08:42:33PM +0100, George Dunlap wrote:
> init_xen_l4_slots() puts the domain's per-domain L3 in slot 260 of the
> L4s it fills. Since fcf91ce9de66 ("x86/pv: don't populate the GDT/LDT
> L3 slot at domain creation") a PV domain gets that L3 with its first
> vCPU (pv_create_gdt_ldt_l1tab()). Only mapcache_domain_init() makes it
> earlier, with the domain, and release builds skip that unless memory
> hotplug is enabled or the host's memory reaches past the part of the
> directmap PV guests can see. So in release builds on most hosts, an L4
> promoted before XEN_DOMCTL_max_vcpus has created the domain's vCPUs gets
> a slot 260 entry made from a NULL page (0x803e900000000063), pointing
> far beyond the host's memory.
Won't it be more reliable long term to instead return an error from
init_xen_l4_slots() when the per-domain L3 page is not present?
It does seem like PV vCPU creation could change yet again in a way
that the per-domain L3 is not allocated there, and then the check
added in this patch will be moot. If we need the per-domain
page-table to be allocated by this point, let's test for that, not
indirectly testing whether vCPU 0 has been created.
I also wonder, this is for PV only, but I assume for correctness
purposes we should do the same check for HVM monitor tables also?
Thanks, Roger.
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [PATCH v4 02/30] x86/mm: refuse to promote an L4 while the domain has no vCPU
2026-10-23 7:05 ` Roger Pau Monné
@ 2026-10-26 10:28 ` George Dunlap
0 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-26 10:28 UTC (permalink / raw)
To: Roger Pau Monné; +Cc: xen-devel, Jan Beulich, Andrew Cooper, Teddy Astie
On Fri, Oct 23, 2026 at 8:06 AM Roger Pau Monné <roger@xenproject.org> wrote:
>
> On Mon, Oct 19, 2026 at 08:42:33PM +0100, George Dunlap wrote:
> > init_xen_l4_slots() puts the domain's per-domain L3 in slot 260 of the
> > L4s it fills. Since fcf91ce9de66 ("x86/pv: don't populate the GDT/LDT
> > L3 slot at domain creation") a PV domain gets that L3 with its first
> > vCPU (pv_create_gdt_ldt_l1tab()). Only mapcache_domain_init() makes it
> > earlier, with the domain, and release builds skip that unless memory
> > hotplug is enabled or the host's memory reaches past the part of the
> > directmap PV guests can see. So in release builds on most hosts, an L4
> > promoted before XEN_DOMCTL_max_vcpus has created the domain's vCPUs gets
> > a slot 260 entry made from a NULL page (0x803e900000000063), pointing
> > far beyond the host's memory.
>
> Won't it be more reliable long term to instead return an error from
> init_xen_l4_slots() when the per-domain L3 page is not present?
>
> It does seem like PV vCPU creation could change yet again in a way
> that the per-domain L3 is not allocated there, and then the check
> added in this patch will be moot. If we need the per-domain
> page-table to be allocated by this point, let's test for that, not
> indirectly testing whether vCPU 0 has been created.
>
> I also wonder, this is for PV only, but I assume for correctness
> purposes we should do the same check for HVM monitor tables also?
Eventually we modify init_xen_l4_slots to take a `struct vcpu*`; if
there are no vcpus, we can't pass it anything. For vcpu_pt domains,
that would technically be fine, I think, since the L3 is written in on
context switch anyway. But we would still need some sort of fall-back
if d->vcpus[0] == NULL but d->arch.perdomain_l3_pg != NULL. Is it
worth trying to figure out how to handle that case, just to save us
time in case we change the vCPU creation again?
We could change this patch to check `d->arch.perdomain_l3_pg` instead;
if we end up backporting this it certainly does make the check more
robust. But we'd still have to add the vcpu check later.
-George
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v4 03/30] x86/mm: purge unneeded destroy_perdomain_mapping()
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
2026-10-19 19:42 ` [PATCH v4 01/30] x86/PoD: map one page at a time in p2m_pod_zero_check() George Dunlap
2026-10-19 19:42 ` [PATCH v4 02/30] x86/mm: refuse to promote an L4 while the domain has no vCPU George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 04/30] x86/mm: prepare destroy_perdomain_mapping() for per-vCPU perdomain areas George Dunlap
` (26 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
We want to change per-domain mappings to be per-vCPU mappings. In
preparation for that, we want to arrange that
destroy_perdomain_mapping() work either with a single perdomain area,
or with a per-vCPU perdomain area.
There are two calls made from domain-scoped contexts; both calls turn
out to be unnecessary:
- destroy_perdomain_mapping() is not logically the undo of
create_perdomain_mapping(), as the name and its use in
hvm_domain_initialise() suggest. create_ allocates a per-domain L3,
but destroy_ tears down mappings without freeing it; and since the
call here passes nr == 0, it tears down nothing at all. The
per-domain L3 page is actually freed by free_perdomain_mappings(),
which hvm_domain_initialise()'s caller, arch_domain_create(),
already invokes on its failure path.
- The call in pv_domain_destroy() is redundant at both of its call
sites. arch_domain_destroy() unconditionally calls
free_perdomain_mappings(), which tears down the same entries and
additionally frees the page-table structures; and the other caller,
the error path of pv_domain_initialise(), returns into
arch_domain_create()'s failure path, which invokes
free_perdomain_mappings() as well.
One vCPU-scoped caller, pv_destroy_gdt_ldt_l1tab(), is covered the
same way: a vCPU is only ever destroyed together with its domain, or
on the failure path of creating it, and both routes end in
free_perdomain_mappings(); the GDT/LDT slots hold no area-owned pages,
so there is nothing else for the call to do. While we're getting rid
of unneccessary callers, drop this one too.
Note that free_compat_arg_xlat() keeps its explicit teardown:
switch_compat() calls it on the failure path. While in practice most
toolstacks will immediately destroy the domain, in theory a toolstack
could let it live on as a 64-bit guest. Release the resources
immediately rather than waiting for domain destruction.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
Changes in v4:
- Remove an unnecessary comment (Jan).
Changes in v3:
- Note that pv_domain_destroy()'s other call site, the error path of
pv_domain_initialise(), is covered the same way (Jan).
- Also drop pv_destroy_gdt_ldt_l1tab(), the per-vCPU teardown call
(Jan), and comment on why free_compat_arg_xlat() keeps its own. The
Reviewed-by tags given for the previous form are dropped.
Changes in v2:
- Added to the series
Changes since the previously posted version:
- Reworked the commit message to make it more clear how it fits in
with the larger series. No functional change.
---
xen/arch/x86/hvm/hvm.c | 1 -
xen/arch/x86/pv/domain.c | 10 ----------
2 files changed, 11 deletions(-)
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index ef9c7cd057..6aefe91e02 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -730,7 +730,6 @@ int hvm_domain_initialise(struct domain *d,
XFREE(d->arch.hvm.irq);
fail0:
hvm_destroy_cacheattr_region_list(d);
- destroy_perdomain_mapping(d, PERDOMAIN_VIRT_START, 0);
fail:
hvm_domain_relinquish_resources(d);
XFREE(d->arch.hvm.io_handler);
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index 0c42ae58aa..28ea0742ab 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -319,12 +319,6 @@ static int pv_create_gdt_ldt_l1tab(struct vcpu *v)
NULL);
}
-static void pv_destroy_gdt_ldt_l1tab(struct vcpu *v)
-{
- destroy_perdomain_mapping(v->domain, GDT_VIRT_START(v),
- 1U << GDT_LDT_VCPU_SHIFT);
-}
-
void pv_vcpu_destroy(struct vcpu *v)
{
if ( is_pv_32bit_vcpu(v) )
@@ -333,7 +327,6 @@ void pv_vcpu_destroy(struct vcpu *v)
release_compat_l4(v);
}
- pv_destroy_gdt_ldt_l1tab(v);
XFREE(v->arch.pv.trap_ctxt);
}
@@ -381,9 +374,6 @@ void pv_domain_destroy(struct domain *d)
{
pv_l1tf_domain_destroy(d);
- destroy_perdomain_mapping(d, GDT_LDT_VIRT_START,
- GDT_LDT_MBYTES << (20 - PAGE_SHIFT));
-
XFREE(d->arch.pv.cpuidmasks);
FREE_XENHEAP_PAGE(d->arch.pv.gdt_ldt_l1tab);
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 04/30] x86/mm: prepare destroy_perdomain_mapping() for per-vCPU perdomain areas
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (2 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 03/30] x86/mm: purge unneeded destroy_perdomain_mapping() George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 05/30] x86/domain_page: drop redundant create_perdomain_mapping() call George Dunlap
` (25 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Roger Pau Monné, Jan Beulich,
Andrew Cooper, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
We want to change per-domain mappings to be per-vCPU mappings. In
preparation for that, we want to arrange that
destroy_perdomain_mapping() work either with a single perdomain area,
or with a per-vCPU perdomain area.
The one remaining caller, free_compat_arg_xlat(), is already in a vCPU
context, so we just need to change the parameter from a domain pointer
to a vCPU pointer.
No functional change.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Roger Pau Monné <roger@xenproject.org>
---
Changes in v4:
- Say "No functional change." (Roger).
Changes in v3:
- Rebased onto staging without the GDT/LDT conversions (for the
HVM-only series): free_compat_arg_xlat() is the only caller left.
- Drop the linear-page-table fast path altogether (Jan): its only user
would have been guest-driven LDT teardown, and without it the
function does not depend on the linear mapping being available.
Changes in v2:
- Added to the series
Changes since the previously posted version:
- The fast path is gone (v3), and with it the previously posted
version's sync_local_execstate() call.
- Reword and retitle for clarity (was: "x86/mm: switch
destroy_perdomain_mapping() parameter from domain to vCPU").
---
xen/arch/x86/include/asm/mm.h | 2 +-
xen/arch/x86/mm.c | 3 ++-
xen/arch/x86/x86_64/mm.c | 2 +-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index aa515efbee..20a1d27cd0 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -609,7 +609,7 @@ int compat_arch_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg);
int create_perdomain_mapping(struct domain *d, unsigned long va,
unsigned int nr, l1_pgentry_t **pl1tab,
struct page_info **ppg);
-void destroy_perdomain_mapping(struct domain *d, unsigned long va,
+void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr);
void free_perdomain_mappings(struct domain *d);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 3f6f9f3bd4..70b4cc87ae 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -6361,10 +6361,11 @@ int create_perdomain_mapping(struct domain *d, unsigned long va,
return rc;
}
-void destroy_perdomain_mapping(struct domain *d, unsigned long va,
+void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr)
{
const l3_pgentry_t *l3tab, *pl3e;
+ const struct domain *d = v->domain;
ASSERT(va >= PERDOMAIN_VIRT_START &&
va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index 8eadab7933..f467ed426b 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -739,7 +739,7 @@ int setup_compat_arg_xlat(struct vcpu *v)
void free_compat_arg_xlat(struct vcpu *v)
{
- destroy_perdomain_mapping(v->domain, ARG_XLAT_START(v),
+ destroy_perdomain_mapping(v, ARG_XLAT_START(v),
PFN_UP(COMPAT_ARG_XLAT_SIZE));
}
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 05/30] x86/domain_page: drop redundant create_perdomain_mapping() call
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (3 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 04/30] x86/mm: prepare destroy_perdomain_mapping() for per-vCPU perdomain areas George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 06/30] x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain areas George Dunlap
` (24 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Roger Pau Monné, Jan Beulich,
Andrew Cooper, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
We want to change per-domain mappings to be per-vCPU mappings. In
preparation for that, we want to arrange that
create_perdomain_mapping() work either with a single perdomain area,
or with a per-vCPU perdomain area.
One of the current calls in a domain context turns out to be
unnecessary:
mapcache_domain_init() pre-plumbs L1 tables over the whole
inuse/garbage bitmap range (sized for the full MAPCACHE_ENTRIES
capacity), without populating any data pages. The plumbing is
redundant: mapcache_vcpu_init() installs the bitmap pages the domain
will actually use with populating calls, which allocate any missing
page-table structure on demand -- and since d->max_vcpus is fixed
before any vCPU is created, the range those calls cover never grows.
The pre-plumbed tail beyond it backs virtual addresses that are never
populated at all.
In debug builds, and on hosts with memory hotplug or with more memory
than the PV directmap covers, the call was also what first allocated a
PV domain's per-domain L3. Release builds on other hosts already
allocated it only with the first vCPU.
Drop the call; now we always allocate only with the first vCPU.
With the only fallible operation gone, mapcache_domain_init() becomes
void, and arch_domain_create()'s error handling for it goes away.
The per-domain-mapping work in mapcache_vcpu_init() stays where it is
rather than moving into mapcache_domain_init() along with this change,
ready for a later patch which introduces per-vCPU mapcaches.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Roger Pau Monné <roger@xenproject.org>
---
Changes in v3:
- Rebased onto staging without the GDT/LDT conversions (for the
HVM-only series); no functional difference.
- Say why the per-vCPU part stays in mapcache_vcpu_init() (Jan).
- Say that the call also first allocated a PV domain's per-domain L3 in
some builds, which all builds now do with the first vCPU.
Changes in v2:
- Added to the series (split out of the following patch).
Changes since the previously posted version:
- Split out of "x86/mm: switch {create,destroy}_perdomain_mapping()
domain parameter to vCPU", where the removal was folded into the
parameter switch without its own rationale.
---
xen/arch/x86/domain.c | 3 +--
xen/arch/x86/domain_page.c | 8 ++------
xen/arch/x86/include/asm/domain.h | 2 +-
3 files changed, 4 insertions(+), 9 deletions(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 368e4deff3..8e17fbe6d9 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -908,8 +908,7 @@ int arch_domain_create(struct domain *d,
}
else if ( is_pv_domain(d) )
{
- if ( (rc = mapcache_domain_init(d)) != 0 )
- goto fail;
+ mapcache_domain_init(d);
if ( (rc = pv_domain_initialise(d)) != 0 )
goto fail;
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 5744754f11..f02be1675b 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -223,7 +223,7 @@ void unmap_domain_page(const void *ptr)
local_irq_restore(flags);
}
-int mapcache_domain_init(struct domain *d)
+void mapcache_domain_init(struct domain *d)
{
struct mapcache_domain *dcache = &d->arch.pv.mapcache;
unsigned int bitmap_pages;
@@ -232,7 +232,7 @@ int mapcache_domain_init(struct domain *d)
#ifdef NDEBUG
if ( !mem_hotplug && max_page <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
- return 0;
+ return;
#endif
BUILD_BUG_ON(MAPCACHE_VIRT_END + PAGE_SIZE * (3 +
@@ -244,10 +244,6 @@ int mapcache_domain_init(struct domain *d)
(bitmap_pages + 1) * PAGE_SIZE / sizeof(long);
spin_lock_init(&dcache->lock);
-
- return create_perdomain_mapping(d, (unsigned long)dcache->inuse,
- 2 * bitmap_pages + 1,
- NIL(l1_pgentry_t *), NULL);
}
int mapcache_vcpu_init(struct vcpu *v)
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 2d0a915410..212daae1f5 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -89,7 +89,7 @@ struct mapcache_domain {
unsigned long *garbage;
};
-int mapcache_domain_init(struct domain *d);
+void mapcache_domain_init(struct domain *d);
int mapcache_vcpu_init(struct vcpu *v);
/* x86/64: toggle guest between kernel and user modes. */
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 06/30] x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain areas
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (4 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 05/30] x86/domain_page: drop redundant create_perdomain_mapping() call George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 07/30] x86/spec-ctrl: introduce Address Space Isolation command line option George Dunlap
` (23 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
We want to change per-domain mappings to be per-vCPU mappings. In
preparation for that, we want to arrange that
create_perdomain_mapping() work either with a single perdomain area,
or with a per-vCPU perdomain area.
Most of the remaining callers are already in a vCPU context. This is
no accident: the perdomain area has always been laid out in per-vCPU
slices -- each vCPU has its own GDT/LDT window, its own
COMPAT_ARG_XLAT pages, its own window of mapcache entries -- and each
vCPU's slice is set up as that vCPU is created. For these callers, we
just need to change the parameter from a domain pointer to a vCPU
pointer. Once the perdomain area itself becomes per-vCPU, the same
calls will populate the owning vCPU's own area rather than slices of a
shared one.
One exception is the call in hvm_domain_initialise(). An HVM vCPU's
monitor table is created during vCPU initialisation, and
init_xen_l4_slots() stamps the perdomain slot into it at that point --
far earlier than for PV, where the Xen slots are written only once
guest page tables are built. hvm_domain_initialise() therefore had an
explicit create_perdomain_mapping() call just to make the perdomain
root exist ahead of that. Move it to arch_vcpu_create(), covering HVM
and PV alike. With a single shared area, the call allocates at most
once per domain; but once each vCPU has its own perdomain area, this
is the call that will allocate every vCPU's root before any page
tables referencing it are built. For PV guests nothing observable
changes: the root was already being created during vCPU creation as a
side effect (by mapcache_vcpu_init(), or failing that
pv_create_gdt_ldt_l1tab()); it now merely becomes explicit.
Note that we cannot yet do a parallel movement of
free_perdomain_mappings(): the per-domain page-table hierarchy is
still a single domain-wide structure shared by all vCPUs, so tearing
it down from a per-vCPU path would pull the mappings out from under
sibling vCPUs (e.g. on a partially failed, retryable
XEN_DOMCTL_max_vcpus), and vCPU-create error paths can rely on domain
destruction to free a partially set up hierarchy. Once a domain's area
is per-vCPU, a later change gives its vCPUs a teardown of their own; the
domain-wide teardown stays for domains with a shared area, PV ones
included.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
Changes in v4:
- Message: drop a sentence repeating its paragraph, and say that the
domain-wide teardown stays for domains with a shared area (Roger).
- End the comment in arch_vcpu_create() at the colon (Roger).
Changes in v3:
- Rebased onto staging without the GDT/LDT conversions (for the
HVM-only series): the capture parameters stay, as the PV GDT/LDT
stash still uses them.
Changes in v2:
- Added to the series
Changes since the previously posted version:
- Keep free_perdomain_mappings() (and hence perdomain teardown)
domain-scoped.
- Keep the idle domain without a perdomain area.
- Retitle (was: "x86/mm: switch {create,destroy}_perdomain_mapping()
domain parameter to vCPU"); destroy_perdomain_mapping() was switched
in a separate patch.
- Split the removal of mapcache_domain_init()'s redundant
create_perdomain_mapping() call into its own (preceding) patch.
---
xen/arch/x86/domain.c | 8 ++++++++
xen/arch/x86/domain_page.c | 6 +++---
xen/arch/x86/hvm/hvm.c | 5 -----
xen/arch/x86/include/asm/mm.h | 2 +-
xen/arch/x86/mm.c | 3 ++-
xen/arch/x86/pv/domain.c | 2 +-
xen/arch/x86/x86_64/mm.c | 2 +-
7 files changed, 16 insertions(+), 12 deletions(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 8e17fbe6d9..94a1ea904a 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -517,6 +517,14 @@ int arch_vcpu_create(struct vcpu *v)
if ( !is_idle_domain(d) )
{
+ /*
+ * Make sure the per-domain L3 exists ahead of any consumer (e.g.
+ * init_xen_l4_slots() for the HVM monitor tables).
+ */
+ rc = create_perdomain_mapping(v, PERDOMAIN_VIRT_START, 0, NULL, NULL);
+ if ( rc )
+ return rc;
+
paging_vcpu_init(v);
if ( (rc = vcpu_init_fpu(v)) != 0 )
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index f02be1675b..f3e65e2d44 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -260,15 +260,15 @@ int mapcache_vcpu_init(struct vcpu *v)
if ( ents > dcache->entries )
{
/* Populate page tables. */
- int rc = create_perdomain_mapping(d, MAPCACHE_VIRT_START, ents,
+ int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START, ents,
NIL(l1_pgentry_t *), NULL);
/* Populate bit maps. */
if ( !rc )
- rc = create_perdomain_mapping(d, (unsigned long)dcache->inuse,
+ rc = create_perdomain_mapping(v, (unsigned long)dcache->inuse,
nr, NULL, NIL(struct page_info *));
if ( !rc )
- rc = create_perdomain_mapping(d, (unsigned long)dcache->garbage,
+ rc = create_perdomain_mapping(v, (unsigned long)dcache->garbage,
nr, NULL, NIL(struct page_info *));
if ( rc )
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 6aefe91e02..71da15c0a4 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -620,10 +620,6 @@ int hvm_domain_initialise(struct domain *d,
INIT_LIST_HEAD(&d->arch.hvm.mmcfg_regions);
INIT_LIST_HEAD(&d->arch.hvm.msix_tables);
- rc = create_perdomain_mapping(d, PERDOMAIN_VIRT_START, 0, NULL, NULL);
- if ( rc )
- goto fail;
-
hvm_init_cacheattr_region_list(d);
rc = paging_enable(d, PG_refcounts|PG_translate|PG_external);
@@ -730,7 +726,6 @@ int hvm_domain_initialise(struct domain *d,
XFREE(d->arch.hvm.irq);
fail0:
hvm_destroy_cacheattr_region_list(d);
- fail:
hvm_domain_relinquish_resources(d);
XFREE(d->arch.hvm.io_handler);
XFREE(d->arch.hvm.pl_time);
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 20a1d27cd0..c9085214fd 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -606,7 +606,7 @@ int compat_arch_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg);
#define NIL(type) ((type *)-sizeof(type))
#define IS_NIL(ptr) (!((uintptr_t)(ptr) + sizeof(*(ptr))))
-int create_perdomain_mapping(struct domain *d, unsigned long va,
+int create_perdomain_mapping(struct vcpu *v, unsigned long va,
unsigned int nr, l1_pgentry_t **pl1tab,
struct page_info **ppg);
void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 70b4cc87ae..c6bd53354b 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -6238,10 +6238,11 @@ static bool perdomain_l1e_needs_freeing(l1_pgentry_t l1e)
(_PAGE_PRESENT | _PAGE_AVAIL0);
}
-int create_perdomain_mapping(struct domain *d, unsigned long va,
+int create_perdomain_mapping(struct vcpu *v, unsigned long va,
unsigned int nr, l1_pgentry_t **pl1tab,
struct page_info **ppg)
{
+ struct domain *d = v->domain;
struct page_info *pg;
l3_pgentry_t *l3tab;
l2_pgentry_t *l2tab;
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index 28ea0742ab..f3c711b85b 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -313,7 +313,7 @@ int switch_compat(struct domain *d)
static int pv_create_gdt_ldt_l1tab(struct vcpu *v)
{
- return create_perdomain_mapping(v->domain, GDT_VIRT_START(v),
+ return create_perdomain_mapping(v, GDT_VIRT_START(v),
1U << GDT_LDT_VCPU_SHIFT,
v->domain->arch.pv.gdt_ldt_l1tab,
NULL);
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index f467ed426b..7b04537066 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -732,7 +732,7 @@ void __init zap_low_mappings(void)
int setup_compat_arg_xlat(struct vcpu *v)
{
- return create_perdomain_mapping(v->domain, ARG_XLAT_START(v),
+ return create_perdomain_mapping(v, ARG_XLAT_START(v),
PFN_UP(COMPAT_ARG_XLAT_SIZE),
NULL, NIL(struct page_info *));
}
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 07/30] x86/spec-ctrl: introduce Address Space Isolation command line option
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (5 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 06/30] x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain areas George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-23 7:45 ` Roger Pau Monné
2026-10-19 19:42 ` [PATCH v4 08/30] x86/mm: have init_xen_l4_slots() take a vCPU in place of the domain George Dunlap
` (22 subsequent siblings)
29 siblings, 1 reply; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Andrew Cooper, Anthony PERARD, Michal Orzel,
Jan Beulich, Julien Grall, Roger Pau Monné,
Stefano Stabellini, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
Introduce the `asi=` command line option, and the opt_vcpu_pt_hvm and
opt_pvh_vcpu_pt_hwdom knobs plus the per-domain d->arch.asi.vcpu_pt
setting they control. The option is introduced ahead of the
functionality it enables, so that the newly added code can be keyed on
it from the start; all knobs currently default to off, and enabling any
of them adds a boot-time warning that the feature is not functional and
is meant for development only.
The mechanisms this option controls apply to HVM domains: an HVM vCPU
already runs on its own monitor table, so a per-vCPU per-domain area
needs no work at context switch. d->arch.asi.vcpu_pt is therefore only
ever set for HVM domains, a PVH hardware domain included; PV domains,
a PV hardware domain included, are unaffected.
Besides the plain boolean, `pv=`, `hvm=` and `dom0=` select the features
for PV domains or HVM domains other than the hardware domain, or for the
hardware domain, PV or PVH. None of the features applies to PV domains,
so `pv=` has no effect, and `dom0=` acts on a PVH hardware domain only.
The boot log gains "ASI features for ..." lines for Dom0 and HVM
domains, so hardware-domain-only configurations remain visible.
Further per-mechanism tokens arrive with their mechanisms in later
patches.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- Docs: "domains" throughout, not "guests" (Roger).
- Group the per-domain settings as one-bit fields of d->arch.asi
(Roger).
- Rename opt_vcpu_pt_hwdom to opt_pvh_vcpu_pt_hwdom: it acts on a PVH
hardware domain only (Jan).
- parse_asi(): break in the success branch and fall through to
default:, rather than a default: label after an else (MISRA C:2012
Rule 16.2; Jan).
- Message: enabling a knob adds a boot-time warning; it does not taint
(Roger).
- Add the pv= and dom0= sub-options here, beside hvm= (Roger).
Changes in v3:
- HVM only: drop the PV knob and its interaction with XPTI. vCPU-PT
applies to HVM domains and to a PVH hardware domain; the Dom0 boot
log line says None for a PV one.
Changes in v2:
- Added to the series
Changes since the previously posted version:
- Include the hardware domain in the development warning and the boot
log summary.
- Make the opt_vcpu_pt_* knobs plain booleans preinitialised to false,
dropping the late -1 resolution.
- Documentation: mention possible protection against unmitigated
attacks, and state that hvm= does not affect the hardware domain.
- Rewrite the commit message.
---
docs/misc/xen-command-line.pandoc | 29 ++++++++++
xen/arch/x86/include/asm/domain.h | 6 ++
xen/arch/x86/include/asm/spec_ctrl.h | 2 +
xen/arch/x86/spec_ctrl.c | 82 ++++++++++++++++++++++++++++
4 files changed, 119 insertions(+)
diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
index a6f08463d1..839d06da26 100644
--- a/docs/misc/xen-command-line.pandoc
+++ b/docs/misc/xen-command-line.pandoc
@@ -202,6 +202,35 @@ to appropriate auditing by Xen. Argo is disabled by default.
This option is disabled by default, to protect domains from a DoS by a
buggy or malicious other domain spamming the ring.
+### asi (x86)
+> `= List of [ <bool>, {pv,hvm,dom0}=<bool>,
+> vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
+
+> Default: `false`
+
+Offers control over whether the hypervisor will engage in Address Space
+Isolation, by not having potentially sensitive information permanently mapped
+in the VMM page-tables. Using this option might avoid the need to apply
+mitigations for certain speculative related attacks, at the cost of mapping
+sensitive information on-demand. It might also offer some protection against
+unmitigated speculation-related attacks.
+
+The mechanisms currently implemented apply to HVM domains, including a PVH
+hardware domain. PV domains, including a PV hardware domain, are unaffected.
+
+* `pv=`, `hvm=` and `dom0=` enable the features for PV domains or HVM domains
+ other than the hardware domain, or for the hardware domain, PV or PVH. The
+ plain boolean does all three: `asi=1,dom0=0` leaves the hardware domain
+ out. The forms naming no domain type (an un-suffixed `vcpu-pt=<bool>`)
+ cover the hardware domain too.
+
+**WARNING: manual de-selection of enabled options will invalidate any
+protection offered by the feature. The fine grained options provided below
+are meant to be used for debugging purposes only.**
+
+* `vcpu-pt` gives each vCPU its own per-domain area: the per-domain slot of
+ each vCPU's monitor table maps a region private to that vCPU.
+
### asid (x86)
> `= <boolean>`
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 212daae1f5..12f79feb42 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -474,6 +474,12 @@ struct arch_domain
/* Don't unconditionally inject #GP for unhandled MSRs. */
bool msr_relaxed;
+ /* Fine grained ASI related options. */
+ struct {
+ /* Give each vCPU its own per-domain area (HVM only). */
+ bool vcpu_pt:1;
+ } asi;
+
/* Emulated devices enabled bitmap. */
uint32_t emulation_flags;
} __cacheline_aligned;
diff --git a/xen/arch/x86/include/asm/spec_ctrl.h b/xen/arch/x86/include/asm/spec_ctrl.h
index 8f82533c41..d84f985e82 100644
--- a/xen/arch/x86/include/asm/spec_ctrl.h
+++ b/xen/arch/x86/include/asm/spec_ctrl.h
@@ -87,6 +87,8 @@ extern uint8_t default_scf;
extern int8_t opt_xpti_hwdom, opt_xpti_domu;
+extern bool opt_pvh_vcpu_pt_hwdom, opt_vcpu_pt_hvm;
+
extern bool cpu_has_bug_l1tf;
extern int8_t opt_pv_l1tf_hwdom, opt_pv_l1tf_domu;
extern bool opt_bp_spec_reduce;
diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c
index bc8538a56f..66c878212a 100644
--- a/xen/arch/x86/spec_ctrl.c
+++ b/xen/arch/x86/spec_ctrl.c
@@ -86,6 +86,10 @@ bool __ro_after_init opt_bp_spec_reduce = true;
static bool __initdata opt_ibpb_alt;
+/* Give each vCPU of an HVM domain its own per-domain area (vCPU-PT). */
+bool __ro_after_init opt_vcpu_pt_hvm;
+bool __ro_after_init opt_pvh_vcpu_pt_hwdom;
+
static int __init cf_check parse_spec_ctrl(const char *s)
{
const char *ss;
@@ -487,6 +491,69 @@ static int __init cf_check parse_pv_l1tf(const char *s)
}
custom_param("pv-l1tf", parse_pv_l1tf);
+static int __init cf_check parse_asi(const char *s)
+{
+ const char *ss;
+ int val, rc = 0;
+
+ /* Interpret 'asi' alone in its positive boolean form. */
+ if ( *s == '\0' )
+ opt_pvh_vcpu_pt_hwdom = opt_vcpu_pt_hvm = true;
+
+ do {
+ ss = strchr(s, ',');
+ if ( !ss )
+ ss = strchr(s, '\0');
+
+ val = parse_bool(s, ss);
+ switch ( val )
+ {
+ case 0:
+ case 1:
+ opt_pvh_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
+ break;
+
+ default:
+ if ( (val = parse_boolean("pv", s, ss)) >= 0 )
+ /* PV domains are unaffected. */;
+ else if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
+ opt_vcpu_pt_hvm = val;
+ else if ( (val = parse_boolean("dom0", s, ss)) >= 0 )
+ opt_pvh_vcpu_pt_hwdom = val;
+ else if ( (val = parse_boolean("vcpu-pt", s, ss)) != -1 )
+ {
+ switch ( val )
+ {
+ case 1:
+ case 0:
+ opt_pvh_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
+ break;
+
+ case -2:
+ s += strlen("vcpu-pt=");
+ if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
+ {
+ opt_vcpu_pt_hvm = val;
+ break;
+ }
+ fallthrough;
+ default:
+ rc = -EINVAL;
+ break;
+ }
+ }
+ else if ( *s )
+ rc = -EINVAL;
+ break;
+ }
+
+ s = ss + 1;
+ } while ( *ss );
+
+ return rc;
+}
+custom_param("asi", parse_asi);
+
static void __init print_details(enum ind_thunk thunk)
{
unsigned int _7d0 = 0, _7d2 = 0, e8b = 0, e21a = 0, e21c = 0, max = 0, tmp;
@@ -680,6 +747,13 @@ static void __init print_details(enum ind_thunk thunk)
opt_pv_l1tf_hwdom ? "enabled" : "disabled",
opt_pv_l1tf_domu ? "enabled" : "disabled");
#endif
+
+ printk(" ASI features for Dom0:%s\n",
+ opt_dom0_pvh && opt_pvh_vcpu_pt_hwdom ? " vCPU-PT" : " None");
+#ifdef CONFIG_HVM
+ printk(" ASI features for HVM VMs:%s\n",
+ opt_vcpu_pt_hvm ? " vCPU-PT" : " None");
+#endif
}
static bool __init check_smt_enabled(void)
@@ -1866,6 +1940,9 @@ void spec_ctrl_init_domain(struct domain *d)
if ( pv )
d->arch.pv.xpti = is_hardware_domain(d) ? opt_xpti_hwdom
: opt_xpti_domu;
+
+ d->arch.asi.vcpu_pt = !pv && (is_hardware_domain(d) ? opt_pvh_vcpu_pt_hwdom
+ : opt_vcpu_pt_hvm);
}
void __init init_speculation_mitigations(void)
@@ -2158,6 +2235,11 @@ void __init init_speculation_mitigations(void)
hw_smt_enabled && default_xen_spec_ctrl )
setup_force_cpu_cap(X86_FEATURE_SC_MSR_IDLE);
+ if ( opt_pvh_vcpu_pt_hwdom || opt_vcpu_pt_hvm )
+ warning_add(
+ "Address Space Isolation is not functional, this option is\n"
+ "intended to be used only for development purposes.\n");
+
xpti_init_default();
l1tf_calculations();
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* Re: [PATCH v4 07/30] x86/spec-ctrl: introduce Address Space Isolation command line option
2026-10-19 19:42 ` [PATCH v4 07/30] x86/spec-ctrl: introduce Address Space Isolation command line option George Dunlap
@ 2026-10-23 7:45 ` Roger Pau Monné
2026-10-26 10:40 ` George Dunlap
0 siblings, 1 reply; 39+ messages in thread
From: Roger Pau Monné @ 2026-10-23 7:45 UTC (permalink / raw)
To: George Dunlap
Cc: xen-devel, Roger Pau Monné, Andrew Cooper, Anthony PERARD,
Michal Orzel, Jan Beulich, Julien Grall, Stefano Stabellini,
Teddy Astie
On Mon, Oct 19, 2026 at 08:42:38PM +0100, George Dunlap wrote:
> From: Roger Pau Monné <roger.pau@citrix.com>
>
> Introduce the `asi=` command line option, and the opt_vcpu_pt_hvm and
> opt_pvh_vcpu_pt_hwdom knobs plus the per-domain d->arch.asi.vcpu_pt
> setting they control. The option is introduced ahead of the
> functionality it enables, so that the newly added code can be keyed on
> it from the start; all knobs currently default to off, and enabling any
> of them adds a boot-time warning that the feature is not functional and
> is meant for development only.
>
> The mechanisms this option controls apply to HVM domains: an HVM vCPU
> already runs on its own monitor table, so a per-vCPU per-domain area
> needs no work at context switch. d->arch.asi.vcpu_pt is therefore only
> ever set for HVM domains, a PVH hardware domain included; PV domains,
> a PV hardware domain included, are unaffected.
>
> Besides the plain boolean, `pv=`, `hvm=` and `dom0=` select the features
> for PV domains or HVM domains other than the hardware domain, or for the
> hardware domain, PV or PVH. None of the features applies to PV domains,
> so `pv=` has no effect, and `dom0=` acts on a PVH hardware domain only.
>
> The boot log gains "ASI features for ..." lines for Dom0 and HVM
> domains, so hardware-domain-only configurations remain visible.
>
> Further per-mechanism tokens arrive with their mechanisms in later
> patches.
>
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
I don't have a strong opinion on the comments below, I think it would
be nice to adjust some as they likely lead to less churn in the
followup patches.
> ---
> Changes in v4:
> - Docs: "domains" throughout, not "guests" (Roger).
> - Group the per-domain settings as one-bit fields of d->arch.asi
> (Roger).
> - Rename opt_vcpu_pt_hwdom to opt_pvh_vcpu_pt_hwdom: it acts on a PVH
> hardware domain only (Jan).
I would leave opt_vcpu_pt_hwdom as-is. The fact it only acts on a PVH
hardware domain is a transient implementation short-coming. I don't
think it's worth introducing a separate opt_pv_vcpu_pt_hwdom when PV
dom0 support is added, or having to rename opt_pvh_vcpu_pt_hwdom ->
opt_vcpu_pt_hwdom.
> - parse_asi(): break in the success branch and fall through to
> default:, rather than a default: label after an else (MISRA C:2012
> Rule 16.2; Jan).
> - Message: enabling a knob adds a boot-time warning; it does not taint
> (Roger).
> - Add the pv= and dom0= sub-options here, beside hvm= (Roger).
>
> Changes in v3:
> - HVM only: drop the PV knob and its interaction with XPTI. vCPU-PT
> applies to HVM domains and to a PVH hardware domain; the Dom0 boot
> log line says None for a PV one.
>
> Changes in v2:
> - Added to the series
>
> Changes since the previously posted version:
> - Include the hardware domain in the development warning and the boot
> log summary.
> - Make the opt_vcpu_pt_* knobs plain booleans preinitialised to false,
> dropping the late -1 resolution.
> - Documentation: mention possible protection against unmitigated
> attacks, and state that hvm= does not affect the hardware domain.
> - Rewrite the commit message.
> ---
> docs/misc/xen-command-line.pandoc | 29 ++++++++++
> xen/arch/x86/include/asm/domain.h | 6 ++
> xen/arch/x86/include/asm/spec_ctrl.h | 2 +
> xen/arch/x86/spec_ctrl.c | 82 ++++++++++++++++++++++++++++
> 4 files changed, 119 insertions(+)
>
> diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
> index a6f08463d1..839d06da26 100644
> --- a/docs/misc/xen-command-line.pandoc
> +++ b/docs/misc/xen-command-line.pandoc
> @@ -202,6 +202,35 @@ to appropriate auditing by Xen. Argo is disabled by default.
> This option is disabled by default, to protect domains from a DoS by a
> buggy or malicious other domain spamming the ring.
>
> +### asi (x86)
> +> `= List of [ <bool>, {pv,hvm,dom0}=<bool>,
> +> vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
> +
> +> Default: `false`
> +
> +Offers control over whether the hypervisor will engage in Address Space
> +Isolation, by not having potentially sensitive information permanently mapped
> +in the VMM page-tables. Using this option might avoid the need to apply
> +mitigations for certain speculative related attacks, at the cost of mapping
> +sensitive information on-demand. It might also offer some protection against
> +unmitigated speculation-related attacks.
> +
> +The mechanisms currently implemented apply to HVM domains, including a PVH
> +hardware domain. PV domains, including a PV hardware domain, are unaffected.
I think we might want to drop this. It's an incomplete in-development
feature: we don't need to mention explicitly what is and what is not
implemented in the command line documentation for the time being.
> +
> +* `pv=`, `hvm=` and `dom0=` enable the features for PV domains or HVM domains
> + other than the hardware domain, or for the hardware domain, PV or PVH. The
> + plain boolean does all three: `asi=1,dom0=0` leaves the hardware domain
> + out. The forms naming no domain type (an un-suffixed `vcpu-pt=<bool>`)
> + cover the hardware domain too.
> +
> +**WARNING: manual de-selection of enabled options will invalidate any
> +protection offered by the feature. The fine grained options provided below
> +are meant to be used for debugging purposes only.**
> +
> +* `vcpu-pt` gives each vCPU its own per-domain area: the per-domain slot of
> + each vCPU's monitor table maps a region private to that vCPU.
> +
> ### asid (x86)
> > `= <boolean>`
>
> diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
> index 212daae1f5..12f79feb42 100644
> --- a/xen/arch/x86/include/asm/domain.h
> +++ b/xen/arch/x86/include/asm/domain.h
> @@ -474,6 +474,12 @@ struct arch_domain
> /* Don't unconditionally inject #GP for unhandled MSRs. */
> bool msr_relaxed;
>
> + /* Fine grained ASI related options. */
> + struct {
> + /* Give each vCPU its own per-domain area (HVM only). */
I would drop the HVM only part, it's a transient implementation
limitation, not a design one.
> + bool vcpu_pt:1;
> + } asi;
> +
> /* Emulated devices enabled bitmap. */
> uint32_t emulation_flags;
> } __cacheline_aligned;
> diff --git a/xen/arch/x86/include/asm/spec_ctrl.h b/xen/arch/x86/include/asm/spec_ctrl.h
> index 8f82533c41..d84f985e82 100644
> --- a/xen/arch/x86/include/asm/spec_ctrl.h
> +++ b/xen/arch/x86/include/asm/spec_ctrl.h
> @@ -87,6 +87,8 @@ extern uint8_t default_scf;
>
> extern int8_t opt_xpti_hwdom, opt_xpti_domu;
>
> +extern bool opt_pvh_vcpu_pt_hwdom, opt_vcpu_pt_hvm;
> +
> extern bool cpu_has_bug_l1tf;
> extern int8_t opt_pv_l1tf_hwdom, opt_pv_l1tf_domu;
> extern bool opt_bp_spec_reduce;
> diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c
> index bc8538a56f..66c878212a 100644
> --- a/xen/arch/x86/spec_ctrl.c
> +++ b/xen/arch/x86/spec_ctrl.c
> @@ -86,6 +86,10 @@ bool __ro_after_init opt_bp_spec_reduce = true;
>
> static bool __initdata opt_ibpb_alt;
>
> +/* Give each vCPU of an HVM domain its own per-domain area (vCPU-PT). */
> +bool __ro_after_init opt_vcpu_pt_hvm;
> +bool __ro_after_init opt_pvh_vcpu_pt_hwdom;
> +
> static int __init cf_check parse_spec_ctrl(const char *s)
> {
> const char *ss;
> @@ -487,6 +491,69 @@ static int __init cf_check parse_pv_l1tf(const char *s)
> }
> custom_param("pv-l1tf", parse_pv_l1tf);
>
> +static int __init cf_check parse_asi(const char *s)
> +{
> + const char *ss;
> + int val, rc = 0;
> +
> + /* Interpret 'asi' alone in its positive boolean form. */
> + if ( *s == '\0' )
> + opt_pvh_vcpu_pt_hwdom = opt_vcpu_pt_hvm = true;
> +
> + do {
> + ss = strchr(s, ',');
> + if ( !ss )
> + ss = strchr(s, '\0');
> +
> + val = parse_bool(s, ss);
> + switch ( val )
> + {
> + case 0:
> + case 1:
> + opt_pvh_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
> + break;
> +
> + default:
> + if ( (val = parse_boolean("pv", s, ss)) >= 0 )
> + /* PV domains are unaffected. */;
You might want to print a message saying that the option is ignored
for PV? It will go into the console buffer at least.
> + else if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
> + opt_vcpu_pt_hvm = val;
> + else if ( (val = parse_boolean("dom0", s, ss)) >= 0 )
> + opt_pvh_vcpu_pt_hwdom = val;
> + else if ( (val = parse_boolean("vcpu-pt", s, ss)) != -1 )
> + {
> + switch ( val )
> + {
> + case 1:
> + case 0:
> + opt_pvh_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
> + break;
> +
> + case -2:
> + s += strlen("vcpu-pt=");
> + if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
> + {
> + opt_vcpu_pt_hvm = val;
> + break;
> + }
> + fallthrough;
> + default:
> + rc = -EINVAL;
> + break;
> + }
> + }
> + else if ( *s )
> + rc = -EINVAL;
> + break;
> + }
> +
> + s = ss + 1;
> + } while ( *ss );
> +
> + return rc;
> +}
> +custom_param("asi", parse_asi);
> +
> static void __init print_details(enum ind_thunk thunk)
> {
> unsigned int _7d0 = 0, _7d2 = 0, e8b = 0, e21a = 0, e21c = 0, max = 0, tmp;
> @@ -680,6 +747,13 @@ static void __init print_details(enum ind_thunk thunk)
> opt_pv_l1tf_hwdom ? "enabled" : "disabled",
> opt_pv_l1tf_domu ? "enabled" : "disabled");
> #endif
> +
> + printk(" ASI features for Dom0:%s\n",
> + opt_dom0_pvh && opt_pvh_vcpu_pt_hwdom ? " vCPU-PT" : " None");
> +#ifdef CONFIG_HVM
> + printk(" ASI features for HVM VMs:%s\n",
> + opt_vcpu_pt_hvm ? " vCPU-PT" : " None");
> +#endif
opt_vcpu_pt_hvm is defined unconditionally, so you might as well use:
if ( IS_ENABLED(CONFIG_HVM) )
printk(" ASI features for HVM VMs:%s\n", ...
Which is a bit nicer than the ifdefs?
> }
>
> static bool __init check_smt_enabled(void)
> @@ -1866,6 +1940,9 @@ void spec_ctrl_init_domain(struct domain *d)
> if ( pv )
> d->arch.pv.xpti = is_hardware_domain(d) ? opt_xpti_hwdom
> : opt_xpti_domu;
> +
> + d->arch.asi.vcpu_pt = !pv && (is_hardware_domain(d) ? opt_pvh_vcpu_pt_hwdom
> + : opt_vcpu_pt_hvm);
> }
>
> void __init init_speculation_mitigations(void)
> @@ -2158,6 +2235,11 @@ void __init init_speculation_mitigations(void)
> hw_smt_enabled && default_xen_spec_ctrl )
> setup_force_cpu_cap(X86_FEATURE_SC_MSR_IDLE);
>
> + if ( opt_pvh_vcpu_pt_hwdom || opt_vcpu_pt_hvm )
> + warning_add(
> + "Address Space Isolation is not functional, this option is\n"
> + "intended to be used only for development purposes.\n");
I wonder, could we add the warning in parse_asi() itself? I know if
would then also trigger for asi=0, but would avoid having to remember
to expand the condition here every time a new option is added?
Thanks, Roger.
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [PATCH v4 07/30] x86/spec-ctrl: introduce Address Space Isolation command line option
2026-10-23 7:45 ` Roger Pau Monné
@ 2026-10-26 10:40 ` George Dunlap
0 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-26 10:40 UTC (permalink / raw)
To: Roger Pau Monné
Cc: xen-devel, Roger Pau Monné, Andrew Cooper, Anthony PERARD,
Michal Orzel, Jan Beulich, Julien Grall, Stefano Stabellini,
Teddy Astie
On Fri, Oct 23, 2026 at 8:45 AM Roger Pau Monné <roger@xenproject.org> wrote:
> > ---
> > Changes in v4:
> > - Docs: "domains" throughout, not "guests" (Roger).
> > - Group the per-domain settings as one-bit fields of d->arch.asi
> > (Roger).
> > - Rename opt_vcpu_pt_hwdom to opt_pvh_vcpu_pt_hwdom: it acts on a PVH
> > hardware domain only (Jan).
>
> I would leave opt_vcpu_pt_hwdom as-is. The fact it only acts on a PVH
> hardware domain is a transient implementation short-coming. I don't
> think it's worth introducing a separate opt_pv_vcpu_pt_hwdom when PV
> dom0 support is added, or having to rename opt_pvh_vcpu_pt_hwdom ->
> opt_vcpu_pt_hwdom.
Yes, I think you're right; in the end an option which says, "Turn on
vcpu_pt if dom0 a PVH domain, but not if it's a PV domain" is terribly
useful.
> > diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
> > index a6f08463d1..839d06da26 100644
> > --- a/docs/misc/xen-command-line.pandoc
> > +++ b/docs/misc/xen-command-line.pandoc
> > @@ -202,6 +202,35 @@ to appropriate auditing by Xen. Argo is disabled by default.
> > This option is disabled by default, to protect domains from a DoS by a
> > buggy or malicious other domain spamming the ring.
> >
> > +### asi (x86)
> > +> `= List of [ <bool>, {pv,hvm,dom0}=<bool>,
> > +> vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
> > +
> > +> Default: `false`
> > +
> > +Offers control over whether the hypervisor will engage in Address Space
> > +Isolation, by not having potentially sensitive information permanently mapped
> > +in the VMM page-tables. Using this option might avoid the need to apply
> > +mitigations for certain speculative related attacks, at the cost of mapping
> > +sensitive information on-demand. It might also offer some protection against
> > +unmitigated speculation-related attacks.
> > +
> > +The mechanisms currently implemented apply to HVM domains, including a PVH
> > +hardware domain. PV domains, including a PV hardware domain, are unaffected.
>
> I think we might want to drop this. It's an incomplete in-development
> feature: we don't need to mention explicitly what is and what is not
> implemented in the command line documentation for the time being.
This seems to me a kind of difference in taste. I prefer the
documentation to be always accurate according to that state in the
tree; having the documentation change as the code changes helps you
see at a big picture how things are progressing. In any case, I doubt
we'll get vcpu_pt implemented by the next release, so we'll want a
statement like this in there anyway, won't we?
> > +* `pv=`, `hvm=` and `dom0=` enable the features for PV domains or HVM domains
> > + other than the hardware domain, or for the hardware domain, PV or PVH. The
> > + plain boolean does all three: `asi=1,dom0=0` leaves the hardware domain
> > + out. The forms naming no domain type (an un-suffixed `vcpu-pt=<bool>`)
> > + cover the hardware domain too.
> > +
> > +**WARNING: manual de-selection of enabled options will invalidate any
> > +protection offered by the feature. The fine grained options provided below
> > +are meant to be used for debugging purposes only.**
> > +
> > +* `vcpu-pt` gives each vCPU its own per-domain area: the per-domain slot of
> > + each vCPU's monitor table maps a region private to that vCPU.
> > +
> > ### asid (x86)
> > > `= <boolean>`
> >
> > diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
> > index 212daae1f5..12f79feb42 100644
> > --- a/xen/arch/x86/include/asm/domain.h
> > +++ b/xen/arch/x86/include/asm/domain.h
> > @@ -474,6 +474,12 @@ struct arch_domain
> > /* Don't unconditionally inject #GP for unhandled MSRs. */
> > bool msr_relaxed;
> >
> > + /* Fine grained ASI related options. */
> > + struct {
> > + /* Give each vCPU its own per-domain area (HVM only). */
>
> I would drop the HVM only part, it's a transient implementation
> limitation, not a design one.
Same idea: It's true as of this patch, and it's likely to be true in
the next release. Better to document it rather than have someone
search down the implementation and determine that it's ignored for PV.
> > @@ -487,6 +491,69 @@ static int __init cf_check parse_pv_l1tf(const char *s)
> > }
> > custom_param("pv-l1tf", parse_pv_l1tf);
> >
> > +static int __init cf_check parse_asi(const char *s)
> > +{
> > + const char *ss;
> > + int val, rc = 0;
> > +
> > + /* Interpret 'asi' alone in its positive boolean form. */
> > + if ( *s == '\0' )
> > + opt_pvh_vcpu_pt_hwdom = opt_vcpu_pt_hvm = true;
> > +
> > + do {
> > + ss = strchr(s, ',');
> > + if ( !ss )
> > + ss = strchr(s, '\0');
> > +
> > + val = parse_bool(s, ss);
> > + switch ( val )
> > + {
> > + case 0:
> > + case 1:
> > + opt_pvh_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
> > + break;
> > +
> > + default:
> > + if ( (val = parse_boolean("pv", s, ss)) >= 0 )
> > + /* PV domains are unaffected. */;
>
> You might want to print a message saying that the option is ignored
> for PV? It will go into the console buffer at least.
Yes, I think so.
> > @@ -680,6 +747,13 @@ static void __init print_details(enum ind_thunk thunk)
> > opt_pv_l1tf_hwdom ? "enabled" : "disabled",
> > opt_pv_l1tf_domu ? "enabled" : "disabled");
> > #endif
> > +
> > + printk(" ASI features for Dom0:%s\n",
> > + opt_dom0_pvh && opt_pvh_vcpu_pt_hwdom ? " vCPU-PT" : " None");
> > +#ifdef CONFIG_HVM
> > + printk(" ASI features for HVM VMs:%s\n",
> > + opt_vcpu_pt_hvm ? " vCPU-PT" : " None");
> > +#endif
>
> opt_vcpu_pt_hvm is defined unconditionally, so you might as well use:
>
> if ( IS_ENABLED(CONFIG_HVM) )
> printk(" ASI features for HVM VMs:%s\n", ...
>
> Which is a bit nicer than the ifdefs?
Ack
> > @@ -2158,6 +2235,11 @@ void __init init_speculation_mitigations(void)
> > hw_smt_enabled && default_xen_spec_ctrl )
> > setup_force_cpu_cap(X86_FEATURE_SC_MSR_IDLE);
> >
> > + if ( opt_pvh_vcpu_pt_hwdom || opt_vcpu_pt_hvm )
> > + warning_add(
> > + "Address Space Isolation is not functional, this option is\n"
> > + "intended to be used only for development purposes.\n");
>
> I wonder, could we add the warning in parse_asi() itself? I know if
> would then also trigger for asi=0, but would avoid having to remember
> to expand the condition here every time a new option is added?
I'm OK with that.
-George
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v4 08/30] x86/mm: have init_xen_l4_slots() take a vCPU in place of the domain
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (6 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 07/30] x86/spec-ctrl: introduce Address Space Isolation command line option George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 09/30] x86/hvm: introduce per-vCPU L3 page-table George Dunlap
` (21 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie,
Tim Deegan
A later change gives the vCPUs of some domains per-domain areas of their
own, and slot 260 of a table private to one vCPU must then hold that
vCPU's L3. init_xen_l4_slots(), which fills slot 260, needs the vCPU
for that.
Have it take a vCPU in place of the domain. A table private to one vCPU
passes that vCPU; tables the domain's vCPUs share -- promoted PV guest
L4s, PV shadow L4s, and the PV dom0 L4 under construction -- pass any of
them. promote_l4_table() passes vCPU 0, whose existence it already
checks, and its call is reflowed.
No functional change: slot 260 still gets the domain's per-domain L3.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- New in this version, split out of "x86/hvm: introduce per-vCPU L3
page-table": init_xen_l4_slots() takes a vCPU in place of the domain
(Roger, Jan).
- Reflow promote_l4_table()'s call to init_xen_l4_slots() (Jan).
---
xen/arch/x86/include/asm/mm.h | 2 +-
xen/arch/x86/mm.c | 16 +++++++++++-----
xen/arch/x86/mm/hap/hap.c | 2 +-
xen/arch/x86/mm/shadow/hvm.c | 2 +-
xen/arch/x86/mm/shadow/multi.c | 2 +-
xen/arch/x86/pv/dom0_build.c | 2 +-
xen/arch/x86/pv/domain.c | 2 +-
7 files changed, 17 insertions(+), 11 deletions(-)
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index c9085214fd..de45a111f3 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -370,7 +370,7 @@ int devalidate_page(struct page_info *page, unsigned long type,
void init_xen_pae_l2_slots(l2_pgentry_t *l2t, const struct domain *d);
void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
- const struct domain *d, mfn_t sl4mfn, bool ro_mpt);
+ const struct vcpu *v, mfn_t sl4mfn, bool ro_mpt);
bool fill_ro_mpt(mfn_t mfn);
void zap_ro_mpt(mfn_t mfn);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index c6bd53354b..ac074cc62d 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -1687,14 +1687,18 @@ static int promote_l3_table(struct page_info *page)
* This function must write all ROOT_PAGETABLE_PV_XEN_SLOTS, to clobber any
* values a guest may have left there from promote_l4_table().
*
- * l4t, l4mfn, and d are mandatory, but l4mfn doesn't need to be the mfn under
+ * l4t, l4mfn, and v are mandatory, but l4mfn doesn't need to be the mfn under
* *l4t. All other parameters are optional and will either fill or zero the
* appropriate slots. Pagetables not shared with guests will gain the
* extended directmap.
+ *
+ * v is the vCPU the table is private to or, for a table the domain's vCPUs
+ * share, any of them.
*/
void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
- const struct domain *d, mfn_t sl4mfn, bool ro_mpt)
+ const struct vcpu *v, mfn_t sl4mfn, bool ro_mpt)
{
+ const struct domain *d = v->domain;
/*
* PV vcpus need a shortened directmap. HVM and Idle vcpus get the full
* directmap.
@@ -1797,6 +1801,8 @@ void zap_ro_mpt(mfn_t mfn)
static int promote_l4_table(struct page_info *page)
{
struct domain *d = page_get_owner(page);
+ /* Any vCPU will do: a PV domain's vCPUs all use the domain-wide L3. */
+ const struct vcpu *v = domain_vcpu(d, 0);
mfn_t l4mfn = page_to_mfn(page);
l4_pgentry_t *pl4e;
unsigned int i;
@@ -1808,7 +1814,7 @@ static int promote_l4_table(struct page_info *page)
* PV domain may get that only with its first vCPU. The domain has no
* vCPUs before XEN_DOMCTL_max_vcpus.
*/
- if ( !domain_vcpu(d, 0) )
+ if ( !v )
return -EINVAL;
pl4e = map_domain_page(l4mfn);
@@ -1887,8 +1893,8 @@ static int promote_l4_table(struct page_info *page)
if ( !rc )
{
- init_xen_l4_slots(pl4e, l4mfn,
- d, INVALID_MFN, VM_ASSIST(d, m2p_strict));
+ init_xen_l4_slots(pl4e, l4mfn, v, INVALID_MFN,
+ VM_ASSIST(d, m2p_strict));
atomic_inc(&d->arch.pv.nr_l4_pages);
}
unmap_domain_page(pl4e);
diff --git a/xen/arch/x86/mm/hap/hap.c b/xen/arch/x86/mm/hap/hap.c
index 0ede4181a0..aba4f77df9 100644
--- a/xen/arch/x86/mm/hap/hap.c
+++ b/xen/arch/x86/mm/hap/hap.c
@@ -407,7 +407,7 @@ static mfn_t hap_make_monitor_table(struct vcpu *v)
m4mfn = page_to_mfn(pg);
l4e = map_domain_page(m4mfn);
- init_xen_l4_slots(l4e, m4mfn, d, INVALID_MFN, false);
+ init_xen_l4_slots(l4e, m4mfn, v, INVALID_MFN, false);
unmap_domain_page(l4e);
return m4mfn;
diff --git a/xen/arch/x86/mm/shadow/hvm.c b/xen/arch/x86/mm/shadow/hvm.c
index e6fb97c4b6..0b23326214 100644
--- a/xen/arch/x86/mm/shadow/hvm.c
+++ b/xen/arch/x86/mm/shadow/hvm.c
@@ -760,7 +760,7 @@ mfn_t sh_make_monitor_table(const struct vcpu *v, unsigned int shadow_levels)
* shadow-linear mapping will either be inserted below when creating
* lower level monitor tables, or later in sh_update_cr3().
*/
- init_xen_l4_slots(l4e, m4mfn, d, INVALID_MFN, false);
+ init_xen_l4_slots(l4e, m4mfn, v, INVALID_MFN, false);
if ( shadow_levels < 4 )
{
diff --git a/xen/arch/x86/mm/shadow/multi.c b/xen/arch/x86/mm/shadow/multi.c
index 7f72b4b7b2..ebbf4a12ce 100644
--- a/xen/arch/x86/mm/shadow/multi.c
+++ b/xen/arch/x86/mm/shadow/multi.c
@@ -974,7 +974,7 @@ sh_make_shadow(struct vcpu *v, mfn_t gmfn, u32 shadow_type)
BUILD_BUG_ON(sizeof(l4_pgentry_t) != sizeof(shadow_l4e_t));
- init_xen_l4_slots(l4t, gmfn, d, smfn, (!is_pv_32bit_domain(d) &&
+ init_xen_l4_slots(l4t, gmfn, v, smfn, (!is_pv_32bit_domain(d) &&
VM_ASSIST(d, m2p_strict)));
unmap_domain_page(l4t);
}
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index 24f2fa0b7f..dcb1d407ec 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -725,7 +725,7 @@ static int __init dom0_construct(const struct boot_domain *bd)
l4start = l4tab = __va(mpt_alloc); mpt_alloc += PAGE_SIZE;
clear_page(l4tab);
init_xen_l4_slots(l4tab, _mfn(virt_to_mfn(l4start)),
- d, INVALID_MFN, true);
+ v, INVALID_MFN, true);
v->arch.guest_table = pagetable_from_paddr(__pa(l4start));
}
else
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index f3c711b85b..f139d8f83c 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -127,7 +127,7 @@ static int setup_compat_l4(struct vcpu *v)
mfn = page_to_mfn(pg);
l4tab = map_domain_page(mfn);
clear_page(l4tab);
- init_xen_l4_slots(l4tab, mfn, v->domain, INVALID_MFN, false);
+ init_xen_l4_slots(l4tab, mfn, v, INVALID_MFN, false);
unmap_domain_page(l4tab);
/* This page needs to look like a pagetable so that it can be shadowed */
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 09/30] x86/hvm: introduce per-vCPU L3 page-table
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (7 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 08/30] x86/mm: have init_xen_l4_slots() take a vCPU in place of the domain George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 10/30] x86/domain_page: drop redundant is_pv_vcpu() checks George Dunlap
` (20 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
The per-domain area is currently a single domain-wide structure: one L3,
referenced from every root page-table associated with the domain, so
every mapping in it is visible to every vCPU of the domain. Its
contents are already laid out in per-vCPU slices (each vCPU's
COMPAT_ARG_XLAT pages, and for PV each vCPU's GDT/LDT window and
mapcache entries), but the visibility is domain-wide. Meanwhile, much
of the per-vCPU state Xen maintains -- the VMCB, the VMX MSR load/save
areas, FPU/XSAVE state -- lives in always-mapped memory.
Allow the "per-domain" area to be per-vCPU instead ("VCPU-PT") for HVM
domains. HVM monitor tables are already per-vCPU, so this needs nothing
at context switch: every monitor table carries its vCPU's own L3 from
creation, and the per-vCPU mappings are isolated from other running
vCPUs at once. We will later build on this, adding a per-vCPU mapcache
and per-vCPU mapped areas, into which we can move the vCPU state
mentioned above.
Add pervcpu_l3_pg to the arch_vcpu struct, to correspond to the
perdomain_l3_pg in the domain struct. (We retain both so that we can
switch between per-domain and per-vCPU on a domain-by-domain basis; PV
domains always use the domain-wide one.)
In {create,destroy}_perdomain_mapping(), if d->arch.asi.vcpu_pt, use
pervcpu_l3_pg as the per-domain L3 (allocating it for a vCPU if it's
NULL, just as we allocate for a domain in !vcpu_pt mode); otherwise,
use perdomain_l3_pg. Introduce a helper, perdomain_l3(), to
consistently choose the correct one.
Introduce free_pervcpu_mappings() to free this tree, called from
arch_vcpu_destroy() on normal teardown. Since the vcpu structure holds
the only reference to pervcpu_l3_pg, arch_vcpu_create() must also call
it on its error paths: nothing else records the allocation once the
vcpu struct is torn down. The domain-wide free_perdomain_mappings()
keeps covering non-vCPU-PT domains.
init_xen_l4_slots() is given the vCPU a table is private to or, for a
table the domain's vCPUs share, any of them. Have it install that
vCPU's perdomain_l3() in slot 260: the tables a domain's vCPUs share are
PV ones, and a PV domain's vCPUs all use the domain-wide L3.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-fable-5-1, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- Rebase over earlier changes
- d->arch.vcpu_pt is now d->arch.asi.vcpu_pt.
- init_xen_l4_slots() taking a vCPU in place of the domain, and
promote_l4_table() failing while the domain has none, are now patches
of their own (Roger, Jan).
- Drop the fail_early label: every cleanup step after fail: is safe
before its setup has run (Roger).
- free_pervcpu_mappings(): clear the pointer, then free, as XFREE()
does (Roger).
- Likewise in free_perdomain_mappings().
Changes in v3:
- HVM only: PV domains keep the domain-wide area, so the PV parts go
(the single-vCPU limit, the shadow refusals, the XPTI slot install).
init_xen_l4_slots() takes the vCPU as an extra, optional argument
rather than in place of the domain, so PV guest L4s, which no vCPU
owns, are promoted exactly as before.
- Retitle (was: "x86/mm: introduce per-vCPU L3 page-table").
- free_compat_arg_xlat()'s comment names free_pervcpu_mappings() as the
teardown for a vCPU-PT domain.
Changes in v2:
- Added to the series
Changes since the previously posted version:
- Expand the commit message with the motivation and the design
rationale.
- Keep domain-wide freeing intact and introduce a vCPU-scoped
free_pervcpu_mappings() instead of re-scoping
free_perdomain_mappings(); fix the arch_vcpu_create() error-path
leaks of a partially built per-vCPU hierarchy.
- Add a perdomain_l3() helper for the root selection, rather than
open-coding the vcpu_pt choice (and testing both root pointers) at
each site.
---
xen/arch/x86/domain.c | 9 +++--
xen/arch/x86/include/asm/domain.h | 4 ++
xen/arch/x86/include/asm/mm.h | 2 +
xen/arch/x86/mm.c | 66 +++++++++++++++++++++++--------
4 files changed, 61 insertions(+), 20 deletions(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 94a1ea904a..7a4da203d4 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -513,7 +513,7 @@ int arch_vcpu_create(struct vcpu *v)
rc = mapcache_vcpu_init(v);
if ( rc )
- return rc;
+ goto fail;
if ( !is_idle_domain(d) )
{
@@ -523,12 +523,12 @@ int arch_vcpu_create(struct vcpu *v)
*/
rc = create_perdomain_mapping(v, PERDOMAIN_VIRT_START, 0, NULL, NULL);
if ( rc )
- return rc;
+ goto fail;
paging_vcpu_init(v);
if ( (rc = vcpu_init_fpu(v)) != 0 )
- return rc;
+ goto fail;
vmce_init_vcpu(v);
@@ -576,6 +576,7 @@ int arch_vcpu_create(struct vcpu *v)
vcpu_destroy_fpu(v);
xfree(v->arch.msrs);
v->arch.msrs = NULL;
+ free_pervcpu_mappings(v);
return rc;
}
@@ -596,6 +597,8 @@ void arch_vcpu_destroy(struct vcpu *v)
pv_vcpu_destroy(v);
else
ASSERT_UNREACHABLE();
+
+ free_pervcpu_mappings(v);
}
int arch_sanitise_domain_config(const struct xen_domctl_createdomain *config)
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 12f79feb42..4140a92f15 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -332,6 +332,7 @@ struct monitor_write_data {
struct arch_domain
{
+ /* Domain-wide L3 for the per-domain slot; NULL in a vCPU-PT domain. */
struct page_info *perdomain_l3_pg;
/* I/O-port admin-specified access capabilities. */
@@ -688,6 +689,9 @@ struct arch_vcpu
struct vcpu_msrs *msrs;
+ /* Per-vCPU L3 for the per-domain slot; NULL in a non-vCPU-PT domain. */
+ struct page_info *pervcpu_l3_pg;
+
struct {
bool next_interrupt_enabled;
} monitor;
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index de45a111f3..12d1c06c12 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -369,6 +369,7 @@ int devalidate_page(struct page_info *page, unsigned long type,
int preemptible);
void init_xen_pae_l2_slots(l2_pgentry_t *l2t, const struct domain *d);
+struct page_info *perdomain_l3(const struct vcpu *v);
void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
const struct vcpu *v, mfn_t sl4mfn, bool ro_mpt);
bool fill_ro_mpt(mfn_t mfn);
@@ -612,6 +613,7 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr);
void free_perdomain_mappings(struct domain *d);
+void free_pervcpu_mappings(struct vcpu *v);
void __iomem *ioremap_wc(paddr_t pa, size_t len);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index ac074cc62d..362a88a300 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -1681,6 +1681,18 @@ static int promote_l3_table(struct page_info *page)
}
#endif /* CONFIG_PV */
+/*
+ * The root of the per-domain area in use by @v: the vCPU's own L3 for a
+ * vCPU-PT domain, the domain-wide one otherwise.
+ */
+struct page_info *perdomain_l3(const struct vcpu *v)
+{
+ const struct domain *d = v->domain;
+
+ return d->arch.asi.vcpu_pt ? v->arch.pervcpu_l3_pg
+ : d->arch.perdomain_l3_pg;
+}
+
/*
* Fill an L4 with Xen entries.
*
@@ -1693,7 +1705,8 @@ static int promote_l3_table(struct page_info *page)
* extended directmap.
*
* v is the vCPU the table is private to or, for a table the domain's vCPUs
- * share, any of them.
+ * share, any of them: slot 260 gets v's per-domain area (perdomain_l3()),
+ * which in a domain whose vCPUs share tables (PV) is the domain-wide one.
*/
void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
const struct vcpu *v, mfn_t sl4mfn, bool ro_mpt)
@@ -1726,7 +1739,7 @@ void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
/* Slot 260: Per-domain mappings. */
l4t[l4_table_offset(PERDOMAIN_VIRT_START)] =
- l4e_from_page(d->arch.perdomain_l3_pg, __PAGE_HYPERVISOR_RW);
+ l4e_from_page(perdomain_l3(v), __PAGE_HYPERVISOR_RW);
/* Slot 4: Per-domain mappings mirror. */
BUILD_BUG_ON(IS_ENABLED(CONFIG_PV32) &&
@@ -6249,7 +6262,7 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
struct page_info **ppg)
{
struct domain *d = v->domain;
- struct page_info *pg;
+ struct page_info *pg, *l3_pg = perdomain_l3(v);
l3_pgentry_t *l3tab;
l2_pgentry_t *l2tab;
l1_pgentry_t *l1tab;
@@ -6258,14 +6271,17 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
ASSERT(va >= PERDOMAIN_VIRT_START &&
va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
- if ( !d->arch.perdomain_l3_pg )
+ if ( !l3_pg )
{
pg = alloc_domheap_page(d, MEMF_no_owner);
if ( !pg )
return -ENOMEM;
l3tab = __map_domain_page(pg);
clear_page(l3tab);
- d->arch.perdomain_l3_pg = pg;
+ if ( d->arch.asi.vcpu_pt )
+ v->arch.pervcpu_l3_pg = pg;
+ else
+ d->arch.perdomain_l3_pg = pg;
if ( !nr )
{
unmap_domain_page(l3tab);
@@ -6275,7 +6291,7 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
else if ( !nr )
return 0;
else
- l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
+ l3tab = __map_domain_page(l3_pg);
ASSERT(!l3_table_offset(va ^ (va + nr * PAGE_SIZE - 1)));
@@ -6372,16 +6388,16 @@ void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr)
{
const l3_pgentry_t *l3tab, *pl3e;
- const struct domain *d = v->domain;
+ struct page_info *l3_pg = perdomain_l3(v);
ASSERT(va >= PERDOMAIN_VIRT_START &&
va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
ASSERT(!nr || !l3_table_offset(va ^ (va + nr * PAGE_SIZE - 1)));
- if ( !d->arch.perdomain_l3_pg )
+ if ( !l3_pg )
return;
- l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
+ l3tab = __map_domain_page(l3_pg);
pl3e = l3tab + l3_table_offset(va);
if ( l3e_get_flags(*pl3e) & _PAGE_PRESENT )
@@ -6420,16 +6436,11 @@ void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unmap_domain_page(l3tab);
}
-void free_perdomain_mappings(struct domain *d)
+static void free_perdomain_l3(struct page_info *l3pg)
{
- l3_pgentry_t *l3tab;
+ l3_pgentry_t *l3tab = __map_domain_page(l3pg);
unsigned int i;
- if ( !d->arch.perdomain_l3_pg )
- return;
-
- l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
-
for ( i = 0; i < PERDOMAIN_SLOTS; ++i)
if ( l3e_get_flags(l3tab[i]) & _PAGE_PRESENT )
{
@@ -6465,8 +6476,29 @@ void free_perdomain_mappings(struct domain *d)
}
unmap_domain_page(l3tab);
- free_domheap_page(d->arch.perdomain_l3_pg);
+ free_domheap_page(l3pg);
+}
+
+void free_perdomain_mappings(struct domain *d)
+{
+ struct page_info *l3pg = d->arch.perdomain_l3_pg;
+
+ if ( !l3pg )
+ return;
+
d->arch.perdomain_l3_pg = NULL;
+ free_perdomain_l3(l3pg);
+}
+
+void free_pervcpu_mappings(struct vcpu *v)
+{
+ struct page_info *l3pg = v->arch.pervcpu_l3_pg;
+
+ if ( !l3pg )
+ return;
+
+ v->arch.pervcpu_l3_pg = NULL;
+ free_perdomain_l3(l3pg);
}
static void write_sss_token(unsigned long *ptr)
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 10/30] x86/domain_page: drop redundant is_pv_vcpu() checks
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (8 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 09/30] x86/hvm: introduce per-vCPU L3 page-table George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 11/30] x86/domain_page: prepare the mapcache for per-vCPU accounting George Dunlap
` (19 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
The is_pv_vcpu() checks after mapcache_current_vcpu() never fail: they
have been redundant since 622c9a5ba95d ("x86/mm: accurately track
which vCPU page-tables are loaded") made mapcache_current_vcpu() do
the check itself. mapcache_current_vcpu() is now guaranteed to return
a PV vCPU with a caching structure (which is what is needed for the
subsequent dcache->inuse check): either a scheduled PV vCPU, or this
CPU's idle vCPU (possibly after finishing a lazy context switch).
Drop the checks; their speculation barriers (evaluate_nospec()) go
too.
mapcache_current_vcpu() returns NULL if this CPU's pgtable_vcpu is not
yet set (early boot) or is not a PV vCPU. Otherwise it reads
pgtable_vcpu again and returns that. In between, pgtable_vcpu changes
only when a lazy context switch from a PV vCPU is completed, by
mapcache_current_vcpu() itself (sync_local_execstate()) or first by an
interrupt such as a FLUSH_VCPU_STATE IPI; completing it loads the idle
vCPU's page-tables and sets pgtable_vcpu to the idle vCPU. No other
writer of pgtable_vcpu runs in an interrupt that returns to the code it
interrupted. During an EFI runtime call pgtable_vcpu is the idle
vCPU: efi_rs_enter() completes any lazy switch, then loads EFI's
page-tables, which carry the directmap. So the vCPU returned is either
the PV vCPU checked or this CPU's idle vCPU.
The idle vCPU passes is_pv_vcpu(), as the idle domain is created
without XEN_DOMCTL_CDF_hvm, but its mapcache is never set up:
domain_create() returns for the idle domain before calling
arch_domain_create(), which is what calls mapcache_domain_init(). With
its dcache->inuse NULL, map_domain_page() uses the directmap, and
unmap_domain_page()'s ASSERT() on dcache->inuse still catches a
mapcache address unmapped with no mapcache in use. In builds without
CONFIG_PV, is_pv_vcpu() is always false and mapcache_current_vcpu()
always returns NULL.
No functional change.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- New in this version, split out of "x86/domain_page: prepare the
mapcache for per-vCPU accounting", with a justification of its own.
NB I've included Claude's complete analysis in the commit message.
I've reworded the commit message so that the middle two paragraphs can
be dropped, or reworded more briefly on commit.
---
xen/arch/x86/domain_page.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index f3e65e2d44..ed68483695 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -74,7 +74,7 @@ void *map_domain_page(mfn_t mfn)
#endif
v = mapcache_current_vcpu();
- if ( !v || !is_pv_vcpu(v) )
+ if ( !v )
return mfn_to_virt(mfn_x(mfn));
dcache = &v->domain->arch.pv.mapcache;
@@ -179,7 +179,7 @@ void unmap_domain_page(const void *ptr)
ASSERT(va >= MAPCACHE_VIRT_START && va < MAPCACHE_VIRT_END);
v = mapcache_current_vcpu();
- ASSERT(v && is_pv_vcpu(v));
+ ASSERT(v);
dcache = &v->domain->arch.pv.mapcache;
ASSERT(dcache->inuse);
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 11/30] x86/domain_page: prepare the mapcache for per-vCPU accounting
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (9 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 10/30] x86/domain_page: drop redundant is_pv_vcpu() checks George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 12/30] x86/domain_page: move struct mapcache_vcpu to struct arch_vcpu George Dunlap
` (18 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
Currently, only PV domains have a mapcache, which is domain-wide one.
In preparation for enabling per-vCPU mapcaches for HVM domains, split
the accounting out into a struct mapcache, embedded in struct
mapcache_domain.
Introduce has_mapcache(), for whether a vCPU has a mapcache, and
vcpu_mapcache(), which returns the accounting for a vCPU's mapcache
and, in *dcache, the domain-wide mapcache whose lock and TLB epoch the
caller has to take into account. For now has_mapcache() is
is_pv_vcpu(), and vcpu_mapcache() always returns the domain's
accounting.
Move map_domain_page()'s handling of the domain-wide lock and TLB epoch
into three helpers: mapcache_enter() takes the lock and catches up with
a wrap another vCPU caused, mapcache_new_epoch() starts a new epoch
after a wrap, and mapcache_exit() drops the lock. A per-vCPU
mapcache can then skip them without map_domain_page() being
re-indented.
struct mapcache_domain keeps its size (40 bytes, x86_64 debug build),
and so do the structures containing it.
No functional change.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- Reword has_mapcache()'s comment (Roger).
- Move the comment on the domain-wide lock and TLB epoch into the
function body (Roger).
- Rename mapcache_lock() / mapcache_unlock() to mapcache_enter() /
mapcache_exit() (Roger).
- mapcache_new_epoch(): assert that the domain-wide lock is held
(Roger).
- mapcache_vcpu_init(): drop the comment on the early return for a
disabled domain-wide mapcache (Jan).
- mapcache_vcpu_init(): initialise ents and nr at their declarations,
as before (Jan).
- Dropping the redundant is_pv_vcpu() checks in map_domain_page() and
unmap_domain_page() is now a patch of its own (Jan).
Changes in v3:
- New in this version, split out of "x86/hvm: introduce per-vCPU
mapcache for vCPU-PT domains" for review.
---
xen/arch/x86/domain_page.c | 137 ++++++++++++++++++++----------
xen/arch/x86/include/asm/domain.h | 19 +++--
2 files changed, 102 insertions(+), 54 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index ed68483695..e4ea7a254f 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -18,17 +18,19 @@
#include <asm/hardirq.h>
#include <asm/setup.h>
+/* Whether the vCPU supports mapcache usage, might not be in use. */
+static bool has_mapcache(const struct vcpu *v)
+{
+ return is_pv_vcpu(v);
+}
+
static inline struct vcpu *mapcache_current_vcpu(void)
{
const struct vcpu *v = this_cpu(pgtable_vcpu);
const struct vcpu *curr = current;
- /*
- * During early boot pgtable_vcpu is not set, callers must handle NULL.
- * Non-PV domains don't have a mapcache, the directmap covers all physical
- * address space.
- */
- if ( !v || !is_pv_vcpu(v) )
+ /* During early boot pgtable_vcpu is not set, callers must handle NULL. */
+ if ( !v || !has_mapcache(v) )
return NULL;
/*
@@ -54,6 +56,52 @@ static inline struct vcpu *mapcache_current_vcpu(void)
return ACCESS_ONCE(this_cpu(pgtable_vcpu));
}
+/*
+ * The accounting for v's mapcache: the domain's, with *dcache set so the
+ * caller takes the domain-wide lock and TLB-epoch machinery into account.
+ */
+static struct mapcache *vcpu_mapcache(struct vcpu *v,
+ struct mapcache_domain **dcache)
+{
+ struct domain *d = v->domain;
+
+ *dcache = &d->arch.pv.mapcache;
+ return &(*dcache)->cache;
+}
+
+static void mapcache_enter(struct mapcache_domain *dcache,
+ struct mapcache_vcpu *vcache)
+{
+ /* The domain's vCPUs share the mapcache's slots. */
+ spin_lock(&dcache->lock);
+
+ /* Has some other CPU caused a wrap? We must flush if so. */
+ if ( unlikely(dcache->epoch != vcache->shadow_epoch) )
+ {
+ vcache->shadow_epoch = dcache->epoch;
+ if ( NEED_FLUSH(this_cpu(tlbflush_time), dcache->tlbflush_timestamp) )
+ {
+ perfc_incr(domain_page_tlb_flush);
+ flush_tlb_local();
+ }
+ }
+}
+
+/* A wrap has just flushed this CPU's TLB: start a new epoch. */
+static void mapcache_new_epoch(struct mapcache_domain *dcache,
+ struct mapcache_vcpu *vcache)
+{
+ ASSERT(spin_is_locked(&dcache->lock));
+
+ vcache->shadow_epoch = ++dcache->epoch;
+ dcache->tlbflush_timestamp = tlbflush_current_time();
+}
+
+static void mapcache_exit(struct mapcache_domain *dcache)
+{
+ spin_unlock(&dcache->lock);
+}
+
#define mapcache_l2_entry(e) ((e) >> PAGETABLE_ORDER)
#define MAPCACHE_L2_ENTRIES (mapcache_l2_entry(MAPCACHE_ENTRIES - 1) + 1)
#define MAPCACHE_L1ENT(idx) \
@@ -66,6 +114,7 @@ void *map_domain_page(mfn_t mfn)
struct vcpu *v;
struct mapcache_domain *dcache;
struct mapcache_vcpu *vcache;
+ struct mapcache *cache;
struct vcpu_maphash_entry *hashent;
#ifdef NDEBUG
@@ -77,9 +126,9 @@ void *map_domain_page(mfn_t mfn)
if ( !v )
return mfn_to_virt(mfn_x(mfn));
- dcache = &v->domain->arch.pv.mapcache;
+ cache = vcpu_mapcache(v, &dcache);
vcache = &v->arch.pv.mapcache;
- if ( !dcache->inuse )
+ if ( !cache->inuse )
return mfn_to_virt(mfn_x(mfn));
perfc_incr(map_domain_page_count);
@@ -90,41 +139,30 @@ void *map_domain_page(mfn_t mfn)
if ( hashent->mfn == mfn_x(mfn) )
{
idx = hashent->idx;
- ASSERT(idx < dcache->entries);
+ ASSERT(idx < cache->entries);
hashent->refcnt++;
ASSERT(hashent->refcnt);
ASSERT(mfn_eq(l1e_get_mfn(MAPCACHE_L1ENT(idx)), mfn));
goto out;
}
- spin_lock(&dcache->lock);
+ mapcache_enter(dcache, vcache);
- /* Has some other CPU caused a wrap? We must flush if so. */
- if ( unlikely(dcache->epoch != vcache->shadow_epoch) )
- {
- vcache->shadow_epoch = dcache->epoch;
- if ( NEED_FLUSH(this_cpu(tlbflush_time), dcache->tlbflush_timestamp) )
- {
- perfc_incr(domain_page_tlb_flush);
- flush_tlb_local();
- }
- }
-
- idx = find_next_zero_bit(dcache->inuse, dcache->entries, dcache->cursor);
- if ( unlikely(idx >= dcache->entries) )
+ idx = find_next_zero_bit(cache->inuse, cache->entries, cache->cursor);
+ if ( unlikely(idx >= cache->entries) )
{
unsigned long accum = 0, prev = 0;
/* /First/, clean the garbage map and update the inuse list. */
- for ( i = 0; i < BITS_TO_LONGS(dcache->entries); i++ )
+ for ( i = 0; i < BITS_TO_LONGS(cache->entries); i++ )
{
accum |= prev;
- dcache->inuse[i] &= ~xchg(&dcache->garbage[i], 0);
- prev = ~dcache->inuse[i];
+ cache->inuse[i] &= ~xchg(&cache->garbage[i], 0);
+ prev = ~cache->inuse[i];
}
- if ( accum | (prev & BITMAP_LAST_WORD_MASK(dcache->entries)) )
- idx = find_first_zero_bit(dcache->inuse, dcache->entries);
+ if ( accum | (prev & BITMAP_LAST_WORD_MASK(cache->entries)) )
+ idx = find_first_zero_bit(cache->inuse, cache->entries);
else
{
/* Replace a hash entry instead. */
@@ -144,19 +182,18 @@ void *map_domain_page(mfn_t mfn)
i = 0;
} while ( i != MAPHASH_HASHFN(mfn_x(mfn)) );
}
- BUG_ON(idx >= dcache->entries);
+ BUG_ON(idx >= cache->entries);
/* /Second/, flush TLBs. */
perfc_incr(domain_page_tlb_flush);
flush_tlb_local();
- vcache->shadow_epoch = ++dcache->epoch;
- dcache->tlbflush_timestamp = tlbflush_current_time();
+ mapcache_new_epoch(dcache, vcache);
}
- set_bit(idx, dcache->inuse);
- dcache->cursor = idx + 1;
+ set_bit(idx, cache->inuse);
+ cache->cursor = idx + 1;
- spin_unlock(&dcache->lock);
+ mapcache_exit(dcache);
l1e_write(&MAPCACHE_L1ENT(idx), l1e_from_mfn(mfn, __PAGE_HYPERVISOR_RW));
@@ -170,6 +207,7 @@ void unmap_domain_page(const void *ptr)
unsigned int idx;
struct vcpu *v;
struct mapcache_domain *dcache;
+ struct mapcache *cache;
unsigned long va = (unsigned long)ptr, mfn, flags;
struct vcpu_maphash_entry *hashent;
@@ -181,8 +219,8 @@ void unmap_domain_page(const void *ptr)
v = mapcache_current_vcpu();
ASSERT(v);
- dcache = &v->domain->arch.pv.mapcache;
- ASSERT(dcache->inuse);
+ cache = vcpu_mapcache(v, &dcache);
+ ASSERT(cache->inuse);
idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
mfn = l1e_get_pfn(MAPCACHE_L1ENT(idx));
@@ -205,7 +243,7 @@ void unmap_domain_page(const void *ptr)
hashent->mfn);
l1e_write(&MAPCACHE_L1ENT(hashent->idx), l1e_empty());
/* /Second/, mark as garbage. */
- set_bit(hashent->idx, dcache->garbage);
+ set_bit(hashent->idx, cache->garbage);
}
/* Add newly-freed mapping to the maphash. */
@@ -217,7 +255,7 @@ void unmap_domain_page(const void *ptr)
/* /First/, zap the PTE. */
l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
/* /Second/, mark as garbage. */
- set_bit(idx, dcache->garbage);
+ set_bit(idx, cache->garbage);
}
local_irq_restore(flags);
@@ -239,9 +277,9 @@ void mapcache_domain_init(struct domain *d)
2 * PFN_UP(BITS_TO_LONGS(MAPCACHE_ENTRIES) * sizeof(long))) >
MAPCACHE_VIRT_START + (PERDOMAIN_SLOT_MBYTES << 20));
bitmap_pages = PFN_UP(BITS_TO_LONGS(MAPCACHE_ENTRIES) * sizeof(long));
- dcache->inuse = (void *)MAPCACHE_VIRT_END + PAGE_SIZE;
- dcache->garbage = dcache->inuse +
- (bitmap_pages + 1) * PAGE_SIZE / sizeof(long);
+ dcache->cache.inuse = (void *)MAPCACHE_VIRT_END + PAGE_SIZE;
+ dcache->cache.garbage = dcache->cache.inuse +
+ (bitmap_pages + 1) * PAGE_SIZE / sizeof(long);
spin_lock_init(&dcache->lock);
}
@@ -249,15 +287,20 @@ void mapcache_domain_init(struct domain *d)
int mapcache_vcpu_init(struct vcpu *v)
{
struct domain *d = v->domain;
- struct mapcache_domain *dcache = &d->arch.pv.mapcache;
+ struct mapcache_domain *dcache;
+ struct mapcache *cache;
unsigned long i;
unsigned int ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
unsigned int nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
- if ( !is_pv_vcpu(v) || !dcache->inuse )
+ if ( !has_mapcache(v) )
+ return 0;
+
+ cache = vcpu_mapcache(v, &dcache);
+ if ( !cache->inuse )
return 0;
- if ( ents > dcache->entries )
+ if ( ents > cache->entries )
{
/* Populate page tables. */
int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START, ents,
@@ -265,16 +308,16 @@ int mapcache_vcpu_init(struct vcpu *v)
/* Populate bit maps. */
if ( !rc )
- rc = create_perdomain_mapping(v, (unsigned long)dcache->inuse,
+ rc = create_perdomain_mapping(v, (unsigned long)cache->inuse,
nr, NULL, NIL(struct page_info *));
if ( !rc )
- rc = create_perdomain_mapping(v, (unsigned long)dcache->garbage,
+ rc = create_perdomain_mapping(v, (unsigned long)cache->garbage,
nr, NULL, NIL(struct page_info *));
if ( rc )
return rc;
- dcache->entries = ents;
+ cache->entries = ents;
}
/* Mark all maphash entries as not in use. */
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 4140a92f15..542c6b5eca 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -57,6 +57,16 @@ struct trap_bounce {
unsigned long eip;
};
+struct mapcache {
+ /* The number of array entries, and a cursor into the array. */
+ unsigned int entries;
+ unsigned int cursor;
+
+ /* Which mappings are in use, and which are garbage to reap next epoch? */
+ unsigned long *inuse;
+ unsigned long *garbage;
+};
+
#define MAPHASH_ENTRIES 8
#define MAPHASH_HASHFN(pfn) ((pfn) & (MAPHASH_ENTRIES-1))
#define MAPHASHENT_NOTINUSE ((u32)~0U)
@@ -73,10 +83,6 @@ struct mapcache_vcpu {
};
struct mapcache_domain {
- /* The number of array entries, and a cursor into the array. */
- unsigned int entries;
- unsigned int cursor;
-
/* Protects map_domain_page(). */
spinlock_t lock;
@@ -84,9 +90,8 @@ struct mapcache_domain {
unsigned int epoch;
u32 tlbflush_timestamp;
- /* Which mappings are in use, and which are garbage to reap next epoch? */
- unsigned long *inuse;
- unsigned long *garbage;
+ /* Accounting of the domain-wide mapcache. */
+ struct mapcache cache;
};
void mapcache_domain_init(struct domain *d);
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 12/30] x86/domain_page: move struct mapcache_vcpu to struct arch_vcpu
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (10 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 11/30] x86/domain_page: prepare the mapcache for per-vCPU accounting George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 13/30] x86/domain_page: don't unmap a mapcache entry that is not present George Dunlap
` (17 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Roger Pau Monné, Jan Beulich,
Andrew Cooper, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
struct mapcache_vcpu, the per-vCPU part of the mapcache (the maphash
and the vCPU's shadow of the domain's TLB epoch), lives in struct
pv_vcpu, as only PV vCPUs have a mapcache. In preparation for
implementing mapcaches for HVM domains, move it to struct arch_vcpu:
v->arch.pv.mapcache becomes v->arch.mapcache.
struct pv_vcpu shrinks from 512 to 376 bytes, and struct vcpu grows
from 2880 to 3008 bytes (x86_64, debug build), within its page: from
here on HVM vCPUs carry the 136-byte struct mapcache_vcpu too, ahead of
its use by those of vCPU-PT domains.
No functional change.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Roger Pau Monné <roger@xenproject.org>
---
Changes in v3:
- New in this version, split out of "x86/hvm: introduce per-vCPU
mapcache for vCPU-PT domains" for review.
---
xen/arch/x86/domain_page.c | 6 +++---
xen/arch/x86/include/asm/domain.h | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index e4ea7a254f..205d7b832a 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -127,7 +127,7 @@ void *map_domain_page(mfn_t mfn)
return mfn_to_virt(mfn_x(mfn));
cache = vcpu_mapcache(v, &dcache);
- vcache = &v->arch.pv.mapcache;
+ vcache = &v->arch.mapcache;
if ( !cache->inuse )
return mfn_to_virt(mfn_x(mfn));
@@ -224,7 +224,7 @@ void unmap_domain_page(const void *ptr)
idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
mfn = l1e_get_pfn(MAPCACHE_L1ENT(idx));
- hashent = &v->arch.pv.mapcache.hash[MAPHASH_HASHFN(mfn)];
+ hashent = &v->arch.mapcache.hash[MAPHASH_HASHFN(mfn)];
local_irq_save(flags);
@@ -324,7 +324,7 @@ int mapcache_vcpu_init(struct vcpu *v)
BUILD_BUG_ON(MAPHASHENT_NOTINUSE < MAPCACHE_ENTRIES);
for ( i = 0; i < MAPHASH_ENTRIES; i++ )
{
- struct vcpu_maphash_entry *hashent = &v->arch.pv.mapcache.hash[i];
+ struct vcpu_maphash_entry *hashent = &v->arch.mapcache.hash[i];
hashent->mfn = ~0UL; /* never valid to map */
hashent->idx = MAPHASHENT_NOTINUSE;
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 542c6b5eca..45d308407e 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -545,9 +545,6 @@ struct arch_domain
struct pv_vcpu
{
- /* map_domain_page() mapping cache. */
- struct mapcache_vcpu mapcache;
-
unsigned int vgc_flags;
struct trap_info *trap_ctxt;
@@ -643,6 +640,9 @@ struct arch_vcpu
#define async_exception_state(t) async_exception_state[(t)-1]
uint8_t async_exception_mask;
+ /* map_domain_page() mapping cache. */
+ struct mapcache_vcpu mapcache;
+
/* Virtual Machine Extensions */
union {
struct pv_vcpu pv;
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 13/30] x86/domain_page: don't unmap a mapcache entry that is not present
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (11 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 12/30] x86/domain_page: move struct mapcache_vcpu to struct arch_vcpu George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 14/30] x86/hvm: introduce per-vCPU mapcache for vCPU-PT domains George Dunlap
` (16 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel; +Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper, Teddy Astie
Mappings must be torn down by the vCPU that created them, and only
once. Have unmap_domain_page() check that the entry it is handed is
present and bail out otherwise (ASSERT_UNREACHABLE()), rather than read
an MFN out of an empty PTE and act on it, as a second unmap of an entry
the first one zapped would. (A second unmap of an entry the maphash
kept is caught, as before, only by the refcount ASSERT() in debug
builds.)
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Roger Pau Monné <roger@xenproject.org>
---
Changes in v4:
- Reword the comment on the new check (Roger).
Changes in v3:
- New in this version, split out of "x86/hvm: introduce per-vCPU
mapcache for vCPU-PT domains" for review.
---
xen/arch/x86/domain_page.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 205d7b832a..47ccefa4e9 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -223,6 +223,15 @@ void unmap_domain_page(const void *ptr)
ASSERT(cache->inuse);
idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
+ /*
+ * Sanity check: an entry that isn't present means a double or stray
+ * unmap. Play it safe: don't act on the MFN in an empty PTE.
+ */
+ if ( !(l1e_get_flags(MAPCACHE_L1ENT(idx)) & _PAGE_PRESENT) )
+ {
+ ASSERT_UNREACHABLE();
+ return;
+ }
mfn = l1e_get_pfn(MAPCACHE_L1ENT(idx));
hashent = &v->arch.mapcache.hash[MAPHASH_HASHFN(mfn)];
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 14/30] x86/hvm: introduce per-vCPU mapcache for vCPU-PT domains
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (12 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 13/30] x86/domain_page: don't unmap a mapcache entry that is not present George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 15/30] x86/mm: let Xen's page-table walker operate on a given root George Dunlap
` (15 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
Add a struct mapcache to struct mapcache_vcpu for the per-vCPU
accounting, with MAPCACHE_PERVCPU_ENTRIES (64) slots. Its inuse and
garbage bitmaps are then an unsigned long each: declare them inline
in struct mapcache_vcpu, rather than mapping them in the per-domain
area as the domain-wide bitmaps are.
A per-vCPU mapcache is only ever used by its own vCPU, and so by one
pCPU at a time. We can therefore skip both the lock and the TLB-epoch
tracking the domain-wide mapcache needs; a wrap still flushes the
local TLB.
struct mapcache_vcpu grows from 136 to 176 bytes, and struct vcpu from
3008 to 3072 bytes (x86_64, debug build), within its page.
Unlike a domain-wide mapcache, a per-vCPU one cannot, by design,
borrow slots from other vCPUs: MAPCACHE_PERVCPU_ENTRIES (64) is the
most single-page mappings such a vCPU can hold at once, interrupt
handlers' included, and one more hits the BUG_ON() in
map_domain_page(). The PoD zero-page sweep held up to 16 on top of
its callers' until "x86/PoD: map one page at a time in
p2m_pod_zero_check()". A domain-wide mapcache keeps
MAPCACHE_VCPU_ENTRIES (16) slots per vCPU; a per-vCPU one gets 64,
for a wider margin. The extra slots take no memory: the area is a
whole L1 page whatever the number of slots, and each bitmap is still
one unsigned long. Wraps, each of which flushes the local TLB, also
come less often.
mapcache_current_vcpu()'s lazy-context-switch handling
(sync_local_execstate() plus a pgtable_vcpu re-read) now applies to
any vCPU with a mapcache: when the idle vCPU runs lazily on an HVM
vCPU's monitor table, the full switch happens before the idle context
uses a mapcache.
In release builds map_domain_page() still tries the directmap first,
for any vCPU; debug builds exercise the new mapcache.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- d->arch.vcpu_pt is now d->arch.asi.vcpu_pt.
- mapcache_lock() / mapcache_unlock() are now mapcache_enter() /
mapcache_exit(). has_mapcache()'s comment no longer changes, and the
early return in mapcache_enter() has a one-line comment instead of a
sentence added to the block (Roger).
- mapcache_vcpu_init(): ents and nr become assignments here, ents now
depending on the kind of mapcache (Jan).
- A per-vCPU mapcache gets 64 slots (MAPCACHE_PERVCPU_ENTRIES) in
place of MAPCACHE_VCPU_ENTRIES (16), which still sizes the
domain-wide mapcache; its bitmaps, now full, are described as an
unsigned long each (Roger).
- Move the comment on why a per-vCPU mapcache needs only a local flush
from map_domain_page()'s TLB flush to mapcache_new_epoch()'s early
return, as one line (Roger).
Changes in v3:
- HVM only, and per-vCPU from the start: the HVM enablement of "x86:
Initialize the mapcache for PV, HVM, and idle domains" (from the
directmap removal series) is folded in, without the idle domain and
without a domain-wide HVM mapcache.
- Retitle (was: "x86/mm: introduce a per-vCPU mapcache when using ASI").
- Say which second unmaps the not-present check in unmap_domain_page()
catches: those of an entry the first unmap zapped.
- State the budget of single-page mappings a vCPU can hold at once.
- The mapcache accounting refactor, the move of struct mapcache_vcpu
to struct arch_vcpu and the not-present check in unmap_domain_page()
are split out, for review, into the three patches before this one.
---
xen/arch/x86/domain_page.c | 61 +++++++++++++++++++++++--------
xen/arch/x86/include/asm/config.h | 2 +
xen/arch/x86/include/asm/domain.h | 6 +++
3 files changed, 53 insertions(+), 16 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 47ccefa4e9..770fee80a4 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -21,7 +21,7 @@
/* Whether the vCPU supports mapcache usage, might not be in use. */
static bool has_mapcache(const struct vcpu *v)
{
- return is_pv_vcpu(v);
+ return is_pv_vcpu(v) || v->domain->arch.asi.vcpu_pt;
}
static inline struct vcpu *mapcache_current_vcpu(void)
@@ -34,10 +34,10 @@ static inline struct vcpu *mapcache_current_vcpu(void)
return NULL;
/*
- * If we are in a lazy context-switch state from a PV vCPU do a full switch
- * to the idle vCPU now, otherwise an incoming FLUSH_VCPU_STATE IPI would
- * change the page tables under our feet an invalidate any in-use mapcache
- * entries.
+ * If we are in a lazy context-switch state from a vCPU with a mapcache do
+ * a full switch to the idle vCPU now, otherwise an incoming
+ * FLUSH_VCPU_STATE IPI would change the page tables under our feet and
+ * invalidate any in-use mapcache entries.
*/
if ( unlikely(this_cpu(curr_vcpu) != curr) )
{
@@ -57,14 +57,21 @@ static inline struct vcpu *mapcache_current_vcpu(void)
}
/*
- * The accounting for v's mapcache: the domain's, with *dcache set so the
- * caller takes the domain-wide lock and TLB-epoch machinery into account.
+ * The accounting for v's mapcache: its own for a vCPU-PT domain, the domain's
+ * otherwise, in which case *dcache is set so the caller takes the domain-wide
+ * lock and TLB-epoch machinery into account.
*/
static struct mapcache *vcpu_mapcache(struct vcpu *v,
struct mapcache_domain **dcache)
{
struct domain *d = v->domain;
+ if ( d->arch.asi.vcpu_pt )
+ {
+ *dcache = NULL;
+ return &v->arch.mapcache.cache;
+ }
+
*dcache = &d->arch.pv.mapcache;
return &(*dcache)->cache;
}
@@ -72,6 +79,10 @@ static struct mapcache *vcpu_mapcache(struct vcpu *v,
static void mapcache_enter(struct mapcache_domain *dcache,
struct mapcache_vcpu *vcache)
{
+ /* Only its own vCPU uses a per-vCPU mapcache: no lock or epoch needed. */
+ if ( !dcache )
+ return;
+
/* The domain's vCPUs share the mapcache's slots. */
spin_lock(&dcache->lock);
@@ -91,6 +102,10 @@ static void mapcache_enter(struct mapcache_domain *dcache,
static void mapcache_new_epoch(struct mapcache_domain *dcache,
struct mapcache_vcpu *vcache)
{
+ /* Per-vCPU: no epoch, as page-table loads flush its non-global entries. */
+ if ( !dcache )
+ return;
+
ASSERT(spin_is_locked(&dcache->lock));
vcache->shadow_epoch = ++dcache->epoch;
@@ -99,7 +114,8 @@ static void mapcache_new_epoch(struct mapcache_domain *dcache,
static void mapcache_exit(struct mapcache_domain *dcache)
{
- spin_unlock(&dcache->lock);
+ if ( dcache )
+ spin_unlock(&dcache->lock);
}
#define mapcache_l2_entry(e) ((e) >> PAGETABLE_ORDER)
@@ -299,15 +315,25 @@ int mapcache_vcpu_init(struct vcpu *v)
struct mapcache_domain *dcache;
struct mapcache *cache;
unsigned long i;
- unsigned int ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
- unsigned int nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
+ unsigned int ents, nr;
if ( !has_mapcache(v) )
return 0;
cache = vcpu_mapcache(v, &dcache);
- if ( !cache->inuse )
+ if ( !dcache )
+ {
+ /* Per-vCPU bitmaps are an unsigned long each, in the vCPU itself. */
+ cache->inuse = v->arch.mapcache.inuse;
+ cache->garbage = v->arch.mapcache.garbage;
+ ents = MAPCACHE_PERVCPU_ENTRIES;
+ }
+ else if ( !cache->inuse )
return 0;
+ else
+ ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
+
+ nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
if ( ents > cache->entries )
{
@@ -315,13 +341,16 @@ int mapcache_vcpu_init(struct vcpu *v)
int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START, ents,
NIL(l1_pgentry_t *), NULL);
- /* Populate bit maps. */
- if ( !rc )
+ /* Populate the domain-wide bit maps (kept in the per-domain area). */
+ if ( !rc && dcache )
+ {
rc = create_perdomain_mapping(v, (unsigned long)cache->inuse,
nr, NULL, NIL(struct page_info *));
- if ( !rc )
- rc = create_perdomain_mapping(v, (unsigned long)cache->garbage,
- nr, NULL, NIL(struct page_info *));
+ if ( !rc )
+ rc = create_perdomain_mapping(v, (unsigned long)cache->garbage,
+ nr, NULL,
+ NIL(struct page_info *));
+ }
if ( rc )
return rc;
diff --git a/xen/arch/x86/include/asm/config.h b/xen/arch/x86/include/asm/config.h
index aebd3e8f45..ee84c57432 100644
--- a/xen/arch/x86/include/asm/config.h
+++ b/xen/arch/x86/include/asm/config.h
@@ -279,6 +279,8 @@ extern unsigned long xen_phys_start;
/* map_domain_page() map cache. The second per-domain-mapping sub-area. */
#define MAPCACHE_VCPU_ENTRIES (CONFIG_PAGING_LEVELS * CONFIG_PAGING_LEVELS)
+/* Single-page slots of a per-vCPU mapcache (vCPU-PT domains). */
+#define MAPCACHE_PERVCPU_ENTRIES 64
#define MAPCACHE_ENTRIES (MAX_VIRT_CPUS * MAPCACHE_VCPU_ENTRIES)
#define MAPCACHE_VIRT_START PERDOMAIN_VIRT_SLOT(1)
#define MAPCACHE_VIRT_END (MAPCACHE_VIRT_START + \
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 45d308407e..7076e38848 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -80,6 +80,12 @@ struct mapcache_vcpu {
uint32_t idx;
uint32_t refcnt;
} hash[MAPHASH_ENTRIES];
+
+ /* Accounting of a per-vCPU mapcache (vCPU-PT domains). */
+ struct mapcache cache;
+ /* Backing for cache.{inuse,garbage}. */
+ DECLARE_BITMAP(inuse, MAPCACHE_PERVCPU_ENTRIES);
+ DECLARE_BITMAP(garbage, MAPCACHE_PERVCPU_ENTRIES);
};
struct mapcache_domain {
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 15/30] x86/mm: let Xen's page-table walker operate on a given root
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (13 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 14/30] x86/hvm: introduce per-vCPU mapcache for vCPU-PT domains George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 16/30] x86/mm: build and maintain a sparse view of the directmap George Dunlap
` (14 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
map_pages_to_xen() and modify_xen_mappings() maintain Xen's own mappings,
walking from idle_pg_table, whose Xen slots every other root page-table
copies. A later patch introduces a second, sparse directmap hierarchy:
page tables for the directmap virtual address range that some guest
contexts use in place of idle_pg_table's. It needs the same maintenance
at the same virtual addresses.
Give virt_to_xen_l{3,2,1}e() the L4 to walk from, and move the bodies of
map_pages_to_xen() and modify_xen_mappings() into map_pages_in() and
modify_mappings_in(), which take the root as well. The public functions
pass idle_pg_table. The EFI runtime-services page-tables mirror
idle_pg_table's L4 entries only, so a newly allocated L3 is propagated
to them only when the root is idle_pg_table.
TLB flushing is unaffected: flushes are by virtual address and go to
every online CPU, which covers both hierarchies.
No functional change.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- Message: name the second hierarchy as sparse; flushes reach every CPU
(Roger).
Changes in v3:
- New in this version.
---
xen/arch/x86/mm.c | 55 ++++++++++++++++++++++++++++++++++-------------
1 file changed, 40 insertions(+), 15 deletions(-)
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 362a88a300..514e121c8b 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -5229,13 +5229,14 @@ static DEFINE_SPINLOCK(map_pgdir_lock);
/*
* For virt_to_xen_lXe() functions, they take a linear address and return a
- * pointer to Xen's LX entry. Caller needs to unmap the pointer.
+ * pointer to Xen's LX entry in the page-table hierarchy rooted at @root
+ * (idle_pg_table for Xen's own mappings). Caller needs to unmap the pointer.
*/
-static l3_pgentry_t *virt_to_xen_l3e(unsigned long v)
+static l3_pgentry_t *virt_to_xen_l3e(l4_pgentry_t *root, unsigned long v)
{
l4_pgentry_t *pl4e;
- pl4e = &idle_pg_table[l4_table_offset(v)];
+ pl4e = &root[l4_table_offset(v)];
if ( !(l4e_get_flags(*pl4e) & _PAGE_PRESENT) )
{
bool locking = system_state > SYS_STATE_boot;
@@ -5251,7 +5252,8 @@ static l3_pgentry_t *virt_to_xen_l3e(unsigned long v)
l4_pgentry_t l4e = l4e_from_mfn(l3mfn, __PAGE_HYPERVISOR);
l4e_write(pl4e, l4e);
- efi_update_l4_pgtable(l4_table_offset(v), l4e);
+ if ( root == idle_pg_table )
+ efi_update_l4_pgtable(l4_table_offset(v), l4e);
l3mfn = INVALID_MFN;
}
if ( locking )
@@ -5262,11 +5264,11 @@ static l3_pgentry_t *virt_to_xen_l3e(unsigned long v)
return map_l3t_from_l4e(*pl4e) + l3_table_offset(v);
}
-static l2_pgentry_t *virt_to_xen_l2e(unsigned long v)
+static l2_pgentry_t *virt_to_xen_l2e(l4_pgentry_t *root, unsigned long v)
{
l3_pgentry_t *pl3e, l3e;
- pl3e = virt_to_xen_l3e(v);
+ pl3e = virt_to_xen_l3e(root, v);
if ( !pl3e )
return NULL;
@@ -5300,11 +5302,11 @@ static l2_pgentry_t *virt_to_xen_l2e(unsigned long v)
return map_l2t_from_l3e(l3e) + l2_table_offset(v);
}
-static l1_pgentry_t *virt_to_xen_l1e(unsigned long v)
+static l1_pgentry_t *virt_to_xen_l1e(l4_pgentry_t *root, unsigned long v)
{
l2_pgentry_t *pl2e, l2e;
- pl2e = virt_to_xen_l2e(v);
+ pl2e = virt_to_xen_l2e(root, v);
if ( !pl2e )
return NULL;
@@ -5380,7 +5382,7 @@ mfn_t xen_map_to_mfn(unsigned long va)
bool locking = system_state > SYS_STATE_boot;
unsigned int l2_offset = l2_table_offset(va);
unsigned int l1_offset = l1_table_offset(va);
- const l3_pgentry_t *pl3e = virt_to_xen_l3e(va);
+ const l3_pgentry_t *pl3e = virt_to_xen_l3e(idle_pg_table, va);
const l2_pgentry_t *pl2e = NULL;
const l1_pgentry_t *pl1e = NULL;
struct page_info *l3page;
@@ -5420,7 +5422,12 @@ mfn_t xen_map_to_mfn(unsigned long va)
return ret;
}
-int map_pages_to_xen(
+/*
+ * map_pages_to_xen() in the hierarchy rooted at @root: idle_pg_table's, or
+ * another hierarchy sharing its virtual addresses.
+ */
+static int map_pages_in(
+ l4_pgentry_t *root,
unsigned long virt,
mfn_t mfn,
unsigned long nr_mfns,
@@ -5471,7 +5478,7 @@ int map_pages_to_xen(
UNMAP_DOMAIN_PAGE(pl3e);
UNMAP_DOMAIN_PAGE(pl2e);
- pl3e = virt_to_xen_l3e(virt);
+ pl3e = virt_to_xen_l3e(root, virt);
if ( !pl3e )
goto out;
@@ -5595,7 +5602,7 @@ int map_pages_to_xen(
free_xen_pagetable(l2mfn);
}
- pl2e = virt_to_xen_l2e(virt);
+ pl2e = virt_to_xen_l2e(root, virt);
if ( !pl2e )
goto out;
@@ -5640,7 +5647,7 @@ int map_pages_to_xen(
/* Normal page mapping. */
if ( !(l2e_get_flags(*pl2e) & _PAGE_PRESENT) )
{
- pl1e = virt_to_xen_l1e(virt);
+ pl1e = virt_to_xen_l1e(root, virt);
if ( pl1e == NULL )
goto out;
}
@@ -5825,6 +5832,15 @@ int map_pages_to_xen(
return rc;
}
+int map_pages_to_xen(
+ unsigned long virt,
+ mfn_t mfn,
+ unsigned long nr_mfns,
+ pte_attr_t flags)
+{
+ return map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
+}
+
int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
{
return map_pages_to_xen(virt, INVALID_MFN, nr_mfns, MAP_SMALL_PAGES);
@@ -5841,8 +5857,12 @@ int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
* flags are ignored.
*
* It is an error to call with present flags over an unpopulated range.
+ *
+ * modify_mappings_in() does so in the hierarchy rooted at @root, as
+ * map_pages_in() does.
*/
-int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
+static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
+ unsigned long e, pte_attr_t nf)
{
bool locking = system_state > SYS_STATE_boot;
l3_pgentry_t *pl3e = NULL;
@@ -5869,7 +5889,7 @@ int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
UNMAP_DOMAIN_PAGE(pl2e);
UNMAP_DOMAIN_PAGE(pl3e);
- pl3e = virt_to_xen_l3e(v);
+ pl3e = virt_to_xen_l3e(root, v);
if ( !pl3e )
goto out;
@@ -6120,6 +6140,11 @@ int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
#undef flush_area
+int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
+{
+ return modify_mappings_in(idle_pg_table, s, e, nf);
+}
+
int destroy_xen_mappings(unsigned long s, unsigned long e)
{
return modify_xen_mappings(s, e, _PAGE_NONE);
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 16/30] x86/mm: build and maintain a sparse view of the directmap
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (14 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 15/30] x86/mm: let Xen's page-table walker operate on a given root George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 17/30] x86/domain_page: key the directmap fast paths on the loaded view George Dunlap
` (13 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie,
Anthony PERARD, Michal Orzel, Julien Grall, Stefano Stabellini
The directmap maps all of RAM, and every root page-table carries it, so
while any guest context runs, all guest memory is reachable from Xen's
address space. For ASI, guest contexts are to run on a view of the
directmap which maps only what Xen must reach through it at all times,
leaving guest memory to transient mappings (map_domain_page()).
Rather than remove the directmap for everyone, keep the full directmap
as it is, for the idle domain and for guest contexts not using ASI, and
build a second, sparse view: page-tables for the same virtual addresses
(the part of the directmap PV guests can see, where everything Xen
reaches through the directmap lives), mapping a subset of what the full
directmap maps, each entry a copy of the full directmap's. Boot and the
idle domain keep the full directmap, so nothing in them needs
converting.
The sparse view maps the xenheap, and memory never handed to the heap
allocator (Xen's image, boot-time allocations, firmware tables). It is
built once the boot allocator has handed its memory to the heap, from
the full directmap minus the ranges the heap was given, plus the xenheap
allocations made meanwhile. The page-tables then come from the heap,
and no other CPU is up. The sparse view has an L3 for each of its L4
slots from the start, so its L4 entries never change and root
page-tables can take copies of them. It is then kept in step:
- xenheap allocations are added, and taken out again before the pages
are freed. If that fails the pages stay mapped, and are leaked with
a warning rather than reused;
- memory handed to the heap later (boot modules, .init) is taken out,
and is not handed over if that fails;
- a PV dom0's initrd, assigned to dom0 in place, is taken out;
- page-tables from the boot allocator, which the sparse view maps with
every other boot-time allocation, are taken out when idle_pg_table's
hierarchy frees them, and are not freed if that fails, so that the
sparse view never keeps a page the heap may hand out;
- changes to the full directmap through map_pages_to_xen() and
modify_xen_mappings() are mirrored. After boot, map_pages_to_xen()
maps into the directmap only memory Xen keeps (stack guard and shadow
stack pages, firmware tables); memory hot-add maps the new range into
the full directmap only, so that the sparse view never needs
page-table pages for it.
modify_xen_mappings_lite() needs no change: it only changes the mappings
of Xen's image, in L4 slot 261, which every root page-table shares with
idle_pg_table, outside the range the sparse view replaces.
Common code gains four hooks for this, arch_heap_pages_init(),
arch_xenheap_needs_scrub() and arch_xenheap_pages_{alloc,free}(), no-ops
unless CONFIG_SPARSE_DIRECTMAP is selected. A failed
arch_xenheap_pages_alloc() may have mapped part of the pages:
alloc_xenheap_pages() then takes them out through
arch_xenheap_pages_free(), as a free would, before freeing them, and
leaks them if that fails too. xen_mapping_lookup(), built only with
CONFIG_SPARSE_DIRECTMAP, reads an entry of either hierarchy under the
L3 table lock.
alloc_xenheap_pages() accepts pages still awaiting scrubbing
(MEMF_no_scrub). That is harmless while the directmap maps all memory
anyway, but the sparse view would make such a page's contents, a dying
domain's data for instance, reachable from every context on it until
overwritten. With a sparse view configured, arch_xenheap_needs_scrub()
makes xenheap allocations scrub those pages first, as domain heap
allocations do; clean pages need nothing extra.
The allocator may still have to scrub a page before returning it; this
will be mapped with map_domain_page(). This is fine for HVM domains
on the sparse view, as the mapcache is per-vCPU, so no other vCPU can
read unscrubbed data. PV domains' mapcache is still per-domain, so
for PV domains on the sparse view, this opens up a window where other
vCPUs could potentially speculatively access the unscrubbed data.
This would be fixed by having per-vCPU mapcaches for PV domains, or
synchronously scrubbing pages on free. Neither are in the scope of the
current work.
arch_mfns_in_directmap() says whether an MFN range is reachable through
the directmap; it now answers for every context, so false once a sparse
view is configured. Its one caller, init_node_heap(), then keeps a NUMA
node's heap metadata in the xenheap instead of carving it out of the
node's memory through the directmap: carved out of a range handed to the
heap, it would be left out of the sparse view, which every heap
allocation needs it in.
The sparse view is built only when some domain type is to use it, which
nothing can request yet: this patch changes nothing on its own.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- Message: "domain type", not "guest type".
- Wrap a line that reached 80 columns.
- Memory hot-add maps the new range into the full directmap only.
- Define opt_sparse_dmap_hwdom here, beside the PV and HVM knobs
(Roger).
- add_to_sparse_view(): assert that the lookup finds the MFN expected
(Roger).
- cmp_mfn_range(): explicit return values, as in cmp_node() (Roger).
- Comments and variable name changes in sparse_dmap_init()'s loop over
the heap's ranges (Roger).
- Message: drop "(NUMA node heap metadata)" (Roger).
- Message: drop the reference to the directmap removal series (Roger).
- Message: modify_xen_mappings_lite() needs no change (Roger).
- Message: note the limit of scrubbing at allocation for PV domains
(Roger).
- Message: only boot and the idle domain keep the full directmap.
- Set the NX bit in this patch, rather than adding it in a later patch.
Changes in v3:
- New in this version.
---
xen/arch/x86/Kconfig | 1 +
xen/arch/x86/Makefile | 1 +
xen/arch/x86/include/asm/mm.h | 16 +-
xen/arch/x86/include/asm/sparse-directmap.h | 67 +++
xen/arch/x86/mm.c | 125 +++++-
xen/arch/x86/pv/dom0_build.c | 4 +
xen/arch/x86/setup.c | 4 +
xen/arch/x86/sparse-directmap.c | 431 ++++++++++++++++++++
xen/arch/x86/x86_64/mm.c | 10 +-
xen/common/Kconfig | 18 +
xen/common/page_alloc.c | 41 +-
xen/include/xen/mm.h | 35 ++
12 files changed, 737 insertions(+), 16 deletions(-)
create mode 100644 xen/arch/x86/include/asm/sparse-directmap.h
create mode 100644 xen/arch/x86/sparse-directmap.c
diff --git a/xen/arch/x86/Kconfig b/xen/arch/x86/Kconfig
index e5535ac484..cd72102ccc 100644
--- a/xen/arch/x86/Kconfig
+++ b/xen/arch/x86/Kconfig
@@ -31,6 +31,7 @@ config X86
select HAS_SCHED_GRANULARITY
select HAS_SHARED_INFO
imply HAS_SOFT_RESET
+ select HAS_SPARSE_DIRECTMAP
select HAS_UBSAN
select HAS_VMAP
select HAS_VPCI if HVM
diff --git a/xen/arch/x86/Makefile b/xen/arch/x86/Makefile
index 2be90ea098..86adf042e9 100644
--- a/xen/arch/x86/Makefile
+++ b/xen/arch/x86/Makefile
@@ -63,6 +63,7 @@ obj-y += shutdown.o
obj-y += smp.o
obj-y += smpboot.o
obj-y += spec_ctrl.o
+obj-$(CONFIG_SPARSE_DIRECTMAP) += sparse-directmap.o
obj-y += srat.o
obj-y += string.o
obj-$(CONFIG_SYSCTL) += sysctl.o
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 12d1c06c12..7612cf66ef 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -7,6 +7,7 @@
#include <xen/rwlock.h>
#include <asm/io.h>
#include <asm/page.h>
+#include <asm/sparse-directmap.h>
#include <asm/uaccess.h>
/*
@@ -597,6 +598,15 @@ mfn_t alloc_xen_pagetable(void);
void free_xen_pagetable(mfn_t mfn);
void *alloc_mapped_pagetable(mfn_t *pmfn);
+int map_pages_in(l4_pgentry_t *root, unsigned long virt, mfn_t mfn,
+ unsigned long nr_mfns, pte_attr_t flags);
+int modify_mappings_in(l4_pgentry_t *root, unsigned long s, unsigned long e,
+ pte_attr_t nf);
+#ifdef CONFIG_SPARSE_DIRECTMAP
+mfn_t xen_mapping_lookup(l4_pgentry_t *root, unsigned long va,
+ pte_attr_t *flags, unsigned long *nr);
+#endif
+
int __sync_local_execstate(void);
/* Arch-specific portion of memory_op hypercall. */
@@ -657,12 +667,16 @@ void write_32bit_pse_identmap(uint32_t *l2);
/*
* x86 maps part of physical memory via the directmap region.
- * Return whether the range of MFN falls in the directmap region.
+ * Return whether the range of MFN falls in the directmap region in every
+ * context: never, with a sparse directmap view configured.
*/
static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)
{
unsigned long eva = min(DIRECTMAP_VIRT_END, HYPERVISOR_VIRT_END);
+ if ( sparse_dmap_configured() )
+ return false;
+
return (mfn + nr) <= (virt_to_mfn(eva - 1) + 1);
}
diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
new file mode 100644
index 0000000000..ea14d44334
--- /dev/null
+++ b/xen/arch/x86/include/asm/sparse-directmap.h
@@ -0,0 +1,67 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef X86_SPARSE_DIRECTMAP_H
+#define X86_SPARSE_DIRECTMAP_H
+
+#include <xen/mm-frame.h>
+#include <xen/mm-types.h>
+#include <asm/page.h>
+
+#ifdef CONFIG_SPARSE_DIRECTMAP
+
+/*
+ * The part of the directmap the sparse view covers: the part PV guests can
+ * see, which is where everything Xen reaches through the directmap lives.
+ * SPARSE_DMAP_END_MFN is the first MFN past it.
+ */
+#define SPARSE_DMAP_START DIRECTMAP_VIRT_START
+#define SPARSE_DMAP_END HYPERVISOR_VIRT_END
+#define SPARSE_DMAP_END_MFN (virt_to_mfn(SPARSE_DMAP_END - 1) + 1)
+
+/*
+ * Which domains use the sparse view: PV domains and HVM domains other than
+ * the hardware domain, and the hardware domain, PV or PVH.
+ */
+extern bool opt_sparse_dmap_pv, opt_sparse_dmap_hvm, opt_sparse_dmap_hwdom;
+
+/* Root of the sparse view, once built (NULL before, or if not configured). */
+extern l4_pgentry_t *sparse_dmap_root;
+
+static inline bool sparse_dmap_configured(void)
+{
+ return opt_sparse_dmap_pv || opt_sparse_dmap_hvm || opt_sparse_dmap_hwdom;
+}
+
+void sparse_dmap_init(void);
+int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
+ pte_attr_t flags);
+int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
+ pte_attr_t nf);
+int sparse_dmap_remove(mfn_t mfn, unsigned long nr);
+int sparse_dmap_drop_pagetable(mfn_t mfn);
+
+#else /* !CONFIG_SPARSE_DIRECTMAP */
+
+static inline bool sparse_dmap_configured(void) { return false; }
+static inline void sparse_dmap_init(void) {}
+static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
+ unsigned long nr, pte_attr_t flags)
+{
+ return 0;
+}
+static inline int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
+ pte_attr_t nf)
+{
+ return 0;
+}
+static inline int sparse_dmap_remove(mfn_t mfn, unsigned long nr)
+{
+ return 0;
+}
+static inline int sparse_dmap_drop_pagetable(mfn_t mfn)
+{
+ return 0;
+}
+
+#endif /* CONFIG_SPARSE_DIRECTMAP */
+
+#endif /* X86_SPARSE_DIRECTMAP_H */
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 514e121c8b..7e3009251c 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -127,6 +127,7 @@
#include <asm/setup.h>
#include <asm/shadow.h>
#include <asm/shared.h>
+#include <asm/sparse-directmap.h>
#include <asm/trampoline.h>
#include <asm/traps.h>
#include <asm/x86_emulate.h>
@@ -5422,11 +5423,101 @@ mfn_t xen_map_to_mfn(unsigned long va)
return ret;
}
+#ifdef CONFIG_SPARSE_DIRECTMAP
+/*
+ * Look @va up in the hierarchy rooted at @root, without allocating anything.
+ * Returns the MFN mapped at @va, or INVALID_MFN, with the mapping's flags in
+ * L1 form in *flags, and in *nr the number of pages from @va to the end of
+ * the entry that maps @va (or that finds it unmapped).
+ */
+mfn_t xen_mapping_lookup(l4_pgentry_t *root, unsigned long va,
+ pte_attr_t *flags, unsigned long *nr)
+{
+ bool locking = system_state > SYS_STATE_boot;
+ l4_pgentry_t l4e = root[l4_table_offset(va)];
+ const l3_pgentry_t *pl3e = NULL;
+ const l2_pgentry_t *pl2e = NULL;
+ const l1_pgentry_t *pl1e = NULL;
+ struct page_info *l3page;
+ unsigned int shift = L4_PAGETABLE_SHIFT;
+ mfn_t ret = INVALID_MFN;
+
+ L3T_INIT(l3page);
+ *flags = 0;
+
+ if ( !(l4e_get_flags(l4e) & _PAGE_PRESENT) )
+ goto out;
+
+ pl3e = map_l3t_from_l4e(l4e) + l3_table_offset(va);
+ l3page = l4e_get_page(l4e);
+ L3T_LOCK(l3page);
+
+ shift = L3_PAGETABLE_SHIFT;
+ if ( !(l3e_get_flags(*pl3e) & _PAGE_PRESENT) )
+ goto out;
+ if ( l3e_get_flags(*pl3e) & _PAGE_PSE )
+ {
+ /* The superpage PAT bit sits among the address bits. */
+ *flags = lNf_to_l1f(l3e_get_flags(*pl3e)) |
+ (l3e_get_intpte(*pl3e) & _PAGE_PSE_PAT ? _PAGE_PAT : 0);
+ ret = _mfn((l3e_get_pfn(*pl3e) &
+ ~((1UL << (2 * PAGETABLE_ORDER)) - 1)) +
+ (PFN_DOWN(va) & ((1UL << (2 * PAGETABLE_ORDER)) - 1)));
+ goto out;
+ }
+
+ pl2e = map_l2t_from_l3e(*pl3e) + l2_table_offset(va);
+ shift = L2_PAGETABLE_SHIFT;
+ if ( !(l2e_get_flags(*pl2e) & _PAGE_PRESENT) )
+ goto out;
+ if ( l2e_get_flags(*pl2e) & _PAGE_PSE )
+ {
+ *flags = lNf_to_l1f(l2e_get_flags(*pl2e)) |
+ (l2e_get_intpte(*pl2e) & _PAGE_PSE_PAT ? _PAGE_PAT : 0);
+ ret = _mfn((l2e_get_pfn(*pl2e) & ~((1UL << PAGETABLE_ORDER) - 1)) +
+ l1_table_offset(va));
+ goto out;
+ }
+
+ pl1e = map_l1t_from_l2e(*pl2e) + l1_table_offset(va);
+ shift = PAGE_SHIFT;
+ if ( l1e_get_flags(*pl1e) & _PAGE_PRESENT )
+ {
+ *flags = l1e_get_flags(*pl1e);
+ ret = l1e_get_mfn(*pl1e);
+ }
+
+ out:
+ L3T_UNLOCK(l3page);
+ unmap_domain_page(pl1e);
+ unmap_domain_page(pl2e);
+ unmap_domain_page(pl3e);
+ *nr = ((((va >> shift) + 1) << shift) - va) >> PAGE_SHIFT;
+ return ret;
+}
+#endif /* CONFIG_SPARSE_DIRECTMAP */
+
+/*
+ * Free a page-table page that the hierarchy rooted at @root has stopped
+ * using. One of idle_pg_table's may be in the sparse directmap view: it is
+ * taken out first, under the sparse view's L3 lock inside an L3 lock of
+ * idle_pg_table's hierarchy (the only order in which the two nest), and
+ * kept if that fails. The sparse view's own page-tables are not in it, and
+ * are freed directly: the caller may hold the sparse view's L3 lock.
+ */
+static void free_pagetable_in(const l4_pgentry_t *root, mfn_t mfn)
+{
+ if ( root == idle_pg_table && sparse_dmap_drop_pagetable(mfn) )
+ return;
+
+ free_xen_pagetable(mfn);
+}
+
/*
* map_pages_to_xen() in the hierarchy rooted at @root: idle_pg_table's, or
* another hierarchy sharing its virtual addresses.
*/
-static int map_pages_in(
+int map_pages_in(
l4_pgentry_t *root,
unsigned long virt,
mfn_t mfn,
@@ -5531,10 +5622,10 @@ static int map_pages_in(
ol2e = l2t[i];
if ( (l2e_get_flags(ol2e) & _PAGE_PRESENT) &&
!(l2e_get_flags(ol2e) & _PAGE_PSE) )
- free_xen_pagetable(l2e_get_mfn(ol2e));
+ free_pagetable_in(root, l2e_get_mfn(ol2e));
}
unmap_domain_page(l2t);
- free_xen_pagetable(l3e_get_mfn(ol3e));
+ free_pagetable_in(root, l3e_get_mfn(ol3e));
}
}
@@ -5632,7 +5723,7 @@ static int map_pages_in(
flush_flags(l1e_get_flags(l1t[i]));
flush_area(virt, flush_flags);
unmap_domain_page(l1t);
- free_xen_pagetable(l2e_get_mfn(ol2e));
+ free_pagetable_in(root, l2e_get_mfn(ol2e));
}
}
@@ -5764,7 +5855,7 @@ static int map_pages_in(
flush_area(virt - PAGE_SIZE,
FLUSH_TLB_GLOBAL |
FLUSH_ORDER(PAGETABLE_ORDER));
- free_xen_pagetable(l2e_get_mfn(ol2e));
+ free_pagetable_in(root, l2e_get_mfn(ol2e));
}
else if ( locking )
spin_unlock(&map_pgdir_lock);
@@ -5811,7 +5902,7 @@ static int map_pages_in(
flush_area(virt - PAGE_SIZE,
FLUSH_TLB_GLOBAL |
FLUSH_ORDER(2*PAGETABLE_ORDER));
- free_xen_pagetable(l3e_get_mfn(ol3e));
+ free_pagetable_in(root, l3e_get_mfn(ol3e));
}
else if ( locking )
spin_unlock(&map_pgdir_lock);
@@ -5838,7 +5929,12 @@ int map_pages_to_xen(
unsigned long nr_mfns,
pte_attr_t flags)
{
- return map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
+ int rc = map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
+
+ if ( !rc )
+ rc = sparse_dmap_mirror_map(virt, mfn, nr_mfns, flags);
+
+ return rc;
}
int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
@@ -5861,8 +5957,8 @@ int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
* modify_mappings_in() does so in the hierarchy rooted at @root, as
* map_pages_in() does.
*/
-static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
- unsigned long e, pte_attr_t nf)
+int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
+ unsigned long e, pte_attr_t nf)
{
bool locking = system_state > SYS_STATE_boot;
l3_pgentry_t *pl3e = NULL;
@@ -6072,7 +6168,7 @@ static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
if ( locking )
spin_unlock(&map_pgdir_lock);
flush_area(NULL, FLUSH_TLB_GLOBAL); /* flush before free */
- free_xen_pagetable(l1mfn);
+ free_pagetable_in(root, l1mfn);
}
else if ( locking )
spin_unlock(&map_pgdir_lock);
@@ -6116,7 +6212,7 @@ static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
if ( locking )
spin_unlock(&map_pgdir_lock);
flush_area(NULL, FLUSH_TLB_GLOBAL); /* flush before free */
- free_xen_pagetable(l2mfn);
+ free_pagetable_in(root, l2mfn);
}
else if ( locking )
spin_unlock(&map_pgdir_lock);
@@ -6142,7 +6238,12 @@ static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
{
- return modify_mappings_in(idle_pg_table, s, e, nf);
+ int rc = modify_mappings_in(idle_pg_table, s, e, nf);
+
+ if ( !rc )
+ rc = sparse_dmap_mirror_modify(s, e, nf);
+
+ return rc;
}
int destroy_xen_mappings(unsigned long s, unsigned long e)
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index dcb1d407ec..f18959f3b4 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -22,6 +22,7 @@
#include <asm/pv/mm.h>
#include <asm/setup.h>
#include <asm/shadow.h>
+#include <asm/sparse-directmap.h>
/* Allow ring-3 access in long mode as guest cannot use ring 1 ... */
#define BASE_PROT (_PAGE_PRESENT|_PAGE_RW|_PAGE_ACCESSED|_PAGE_USER)
@@ -650,6 +651,9 @@ static int __init dom0_construct(const struct boot_domain *bd)
{
if ( assign_pages(mfn_to_page(_mfn(initrd_mfn)), count, d, 0) )
BUG();
+ /* The initrd is dom0's now: out of the sparse directmap. */
+ if ( sparse_dmap_remove(_mfn(initrd_mfn), PFN_UP(initrd_len)) )
+ panic("Cannot unmap dom0's initrd\n");
/*
* We have mapped the initrd directly into dom0, and assigned the
* pages. Tell the boot_module handling that we've freed it, so the
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index fef3c2dd44..f95aa36e05 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -54,6 +54,7 @@
#include <asm/spec_ctrl.h>
#include <asm/stubs.h>
#include <asm/tboot.h>
+#include <asm/sparse-directmap.h>
#include <asm/trampoline.h>
#include <asm/traps.h>
@@ -1944,6 +1945,9 @@ void asmlinkage __init noreturn __start_xen(void)
system_state = SYS_STATE_boot;
+ /* The heap now has all memory: page-tables come from it from here on. */
+ sparse_dmap_init();
+
bsp_stack = cpu_alloc_stack(0); /* Needs to know IDT vs FRED */
if ( !bsp_stack )
panic("No memory for BSP stack\n");
diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
new file mode 100644
index 0000000000..d35a8d6979
--- /dev/null
+++ b/xen/arch/x86/sparse-directmap.c
@@ -0,0 +1,431 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * A sparse view of the directmap.
+ *
+ * Xen maps all of RAM at DIRECTMAP_VIRT_START, and every root page-table
+ * carries those mappings. The sparse view is a second set of page-tables
+ * for the same virtual addresses (the part of the directmap PV guests can
+ * see), mapping only what Xen needs to reach through the directmap at all
+ * times:
+ *
+ * - Memory never handed to the heap: Xen's image, boot-time allocations,
+ * firmware tables. Not a PV dom0's initrd, used in place, nor the frame
+ * table and M2P pages carved from hot-added memory.
+ * - Memory handed to the heap, then allocated to the xenheap.
+ *
+ * The rest of the heap's memory, where guest memory comes from, is not
+ * mapped. Every mapping in the sparse view is either identical to the
+ * full directmap's at the same address, or empty.
+ *
+ * Guest contexts selected on the command line run on root page-tables
+ * carrying the sparse view's L3s in the directmap slots.
+ */
+
+#include <xen/domain_page.h>
+#include <xen/init.h>
+#include <xen/lib.h>
+#include <xen/mm.h>
+#include <xen/numa.h>
+#include <xen/sort.h>
+
+#include <asm/page.h>
+#include <asm/sparse-directmap.h>
+
+bool __ro_after_init opt_sparse_dmap_pv;
+bool __ro_after_init opt_sparse_dmap_hvm;
+bool __ro_after_init opt_sparse_dmap_hwdom;
+
+/* Root of the sparse view: only ever walked and copied from, never loaded. */
+static l4_pgentry_t __aligned(PAGE_SIZE) sparse_l4[L4_PAGETABLE_ENTRIES];
+l4_pgentry_t *__ro_after_init sparse_dmap_root;
+
+/*
+ * Before the sparse view is built: the MFN ranges handed to the heap, which
+ * it leaves out, and the xenheap allocations made since, which it puts back.
+ */
+struct mfn_range {
+ unsigned long s, e;
+};
+static struct mfn_range early_heap[256];
+static unsigned int nr_early_heap;
+static struct mfn_range early_xenheap[2 * MAX_NUMNODES];
+static unsigned int nr_early_xenheap;
+static bool early_overflow;
+
+/*
+ * Clip [*mfn, *mfn + *nr) to what the sparse view covers; false if nothing is
+ * left.
+ */
+static bool clip_to_sparse_view(unsigned long *mfn, unsigned long *nr)
+{
+ unsigned long end = SPARSE_DMAP_END_MFN;
+
+ if ( *mfn >= end )
+ return false;
+ if ( *nr > end - *mfn )
+ *nr = end - *mfn;
+
+ return *nr;
+}
+
+static void early_range_add(struct mfn_range *r, unsigned int *nr,
+ unsigned int max, unsigned long s, unsigned long e)
+{
+ if ( *nr && r[*nr - 1].e == s )
+ r[*nr - 1].e = e;
+ else if ( *nr < max )
+ r[(*nr)++] = (struct mfn_range){ s, e };
+ else
+ early_overflow = true;
+}
+
+static void early_range_del(struct mfn_range *r, unsigned int *nr,
+ unsigned long s, unsigned long e)
+{
+ unsigned int i;
+
+ for ( i = 0; i < *nr; i++ )
+ if ( r[i].s == s && r[i].e == e )
+ {
+ r[i] = r[--*nr];
+ return;
+ }
+}
+
+/*
+ * Copy the full directmap's mappings of MFNs [mfn, mfn + nr) into the sparse
+ * view.
+ */
+static int add_to_sparse_view(l4_pgentry_t *root, unsigned long mfn,
+ unsigned long nr)
+{
+ unsigned long start = (unsigned long)mfn_to_virt(mfn);
+ unsigned long va = start, end = start + (nr << PAGE_SHIFT);
+
+ while ( va < end )
+ {
+ pte_attr_t flags;
+ unsigned long n;
+ mfn_t va_mfn = xen_mapping_lookup(idle_pg_table, va, &flags, &n);
+ int rc;
+
+ ASSERT(mfn_eq(va_mfn, INVALID_MFN) ||
+ mfn_x(va_mfn) == mfn + ((va - start) >> PAGE_SHIFT));
+ n = min(n, (end - va) >> PAGE_SHIFT);
+ if ( !mfn_eq(va_mfn, INVALID_MFN) &&
+ (rc = map_pages_in(root, va, va_mfn, n, flags)) )
+ return rc;
+ va += n << PAGE_SHIFT;
+ }
+
+ return 0;
+}
+
+static int remove_from_sparse_view(unsigned long mfn, unsigned long nr)
+{
+ unsigned long va = (unsigned long)mfn_to_virt(mfn);
+
+ return modify_mappings_in(sparse_dmap_root, va, va + (nr << PAGE_SHIFT),
+ _PAGE_NONE);
+}
+
+/* Memory is about to be handed to the heap allocator. */
+bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
+{
+ unsigned long s = mfn_x(mfn);
+
+ if ( !sparse_dmap_configured() || !clip_to_sparse_view(&s, &nr) )
+ return true;
+
+ if ( !sparse_dmap_root )
+ {
+ early_range_add(early_heap, &nr_early_heap, ARRAY_SIZE(early_heap),
+ s, s + nr);
+ return true;
+ }
+
+ if ( remove_from_sparse_view(s, nr) )
+ {
+ printk(XENLOG_ERR
+ "sparse directmap: cannot unmap MFNs %#lx-%#lx, not freeing\n",
+ s, s + nr - 1);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Xenheap pages enter the sparse view, which guest contexts run on: those
+ * still holding a previous owner's data must be scrubbed first.
+ */
+bool arch_xenheap_needs_scrub(void)
+{
+ return sparse_dmap_configured();
+}
+
+/* Pages have been allocated from the heap as xenheap pages. */
+int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
+{
+ unsigned long s = mfn_x(mfn), nr = 1UL << order;
+
+ if ( !sparse_dmap_configured() )
+ return 0;
+
+ /* Xenheap pages must be reachable through the sparse view. */
+ BUG_ON(s + nr > SPARSE_DMAP_END_MFN);
+
+ if ( !sparse_dmap_root )
+ {
+ early_range_add(early_xenheap, &nr_early_xenheap,
+ ARRAY_SIZE(early_xenheap), s, s + nr);
+ return 0;
+ }
+
+ return add_to_sparse_view(sparse_dmap_root, s, nr);
+}
+
+/*
+ * Xenheap pages are about to be freed. A failure means the pages may still
+ * be reachable through the sparse view, and must not be reused.
+ */
+int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
+{
+ unsigned long s = mfn_x(mfn), nr = 1UL << order;
+
+ if ( !sparse_dmap_configured() )
+ return 0;
+
+ BUG_ON(s + nr > SPARSE_DMAP_END_MFN);
+
+ if ( !sparse_dmap_root )
+ {
+ early_range_del(early_xenheap, &nr_early_xenheap, s, s + nr);
+ return 0;
+ }
+
+ return remove_from_sparse_view(s, nr);
+}
+
+/* Memory leaves Xen's hands without passing through the heap. */
+int sparse_dmap_remove(mfn_t mfn, unsigned long nr)
+{
+ return arch_heap_pages_init(mfn, nr) ? 0 : -ENOMEM;
+}
+
+/*
+ * A page-table page of idle_pg_table's hierarchy is about to be freed. One
+ * the boot allocator provided is in the sparse view, as every boot-time
+ * allocation is: take it out before the heap may hand it on. One allocated
+ * since came from the heap and is not in the sparse view, which a lookup
+ * finds without changing, or flushing, anything. Non-zero: the page may
+ * still be mapped, and must not be freed.
+ */
+int sparse_dmap_drop_pagetable(mfn_t mfn)
+{
+ unsigned long s = mfn_x(mfn), nr = 1, n;
+ pte_attr_t flags;
+ int rc;
+
+ if ( !sparse_dmap_root || !clip_to_sparse_view(&s, &nr) ||
+ mfn_eq(xen_mapping_lookup(sparse_dmap_root,
+ (unsigned long)mfn_to_virt(s), &flags, &n),
+ INVALID_MFN) )
+ return 0;
+
+ rc = remove_from_sparse_view(s, 1);
+ if ( rc )
+ printk(XENLOG_WARNING
+ "Leaking page-table page %"PRI_mfn": still in the sparse directmap\n",
+ mfn_x(mfn));
+
+ return rc;
+}
+
+int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
+ pte_attr_t flags)
+{
+ if ( !sparse_dmap_root || virt < SPARSE_DMAP_START ||
+ virt >= SPARSE_DMAP_END )
+ return 0;
+
+ nr = min(nr, (SPARSE_DMAP_END - virt) >> PAGE_SHIFT);
+
+ return map_pages_in(sparse_dmap_root, virt, mfn, nr, flags);
+}
+
+int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
+ pte_attr_t nf)
+{
+ unsigned long va;
+
+ if ( !sparse_dmap_root || e <= SPARSE_DMAP_START ||
+ s >= SPARSE_DMAP_END )
+ return 0;
+
+ s = max(s, SPARSE_DMAP_START);
+ e = min(e, SPARSE_DMAP_END);
+
+ if ( !(nf & _PAGE_PRESENT) )
+ return modify_mappings_in(sparse_dmap_root, s, e, nf);
+
+ /*
+ * A permission change applies where the sparse view maps anything: it
+ * never creates mappings, and the sparse view need not map all of [s, e).
+ */
+ for ( va = s; va < e; )
+ {
+ pte_attr_t flags;
+ unsigned long n;
+ mfn_t mfn = xen_mapping_lookup(sparse_dmap_root, va, &flags, &n);
+ int rc;
+
+ n = min(n, (e - va) >> PAGE_SHIFT);
+ if ( !mfn_eq(mfn, INVALID_MFN) &&
+ (rc = modify_mappings_in(sparse_dmap_root, va,
+ va + (n << PAGE_SHIFT), nf)) )
+ return rc;
+ va += n << PAGE_SHIFT;
+ }
+
+ return 0;
+}
+
+static int __init cf_check cmp_mfn_range(const void *a, const void *b)
+{
+ const struct mfn_range *l = a, *r = b;
+
+ if ( l->s > r->s )
+ return 1;
+ if ( l->s < r->s )
+ return -1;
+
+ return 0;
+}
+
+static void __init cf_check swap_mfn_range(void *a, void *b)
+{
+ SWAP(*(struct mfn_range *)a, *(struct mfn_range *)b);
+}
+
+/*
+ * Build the sparse view: the full directmap, minus the ranges handed to the
+ * heap so far, plus the xenheap allocations made from them. Called once the
+ * heap has taken over from the boot allocator, so that the page-tables come
+ * from the heap, and before any other CPU is up.
+ */
+void __init sparse_dmap_init(void)
+{
+ unsigned long va, mapped = 0;
+ unsigned int i, heap_idx;
+
+ if ( !sparse_dmap_configured() )
+ return;
+
+ if ( early_overflow )
+ {
+ printk(XENLOG_WARNING
+ "Sparse directmap: too many early heap ranges, disabled\n");
+ opt_sparse_dmap_pv = opt_sparse_dmap_hvm = opt_sparse_dmap_hwdom =
+ false;
+ return;
+ }
+
+ /* An L3 for every slot, so that the L4 entries never change. */
+ for ( va = SPARSE_DMAP_START; va < SPARSE_DMAP_END;
+ va += 1UL << L4_PAGETABLE_SHIFT )
+ {
+ mfn_t l3mfn;
+ void *l3t = alloc_mapped_pagetable(&l3mfn);
+
+ if ( !l3t )
+ panic("Sparse directmap: out of memory\n");
+ unmap_domain_page(l3t);
+ sparse_l4[l4_table_offset(va)] =
+ l4e_from_mfn(l3mfn, __PAGE_HYPERVISOR);
+ }
+
+ sort(early_heap, nr_early_heap, sizeof(*early_heap), cmp_mfn_range,
+ swap_mfn_range);
+
+ /*
+ * The full directmap, leaf by leaf, minus the ranges added to the
+ * heap. Leaves come in address order, which is MFN order in the
+ * directmap, and the heap ranges are sorted and disjoint, so
+ * heap_idx only moves forward.
+ */
+ for ( va = SPARSE_DMAP_START, heap_idx = 0; va < SPARSE_DMAP_END; )
+ {
+ pte_attr_t flags;
+ unsigned long n, s, e;
+
+ /* Get information about the leaf PTE at va. */
+ mfn_t mfn = xen_mapping_lookup(idle_pg_table, va, &flags, &n);
+
+ /* Clip the va range we're considering to SPARSE_DMAP_END. */
+ n = min(n, (SPARSE_DMAP_END - va) >> PAGE_SHIFT);
+ if ( mfn_eq(mfn, INVALID_MFN) )
+ {
+ va += n << PAGE_SHIFT;
+ continue;
+ }
+
+ /*
+ * Walk through the range [mfn, mfn + n) looking for ranges not
+ * handed to the heap.
+ */
+ for ( s = mfn_x(mfn), e = s + n; s < e; )
+ {
+ unsigned long upto = e;
+
+ /* Step past heap ranges that end at or before s. */
+ while ( heap_idx < nr_early_heap && early_heap[heap_idx].e <= s )
+ heap_idx++;
+
+ /* s lies in a heap range: skip to its end. */
+ if ( heap_idx < nr_early_heap && early_heap[heap_idx].s <= s )
+ {
+ s = min(e, early_heap[heap_idx].e);
+ continue;
+ }
+
+ /* Map up to the next heap range, or to the end of the leaf. */
+ if ( heap_idx < nr_early_heap )
+ upto = min(e, early_heap[heap_idx].s);
+
+ if ( map_pages_in(sparse_l4, va + ((s - mfn_x(mfn)) << PAGE_SHIFT),
+ _mfn(s), upto - s, flags) )
+ panic("Sparse directmap: out of memory\n");
+ mapped += upto - s;
+ s = upto;
+ }
+
+ va += n << PAGE_SHIFT;
+ }
+
+ /*
+ * Now add back in ranges from the heap which were allocated to
+ * the xenheap.
+ */
+ for ( i = 0; i < nr_early_xenheap; i++ )
+ {
+ unsigned long nr = early_xenheap[i].e - early_xenheap[i].s;
+
+ if ( add_to_sparse_view(sparse_l4, early_xenheap[i].s, nr) )
+ panic("Sparse directmap: out of memory\n");
+ mapped += nr;
+ }
+
+ /*
+ * Nothing in the directmap is executed: its slots are NX, as the full
+ * directmap's are.
+ */
+ if ( cpu_has_nx )
+ for ( i = l4_table_offset(SPARSE_DMAP_START);
+ i < l4_table_offset(SPARSE_DMAP_END); i++ )
+ l4e_add_flags(sparse_l4[i], _PAGE_NX_BIT);
+
+ sparse_dmap_root = sparse_l4;
+
+ printk(XENLOG_INFO "Sparse directmap: %lu pages mapped\n", mapped);
+}
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index 7b04537066..c5d9dd64a3 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -1231,8 +1231,14 @@ int memory_add(unsigned long spfn, unsigned long epfn, unsigned int pxm)
i = virt_to_mfn(HYPERVISOR_VIRT_END - 1) + 1;
if ( spfn < i )
{
- ret = map_pages_to_xen((unsigned long)mfn_to_virt(spfn), _mfn(spfn),
- min(epfn, i) - spfn, PAGE_HYPERVISOR);
+ /*
+ * Into the full directmap only, not a sparse directmap view: the
+ * frame table and M2P carved from the start of the range are
+ * reached through their own mappings, and the rest goes to the
+ * heap, which the sparse view leaves out.
+ */
+ ret = map_pages_in(idle_pg_table, (unsigned long)mfn_to_virt(spfn),
+ _mfn(spfn), min(epfn, i) - spfn, PAGE_HYPERVISOR);
if ( ret )
goto destroy_directmap;
}
diff --git a/xen/common/Kconfig b/xen/common/Kconfig
index 5b289e444f..8f2f605e45 100644
--- a/xen/common/Kconfig
+++ b/xen/common/Kconfig
@@ -167,6 +167,9 @@ config HAS_STATIC_MEMORY
config HAS_SOFT_RESET
bool
+config HAS_SPARSE_DIRECTMAP
+ bool
+
config HAS_STACK_PROTECTOR
bool
@@ -299,6 +302,21 @@ config SPECULATIVE_HARDEN_LOCK
This option is disabled by default at run time, and needs to be
enabled on the command line.
+
+config SPARSE_DIRECTMAP
+ bool "Sparse directmap view"
+ depends on HAS_SPARSE_DIRECTMAP
+ help
+ Build support for a sparse view of the directmap, alongside the
+ full one: it maps only the memory Xen needs to reach at all times
+ (the Xen heap, Xen's own image, firmware tables), while guest
+ memory is reached through transient mappings. Guest contexts
+ selected with the `asi=` command line option run on the sparse
+ directmap view, narrowing the memory reachable from Xen's address
+ space while they run, and hence the memory disclosure surface of
+ speculative side channels.
+
+ If unsure, say N.
endmenu
menu "Other hardening"
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index fe150eccd7..69e8e982b6 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -1971,6 +1971,9 @@ static void init_heap_pages(
unsigned long i;
bool need_scrub = scrub_debug;
+ if ( !arch_heap_pages_init(page_to_mfn(pg), nr_pages) )
+ return;
+
/*
* Keep MFN 0 away from the buddy allocator to avoid crossing zone
* boundary when merging two buddies.
@@ -2540,10 +2543,33 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags)
if ( !(memflags >> _MEMF_bits) )
memflags |= MEMF_bits(xenheap_bits);
- pg = alloc_domheap_pages(NULL, order, memflags | MEMF_no_scrub);
+ /*
+ * Pages still awaiting scrubbing may do as xenheap pages: the directmap
+ * maps them anyway. Not so if an arch's sparse directmap view makes them
+ * reachable where the rest of the heap's memory is not.
+ */
+ if ( !arch_xenheap_needs_scrub() )
+ memflags |= MEMF_no_scrub;
+
+ pg = alloc_domheap_pages(NULL, order, memflags);
if ( unlikely(pg == NULL) )
return NULL;
+ if ( arch_xenheap_pages_alloc(page_to_mfn(pg), order) )
+ {
+ /*
+ * The hook may have mapped part of the pages before failing: take
+ * them out again as a free would, and leak them if that fails too.
+ */
+ if ( arch_xenheap_pages_free(page_to_mfn(pg), order) )
+ printk(XENLOG_WARNING
+ "Leaking xenheap pages %"PRI_mfn"+%u: still mapped\n",
+ mfn_x(page_to_mfn(pg)), 1U << order);
+ else
+ free_domheap_pages(pg, order);
+ return NULL;
+ }
+
for ( i = 0; i < (1u << order); i++ )
pg[i].count_info |= PGC_xen_heap;
@@ -2562,6 +2588,19 @@ void free_xenheap_pages(void *v, unsigned int order)
pg = virt_to_page(v);
+ /*
+ * If the pages cannot be taken out of an arch's sparse directmap view,
+ * they may remain reachable through it: leak them rather than hand them
+ * out.
+ */
+ if ( arch_xenheap_pages_free(page_to_mfn(pg), order) )
+ {
+ printk(XENLOG_WARNING
+ "Leaking xenheap pages %"PRI_mfn"+%u: still mapped\n",
+ mfn_x(page_to_mfn(pg)), 1U << order);
+ return;
+ }
+
for ( i = 0; i < (1u << order); i++ )
pg[i].count_info &= ~PGC_xen_heap;
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index 153c57f5dd..2bcc45cf58 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -94,6 +94,41 @@ bool scrub_free_pages(void);
#define alloc_xenheap_page() (alloc_xenheap_pages(0,0))
#define free_xenheap_page(v) (free_xenheap_pages(v,0))
+/*
+ * Hooks for an architecture keeping a second, sparse view of its directmap,
+ * which maps the xenheap but not the rest of the heap's memory.
+ */
+#ifdef CONFIG_SPARSE_DIRECTMAP
+/* Memory is about to be handed to the heap. False: don't, keep it out. */
+bool arch_heap_pages_init(mfn_t mfn, unsigned long nr);
+/* Xenheap pages are to be allocated. True: scrub those needing it first. */
+bool arch_xenheap_needs_scrub(void);
+/*
+ * Xenheap pages were allocated. Non-zero: failed, possibly part-way; the
+ * pages go through arch_xenheap_pages_free() before being freed again.
+ */
+int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order);
+/* Xenheap pages are to be freed. Non-zero: failed, don't reuse them. */
+int arch_xenheap_pages_free(mfn_t mfn, unsigned int order);
+#else
+static inline bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
+{
+ return true;
+}
+static inline bool arch_xenheap_needs_scrub(void)
+{
+ return false;
+}
+static inline int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
+{
+ return 0;
+}
+static inline int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
+{
+ return 0;
+}
+#endif
+
/* Free an allocation, and zero the pointer to it. */
#define FREE_XENHEAP_PAGES(p, o) do { \
void *_ptr_ = (p); \
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 17/30] x86/domain_page: key the directmap fast paths on the loaded view
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (15 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 16/30] x86/mm: build and maintain a sparse view of the directmap George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 18/30] x86/domain_page: clear a mapcache slot at its last unmap George Dunlap
` (12 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
map_domain_page() returns directmap addresses whenever it can: for any
MFN the directmap covers in release builds, and for vCPUs without a
mapcache. Once guest contexts may run on the sparse view of the
directmap, modify this to take the fast path returning the directmap
address only when the full directmap is loaded.
The release-build shortcut ahead of the vCPU lookup looks at the vCPU
whose page-tables are loaded, without completing a lazy context
switch: if that vCPU is not on the sparse view, its page-tables map
the page through the directmap, and completing the switch would only
move the CPU to the idle vCPU's page-tables, which map it too.
A vCPU on the sparse view always has a mapcache: HVM ones through
per-vCPU page-tables, PV ones through the domain-wide mapcache, which
mapcache_domain_init() now sets up for a domain using the sparse view in
release builds too, rather than only when the directmap does not cover
all memory.
map_domain_page_global() mappings must be valid in every context: use
vmap() for all of them once a sparse view is configured, as
arch_mfns_in_directmap() then says.
Add d->arch.asi.sparse_dmap for this; nothing sets it yet.
No functional change.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- d->arch.sparse_dmap is now a one-bit field of d->arch.asi, beside
vcpu_pt.
- Wrap a line that reached 80 columns.
- Drop sparse_dmap_active: d->arch.asi.sparse_dmap alone says whether a
domain's root page-tables carry the sparse view (Roger). Builds
without CONFIG_SPARSE_DIRECTMAP compile the test out, as before.
- Drop the second directmap check after the vCPU lookup: the
!cache->inuse case covers it.
Changes in v3:
- New in this version.
---
xen/arch/x86/domain_page.c | 34 ++++++++++++++++++++++++++++---
xen/arch/x86/include/asm/domain.h | 3 +++
2 files changed, 34 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 770fee80a4..6b662706fb 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -56,6 +56,13 @@ static inline struct vcpu *mapcache_current_vcpu(void)
return ACCESS_ONCE(this_cpu(pgtable_vcpu));
}
+/* Whether @v's page-tables carry the sparse view of the directmap. */
+static bool on_sparse_dmap(const struct vcpu *v)
+{
+ return IS_ENABLED(CONFIG_SPARSE_DIRECTMAP) &&
+ v->domain->arch.asi.sparse_dmap;
+}
+
/*
* The accounting for v's mapcache: its own for a vCPU-PT domain, the domain's
* otherwise, in which case *dcache is set so the caller takes the domain-wide
@@ -135,9 +142,19 @@ void *map_domain_page(mfn_t mfn)
#ifdef NDEBUG
if ( mfn_x(mfn) <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
- return mfn_to_virt(mfn_x(mfn));
+ {
+ const struct vcpu *pt_vcpu = this_cpu(pgtable_vcpu);
+
+ /*
+ * The page-tables loaded map @mfn through the directmap, unless they
+ * carry the sparse view.
+ */
+ if ( !pt_vcpu || !on_sparse_dmap(pt_vcpu) )
+ return mfn_to_virt(mfn_x(mfn));
+ }
#endif
+ /* No vCPU (early boot), or no mapcache: the full directmap is loaded. */
v = mapcache_current_vcpu();
if ( !v )
return mfn_to_virt(mfn_x(mfn));
@@ -145,7 +162,10 @@ void *map_domain_page(mfn_t mfn)
cache = vcpu_mapcache(v, &dcache);
vcache = &v->arch.mapcache;
if ( !cache->inuse )
+ {
+ ASSERT(!on_sparse_dmap(v));
return mfn_to_virt(mfn_x(mfn));
+ }
perfc_incr(map_domain_page_count);
@@ -294,7 +314,11 @@ void mapcache_domain_init(struct domain *d)
ASSERT(is_pv_domain(d));
#ifdef NDEBUG
- if ( !mem_hotplug && max_page <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
+ /*
+ * Not needed when the directmap loaded for the domain covers everything.
+ */
+ if ( !d->arch.asi.sparse_dmap && !mem_hotplug &&
+ max_page <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
return;
#endif
@@ -378,8 +402,12 @@ void *map_domain_page_global(mfn_t mfn)
system_state < SYS_STATE_active) ||
local_irq_is_enabled()));
+ /*
+ * Global mappings must hold in every context: a sparse view of the
+ * directmap rules the directmap out.
+ */
#ifdef NDEBUG
- if ( mfn_x(mfn) <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
+ if ( arch_mfns_in_directmap(mfn_x(mfn), 1) )
return mfn_to_virt(mfn_x(mfn));
#endif
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 7076e38848..f84ac15759 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -490,6 +490,9 @@ struct arch_domain
struct {
/* Give each vCPU its own per-domain area (HVM only). */
bool vcpu_pt:1;
+
+ /* The domain's root page-tables carry the sparse directmap view. */
+ bool sparse_dmap:1;
} asi;
/* Emulated devices enabled bitmap. */
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 18/30] x86/domain_page: clear a mapcache slot at its last unmap
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (16 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 17/30] x86/domain_page: key the directmap fast paths on the loaded view George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 19/30] x86/mm: run guest contexts on the sparse directmap view George Dunlap
` (11 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
unmap_domain_page() keeps a recently unmapped slot in the vCPU's
maphash (MAPHASH_ENTRIES, 8 per vCPU) for a quick remap of the same
MFN, and leaves its PTE in place until the entry is displaced by a later
unmap or reclaimed by a full cache. Until then the page stays mapped,
although nothing refers to the mapping and the mapcache holds no
reference to the page. Where the mapcache clears a slot's PTE, it
defers the flush: the slot's translation can stay in the TLB of the CPU
which used it until that CPU next flushes.
While the full directmap is loaded, that reaches nothing it does not
map anyway. On the sparse view of the directmap, Xen reaches
domain-heap memory only through mappings made for the purpose. A
mapping or a translation left behind keeps the page reachable after its
last unmap: while Xen serves another of the guest's contexts on the
same vCPU, and, once the page is freed and handed to another domain,
from a context which has nothing to do with it.
So clear a slot's PTE when its last reference goes, maphash slots
included: the hash entry keeps the slot, not the mapping, and a later
hit maps the page again in the same slot. The slot has mapped nothing
else in between, so that needs no flush. Reusing the slot for another
page still goes through the garbage map, or through the replacement of
a hash entry when the cache is full, both of which flush first.
In a per-vCPU mapcache, as vCPU-PT domains have, also invalidate the
slot's translation on the local CPU, right after clearing the PTE. The
slot has a second address, in the mirror of the per-domain area that
every domain but a 64-bit PV one gets for the compat argument
translation area, and its translation is invalidated too: two INVLPGs.
That leaves no translation on any CPU, as the vCPU's page-tables are
loaded on one CPU at a time, and a CPU loading other page-tables
(switch_cr3_cr4()) flushes non-global translations, the mapcache's
included.
A PV domain's domain-wide mapcache has its PTEs cleared but no
INVLPG, as CPUs running the domain's other vCPUs may hold the slot's
translation too, which a local INVLPG would not reach. For it, as
before, builds with PV support flush a page freed by its owner from
TLBs before the page is handed out again.
The cost is one PTE write per unmap into the maphash and per remap from
it, and in a per-vCPU mapcache two INVLPGs per slot's last unmap. A
second unmap of an entry in the maphash now finds its PTE not present
and is caught by the check unmap_domain_page() has for entries the
first unmap zapped, rather than by the reference count's ASSERT() in
debug builds only.
The change does not depend on asi= or on a sparse view of the directmap
being configured: PV vCPUs use the domain-wide mapcache on the default
command line too. For them it adds the PTE writes above, and leaves up
to MAPHASH_ENTRIES fewer pages per vCPU mapped in the domain-wide area.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- Merge v3's 15/18 ("x86/mm: flush owned pages before reuse with the
sparse directmap view") and 16/18 ("x86/domain_page: keep the
maphash's slots, not their mappings"). Rather than flush pages freed
by their owner before reuse wherever a sparse view is configured,
invalidate a per-vCPU mapcache slot's translations on the local CPU
right after its PTE is cleared (Roger).
- Message: don't restate when PV vCPUs use the mapcache (Roger).
Changes in v3:
- New in this version.
For reviewers: with a slot cleared and invalidated at its last unmap,
a per-vCPU mapcache's maphash saves only the search for a free slot,
and a second slot for a page mapped twice at once. One page in two
slots is fine, and already happens when it is not in the maphash. So
per-vCPU mapcaches could drop the maphash, and with it their garbage
map and wrap-time flush. In v5, or as a follow-up?
---
xen/arch/x86/domain_page.c | 53 +++++++++++++++++++++++++------
xen/arch/x86/include/asm/domain.h | 5 ++-
2 files changed, 48 insertions(+), 10 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 6b662706fb..4198ee7c18 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -130,6 +130,26 @@ static void mapcache_exit(struct mapcache_domain *dcache)
#define MAPCACHE_L1ENT(idx) \
__linear_l1_table[l1_linear_offset(MAPCACHE_VIRT_START + pfn_to_paddr(idx))]
+/*
+ * Clear a slot's PTE. For a per-vCPU mapcache (HVM domains only), then also
+ * drop the slot's translations from this CPU's TLB, at its address and at its
+ * alias in the per-domain mirror, which every HVM domain has. No other CPU
+ * can hold them: the vCPU's page-tables are loaded on one CPU at a time, and
+ * loading other page-tables flushes non-global translations.
+ */
+static void mapcache_zap(const struct mapcache_domain *dcache,
+ unsigned int idx)
+{
+ unsigned long va = MAPCACHE_VIRT_START + pfn_to_paddr(idx);
+
+ l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
+ if ( dcache )
+ return;
+
+ flush_tlb_one_local(va);
+ flush_tlb_one_local(va - PERDOMAIN_VIRT_START + PERDOMAIN_ALT_VIRT_START);
+}
+
void *map_domain_page(mfn_t mfn)
{
unsigned long flags;
@@ -176,6 +196,14 @@ void *map_domain_page(mfn_t mfn)
{
idx = hashent->idx;
ASSERT(idx < cache->entries);
+ /*
+ * An entry the maphash kept holds on to its slot, not to its mapping
+ * (see unmap_domain_page()): map the page again. No flush is needed,
+ * as the slot has mapped nothing else since.
+ */
+ if ( !hashent->refcnt )
+ l1e_write(&MAPCACHE_L1ENT(idx),
+ l1e_from_mfn(mfn, __PAGE_HYPERVISOR_RW));
hashent->refcnt++;
ASSERT(hashent->refcnt);
ASSERT(mfn_eq(l1e_get_mfn(MAPCACHE_L1ENT(idx)), mfn));
@@ -208,8 +236,8 @@ void *map_domain_page(mfn_t mfn)
if ( hashent->idx != MAPHASHENT_NOTINUSE && !hashent->refcnt )
{
idx = hashent->idx;
- ASSERT(l1e_get_pfn(MAPCACHE_L1ENT(idx)) == hashent->mfn);
- l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
+ ASSERT(!(l1e_get_flags(MAPCACHE_L1ENT(idx)) &
+ _PAGE_PRESENT));
hashent->idx = MAPHASHENT_NOTINUSE;
hashent->mfn = ~0UL;
break;
@@ -273,32 +301,39 @@ void unmap_domain_page(const void *ptr)
local_irq_save(flags);
+ /*
+ * The maphash keeps a freed slot, not its mapping: the page may change
+ * hands meanwhile. A domain-wide mapcache's translation left in a TLB
+ * is covered by the flush the allocator makes before reusing an owned
+ * page.
+ */
if ( hashent->idx == idx )
{
ASSERT(hashent->mfn == mfn);
ASSERT(hashent->refcnt);
hashent->refcnt--;
+ if ( !hashent->refcnt )
+ mapcache_zap(dcache, idx);
}
else if ( !hashent->refcnt )
{
+ /* The entry replaced has no mapping left: mark its slot as garbage. */
if ( hashent->idx != MAPHASHENT_NOTINUSE )
{
- /* /First/, zap the PTE. */
- ASSERT(l1e_get_pfn(MAPCACHE_L1ENT(hashent->idx)) ==
- hashent->mfn);
- l1e_write(&MAPCACHE_L1ENT(hashent->idx), l1e_empty());
- /* /Second/, mark as garbage. */
+ ASSERT(!(l1e_get_flags(MAPCACHE_L1ENT(hashent->idx)) &
+ _PAGE_PRESENT));
set_bit(hashent->idx, cache->garbage);
}
- /* Add newly-freed mapping to the maphash. */
+ /* Add the newly-freed slot to the maphash, without its mapping. */
+ mapcache_zap(dcache, idx);
hashent->mfn = mfn;
hashent->idx = idx;
}
else
{
/* /First/, zap the PTE. */
- l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
+ mapcache_zap(dcache, idx);
/* /Second/, mark as garbage. */
set_bit(idx, cache->garbage);
}
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index f84ac15759..0e405157b4 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -74,7 +74,10 @@ struct mapcache_vcpu {
/* Shadow of mapcache_domain.epoch. */
unsigned int shadow_epoch;
- /* Lock-free per-VCPU hash of recently-used mappings. */
+ /*
+ * Lock-free per-VCPU hash of recently-used mappings. An entry without
+ * references keeps its slot but not its mapping (see unmap_domain_page()).
+ */
struct vcpu_maphash_entry {
unsigned long mfn;
uint32_t idx;
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 19/30] x86/mm: run guest contexts on the sparse directmap view
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (17 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 18/30] x86/domain_page: clear a mapcache slot at its last unmap George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 20/30] x86/spec-ctrl: wire the sparse directmap view into the asi= option George Dunlap
` (10 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
In init_xen_l4_slots(), if d->arch.asi.sparse_dmap is set, point the
directmap slots at the sparse view, copying from idle_pg_table only the
Xen slots below them. A root page-table not shared with a PV guest gets
no extended directmap, which the sparse view does not cover.
d->arch.asi.sparse_dmap is decided at domain creation: PV domains all
have a mapcache, HVM domains need per-vCPU page-tables for one. No
option asks for the sparse view yet.
The sparse view is built before any domain is created, so the root
page-tables of every domain using it carry it from their creation,
dom0's included. Dom0's builders reach dom0's memory only through guest
virtual addresses and map_domain_page(). In PV shim mode,
replace_va_mapping() reads the guest's L4 through the directmap; map it
with map_domain_page() instead, as the lower levels already are. Shadow
L4s, dom0=shadow's included, get the sparse view from
init_xen_l4_slots().
The two directmaps differ where the sparse view leaves memory out, so no
translation of the full directmap may remain usable on page-tables
carrying the sparse view. Directmap mappings stay global nevertheless:
every CR3 write which can move a CPU onto the sparse view goes through
switch_cr3_cr4(), which flushes the old page-tables' translations,
global entries included, before it returns. The CR3 writes which keep
entries -- a PV vCPU's switch between its kernel and user page-tables,
XPTI's entry and exit -- stay within one vCPU's page-tables, which carry
the same directmap, or go to XPTI's per-CPU root, which maps only Xen's
own per-CPU pages there. Removals from the sparse view are flushed on
every CPU, global entries included. PV guests running with global
pages, a PV dom0 not using the sparse view among them, so keep their
directmap TLB entries across switches between guest kernel and user
mode.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- d->arch.vcpu_pt and d->arch.sparse_dmap are now d->arch.asi.vcpu_pt
and d->arch.asi.sparse_dmap.
- Drop sparse_dmap_active and sparse_dmap_activate(): root page-tables
carry the sparse view from their creation, dom0's included (Roger).
- The sparse view's slots are now marked NX in "x86/mm: build and
maintain a sparse view of the directmap".
- PV shim: replace_va_mapping() maps the guest's L4 with
map_domain_page().
- Give sparse_dmap_wanted() its hardware-domain case (Roger).
- init_xen_l4_slots() takes the directmap slots from the sparse view,
rather than copying the full directmap's and overwriting them;
sparse_dmap_install() goes (Roger).
- Message: drop the sentence on NX (Roger).
Changes in v3:
- New in this version.
---
xen/arch/x86/include/asm/sparse-directmap.h | 9 +++++++
xen/arch/x86/mm.c | 28 +++++++++++++++++++--
xen/arch/x86/pv/shim.c | 4 ++-
xen/arch/x86/spec_ctrl.c | 7 ++++++
4 files changed, 45 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
index ea14d44334..2f8b94f4c1 100644
--- a/xen/arch/x86/include/asm/sparse-directmap.h
+++ b/xen/arch/x86/include/asm/sparse-directmap.h
@@ -31,6 +31,14 @@ static inline bool sparse_dmap_configured(void)
return opt_sparse_dmap_pv || opt_sparse_dmap_hvm || opt_sparse_dmap_hwdom;
}
+/* Whether a new domain of the given type is to use the sparse view. */
+static inline bool sparse_dmap_wanted(bool pv, bool hwdom)
+{
+ return sparse_dmap_root &&
+ (hwdom ? opt_sparse_dmap_hwdom
+ : pv ? opt_sparse_dmap_pv : opt_sparse_dmap_hvm);
+}
+
void sparse_dmap_init(void);
int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
pte_attr_t flags);
@@ -42,6 +50,7 @@ int sparse_dmap_drop_pagetable(mfn_t mfn);
#else /* !CONFIG_SPARSE_DIRECTMAP */
static inline bool sparse_dmap_configured(void) { return false; }
+static inline bool sparse_dmap_wanted(bool pv, bool hwdom) { return false; }
static inline void sparse_dmap_init(void) {}
static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
unsigned long nr, pte_attr_t flags)
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 7e3009251c..7b5a355e6e 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -1703,7 +1703,8 @@ struct page_info *perdomain_l3(const struct vcpu *v)
* l4t, l4mfn, and v are mandatory, but l4mfn doesn't need to be the mfn under
* *l4t. All other parameters are optional and will either fill or zero the
* appropriate slots. Pagetables not shared with guests will gain the
- * extended directmap.
+ * extended directmap, unless the domain runs on the sparse view of the
+ * directmap, which replaces the directmap slots.
*
* v is the vCPU the table is private to or, for a table the domain's vCPUs
* share, any of them: slot 260 gets v's per-domain area (perdomain_l3()),
@@ -1715,7 +1716,7 @@ void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
const struct domain *d = v->domain;
/*
* PV vcpus need a shortened directmap. HVM and Idle vcpus get the full
- * directmap.
+ * directmap, unless the domain runs on the sparse view (below).
*/
bool short_directmap = !paging_mode_external(d);
@@ -1750,6 +1751,29 @@ void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
l4t[l4_table_offset(PERDOMAIN_VIRT_START)];
/* Slot 261-: text/data/bss, RW M2P, vmap, frametable, directmap. */
+#ifdef CONFIG_SPARSE_DIRECTMAP
+ if ( d->arch.asi.sparse_dmap )
+ {
+ unsigned int i = l4_table_offset(SPARSE_DMAP_START);
+
+ memcpy(&l4t[l4_table_offset(XEN_VIRT_START)],
+ &idle_pg_table[l4_table_offset(XEN_VIRT_START)],
+ (i - l4_table_offset(XEN_VIRT_START)) * sizeof(*l4t));
+
+ /*
+ * Every slot the sparse view covers gets its L3, whatever the full
+ * directmap has there, so that memory added later is covered too. A
+ * PV root has guest slots above; any other root loses the rest of the
+ * directmap, which the sparse view does not cover.
+ */
+ for ( ; i < l4_table_offset(SPARSE_DMAP_END); i++ )
+ l4t[i] = sparse_dmap_root[i];
+ if ( !short_directmap )
+ for ( ; i < l4_table_offset(DIRECTMAP_VIRT_END); i++ )
+ l4t[i] = l4e_empty();
+ }
+ else
+#endif
#ifndef NDEBUG
if ( short_directmap &&
unlikely(root_pgt_pv_xen_slots < ROOT_PAGETABLE_PV_XEN_SLOTS) )
diff --git a/xen/arch/x86/pv/shim.c b/xen/arch/x86/pv/shim.c
index 41000b0523..2a47dbc56c 100644
--- a/xen/arch/x86/pv/shim.c
+++ b/xen/arch/x86/pv/shim.c
@@ -148,12 +148,14 @@ const struct platform_bad_page *__init pv_shim_reserved_pages(unsigned int *size
static void __init replace_va_mapping(struct domain *d, l4_pgentry_t *l4start,
unsigned long va, mfn_t mfn)
{
- l4_pgentry_t l4e = l4start[l4_table_offset(va)];
+ const l4_pgentry_t *l4t = map_domain_page(virt_to_mfn(l4start));
+ l4_pgentry_t l4e = l4t[l4_table_offset(va)];
l3_pgentry_t l3e = l3e_from_l4e(l4e, l3_table_offset(va));
l2_pgentry_t l2e = l2e_from_l3e(l3e, l2_table_offset(va));
l1_pgentry_t *pl1e = map_l1t_from_l2e(l2e) + l1_table_offset(va);
struct page_info *page = mfn_to_page(l1e_get_mfn(*pl1e));
+ unmap_domain_page(l4t);
put_page_and_type(page);
*pl1e = l1e_from_mfn(mfn, (!is_pv_32bit_domain(d) ? L1_PROT
diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c
index 66c878212a..32e6ba28a3 100644
--- a/xen/arch/x86/spec_ctrl.c
+++ b/xen/arch/x86/spec_ctrl.c
@@ -1943,6 +1943,13 @@ void spec_ctrl_init_domain(struct domain *d)
d->arch.asi.vcpu_pt = !pv && (is_hardware_domain(d) ? opt_pvh_vcpu_pt_hwdom
: opt_vcpu_pt_hvm);
+
+ /*
+ * The sparse directmap view needs a mapcache: PV domains all have one,
+ * HVM domains through per-vCPU page-tables.
+ */
+ d->arch.asi.sparse_dmap = (pv || d->arch.asi.vcpu_pt) &&
+ sparse_dmap_wanted(pv, is_hardware_domain(d));
}
void __init init_speculation_mitigations(void)
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 20/30] x86/spec-ctrl: wire the sparse directmap view into the asi= option
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (18 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 19/30] x86/mm: run guest contexts on the sparse directmap view George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 21/30] xen/xmalloc: return an emptied region after dropping the pool lock George Dunlap
` (9 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Andrew Cooper, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné, Stefano Stabellini,
Teddy Astie
Let asi= choose which domains run on the sparse view of the directmap,
when the hypervisor is built with CONFIG_SPARSE_DIRECTMAP:
* `asi` and `asi=<bool>` enable or disable the sparse view, along with
vCPU-PT, for every domain including the hardware domain, PV or PVH.
HVM domains, a PVH hardware domain included, use the sparse view
together with vCPU-PT, which gives them the mapcache it needs; PV
domains all have one. A PV domain on the sparse view keeps its
domain-wide mapcache and per-domain mappings: the sparse view
separates it from other domains, not its vCPUs from each other.
* `asi=hvm=<bool>`, `asi=pv=<bool>` and `asi=dom0=<bool>` do the same
for HVM or PV domains other than the hardware domain, or for the
hardware domain, whatever its type: `asi=1,dom0=0` leaves the
hardware domain out of ASI altogether.
* A new `asi=directmap=<bool>` selects the sparse view for all domains,
as `asi=vcpu-pt=<bool>` selects vCPU-PT; `vcpu-pt=` alone leaves the
sparse view off. On a hypervisor built without
CONFIG_SPARSE_DIRECTMAP the forms enabling all features leave the
sparse view off, and `directmap=` is rejected as invalid, as
compiled-out `spec-ctrl=` sub-options are. A domain type the build
lacks is ignored, so that the sparse view is not built for nothing.
* For now, a PV domain uses one of XPTI and the sparse view, not
both; a later patch lets them combine. Enabled explicitly, for
dom0, domUs or both, XPTI takes precedence: the PV domains it
covers keep the full directmap, and the boot log says so. Left to
its default, XPTI yields: it is disabled for the PV domains using
the sparse view, with a boot-log warning where it would otherwise
be enabled, as the sparse view does not replace it yet (it still
maps the xenheap and Xen's stacks). Each domain's case is settled
at its creation, and again if it becomes 32-bit, which XPTI does
not cover: switch_compat() then puts a domain that XPTI kept off
the sparse view on it, with the domain-wide mapcache, which release
builds may not have set up for it.
The boot log gains a line with the directmap mode, "sparse" once a
sparse view is configured and "full" otherwise, and the "ASI features
for ..." lines, one for PV domUs now among them, say which domains use
the sparse view.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- Docs, comments and message: "domains" throughout, not "guests"
(Roger).
- Follow the rename of opt_vcpu_pt_hwdom to opt_pvh_vcpu_pt_hwdom and
the move of the per-domain settings into d->arch.asi.
- pv= and dom0= move to the patch introducing asi=, beside hvm=;
opt_sparse_dmap_hwdom and sparse_dmap_wanted()'s hardware-domain case
to the patches that add the other knobs and sparse_dmap_wanted()
(Roger).
- directmap= on a build without CONFIG_SPARSE_DIRECTMAP is rejected as
invalid, as compiled-out spec-ctrl= sub-options are, rather than
ignored with a warning (Roger).
- A PV domain that XPTI kept off the sparse view goes on it when it
becomes 32-bit, which ends XPTI; it used to end up with neither.
Changes in v3:
- New in this version.
---
docs/misc/xen-command-line.pandoc | 30 +++++-
xen/arch/x86/include/asm/sparse-directmap.h | 7 ++
xen/arch/x86/pv/domain.c | 16 ++++
xen/arch/x86/spec_ctrl.c | 101 ++++++++++++++++++--
4 files changed, 139 insertions(+), 15 deletions(-)
diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
index 839d06da26..092c858f34 100644
--- a/docs/misc/xen-command-line.pandoc
+++ b/docs/misc/xen-command-line.pandoc
@@ -203,7 +203,7 @@ to appropriate auditing by Xen. Argo is disabled by default.
buggy or malicious other domain spamming the ring.
### asi (x86)
-> `= List of [ <bool>, {pv,hvm,dom0}=<bool>,
+> `= List of [ <bool>, {pv,hvm,dom0}=<bool>, directmap=<bool>,
> vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
> Default: `false`
@@ -215,14 +215,16 @@ mitigations for certain speculative related attacks, at the cost of mapping
sensitive information on-demand. It might also offer some protection against
unmitigated speculation-related attacks.
-The mechanisms currently implemented apply to HVM domains, including a PVH
-hardware domain. PV domains, including a PV hardware domain, are unaffected.
+The mechanisms currently implemented are `vcpu-pt`, for HVM domains including
+a PVH hardware domain, and `directmap`, for all domains including the hardware
+domain, PV or PVH. The plain boolean enables or disables both, wherever they
+apply.
* `pv=`, `hvm=` and `dom0=` enable the features for PV domains or HVM domains
other than the hardware domain, or for the hardware domain, PV or PVH. The
plain boolean does all three: `asi=1,dom0=0` leaves the hardware domain
- out. The forms naming no domain type (an un-suffixed `vcpu-pt=<bool>`)
- cover the hardware domain too.
+ out. The forms naming no domain type (an un-suffixed `vcpu-pt=<bool>`, and
+ `directmap=<bool>`) cover the hardware domain too.
**WARNING: manual de-selection of enabled options will invalidate any
protection offered by the feature. The fine grained options provided below
@@ -231,6 +233,18 @@ are meant to be used for debugging purposes only.**
* `vcpu-pt` gives each vCPU its own per-domain area: the per-domain slot of
each vCPU's monitor table maps a region private to that vCPU.
+* `directmap` (only available when Xen is built with
+ `CONFIG_SPARSE_DIRECTMAP`) runs guest contexts on a sparse view of the
+ directmap, which maps the memory Xen needs to reach at all times (the Xen
+ heap, Xen's own image, firmware tables), while guest memory is mapped
+ transiently. This selection is system-wide; `pv=`, `hvm=` and `dom0=`
+ make it for one domain type, along with the other features. HVM domains use
+ the sparse view only together with `vcpu-pt`, and PV domains only without
+ XPTI, which yields to the sparse view unless enabled explicitly (see
+ `xpti`). The whole-feature forms (`asi`, `asi=<bool>`, and `pv=`, `hvm=`
+ and `dom0=`) set this too, on builds that support it; on other builds they
+ leave it off.
+
### asid (x86)
> `= <boolean>`
@@ -3132,6 +3146,12 @@ Meltdown for all domains.
With `dom0` and `domu` it is possible to control page table isolation
for dom0 or guest domains only.
+With `asi`, XPTI and the sparse directmap view (`asi=directmap`) exclude each
+other for a PV domain. Enabled explicitly, as a whole or with `dom0` or
+`domu`, XPTI keeps the PV domains it covers off the sparse view, and the boot
+log says so. Left to its default, XPTI is disabled for the PV domains using
+the sparse view, with a boot-log warning where it would otherwise be enabled.
+
### xsave (x86)
> `= <boolean>`
diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
index 2f8b94f4c1..db4df82215 100644
--- a/xen/arch/x86/include/asm/sparse-directmap.h
+++ b/xen/arch/x86/include/asm/sparse-directmap.h
@@ -31,6 +31,10 @@ static inline bool sparse_dmap_configured(void)
return opt_sparse_dmap_pv || opt_sparse_dmap_hvm || opt_sparse_dmap_hwdom;
}
+static inline bool sparse_dmap_pv(void) { return opt_sparse_dmap_pv; }
+static inline bool sparse_dmap_hvm(void) { return opt_sparse_dmap_hvm; }
+static inline bool sparse_dmap_hwdom(void) { return opt_sparse_dmap_hwdom; }
+
/* Whether a new domain of the given type is to use the sparse view. */
static inline bool sparse_dmap_wanted(bool pv, bool hwdom)
{
@@ -51,6 +55,9 @@ int sparse_dmap_drop_pagetable(mfn_t mfn);
static inline bool sparse_dmap_configured(void) { return false; }
static inline bool sparse_dmap_wanted(bool pv, bool hwdom) { return false; }
+static inline bool sparse_dmap_pv(void) { return false; }
+static inline bool sparse_dmap_hvm(void) { return false; }
+static inline bool sparse_dmap_hwdom(void) { return false; }
static inline void sparse_dmap_init(void) {}
static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
unsigned long nr, pte_attr_t flags)
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index f139d8f83c..579d6133f5 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -248,6 +248,20 @@ int switch_compat(struct domain *d)
d->arch.has_32bit_shinfo = 1;
d->arch.pv.is_32bit = true;
+ /*
+ * XPTI, cleared below, is for 64-bit domains. A domain it kept off the
+ * sparse directmap view goes on it now if wanted, and the domain-wide
+ * mapcache the view needs is set up for the domain and its vCPUs.
+ */
+ if ( d->arch.pv.xpti && sparse_dmap_wanted(true, is_hardware_domain(d)) )
+ {
+ d->arch.asi.sparse_dmap = true;
+ mapcache_domain_init(d);
+ for_each_vcpu ( d, v )
+ if ( (rc = mapcache_vcpu_init(v)) )
+ goto undo_and_fail;
+ }
+
/*
* For 32bit PV guests the shared_info machine address must fit in a 32-bit
* field within the guest's start_info structure. We might need to free
@@ -301,6 +315,8 @@ int switch_compat(struct domain *d)
undo_and_fail:
d->arch.pv.is_32bit = d->arch.has_32bit_shinfo = false;
+ if ( d->arch.pv.xpti )
+ d->arch.asi.sparse_dmap = false;
for_each_vcpu( d, v )
{
free_compat_arg_xlat(v);
diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c
index 32e6ba28a3..25726978b9 100644
--- a/xen/arch/x86/spec_ctrl.c
+++ b/xen/arch/x86/spec_ctrl.c
@@ -387,6 +387,19 @@ int8_t __ro_after_init opt_xpti_domu = -1;
static __init void xpti_init_default(void)
{
+ /*
+ * XPTI and the sparse directmap view exclude each other for a PV domain
+ * (see spec_ctrl_init_domain()). Enabled explicitly, XPTI takes
+ * precedence: the PV domains it covers stay off the sparse view. Left to
+ * its default, it yields: it is off where the sparse view is asked for.
+ */
+ if ( !opt_dom0_pvh && opt_xpti_hwdom == 1 && sparse_dmap_hwdom() )
+ printk(XENLOG_ERR
+ "XPTI enabled, disabling Dom0 sparse-directmap\n");
+ if ( opt_xpti_domu == 1 && sparse_dmap_pv() )
+ printk(XENLOG_ERR
+ "XPTI enabled, disabling PV DomU sparse-directmap\n");
+
if ( (boot_cpu_data.vendor & (X86_VENDOR_AMD | X86_VENDOR_HYGON)) ||
cpu_has_rdcl_no )
{
@@ -397,10 +410,21 @@ static __init void xpti_init_default(void)
}
else
{
+ bool yield_hwdom = opt_xpti_hwdom < 0 && !opt_dom0_pvh &&
+ sparse_dmap_hwdom();
+ bool yield_domu = opt_xpti_domu < 0 && sparse_dmap_pv();
+
+ /* The default would be on here: say where it yields. */
+ if ( yield_hwdom || yield_domu )
+ printk(XENLOG_WARNING
+ "XPTI defaults off for PV %s on the sparse directmap\n",
+ yield_hwdom ? (yield_domu ? "Dom0 and DomU" : "Dom0")
+ : "DomU");
+
if ( opt_xpti_hwdom < 0 )
- opt_xpti_hwdom = 1;
+ opt_xpti_hwdom = !sparse_dmap_hwdom();
if ( opt_xpti_domu < 0 )
- opt_xpti_domu = 1;
+ opt_xpti_domu = !sparse_dmap_pv();
}
}
@@ -491,6 +515,23 @@ static int __init cf_check parse_pv_l1tf(const char *s)
}
custom_param("pv-l1tf", parse_pv_l1tf);
+/*
+ * Set the sparse directmap view's knob for each domain type given. A type
+ * the build lacks stays off, so that no view is built for it, and without
+ * CONFIG_SPARSE_DIRECTMAP all stay off.
+ */
+static void __init set_sparse_dmap(bool pv, bool hvm, bool hwdom, bool val)
+{
+#ifdef CONFIG_SPARSE_DIRECTMAP
+ if ( pv && IS_ENABLED(CONFIG_PV) )
+ opt_sparse_dmap_pv = val;
+ if ( hvm && IS_ENABLED(CONFIG_HVM) )
+ opt_sparse_dmap_hvm = val;
+ if ( hwdom )
+ opt_sparse_dmap_hwdom = val;
+#endif
+}
+
static int __init cf_check parse_asi(const char *s)
{
const char *ss;
@@ -498,7 +539,10 @@ static int __init cf_check parse_asi(const char *s)
/* Interpret 'asi' alone in its positive boolean form. */
if ( *s == '\0' )
+ {
opt_pvh_vcpu_pt_hwdom = opt_vcpu_pt_hvm = true;
+ set_sparse_dmap(true, true, true, true);
+ }
do {
ss = strchr(s, ',');
@@ -511,15 +555,33 @@ static int __init cf_check parse_asi(const char *s)
case 0:
case 1:
opt_pvh_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
+ set_sparse_dmap(true, true, true, val);
break;
default:
if ( (val = parse_boolean("pv", s, ss)) >= 0 )
- /* PV domains are unaffected. */;
+ set_sparse_dmap(true, false, false, val);
else if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
+ {
opt_vcpu_pt_hvm = val;
+ set_sparse_dmap(false, true, false, val);
+ }
else if ( (val = parse_boolean("dom0", s, ss)) >= 0 )
+ {
opt_pvh_vcpu_pt_hwdom = val;
+ set_sparse_dmap(false, false, true, val);
+ }
+ else if ( (val = parse_boolean("directmap", s, ss)) >= 0 )
+ {
+ /* System-wide: every domain type, the hardware domain too. */
+ if ( IS_ENABLED(CONFIG_SPARSE_DIRECTMAP) )
+ set_sparse_dmap(true, true, true, val);
+ else
+ {
+ no_config_param("SPARSE_DIRECTMAP", "asi", s, ss);
+ rc = -EINVAL;
+ }
+ }
else if ( (val = parse_boolean("vcpu-pt", s, ss)) != -1 )
{
switch ( val )
@@ -748,12 +810,30 @@ static void __init print_details(enum ind_thunk thunk)
opt_pv_l1tf_domu ? "enabled" : "disabled");
#endif
- printk(" ASI features for Dom0:%s\n",
- opt_dom0_pvh && opt_pvh_vcpu_pt_hwdom ? " vCPU-PT" : " None");
+ {
+ bool hwdom_pt = opt_dom0_pvh && opt_pvh_vcpu_pt_hwdom;
+ bool hwdom_sd = sparse_dmap_hwdom() &&
+ (opt_dom0_pvh ? hwdom_pt : !opt_xpti_hwdom);
+
+ printk(" ASI directmap mode: %s\n",
+ sparse_dmap_configured() ? "sparse" : "full");
+ printk(" ASI features for Dom0:%s%s%s\n",
+ hwdom_pt || hwdom_sd ? "" : " None",
+ hwdom_pt ? " vCPU-PT" : "",
+ hwdom_sd ? " sparse-directmap" : "");
#ifdef CONFIG_HVM
- printk(" ASI features for HVM VMs:%s\n",
- opt_vcpu_pt_hvm ? " vCPU-PT" : " None");
+ printk(" ASI features for HVM VMs:%s%s%s\n",
+ opt_vcpu_pt_hvm ? "" : " None",
+ opt_vcpu_pt_hvm ? " vCPU-PT" : "",
+ opt_vcpu_pt_hvm && sparse_dmap_hvm()
+ ? " sparse-directmap" : "");
#endif
+#ifdef CONFIG_PV
+ printk(" ASI features for PV VMs:%s\n",
+ sparse_dmap_pv() && !opt_xpti_domu
+ ? " sparse-directmap" : " None");
+#endif
+ }
}
static bool __init check_smt_enabled(void)
@@ -1946,9 +2026,10 @@ void spec_ctrl_init_domain(struct domain *d)
/*
* The sparse directmap view needs a mapcache: PV domains all have one,
- * HVM domains through per-vCPU page-tables.
+ * HVM domains through per-vCPU page-tables. A PV domain using XPTI
+ * stays on the full directmap.
*/
- d->arch.asi.sparse_dmap = (pv || d->arch.asi.vcpu_pt) &&
+ d->arch.asi.sparse_dmap = (pv ? !d->arch.pv.xpti : d->arch.asi.vcpu_pt) &&
sparse_dmap_wanted(pv, is_hardware_domain(d));
}
@@ -2242,7 +2323,7 @@ void __init init_speculation_mitigations(void)
hw_smt_enabled && default_xen_spec_ctrl )
setup_force_cpu_cap(X86_FEATURE_SC_MSR_IDLE);
- if ( opt_pvh_vcpu_pt_hwdom || opt_vcpu_pt_hvm )
+ if ( opt_pvh_vcpu_pt_hwdom || opt_vcpu_pt_hvm || sparse_dmap_configured() )
warning_add(
"Address Space Isolation is not functional, this option is\n"
"intended to be used only for development purposes.\n");
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 21/30] xen/xmalloc: return an emptied region after dropping the pool lock
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (19 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 20/30] x86/spec-ctrl: wire the sparse directmap view into the asi= option George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 22/30] x86/mm: split a sparse directmap superpage with a page being freed George Dunlap
` (8 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Andrew Cooper, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné, Stefano Stabellini
When a free leaves a whole region of an xmalloc pool empty,
xmem_pool_free() hands the region back through pool->put_mem() with
pool->lock still held. For the xenheap pool that is
free_xenheap_pages(), which takes the heap lock inside the pool lock
and runs arch_xenheap_pages_free(). On x86 with a sparse directmap
view configured, the latter takes the page out of the view, at the
cost of one to three rounds of TLB-flush IPIs to every CPU, while every
other CPU's sub-page xmalloc() and xfree() waits on the pool lock.
By then the region is on no free list and no longer counted in the
pool, so nothing else can reach it. Drop the lock first and return the
region afterwards, as xmem_pool_alloc() already does when it calls
pool->get_mem(), and say at both callbacks that they are called without
the pool lock, so that they keep not relying on it.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
xen/common/xmalloc_tlsf.c | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/xen/common/xmalloc_tlsf.c b/xen/common/xmalloc_tlsf.c
index b43219b4af..caf33f1a04 100644
--- a/xen/common/xmalloc_tlsf.c
+++ b/xen/common/xmalloc_tlsf.c
@@ -106,7 +106,10 @@ struct xmem_pool {
unsigned long used_size;
unsigned long num_regions;
- /* User provided functions for expanding/shrinking pool */
+ /*
+ * User provided functions for expanding/shrinking pool. They are
+ * called without the pool lock held, and must not rely on it.
+ */
xmem_pool_get_memory *get_mem;
xmem_pool_put_memory *put_mem;
@@ -488,17 +491,18 @@ void xmem_pool_free(void *ptr, struct xmem_pool *pool)
if ( (b->prev_hdr == NULL) && ((tmp_b->size & BLOCK_SIZE_MASK) == 0) )
{
- pool->put_mem(b);
pool->num_regions--;
pool->used_size -= BHDR_OVERHEAD; /* sentinel block header */
- goto out;
+ spin_unlock(&pool->lock);
+ pool->put_mem(b);
+ return;
}
INSERT_BLOCK(b, pool, fl, sl);
tmp_b->size |= PREV_FREE;
tmp_b->prev_hdr = b;
- out:
+
spin_unlock(&pool->lock);
}
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 22/30] x86/mm: split a sparse directmap superpage with a page being freed
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (20 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 21/30] xen/xmalloc: return an emptied region after dropping the pool lock George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 23/30] x86/mm: introduce populate_perdomain_mapping() George Dunlap
` (7 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie,
Anthony PERARD, Michal Orzel, Julien Grall, Stefano Stabellini
A xenheap free takes its pages out of the sparse directmap view. Where a
superpage covers them, the superpage has to be split, and
modify_mappings_in() allocates the page-table page for that. The sparse
view has such superpages wherever a xenheap block of 2MiB or more was
mapped whole, and wherever map_pages_in() merged the entries of a 2MiB
region into one, as it does in the full directmap, once all 512 pages
were mapped. The allocation can fail once the hardware domain exists,
and the free then keeps its pages out of the heap for good, with a
warning.
Use one of the pages being freed instead. The sparse view's free hook
makes the first of them what alloc_xen_pagetable() would return, and
passes it to modify_mappings_in(), which uses it, should a split need a
page, before allocating one. The new L1 maps the rest of the superpage,
and the page's own entry is cleared along with the others being freed.
The hook tells free_xenheap_pages() whether it kept the page, and
free_xenheap_pages() then frees the rest of the block as blocks of orders
0 to order - 1. The page goes back to the heap once the L1 has emptied,
as any page-table page of the sparse view does.
One page is enough if the sparse view has no 1GiB pages: a block smaller
than 2MiB lies within one 2MiB page and needs one L1, and a larger block
covers whole 2MiB pages and needs none, where a block within a 1GiB page
could need an L2 as well. So map_pages_in() makes 1GiB pages, by mapping
or by merging, only in idle_pg_table's hierarchy; the sparse view keeps
its 2MiB pages, mapped and merged. A xenheap free then allocates nothing
in the sparse view.
The costs: a 2MiB region split this way can merge again only once it has
emptied, as its L1 is one of its own pages; where the sparse view maps a
whole 1GiB region, which takes memory never handed to the heap and
xenheap memory filling it, contexts on it use 2MiB pages instead of one
1GiB page; and arch_xenheap_pages_free() gains a parameter. The other
removals from the sparse view (memory handed to the heap,
idle_pg_table's boot-time page-tables, and the undo of a failed xenheap
allocation) still allocate when they split a superpage.
The sparse view is correct without this, as a free that cannot split its
superpage fails safe. It is kept apart from "x86/mm: build and maintain a
sparse view of the directmap" so that it can be weighed on its own.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
xen/arch/x86/include/asm/mm.h | 2 +-
xen/arch/x86/mm.c | 34 ++++++++++++++++++++++------
xen/arch/x86/sparse-directmap.c | 40 ++++++++++++++++++++++++++-------
xen/common/page_alloc.c | 14 ++++++++----
xen/include/xen/mm.h | 11 ++++++---
5 files changed, 78 insertions(+), 23 deletions(-)
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 7612cf66ef..af43ae01b0 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -601,7 +601,7 @@ void *alloc_mapped_pagetable(mfn_t *pmfn);
int map_pages_in(l4_pgentry_t *root, unsigned long virt, mfn_t mfn,
unsigned long nr_mfns, pte_attr_t flags);
int modify_mappings_in(l4_pgentry_t *root, unsigned long s, unsigned long e,
- pte_attr_t nf);
+ pte_attr_t nf, mfn_t *spare);
#ifdef CONFIG_SPARSE_DIRECTMAP
mfn_t xen_mapping_lookup(l4_pgentry_t *root, unsigned long va,
pte_attr_t *flags, unsigned long *nr);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 7b5a355e6e..449ae561dc 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -5540,6 +5540,10 @@ static void free_pagetable_in(const l4_pgentry_t *root, mfn_t mfn)
/*
* map_pages_to_xen() in the hierarchy rooted at @root: idle_pg_table's, or
* another hierarchy sharing its virtual addresses.
+ *
+ * Only idle_pg_table's hierarchy gets 1GiB pages: a xenheap free splits a
+ * superpage of the sparse directmap view with one of the pages it frees,
+ * which is enough for a 2MiB page only.
*/
int map_pages_in(
l4_pgentry_t *root,
@@ -5549,6 +5553,7 @@ int map_pages_in(
pte_attr_t flags)
{
bool locking = system_state > SYS_STATE_boot;
+ bool page1gb = cpu_has_page1gb && root == idle_pg_table;
l3_pgentry_t *pl3e = NULL, ol3e;
l2_pgentry_t *pl2e = NULL, ol2e;
l1_pgentry_t *pl1e, ol1e;
@@ -5601,7 +5606,7 @@ int map_pages_in(
L3T_LOCK(current_l3page);
ol3e = *pl3e;
- if ( cpu_has_page1gb &&
+ if ( page1gb &&
IS_L3E_ALIGNED(virt, flags & _PAGE_PRESENT ? mfn : _mfn(0)) &&
nr_mfns >= (1UL << (L3_PAGETABLE_SHIFT - PAGE_SHIFT)) &&
!(flags & (_PAGE_PAT | MAP_SMALL_PAGES)) )
@@ -5887,7 +5892,7 @@ int map_pages_in(
}
check_l3:
- if ( cpu_has_page1gb &&
+ if ( page1gb &&
(flags == PAGE_HYPERVISOR) &&
((nr_mfns == 0) || IS_L3E_ALIGNED(virt, mfn)) )
{
@@ -5966,6 +5971,18 @@ int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
return map_pages_to_xen(virt, INVALID_MFN, nr_mfns, MAP_SMALL_PAGES);
}
+static mfn_t take_or_alloc_pagetable(mfn_t *spare)
+{
+ mfn_t mfn = spare ? *spare : INVALID_MFN;
+
+ if ( mfn_eq(mfn, INVALID_MFN) )
+ return alloc_xen_pagetable();
+
+ *spare = INVALID_MFN;
+
+ return mfn;
+}
+
/*
* Alter the permissions of a range of Xen virtual address space.
*
@@ -5979,10 +5996,13 @@ int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
* It is an error to call with present flags over an unpopulated range.
*
* modify_mappings_in() does so in the hierarchy rooted at @root, as
- * map_pages_in() does.
+ * map_pages_in() does. @spare, if not NULL, may hold a page the caller
+ * gives up for a page-table page, which shattering a superpage then uses
+ * before allocating one. *@spare reads INVALID_MFN once it has been used,
+ * and the page is then no longer the caller's.
*/
int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
- unsigned long e, pte_attr_t nf)
+ unsigned long e, pte_attr_t nf, mfn_t *spare)
{
bool locking = system_state > SYS_STATE_boot;
l3_pgentry_t *pl3e = NULL;
@@ -6046,7 +6066,7 @@ int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
}
/* PAGE1GB: shatter the superpage and fall through. */
- l2mfn = alloc_xen_pagetable();
+ l2mfn = take_or_alloc_pagetable(spare);
if ( mfn_eq(l2mfn, INVALID_MFN) )
goto out;
@@ -6104,7 +6124,7 @@ int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
{
l1_pgentry_t *l1t;
/* PSE: shatter the superpage and try again. */
- mfn_t l1mfn = alloc_xen_pagetable();
+ mfn_t l1mfn = take_or_alloc_pagetable(spare);
if ( mfn_eq(l1mfn, INVALID_MFN) )
goto out;
@@ -6262,7 +6282,7 @@ int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
{
- int rc = modify_mappings_in(idle_pg_table, s, e, nf);
+ int rc = modify_mappings_in(idle_pg_table, s, e, nf, NULL);
if ( !rc )
rc = sparse_dmap_mirror_modify(s, e, nf);
diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
index d35a8d6979..adec474f72 100644
--- a/xen/arch/x86/sparse-directmap.c
+++ b/xen/arch/x86/sparse-directmap.c
@@ -121,12 +121,13 @@ static int add_to_sparse_view(l4_pgentry_t *root, unsigned long mfn,
return 0;
}
-static int remove_from_sparse_view(unsigned long mfn, unsigned long nr)
+static int remove_from_sparse_view(unsigned long mfn, unsigned long nr,
+ mfn_t *spare)
{
unsigned long va = (unsigned long)mfn_to_virt(mfn);
return modify_mappings_in(sparse_dmap_root, va, va + (nr << PAGE_SHIFT),
- _PAGE_NONE);
+ _PAGE_NONE, spare);
}
/* Memory is about to be handed to the heap allocator. */
@@ -144,7 +145,7 @@ bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
return true;
}
- if ( remove_from_sparse_view(s, nr) )
+ if ( remove_from_sparse_view(s, nr, NULL) )
{
printk(XENLOG_ERR
"sparse directmap: cannot unmap MFNs %#lx-%#lx, not freeing\n",
@@ -188,10 +189,18 @@ int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
/*
* Xenheap pages are about to be freed. A failure means the pages may still
* be reachable through the sparse view, and must not be reused.
+ *
+ * If the caller allows, the first page serves as the page-table page that
+ * splitting a superpage of the sparse view needs: the new L1 maps the rest
+ * of the superpage, and the page leaves the sparse view with the others.
+ * One page is enough: the sparse view has no 1GiB pages, a block below 2MiB
+ * lies within one 2MiB page, and a larger one covers whole ones.
*/
-int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
+int arch_xenheap_pages_free(mfn_t mfn, unsigned int order, bool *first_kept)
{
unsigned long s = mfn_x(mfn), nr = 1UL << order;
+ mfn_t spare = INVALID_MFN;
+ int rc;
if ( !sparse_dmap_configured() )
return 0;
@@ -204,7 +213,22 @@ int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
return 0;
}
- return remove_from_sparse_view(s, nr);
+ if ( first_kept )
+ {
+ struct page_info *pg = mfn_to_page(mfn);
+
+ /* Made what alloc_xen_pagetable() would return, in case it is used. */
+ pg->count_info &= ~PGC_xen_heap;
+ page_set_owner(pg, NULL);
+ spare = mfn;
+ }
+
+ rc = remove_from_sparse_view(s, nr, &spare);
+
+ if ( first_kept && mfn_eq(spare, INVALID_MFN) )
+ *first_kept = true;
+
+ return rc;
}
/* Memory leaves Xen's hands without passing through the heap. */
@@ -233,7 +257,7 @@ int sparse_dmap_drop_pagetable(mfn_t mfn)
INVALID_MFN) )
return 0;
- rc = remove_from_sparse_view(s, 1);
+ rc = remove_from_sparse_view(s, 1, NULL);
if ( rc )
printk(XENLOG_WARNING
"Leaking page-table page %"PRI_mfn": still in the sparse directmap\n",
@@ -267,7 +291,7 @@ int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
e = min(e, SPARSE_DMAP_END);
if ( !(nf & _PAGE_PRESENT) )
- return modify_mappings_in(sparse_dmap_root, s, e, nf);
+ return modify_mappings_in(sparse_dmap_root, s, e, nf, NULL);
/*
* A permission change applies where the sparse view maps anything: it
@@ -283,7 +307,7 @@ int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
n = min(n, (e - va) >> PAGE_SHIFT);
if ( !mfn_eq(mfn, INVALID_MFN) &&
(rc = modify_mappings_in(sparse_dmap_root, va,
- va + (n << PAGE_SHIFT), nf)) )
+ va + (n << PAGE_SHIFT), nf, NULL)) )
return rc;
va += n << PAGE_SHIFT;
}
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 69e8e982b6..055e358f3a 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -2561,7 +2561,7 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags)
* The hook may have mapped part of the pages before failing: take
* them out again as a free would, and leak them if that fails too.
*/
- if ( arch_xenheap_pages_free(page_to_mfn(pg), order) )
+ if ( arch_xenheap_pages_free(page_to_mfn(pg), order, NULL) )
printk(XENLOG_WARNING
"Leaking xenheap pages %"PRI_mfn"+%u: still mapped\n",
mfn_x(page_to_mfn(pg)), 1U << order);
@@ -2580,6 +2580,7 @@ void free_xenheap_pages(void *v, unsigned int order)
{
struct page_info *pg;
unsigned int i;
+ bool first_kept = false;
ASSERT_ALLOC_CONTEXT();
@@ -2593,7 +2594,7 @@ void free_xenheap_pages(void *v, unsigned int order)
* they may remain reachable through it: leak them rather than hand them
* out.
*/
- if ( arch_xenheap_pages_free(page_to_mfn(pg), order) )
+ if ( arch_xenheap_pages_free(page_to_mfn(pg), order, &first_kept) )
{
printk(XENLOG_WARNING
"Leaking xenheap pages %"PRI_mfn"+%u: still mapped\n",
@@ -2601,10 +2602,15 @@ void free_xenheap_pages(void *v, unsigned int order)
return;
}
- for ( i = 0; i < (1u << order); i++ )
+ for ( i = first_kept ? 1 : 0; i < (1u << order); i++ )
pg[i].count_info &= ~PGC_xen_heap;
- free_heap_pages(pg, order, true);
+ if ( !first_kept )
+ free_heap_pages(pg, order, true);
+ else
+ /* The rest of the block, as the blocks of orders 0 to order - 1. */
+ for ( i = 0; i < order; i++ )
+ free_heap_pages(pg + (1u << i), i, true);
}
#endif /* CONFIG_SEPARATE_XENHEAP */
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index 2bcc45cf58..debd511caf 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -108,8 +108,12 @@ bool arch_xenheap_needs_scrub(void);
* pages go through arch_xenheap_pages_free() before being freed again.
*/
int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order);
-/* Xenheap pages are to be freed. Non-zero: failed, don't reuse them. */
-int arch_xenheap_pages_free(mfn_t mfn, unsigned int order);
+/*
+ * Xenheap pages are to be freed. Non-zero: failed, don't reuse them. With
+ * @first_kept not NULL, the arch may keep the first page for itself, and
+ * then sets *@first_kept to true: that page is not the caller's to free.
+ */
+int arch_xenheap_pages_free(mfn_t mfn, unsigned int order, bool *first_kept);
#else
static inline bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
{
@@ -123,7 +127,8 @@ static inline int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
{
return 0;
}
-static inline int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
+static inline int arch_xenheap_pages_free(mfn_t mfn, unsigned int order,
+ bool *first_kept)
{
return 0;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 23/30] x86/mm: introduce populate_perdomain_mapping()
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (21 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 22/30] x86/mm: split a sparse directmap superpage with a page being freed George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 24/30] x86/mm: introduce a per-vCPU state window in the perdomain area George Dunlap
` (6 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
The per-domain area has central machinery for building its page-tables
and for managing the backing pages it owns itself:
create_perdomain_mapping() / destroy_perdomain_mapping(), used by the
mapcache and the compat argument-translation area. What the interface
lacks is a way to install a caller's own pages at a chosen address,
which a per-vCPU state window, introduced by a later patch, needs: its
users allocate their pages themselves and map them there.
Introduce populate_perdomain_mapping(v, va, mfn, nr, flags) for this.
It writes the given MFNs, with the given page-table flags, into v's view
of the per-domain area, walking the per-domain page-table structures
with transient mappings, so v need not be current. Callers don't need
to know where the page-tables live, how the area is structured, or
whether it is per-domain or per-vCPU.
We require the range to already have been populated down to the L1
tables by create_perdomain_mapping(). TLB flushing is left to the
caller. A present entry not owned by the area (!_PAGE_AVAIL0) is
replaced. A present entry owned by the area (_PAGE_AVAIL0, installed
by create_perdomain_mapping() itself) is never expected: such a page is
referenced only by the mapping, and no caller replaces area-owned
backing. Should one ever do so, the page is leaked with a message
rather than freed.
The function is not for contexts with interrupts off: its users map
their pages when allocating them.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- Re-introduced into the series.
- HVM only: the users are the state window's, all of which map their
pages at allocation time, so drop the linear-mapping fast path and the
IRQs-off walk that context-switch callers needed, along with the
map_domain_page_irqoff() variants.
---
xen/arch/x86/include/asm/mm.h | 3 ++
xen/arch/x86/mm.c | 62 +++++++++++++++++++++++++++++++++++
2 files changed, 65 insertions(+)
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index af43ae01b0..6d50e17259 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -620,6 +620,9 @@ int compat_arch_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg);
int create_perdomain_mapping(struct vcpu *v, unsigned long va,
unsigned int nr, l1_pgentry_t **pl1tab,
struct page_info **ppg);
+void populate_perdomain_mapping(const struct vcpu *v, unsigned long va,
+ const mfn_t *mfn, unsigned int nr,
+ unsigned int flags);
void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr);
void free_perdomain_mappings(struct domain *d);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 449ae561dc..bf029df566 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -6554,6 +6554,68 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
return rc;
}
+/*
+ * Map @nr pages, @mfn[0..nr-1], at consecutive pages from @va in v's view of
+ * the per-domain area, with page-table @flags. The range must lie within a
+ * single per-domain slot, and must already have been plumbed down to the L1
+ * tables by create_perdomain_mapping(). Present entries are replaced, and
+ * no TLB flushing is done.
+ *
+ * The walk maps the per-domain page-table pages transiently, so v need not
+ * be current. It is not for contexts with interrupts disabled.
+ */
+void populate_perdomain_mapping(const struct vcpu *v, unsigned long va,
+ const mfn_t *mfn, unsigned int nr,
+ unsigned int flags)
+{
+ struct page_info *l3_pg = perdomain_l3(v);
+ const l3_pgentry_t *l3tab;
+ const l2_pgentry_t *l2tab;
+ l1_pgentry_t *l1tab = NULL;
+ unsigned int i;
+
+ ASSERT(va >= PERDOMAIN_VIRT_START &&
+ va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
+ ASSERT(!nr || !l3_table_offset(va ^ (va + nr * PAGE_SIZE - 1)));
+ ASSERT(local_irq_is_enabled());
+
+ if ( !nr )
+ return;
+
+ BUG_ON(!l3_pg);
+ l3tab = __map_domain_page(l3_pg);
+ BUG_ON(!(l3e_get_flags(l3tab[l3_table_offset(va)]) & _PAGE_PRESENT));
+ l2tab = map_l2t_from_l3e(l3tab[l3_table_offset(va)]);
+ unmap_domain_page(l3tab);
+
+ for ( i = 0; i < nr; i++, va += PAGE_SIZE )
+ {
+ l1_pgentry_t *pl1e;
+
+ if ( !l1tab || !l1_table_offset(va) )
+ {
+ const l2_pgentry_t *pl2e = &l2tab[l2_table_offset(va)];
+
+ unmap_domain_page(l1tab);
+ BUG_ON(!(l2e_get_flags(*pl2e) & _PAGE_PRESENT));
+ l1tab = map_l1t_from_l2e(*pl2e);
+ }
+
+ pl1e = &l1tab[l1_table_offset(va)];
+ if ( perdomain_l1e_needs_freeing(*pl1e) )
+ {
+ printk(XENLOG_ERR
+ "%pv: leaking area-owned page %"PRI_mfn" at %#lx\n",
+ v, mfn_x(l1e_get_mfn(*pl1e)), va);
+ ASSERT_UNREACHABLE();
+ }
+ l1e_write(pl1e, l1e_from_mfn(mfn[i], flags));
+ }
+
+ unmap_domain_page(l1tab);
+ unmap_domain_page(l2tab);
+}
+
void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr)
{
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 24/30] x86/mm: introduce a per-vCPU state window in the perdomain area
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (22 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 23/30] x86/mm: introduce populate_perdomain_mapping() George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 25/30] x86/svm: introduce svm_vmcb_map() and svm_vmcb_unmap() George Dunlap
` (5 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
Under Address Space Isolation, guest-owned register state that Xen
keeps in always-mapped memory moves to the domain heap, mapped only in
the contexts that use it. The long-lived per-vCPU structures involved
(the SVM VMCB, the VMX guest MSR load/save list, the guest XSAVE area,
and later the aggregated small per-vCPU allocations) all want the same
shape: one or more domain-heap pages, mapped once at fixed per-vCPU
virtual addresses for the life of the vCPU.
Add a fourth per-domain-mapping sub-area providing that destination: a
small per-vCPU stride of page slots, with a fixed index assigned to
each relocated structure and the page tables created at vCPU creation
time. Consumers allocate their own pages and install them with
populate_perdomain_mapping(); the backing pages remain caller-owned,
so their lifetime stays with the consumer, while the page tables are
torn down with the rest of the perdomain area.
Before freeing their pages, consumers take them out of the window with
unmap_vcpu_state(), on failure paths as at teardown: vCPU creation can
fail once a consumer has mapped its pages, and the domain lives on.
It flushes the domain's dirty CPUs, which include every CPU whose
page-tables carry the window, in the vCPU's own area or in the shared
one; so far that mask is empty on every call, as the vCPU has not run
or the domain is paused. A consumer reaching a page from another
context, through a transient map_domain_page() mapping, drops the
mapping with unmap_vcpu_state_page(), which also flushes this CPU's
translations of it, there and at its alias in the per-domain mirror:
a domain-wide mapcache defers that flush, and the translations would
otherwise outlive the mapping.
For a domain running without per-vCPU page tables the window has the
same domain-wide visibility as the other perdomain sub-areas: the
virtual addresses are keyed on vcpu_id, so the same VA arithmetic
serves both modes. With vcpu-pt enabled, a vCPU's window is simply
absent from its siblings' address spaces.
This is infrastructure for moving the SVM VMCB, the VMX MSR load/save
lists and the guest XSAVE areas off always-mapped memory. Nothing
occupies the area yet; the only behavioural change is the page-table
allocation itself and the failure path that comes with it: vCPU
creation can now fail with -ENOMEM. Both apply to every guest vCPU,
with or without asi=, PV ones included. A domain with per-vCPU page
tables gets an L2 and an L1 per vCPU; any other domain, every PV domain
among them, one L2 plus an L1 per 64 vCPUs: 2 pages for up to 64 vCPUs.
A PV domain's window stays empty until the XSAVE areas move there.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- New in this version.
---
xen/arch/x86/domain.c | 12 +++++++++
xen/arch/x86/include/asm/config.h | 21 ++++++++++++++-
xen/arch/x86/include/asm/mm.h | 2 ++
xen/arch/x86/mm.c | 44 +++++++++++++++++++++++++++++++
4 files changed, 78 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 7a4da203d4..04868a781e 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -525,6 +525,18 @@ int arch_vcpu_create(struct vcpu *v)
if ( rc )
goto fail;
+ BUILD_BUG_ON((MAX_VIRT_CPUS << VCPU_STATE_VA_SHIFT) >
+ (PERDOMAIN_SLOT_MBYTES << 20));
+ BUILD_BUG_ON(VCPU_STATE_IDX_VMCB >= VCPU_STATE_NR_PAGES);
+ BUILD_BUG_ON(VCPU_STATE_IDX_N2VMCB >= VCPU_STATE_NR_PAGES);
+ BUILD_BUG_ON(VCPU_STATE_IDX_VMX_MSR >= VCPU_STATE_NR_PAGES);
+ /* Page tables only; torn down with the rest of the perdomain area. */
+ rc = create_perdomain_mapping(v, VCPU_STATE_START(v),
+ VCPU_STATE_NR_PAGES,
+ NIL(l1_pgentry_t *), NULL);
+ if ( rc != 0 )
+ goto fail;
+
paging_vcpu_init(v);
if ( (rc = vcpu_init_fpu(v)) != 0 )
diff --git a/xen/arch/x86/include/asm/config.h b/xen/arch/x86/include/asm/config.h
index ee84c57432..8a265e25c8 100644
--- a/xen/arch/x86/include/asm/config.h
+++ b/xen/arch/x86/include/asm/config.h
@@ -158,7 +158,7 @@
/* Slot 260: per-domain mappings (including map cache). */
#define PERDOMAIN_VIRT_START (PML4_ADDR(260))
#define PERDOMAIN_SLOT_MBYTES (PML4_ENTRY_BYTES >> (20 + PAGETABLE_ORDER))
-#define PERDOMAIN_SLOTS 3
+#define PERDOMAIN_SLOTS 4
#define PERDOMAIN_VIRT_SLOT(s) (PERDOMAIN_VIRT_START + (s) * \
(PERDOMAIN_SLOT_MBYTES << 20))
/* Slot 4: mirror of per-domain mappings (for compat xlat area accesses). */
@@ -293,6 +293,25 @@ extern unsigned long xen_phys_start;
#define ARG_XLAT_START(v) \
(ARG_XLAT_VIRT_START + ((v)->vcpu_id << ARG_XLAT_VA_SHIFT))
+/*
+ * Per-vCPU state window. The fourth per-domain-mapping sub-area: a small
+ * per-vCPU stride of page slots holding long-lived per-vCPU structures kept
+ * out of always-mapped memory, each mapped once for the life of the vCPU at
+ * a fixed index within the stride.
+ */
+#define VCPU_STATE_SLOT 3
+#define VCPU_STATE_VIRT_START PERDOMAIN_VIRT_SLOT(VCPU_STATE_SLOT)
+#define VCPU_STATE_ORDER 3
+#define VCPU_STATE_NR_PAGES (1U << VCPU_STATE_ORDER)
+#define VCPU_STATE_VA_SHIFT (VCPU_STATE_ORDER + PAGE_SHIFT)
+#define VCPU_STATE_START(v) \
+ (VCPU_STATE_VIRT_START + ((v)->vcpu_id << VCPU_STATE_VA_SHIFT))
+#define VCPU_STATE_ADDR(v, idx) \
+ (VCPU_STATE_START(v) + (idx) * PAGE_SIZE)
+#define VCPU_STATE_IDX_VMCB 0
+#define VCPU_STATE_IDX_N2VMCB 1
+#define VCPU_STATE_IDX_VMX_MSR 2
+
#define ELFSIZE 64
#define ARCH_CRASH_SAVE_VMCOREINFO
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 6d50e17259..6eaea3ba01 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -625,6 +625,8 @@ void populate_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int flags);
void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr);
+void unmap_vcpu_state(const struct vcpu *v, unsigned int idx, unsigned int nr);
+void unmap_vcpu_state_page(const void *va);
void free_perdomain_mappings(struct domain *d);
void free_pervcpu_mappings(struct vcpu *v);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index bf029df566..9c9f3e728c 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -6668,6 +6668,50 @@ void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unmap_domain_page(l3tab);
}
+/*
+ * Take @nr slots of @v's state window, from @idx, out of the per-domain
+ * area, before the caller frees the pages behind them: the pages are the
+ * caller's own, and must not stay mapped once freed. The window's
+ * translations are usable only on CPUs whose page-tables carry it, all in
+ * the domain's dirty mask: with per-vCPU page-tables, the CPU running v or
+ * holding it lazily; otherwise, those of any of the domain's vCPUs.
+ */
+void unmap_vcpu_state(const struct vcpu *v, unsigned int idx, unsigned int nr)
+{
+ const struct domain *d = v->domain;
+
+ /* Beyond the stride is the next vCPU's window, in a shared area. */
+ BUG_ON(idx >= VCPU_STATE_NR_PAGES || nr > VCPU_STATE_NR_PAGES - idx);
+
+ destroy_perdomain_mapping(v, VCPU_STATE_ADDR(v, idx), nr);
+
+ flush_tlb_mask(d->dirty_cpumask);
+}
+
+/*
+ * Drop a transient mapping of a page behind a vCPU's state window, and this
+ * CPU's translations of it: at its address, and at its alias in the
+ * per-domain mirror, which every domain but a 64-bit PV one has (flushing
+ * that address there is harmless). A per-vCPU mapcache flushes both
+ * already, with the slot's last mapping; a domain-wide one defers the
+ * flush, and CPUs running other vCPUs of its domain may have cached the
+ * slot too. A directmap address, where the caller has no mapcache, needs
+ * nothing.
+ */
+void unmap_vcpu_state_page(const void *va)
+{
+ unsigned long addr = (unsigned long)va;
+
+ unmap_domain_page(va);
+
+ if ( addr < DIRECTMAP_VIRT_START )
+ {
+ flush_tlb_one_local(addr);
+ flush_tlb_one_local(addr - PERDOMAIN_VIRT_START +
+ PERDOMAIN_ALT_VIRT_START);
+ }
+}
+
static void free_perdomain_l3(struct page_info *l3pg)
{
l3_pgentry_t *l3tab = __map_domain_page(l3pg);
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 25/30] x86/svm: introduce svm_vmcb_map() and svm_vmcb_unmap()
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (23 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 24/30] x86/mm: introduce a per-vCPU state window in the perdomain area George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 26/30] x86/svm: clear the VMCB clean bits in svm_ctxt_switch_from() George Dunlap
` (4 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Jason Andryuk,
Teddy Astie
Some accessors of a vCPU's VMCB can run with that vCPU paused, from
another vCPU's context: context save/load, the segment and CPL getters,
PAT/TSC/intercept updates reachable from toolstack operations, vm_event,
the 'v' debug key, and construct_vmcb() itself, which runs before the
vCPU ever entered guest context. svm_nested_features_on_efer_update()
is among them: it is reachable remotely via hvm_load_cpu_ctxt() ->
svm_update_guest_efer().
Route them through new svm_vmcb_map/unmap helpers; construct_vmcb()
takes the VMCB from its caller. svm_set_tsc_offset() needs both VMCBs
rather than just the active one, so it maps them individually via
svm_vmcb_map_one(). A vCPU stays in nested guest mode across a
deschedule or a pause, so this path can run remotely with the L2 VMCB
active; guest mode does not imply v == current. Hot vmexit-path
accesses all have v == current and keep dereferencing svm->vmcb
directly.
No functional change: the helpers return the VMCB's existing address.
They prepare for moving VMCBs off always-mapped memory, where a VMCB's
mapping is valid only in its own vCPU's context.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- New in this version.
---
xen/arch/x86/hvm/svm/nestedsvm.c | 4 +-
xen/arch/x86/hvm/svm/svm.c | 133 +++++++++++++++++++++++--------
xen/arch/x86/hvm/svm/vmcb.c | 29 ++++++-
xen/arch/x86/hvm/svm/vmcb.h | 9 +++
4 files changed, 136 insertions(+), 39 deletions(-)
diff --git a/xen/arch/x86/hvm/svm/nestedsvm.c b/xen/arch/x86/hvm/svm/nestedsvm.c
index 2b998e26bc..dc853e5f1a 100644
--- a/xen/arch/x86/hvm/svm/nestedsvm.c
+++ b/xen/arch/x86/hvm/svm/nestedsvm.c
@@ -1554,7 +1554,7 @@ void svm_vmexit_do_clgi(struct cpu_user_regs *regs, struct vcpu *v)
*/
void svm_nested_features_on_efer_update(struct vcpu *v)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
struct nestedsvm *svm = &vcpu_nestedsvm(v);
u32 general2_intercepts;
vintr_t vintr;
@@ -1612,6 +1612,8 @@ void svm_nested_features_on_efer_update(struct vcpu *v)
vmcb_set_general2_intercepts(vmcb, general2_intercepts);
}
}
+
+ svm_vmcb_unmap(v, vmcb);
}
void __init start_nested_svm(struct hvm_function_table *hvm_function_table)
diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index 71e48351f2..241843d8c0 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -107,7 +107,7 @@ static void cf_check svm_cpu_down(void)
static void cf_check svm_update_guest_cr(
struct vcpu *v, unsigned int cr, unsigned int flags)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
uint64_t value;
switch ( cr )
@@ -173,11 +173,13 @@ static void cf_check svm_update_guest_cr(
default:
BUG();
}
+
+ svm_vmcb_unmap(v, vmcb);
}
static void cf_check svm_update_guest_efer(struct vcpu *v)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
unsigned long guest_efer = v->arch.hvm.guest_efer,
xen_efer = read_efer();
@@ -204,6 +206,7 @@ static void cf_check svm_update_guest_efer(struct vcpu *v)
guest_efer |= EFER_SVME;
vmcb_set_efer(vmcb, guest_efer);
+ svm_vmcb_unmap(v, vmcb);
ASSERT(nestedhvm_enabled(v->domain) ||
!nsvm_efer_svm_enabled(v));
@@ -352,7 +355,7 @@ static void svm_restore_dr(struct vcpu *v)
static int svm_vmcb_save(struct vcpu *v, struct hvm_hw_cpu *c)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
if ( vmcb->event_inj.v &&
hvm_event_needs_reinjection(vmcb->event_inj.type,
@@ -362,13 +365,15 @@ static int svm_vmcb_save(struct vcpu *v, struct hvm_hw_cpu *c)
c->error_code = vmcb->event_inj.ec;
}
+ svm_vmcb_unmap(v, vmcb);
+
return 1;
}
static int svm_vmcb_restore(struct vcpu *v, struct hvm_hw_cpu *c)
{
struct page_info *page = NULL;
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb;
struct p2m_domain *p2m = p2m_get_hostp2m(v->domain);
if ( c->pending_valid )
@@ -417,6 +422,8 @@ static int svm_vmcb_restore(struct vcpu *v, struct hvm_hw_cpu *c)
svm_update_guest_cr(v, 0, 0);
svm_update_guest_cr(v, 4, 0);
+ vmcb = svm_vmcb_map(v);
+
if ( !paging_mode_shadow(v->domain) )
{
vmcb_set_np(vmcb, true);
@@ -436,6 +443,8 @@ static int svm_vmcb_restore(struct vcpu *v, struct hvm_hw_cpu *c)
vmcb->event_inj.raw = 0;
vmcb->cleanbits.raw = 0;
+ svm_vmcb_unmap(v, vmcb);
+
paging_update_paging_modes(v);
return 0;
@@ -443,7 +452,7 @@ static int svm_vmcb_restore(struct vcpu *v, struct hvm_hw_cpu *c)
static void svm_save_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ const struct vmcb_struct *vmcb = svm_vmcb_map(v);
data->sysenter_cs = vmcb->sysenter_cs;
data->sysenter_esp = vmcb->sysenter_esp;
@@ -453,11 +462,13 @@ static void svm_save_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
data->msr_star = vmcb->star;
data->msr_cstar = vmcb->cstar;
data->msr_syscall_mask = vmcb->sfmask;
+
+ svm_vmcb_unmap(v, vmcb);
}
static void svm_load_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
vmcb->lstar = data->msr_lstar;
vmcb->star = data->msr_star;
@@ -467,6 +478,9 @@ static void svm_load_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
vmcb->sysenter_cs = data->sysenter_cs;
vmcb->sysenter_esp = data->sysenter_esp;
vmcb->sysenter_eip = data->sysenter_eip;
+
+ svm_vmcb_unmap(v, vmcb);
+
v->arch.hvm.guest_efer = data->msr_efer;
svm_update_guest_efer(v);
}
@@ -492,7 +506,7 @@ static int cf_check svm_load_vmcb_ctxt(struct vcpu *v, struct hvm_hw_cpu *ctxt)
static unsigned cf_check int svm_get_interrupt_shadow(struct vcpu *v)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
unsigned int intr_shadow = 0;
if ( vmcb->int_stat.intr_shadow )
@@ -503,13 +517,15 @@ static unsigned cf_check int svm_get_interrupt_shadow(struct vcpu *v)
: (vmcb_get_general1_intercepts(vmcb) & GENERAL1_INTERCEPT_IRET) )
intr_shadow |= HVM_INTR_SHADOW_NMI;
+ svm_vmcb_unmap(v, vmcb);
+
return intr_shadow;
}
static void cf_check svm_set_interrupt_shadow(
struct vcpu *v, unsigned int intr_shadow)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
bool block_nmi = intr_shadow & HVM_INTR_SHADOW_NMI;
vmcb->int_stat.intr_shadow =
@@ -527,10 +543,13 @@ static void cf_check svm_set_interrupt_shadow(
vmcb_set_general1_intercepts(vmcb, gen1);
}
+
+ svm_vmcb_unmap(v, vmcb);
}
static int cf_check svm_guest_x86_mode(struct vcpu *v)
{
+ /* Current only (see guest_cpu_user_regs() below): no mapping needed. */
struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
if ( unlikely(!(v->arch.hvm.guest_cr[0] & X86_CR0_PE)) )
@@ -544,8 +563,7 @@ static int cf_check svm_guest_x86_mode(struct vcpu *v)
static void cf_check svm_cpuid_policy_changed(struct vcpu *v)
{
- struct svm_vcpu *svm = &v->arch.hvm.svm;
- struct vmcb_struct *vmcb = svm->vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
const struct cpu_policy *cp = v->domain->arch.cpu_policy;
u32 bitmap = vmcb_get_exception_intercepts(vmcb);
@@ -555,6 +573,7 @@ static void cf_check svm_cpuid_policy_changed(struct vcpu *v)
bitmap &= ~(1U << X86_EXC_UD);
vmcb_set_exception_intercepts(vmcb, bitmap);
+ svm_vmcb_unmap(v, vmcb);
/* Give access to MSR_SPEC_CTRL if the guest has been told about it. */
svm_intercept_msr(v, MSR_SPEC_CTRL,
@@ -598,13 +617,18 @@ void svm_sync_vmcb(struct vcpu *v, enum vmcb_sync_state new_state)
static unsigned int cf_check svm_get_cpl(struct vcpu *v)
{
- return vmcb_get_cpl(v->arch.hvm.svm.vmcb);
+ const struct vmcb_struct *vmcb = svm_vmcb_map(v);
+ unsigned int cpl = vmcb_get_cpl(vmcb);
+
+ svm_vmcb_unmap(v, vmcb);
+
+ return cpl;
}
static void cf_check svm_get_segment_register(
struct vcpu *v, enum x86_segment seg, struct segment_register *reg)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
ASSERT((v == current) || !vcpu_runnable(v));
@@ -645,12 +669,14 @@ static void cf_check svm_get_segment_register(
*reg = (struct segment_register){};
break;
}
+
+ svm_vmcb_unmap(v, vmcb);
}
static void cf_check svm_set_segment_register(
struct vcpu *v, enum x86_segment seg, struct segment_register *reg)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
ASSERT((v == current) || !vcpu_runnable(v));
@@ -679,6 +705,7 @@ static void cf_check svm_set_segment_register(
default:
ASSERT_UNREACHABLE();
domain_crash(v->domain);
+ svm_vmcb_unmap(v, vmcb);
return;
}
@@ -716,33 +743,41 @@ static void cf_check svm_set_segment_register(
ASSERT_UNREACHABLE();
break;
}
+
+ svm_vmcb_unmap(v, vmcb);
}
static int cf_check svm_set_guest_pat(struct vcpu *v, u64 gpat)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb;
if ( paging_mode_shadow(v->domain) )
return 0;
+ vmcb = svm_vmcb_map(v);
vmcb_set_g_pat(vmcb, gpat);
+ svm_vmcb_unmap(v, vmcb);
+
return 1;
}
static int cf_check svm_get_guest_pat(struct vcpu *v, u64 *gpat)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ const struct vmcb_struct *vmcb;
if ( paging_mode_shadow(v->domain) )
return 0;
+ vmcb = svm_vmcb_map(v);
*gpat = vmcb_get_g_pat(vmcb);
+ svm_vmcb_unmap(v, vmcb);
+
return 1;
}
static void cf_check svm_set_tsc_offset(struct vcpu *v, uint64_t offset)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
struct vmcb_struct *n1vmcb, *n2vmcb;
uint64_t n2_tsc_offset = 0;
struct domain *d = v->domain;
@@ -750,25 +785,36 @@ static void cf_check svm_set_tsc_offset(struct vcpu *v, uint64_t offset)
if ( !nestedhvm_enabled(d) )
{
vmcb_set_tsc_offset(vmcb, offset);
+ svm_vmcb_unmap(v, vmcb);
return;
}
- n1vmcb = vcpu_nestedhvm(v).nv_n1vmcx;
- n2vmcb = vcpu_nestedhvm(v).nv_n2vmcx;
-
+ /*
+ * A vCPU stays in nested guest mode across a deschedule or a pause, so
+ * this can run remotely with the L2 VMCB active: map both explicitly.
+ */
if ( nestedhvm_vcpu_in_guestmode(v) )
{
+ const struct nestedvcpu *nv = &vcpu_nestedhvm(v);
+
+ n1vmcb = svm_vmcb_map_one(v, nv->nv_n1vmcx, nv->nv_n1vmcx_pa);
+ n2vmcb = svm_vmcb_map_one(v, nv->nv_n2vmcx, nv->nv_n2vmcx_pa);
+
n2_tsc_offset = vmcb_get_tsc_offset(n2vmcb) -
vmcb_get_tsc_offset(n1vmcb);
vmcb_set_tsc_offset(n1vmcb, offset);
+
+ svm_vmcb_unmap(v, n2vmcb);
+ svm_vmcb_unmap(v, n1vmcb);
}
vmcb_set_tsc_offset(vmcb, offset + n2_tsc_offset);
+ svm_vmcb_unmap(v, vmcb);
}
static void cf_check svm_set_rdtsc_exiting(struct vcpu *v, bool enable)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
u32 general1_intercepts = vmcb_get_general1_intercepts(vmcb);
u32 general2_intercepts = vmcb_get_general2_intercepts(vmcb);
@@ -783,13 +829,14 @@ static void cf_check svm_set_rdtsc_exiting(struct vcpu *v, bool enable)
vmcb_set_general1_intercepts(vmcb, general1_intercepts);
vmcb_set_general2_intercepts(vmcb, general2_intercepts);
+ svm_vmcb_unmap(v, vmcb);
}
#ifdef CONFIG_VM_EVENT
static void cf_check svm_set_descriptor_access_exiting(
struct vcpu *v, bool enable)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
u32 general1_intercepts = vmcb_get_general1_intercepts(vmcb);
u32 mask = GENERAL1_INTERCEPT_IDTR_READ | GENERAL1_INTERCEPT_GDTR_READ
| GENERAL1_INTERCEPT_LDTR_READ | GENERAL1_INTERCEPT_TR_READ
@@ -802,6 +849,7 @@ static void cf_check svm_set_descriptor_access_exiting(
general1_intercepts &= ~mask;
vmcb_set_general1_intercepts(vmcb, general1_intercepts);
+ svm_vmcb_unmap(v, vmcb);
}
#endif /* CONFIG_VM_EVENT */
@@ -1360,7 +1408,12 @@ static void cf_check svm_inject_event(const struct x86_event *event)
static bool cf_check svm_event_pending(const struct vcpu *v)
{
- return v->arch.hvm.svm.vmcb->event_inj.v;
+ const struct vmcb_struct *vmcb = svm_vmcb_map(v);
+ bool pending = vmcb->event_inj.v;
+
+ svm_vmcb_unmap(v, vmcb);
+
+ return pending;
}
static void cf_check svm_cpu_dead(unsigned int cpu)
@@ -2318,45 +2371,55 @@ static void cf_check svm_invlpg(struct vcpu *v, unsigned long linear)
static bool cf_check svm_get_pending_event(
struct vcpu *v, struct x86_event *info)
{
- const struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ const struct vmcb_struct *vmcb = svm_vmcb_map(v);
+ bool pending = vmcb->event_inj.v;
- if ( !vmcb->event_inj.v )
- return false;
+ if ( pending )
+ {
+ info->vector = vmcb->event_inj.vector;
+ info->type = vmcb->event_inj.type;
+ info->error_code = vmcb->event_inj.ec;
+ }
- info->vector = vmcb->event_inj.vector;
- info->type = vmcb->event_inj.type;
- info->error_code = vmcb->event_inj.ec;
+ svm_vmcb_unmap(v, vmcb);
- return true;
+ return pending;
}
static uint64_t cf_check svm_get_reg(struct vcpu *v, unsigned int reg)
{
struct vcpu *curr = current;
- const struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ const struct vmcb_struct *vmcb = svm_vmcb_map(v);
struct domain *d = v->domain;
+ uint64_t val = 0;
switch ( reg )
{
case MSR_SPEC_CTRL:
- return vmcb->spec_ctrl;
+ val = vmcb->spec_ctrl;
+ break;
case MSR_SHADOW_GS_BASE:
if ( v == curr )
svm_sync_vmcb(v, vmcb_in_sync);
- return vmcb->kerngsbase;
+ val = vmcb->kerngsbase;
+ break;
default:
printk(XENLOG_G_ERR "%s(%pv, 0x%08x) Bad register\n",
__func__, v, reg);
domain_crash(d);
- return 0;
+ break;
}
+
+ svm_vmcb_unmap(v, vmcb);
+
+ return val;
}
static void cf_check svm_set_reg(struct vcpu *v, unsigned int reg, uint64_t val)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = svm_vmcb_map(v);
struct domain *d = v->domain;
switch ( reg )
@@ -2371,6 +2434,8 @@ static void cf_check svm_set_reg(struct vcpu *v, unsigned int reg, uint64_t val)
domain_crash(d);
break;
}
+
+ svm_vmcb_unmap(v, vmcb);
}
static struct hvm_function_table __initdata_cf_clobber svm_function_table = {
diff --git a/xen/arch/x86/hvm/svm/vmcb.c b/xen/arch/x86/hvm/svm/vmcb.c
index a6f09672a7..4c48bbc76d 100644
--- a/xen/arch/x86/hvm/svm/vmcb.c
+++ b/xen/arch/x86/hvm/svm/vmcb.c
@@ -43,11 +43,25 @@ void free_vmcb(struct vmcb_struct *vmcb)
free_xenheap_page(vmcb);
}
+struct vmcb_struct *svm_vmcb_map_one(const struct vcpu *v,
+ struct vmcb_struct *va, paddr_t pa)
+{
+ return va;
+}
+
+struct vmcb_struct *svm_vmcb_map(const struct vcpu *v)
+{
+ return svm_vmcb_map_one(v, v->arch.hvm.svm.vmcb, v->arch.hvm.svm.vmcb_pa);
+}
+
+void svm_vmcb_unmap(const struct vcpu *v, const struct vmcb_struct *vmcb)
+{
+}
+
/* This function can directly access fields which are covered by clean bits. */
-static int construct_vmcb(struct vcpu *v)
+static int construct_vmcb(struct vcpu *v, struct vmcb_struct *vmcb)
{
struct svm_vcpu *svm = &v->arch.hvm.svm;
- struct vmcb_struct *vmcb = svm->vmcb;
vmcb->_general1_intercepts =
GENERAL1_INTERCEPT_INTR | GENERAL1_INTERCEPT_NMI |
@@ -201,6 +215,7 @@ int svm_create_vmcb(struct vcpu *v)
{
struct nestedvcpu *nv = &vcpu_nestedhvm(v);
struct svm_vcpu *svm = &v->arch.hvm.svm;
+ struct vmcb_struct *vmcb;
int rc;
if ( (nv->nv_n1vmcx == NULL) &&
@@ -211,7 +226,10 @@ int svm_create_vmcb(struct vcpu *v)
}
svm->vmcb = nv->nv_n1vmcx;
- rc = construct_vmcb(v);
+
+ vmcb = svm_vmcb_map(v);
+ rc = construct_vmcb(v, vmcb);
+ svm_vmcb_unmap(v, vmcb);
if ( rc != 0 )
{
free_vmcb(nv->nv_n1vmcx);
@@ -407,6 +425,7 @@ static void cf_check vmcb_dump(unsigned char ch)
{
struct domain *d;
struct vcpu *v;
+ struct vmcb_struct *vmcb;
printk("*********** VMCB Areas **************\n");
@@ -425,7 +444,9 @@ static void cf_check vmcb_dump(unsigned char ch)
continue;
}
printk("\tVCPU %d\n", v->vcpu_id);
- svm_vmcb_dump("key_handler", v->arch.hvm.svm.vmcb);
+ vmcb = svm_vmcb_map(v);
+ svm_vmcb_dump("key_handler", vmcb);
+ svm_vmcb_unmap(v, vmcb);
process_pending_softirqs();
}
diff --git a/xen/arch/x86/hvm/svm/vmcb.h b/xen/arch/x86/hvm/svm/vmcb.h
index 28dae325ce..9d61767ec7 100644
--- a/xen/arch/x86/hvm/svm/vmcb.h
+++ b/xen/arch/x86/hvm/svm/vmcb.h
@@ -559,6 +559,15 @@ struct vmcb_struct {
struct vmcb_struct *alloc_vmcb(void);
void free_vmcb(struct vmcb_struct *vmcb);
+/*
+ * Access a VMCB of a vCPU that may not be current. svm_vmcb_map() takes
+ * the active VMCB, svm_vmcb_map_one() a specific one.
+ */
+struct vmcb_struct *svm_vmcb_map_one(const struct vcpu *v,
+ struct vmcb_struct *va, paddr_t pa);
+struct vmcb_struct *svm_vmcb_map(const struct vcpu *v);
+void svm_vmcb_unmap(const struct vcpu *v, const struct vmcb_struct *vmcb);
+
int svm_create_vmcb(struct vcpu *v);
void svm_destroy_vmcb(struct vcpu *v);
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 26/30] x86/svm: clear the VMCB clean bits in svm_ctxt_switch_from()
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (24 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 25/30] x86/svm: introduce svm_vmcb_map() and svm_vmcb_unmap() George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-21 9:21 ` Ross Lagerwall
2026-10-19 19:42 ` [PATCH v4 27/30] x86/svm: restore guest debug registers in svm_do_resume() George Dunlap
` (3 subsequent siblings)
29 siblings, 1 reply; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Jason Andryuk,
Teddy Astie
A VMCB's clean bits must all be clear when its vCPU next runs after a
deschedule, as other VMCBs run on the pCPU in the meantime.
svm_ctxt_switch_to() clears them as the vCPU is scheduled in; clear them
in svm_ctxt_switch_from() instead, as it is scheduled out.
This is equivalent. While the vCPU is descheduled, accessors reaching
its VMCB only ever clear clean bits: svm_vmexit_handler(), which sets
them all, runs only in the vCPU's own context. So do the nested-SVM
paths that change the active VMCB, nsvm_vcpu_hostrestore() and
nsvm_vcpu_vmentry(), and they clear the incoming VMCB's clean bits
themselves.
This prepares for moving VMCBs off always-mapped memory:
svm_ctxt_switch_to() runs before write_ptbase(), where the incoming
vCPU's VMCB will not be mapped.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- New in this version.
---
xen/arch/x86/hvm/svm/svm.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index 241843d8c0..4e19df9865 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -919,6 +919,8 @@ static void cf_check svm_ctxt_switch_from(struct vcpu *v)
svm_save_dr(v);
svm_tsc_ratio_save(v);
+ v->arch.hvm.svm.vmcb->cleanbits.raw = 0;
+
svm_sync_vmcb(v, vmcb_needs_vmload);
svm_vmload_pa(per_cpu(host_vmcb, cpu));
@@ -938,7 +940,6 @@ static void cf_check svm_ctxt_switch_from(struct vcpu *v)
static void cf_check svm_ctxt_switch_to(struct vcpu *v)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
int cpu = smp_processor_id();
/*
@@ -956,7 +957,6 @@ static void cf_check svm_ctxt_switch_to(struct vcpu *v)
svm_restore_dr(v);
- vmcb->cleanbits.raw = 0;
svm_tsc_ratio_load(v);
if ( cpu_has_msr_tsc_aux )
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* Re: [PATCH v4 26/30] x86/svm: clear the VMCB clean bits in svm_ctxt_switch_from()
2026-10-19 19:42 ` [PATCH v4 26/30] x86/svm: clear the VMCB clean bits in svm_ctxt_switch_from() George Dunlap
@ 2026-10-21 9:21 ` Ross Lagerwall
2026-10-21 16:02 ` George Dunlap
0 siblings, 1 reply; 39+ messages in thread
From: Ross Lagerwall @ 2026-10-21 9:21 UTC (permalink / raw)
To: George Dunlap, xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Jason Andryuk,
Teddy Astie
On 10/19/26 8:42 PM, George Dunlap wrote:
> A VMCB's clean bits must all be clear when its vCPU next runs after a
> deschedule, as other VMCBs run on the pCPU in the meantime.
> svm_ctxt_switch_to() clears them as the vCPU is scheduled in; clear them
> in svm_ctxt_switch_from() instead, as it is scheduled out.
>
> This is equivalent. While the vCPU is descheduled, accessors reaching
> its VMCB only ever clear clean bits: svm_vmexit_handler(), which sets
> them all, runs only in the vCPU's own context. So do the nested-SVM
> paths that change the active VMCB, nsvm_vcpu_hostrestore() and
> nsvm_vcpu_vmentry(), and they clear the incoming VMCB's clean bits
> themselves.
>
> This prepares for moving VMCBs off always-mapped memory:
> svm_ctxt_switch_to() runs before write_ptbase(), where the incoming
> vCPU's VMCB will not be mapped.
>
> Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
> ---
> Changes in v4:
> - New in this version.
> ---
> xen/arch/x86/hvm/svm/svm.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
> index 241843d8c0..4e19df9865 100644
> --- a/xen/arch/x86/hvm/svm/svm.c
> +++ b/xen/arch/x86/hvm/svm/svm.c
> @@ -919,6 +919,8 @@ static void cf_check svm_ctxt_switch_from(struct vcpu *v)
> svm_save_dr(v);
> svm_tsc_ratio_save(v);
>
> + v->arch.hvm.svm.vmcb->cleanbits.raw = 0;
> +
> svm_sync_vmcb(v, vmcb_needs_vmload);
> svm_vmload_pa(per_cpu(host_vmcb, cpu));
>
> @@ -938,7 +940,6 @@ static void cf_check svm_ctxt_switch_from(struct vcpu *v)
>
> static void cf_check svm_ctxt_switch_to(struct vcpu *v)
> {
> - struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
> int cpu = smp_processor_id();
>
> /*
> @@ -956,7 +957,6 @@ static void cf_check svm_ctxt_switch_to(struct vcpu *v)
>
> svm_restore_dr(v);
>
> - vmcb->cleanbits.raw = 0;
> svm_tsc_ratio_load(v);
>
> if ( cpu_has_msr_tsc_aux )
FYI I posted "x86/svm: Don't clear clean bits on every context switch" a few
weeks ago which moves it into svm_do_resume(). I think that is preferable for
the reasons mentioned in the patch and IIUC would still be after the vCPU's
VMCB is mapped.
Ross
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [PATCH v4 26/30] x86/svm: clear the VMCB clean bits in svm_ctxt_switch_from()
2026-10-21 9:21 ` Ross Lagerwall
@ 2026-10-21 16:02 ` George Dunlap
2026-10-21 16:57 ` Andrew Cooper
0 siblings, 1 reply; 39+ messages in thread
From: George Dunlap @ 2026-10-21 16:02 UTC (permalink / raw)
To: Ross Lagerwall
Cc: xen-devel, Jan Beulich, Andrew Cooper, Roger Pau Monné,
Jason Andryuk, Teddy Astie
On Wed, Oct 21, 2026 at 10:22 AM Ross Lagerwall
<ross.lagerwall@citrix.com> wrote:
> FYI I posted "x86/svm: Don't clear clean bits on every context switch" a few
> weeks ago which moves it into svm_do_resume(). I think that is preferable for
> the reasons mentioned in the patch and IIUC would still be after the vCPU's
> VMCB is mapped.
OK; since yours is a single patch with two acks, I'll rebase onto yours.
Thanks,
-George
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [PATCH v4 26/30] x86/svm: clear the VMCB clean bits in svm_ctxt_switch_from()
2026-10-21 16:02 ` George Dunlap
@ 2026-10-21 16:57 ` Andrew Cooper
0 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2026-10-21 16:57 UTC (permalink / raw)
To: George Dunlap, Ross Lagerwall
Cc: Andrew Cooper, xen-devel, Jan Beulich, Roger Pau Monné,
Jason Andryuk, Teddy Astie
On 21/10/2026 5:02 pm, George Dunlap wrote:
> On Wed, Oct 21, 2026 at 10:22 AM Ross Lagerwall
> <ross.lagerwall@citrix.com> wrote:
>> FYI I posted "x86/svm: Don't clear clean bits on every context switch" a few
>> weeks ago which moves it into svm_do_resume(). I think that is preferable for
>> the reasons mentioned in the patch and IIUC would still be after the vCPU's
>> VMCB is mapped.
>
> OK; since yours is a single patch with two acks, I'll rebase onto yours.
I thought I'd already committed it... Apparently not.
~Andrew
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v4 27/30] x86/svm: restore guest debug registers in svm_do_resume()
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (25 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 26/30] x86/svm: clear the VMCB clean bits in svm_ctxt_switch_from() George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:42 ` [PATCH v4 28/30] x86/svm: drop svm_vmcb_dump()'s paddr and the unused VCPU_svm_vmcb George Dunlap
` (2 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Jason Andryuk,
Teddy Astie
svm_ctxt_switch_to() restores the guest's debug registers when its DR7
enables any of them, which also updates the VMCB. Do it in
svm_do_resume() instead, which follows it whenever the vCPU is
scheduled in.
svm_do_resume() also runs when a vCPU keeps running on its pCPU
(continue_running()), without svm_ctxt_switch_to(). Where the guest's
debug registers are already loaded, the restore does nothing:
flag_dr_dirty records that they are, and only svm_save_dr() clears it,
from svm_ctxt_switch_from().
This prepares for moving VMCBs off always-mapped memory:
svm_ctxt_switch_to() runs before write_ptbase(), where the incoming
vCPU's VMCB will not be mapped.
Not yet run with a guest using debug registers.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- New in this version.
---
xen/arch/x86/hvm/svm/svm.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index 4e19df9865..3de8da7ea3 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -955,8 +955,6 @@ static void cf_check svm_ctxt_switch_to(struct vcpu *v)
*/
disable_each_ist(per_cpu(idt, cpu));
- svm_restore_dr(v);
-
svm_tsc_ratio_load(v);
if ( cpu_has_msr_tsc_aux )
@@ -980,6 +978,8 @@ static void noreturn cf_check svm_do_resume(void)
bool vcpu_guestmode = false;
struct vlapic *vlapic = vcpu_vlapic(v);
+ svm_restore_dr(v);
+
if ( nestedhvm_enabled(v->domain) && nestedhvm_vcpu_in_guestmode(v) )
vcpu_guestmode = 1;
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 28/30] x86/svm: drop svm_vmcb_dump()'s paddr and the unused VCPU_svm_vmcb
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (26 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 27/30] x86/svm: restore guest debug registers in svm_do_resume() George Dunlap
@ 2026-10-19 19:42 ` George Dunlap
2026-10-19 19:43 ` [PATCH v4 29/30] x86/svm: allocate VMCBs from the domain heap behind the vCPU state window George Dunlap
2026-10-19 19:43 ` [PATCH v4 30/30] x86/vmx: move the guest MSR load/save list off always-mapped memory George Dunlap
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:42 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Jason Andryuk,
Teddy Astie
svm_vmcb_dump() prints virt_to_maddr() of the VMCB, which is right only
while VMCBs are reached through the directmap. Print the virtual
address alone.
Nothing uses the VCPU_svm_vmcb asm-offset: assembly only consumes
VCPU_svm_vmcb_pa. Drop it.
Both prepare for moving VMCBs off always-mapped memory.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- New in this version.
---
xen/arch/x86/hvm/svm/vmcb.c | 4 ++--
xen/arch/x86/x86_64/asm-offsets.c | 1 -
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/hvm/svm/vmcb.c b/xen/arch/x86/hvm/svm/vmcb.c
index 4c48bbc76d..f5321b03fb 100644
--- a/xen/arch/x86/hvm/svm/vmcb.c
+++ b/xen/arch/x86/hvm/svm/vmcb.c
@@ -280,8 +280,8 @@ void svm_vmcb_dump(const char *from, const struct vmcb_struct *vmcb)
svm_sync_vmcb(curr, vmcb_in_sync);
printk("Dumping guest's current state at %s...\n", from);
- printk("Size of VMCB = %zu, paddr = %"PRIpaddr", vaddr = %p\n",
- sizeof(struct vmcb_struct), virt_to_maddr(vmcb), vmcb);
+ printk("Size of VMCB = %zu, vaddr = %p\n",
+ sizeof(struct vmcb_struct), vmcb);
printk("cr_intercepts = %#x dr_intercepts = %#x "
"exception_intercepts = %#x\n",
diff --git a/xen/arch/x86/x86_64/asm-offsets.c b/xen/arch/x86/x86_64/asm-offsets.c
index 0549fd5e8d..c5259de604 100644
--- a/xen/arch/x86/x86_64/asm-offsets.c
+++ b/xen/arch/x86/x86_64/asm-offsets.c
@@ -118,7 +118,6 @@ void __dummy__(void)
#ifdef CONFIG_HVM
OFFSET(VCPU_svm_vmcb_pa, struct vcpu, arch.hvm.svm.vmcb_pa);
- OFFSET(VCPU_svm_vmcb, struct vcpu, arch.hvm.svm.vmcb);
BLANK();
OFFSET(VCPU_vmx_launched, struct vcpu, arch.hvm.vmx.launched);
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 29/30] x86/svm: allocate VMCBs from the domain heap behind the vCPU state window
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (27 preceding siblings ...)
2026-10-19 19:42 ` [PATCH v4 28/30] x86/svm: drop svm_vmcb_dump()'s paddr and the unused VCPU_svm_vmcb George Dunlap
@ 2026-10-19 19:43 ` George Dunlap
2026-10-19 19:43 ` [PATCH v4 30/30] x86/vmx: move the guest MSR load/save list off always-mapped memory George Dunlap
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:43 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Jason Andryuk,
Teddy Astie
The VMCB holds the full guest register save area, but lives on the
xenheap, where it stays reachable from every context. Move it to the
domain heap and map it at a fixed slot in the owning vCPU's state
window, so that under Address Space Isolation it is only mapped while
that vCPU's page tables are loaded. Hardware is unaffected: VMRUN,
VMLOAD and VMSAVE address the VMCB physically, and vmcb_pa now derives
from the page rather than from virt_to_maddr() on a directmap VA.
A vCPU can have two VMCBs: the primary (nv_n1vmcx) and the nested L2
one (nv_n2vmcx). Both move, taking the first two slots in the window
stride. The nested code's swapping of svm->vmcb / svm->vmcb_pa
between the two works unchanged, now swapping between two window VAs.
free_vmcb() takes a VMCB out of the window with unmap_vcpu_state()
before freeing the page: without per-vCPU page tables the window is in
the domain-wide area, which outlives a vCPU whose creation fails.
The window VA is only valid in the owning vCPU's context.
svm_vmcb_map() and svm_vmcb_map_one() return it for v == current, and
a transient map_domain_page() mapping otherwise, which svm_vmcb_unmap()
drops with unmap_vcpu_state_page(), along with this CPU's translations
of it. svm_ctxt_switch_to() no longer touches the VMCB, as it runs
before write_ptbase(); svm_ctxt_switch_from() keeps dereferencing the
window VA, as the outgoing page tables are still loaded at that point.
Runtime-tested under QEMU's SVM emulation with a PVH dom0 on two vCPUs,
with and without per-vCPU page-tables: VMCB allocation, VMRUN from the
window and context switches. Nested SVM, save/restore, the 'v' debug
key and a guest using debug registers remain to be exercised.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- New in this version.
---
xen/arch/x86/hvm/svm/nestedsvm.c | 8 +++---
xen/arch/x86/hvm/svm/svm.c | 10 ++++++++
xen/arch/x86/hvm/svm/vmcb.c | 43 ++++++++++++++++++++++----------
xen/arch/x86/hvm/svm/vmcb.h | 15 ++++++++---
4 files changed, 56 insertions(+), 20 deletions(-)
diff --git a/xen/arch/x86/hvm/svm/nestedsvm.c b/xen/arch/x86/hvm/svm/nestedsvm.c
index dc853e5f1a..1599a9908d 100644
--- a/xen/arch/x86/hvm/svm/nestedsvm.c
+++ b/xen/arch/x86/hvm/svm/nestedsvm.c
@@ -71,10 +71,9 @@ int cf_check nsvm_vcpu_initialise(struct vcpu *v)
goto err;
memset(msrpm, 0x0, MSRPM_SIZE);
- nv->nv_n2vmcx = alloc_vmcb();
+ nv->nv_n2vmcx = alloc_vmcb(v, VCPU_STATE_IDX_N2VMCB, &nv->nv_n2vmcx_pa);
if ( nv->nv_n2vmcx == NULL )
goto err;
- nv->nv_n2vmcx_pa = virt_to_maddr(nv->nv_n2vmcx);
return 0;
@@ -95,7 +94,10 @@ void cf_check nsvm_vcpu_destroy(struct vcpu *v)
* of l1 vmcb page.
*/
if ( nv->nv_n1vmcx )
+ {
v->arch.hvm.svm.vmcb = nv->nv_n1vmcx;
+ v->arch.hvm.svm.vmcb_pa = nv->nv_n1vmcx_pa;
+ }
if ( svm->ns_cached_msrpm )
{
@@ -113,7 +115,7 @@ void cf_check nsvm_vcpu_destroy(struct vcpu *v)
if ( nv->nv_n2vmcx )
{
- free_vmcb(nv->nv_n2vmcx);
+ free_vmcb(v, VCPU_STATE_IDX_N2VMCB, nv->nv_n2vmcx_pa);
nv->nv_n2vmcx = NULL;
nv->nv_n2vmcx_pa = INVALID_PADDR;
}
diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index 3de8da7ea3..6cb3ff4258 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -265,6 +265,10 @@ static void cf_check svm_enable_msr_interception(struct domain *d, uint32_t msr)
static void svm_save_dr(struct vcpu *v)
{
+ /*
+ * @v is the outgoing vCPU, whose page tables are still loaded: the
+ * VMCB's window VA serves, although current has moved on.
+ */
struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
unsigned int flag_dr_dirty = v->arch.hvm.flag_dr_dirty;
@@ -942,6 +946,12 @@ static void cf_check svm_ctxt_switch_to(struct vcpu *v)
{
int cpu = smp_processor_id();
+ /*
+ * NB we cannot access the VMCB here: this runs before
+ * write_ptbase(), so the incoming vCPU's state window is not
+ * reachable yet.
+ */
+
/*
* This is required, because VMRUN does consistency check and some of the
* DOM0 selectors are pointing to invalid GDT locations, and cause AMD
diff --git a/xen/arch/x86/hvm/svm/vmcb.c b/xen/arch/x86/hvm/svm/vmcb.c
index f5321b03fb..4b398742a9 100644
--- a/xen/arch/x86/hvm/svm/vmcb.c
+++ b/xen/arch/x86/hvm/svm/vmcb.c
@@ -6,6 +6,7 @@
*
*/
+#include <xen/domain_page.h>
#include <xen/init.h>
#include <xen/keyhandler.h>
#include <xen/lib.h>
@@ -23,30 +24,40 @@
#include "svm.h"
#include "vmcb.h"
-struct vmcb_struct *alloc_vmcb(void)
+struct vmcb_struct *alloc_vmcb(struct vcpu *v, unsigned int idx, paddr_t *pa)
{
- struct vmcb_struct *vmcb;
+ struct page_info *pg = alloc_domheap_page(v->domain, MEMF_no_owner);
+ mfn_t mfn;
+
+ ASSERT(idx < VCPU_STATE_NR_PAGES);
- vmcb = alloc_xenheap_page();
- if ( vmcb == NULL )
+ if ( pg == NULL )
{
printk(XENLOG_WARNING "Warning: failed to allocate vmcb.\n");
return NULL;
}
- clear_page(vmcb);
- return vmcb;
+ mfn = page_to_mfn(pg);
+ clear_domain_page(mfn);
+ populate_perdomain_mapping(v, VCPU_STATE_ADDR(v, idx), &mfn, 1,
+ __PAGE_HYPERVISOR_RW);
+ *pa = mfn_to_maddr(mfn);
+
+ return (struct vmcb_struct *)VCPU_STATE_ADDR(v, idx);
}
-void free_vmcb(struct vmcb_struct *vmcb)
+void free_vmcb(struct vcpu *v, unsigned int idx, paddr_t pa)
{
- free_xenheap_page(vmcb);
+ ASSERT(idx < VCPU_STATE_NR_PAGES);
+
+ unmap_vcpu_state(v, idx, 1);
+ free_domheap_page(maddr_to_page(pa));
}
struct vmcb_struct *svm_vmcb_map_one(const struct vcpu *v,
struct vmcb_struct *va, paddr_t pa)
{
- return va;
+ return v == current ? va : map_domain_page(maddr_to_mfn(pa));
}
struct vmcb_struct *svm_vmcb_map(const struct vcpu *v)
@@ -56,6 +67,8 @@ struct vmcb_struct *svm_vmcb_map(const struct vcpu *v)
void svm_vmcb_unmap(const struct vcpu *v, const struct vmcb_struct *vmcb)
{
+ if ( v != current )
+ unmap_vcpu_state_page(vmcb);
}
/* This function can directly access fields which are covered by clean bits. */
@@ -219,26 +232,29 @@ int svm_create_vmcb(struct vcpu *v)
int rc;
if ( (nv->nv_n1vmcx == NULL) &&
- (nv->nv_n1vmcx = alloc_vmcb()) == NULL )
+ (nv->nv_n1vmcx = alloc_vmcb(v, VCPU_STATE_IDX_VMCB,
+ &nv->nv_n1vmcx_pa)) == NULL )
{
printk("Failed to create a new VMCB\n");
return -ENOMEM;
}
svm->vmcb = nv->nv_n1vmcx;
+ svm->vmcb_pa = nv->nv_n1vmcx_pa;
vmcb = svm_vmcb_map(v);
rc = construct_vmcb(v, vmcb);
svm_vmcb_unmap(v, vmcb);
if ( rc != 0 )
{
- free_vmcb(nv->nv_n1vmcx);
+ free_vmcb(v, VCPU_STATE_IDX_VMCB, nv->nv_n1vmcx_pa);
nv->nv_n1vmcx = NULL;
+ nv->nv_n1vmcx_pa = INVALID_PADDR;
svm->vmcb = NULL;
+ svm->vmcb_pa = INVALID_PADDR;
return rc;
}
- svm->vmcb_pa = nv->nv_n1vmcx_pa = virt_to_maddr(svm->vmcb);
return 0;
}
@@ -248,7 +264,7 @@ void svm_destroy_vmcb(struct vcpu *v)
struct svm_vcpu *svm = &v->arch.hvm.svm;
if ( nv->nv_n1vmcx != NULL )
- free_vmcb(nv->nv_n1vmcx);
+ free_vmcb(v, VCPU_STATE_IDX_VMCB, nv->nv_n1vmcx_pa);
if ( svm->msrpm != NULL )
{
@@ -260,6 +276,7 @@ void svm_destroy_vmcb(struct vcpu *v)
nv->nv_n1vmcx = NULL;
nv->nv_n1vmcx_pa = INVALID_PADDR;
svm->vmcb = NULL;
+ svm->vmcb_pa = INVALID_PADDR;
}
static void svm_dump_sel(const char *name, const struct segment_register *s)
diff --git a/xen/arch/x86/hvm/svm/vmcb.h b/xen/arch/x86/hvm/svm/vmcb.h
index 9d61767ec7..d899beb46e 100644
--- a/xen/arch/x86/hvm/svm/vmcb.h
+++ b/xen/arch/x86/hvm/svm/vmcb.h
@@ -556,12 +556,19 @@ struct vmcb_struct {
uint64_t res18[291];
};
-struct vmcb_struct *alloc_vmcb(void);
-void free_vmcb(struct vmcb_struct *vmcb);
+/*
+ * VMCBs are domain-heap pages: hardware addresses them physically, and
+ * Xen maps each one at a fixed slot in the owning vCPU's state window,
+ * so the returned virtual address is valid only in that vCPU's context.
+ * @idx is the VCPU_STATE_IDX_* slot to place the mapping in.
+ */
+struct vmcb_struct *alloc_vmcb(struct vcpu *v, unsigned int idx, paddr_t *pa);
+void free_vmcb(struct vcpu *v, unsigned int idx, paddr_t pa);
/*
- * Access a VMCB of a vCPU that may not be current. svm_vmcb_map() takes
- * the active VMCB, svm_vmcb_map_one() a specific one.
+ * Access a VMCB of a vCPU that may not be current: the window VA for
+ * current, a transient mapping otherwise. svm_vmcb_map() takes the active
+ * VMCB, svm_vmcb_map_one() a specific one.
*/
struct vmcb_struct *svm_vmcb_map_one(const struct vcpu *v,
struct vmcb_struct *va, paddr_t pa);
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 30/30] x86/vmx: move the guest MSR load/save list off always-mapped memory
2026-10-19 19:42 [PATCH v4 00/30] ASI: HVM per-vCPU areas, sparse-view directmap, per-vCPU state area George Dunlap
` (28 preceding siblings ...)
2026-10-19 19:43 ` [PATCH v4 29/30] x86/svm: allocate VMCBs from the domain heap behind the vCPU state window George Dunlap
@ 2026-10-19 19:43 ` George Dunlap
29 siblings, 0 replies; 39+ messages in thread
From: George Dunlap @ 2026-10-19 19:43 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
The guest MSR load/save list holds guest MSR values (SPEC_CTRL, LBR,
PT, vPMU global control, and EFER on hardware without dedicated
controls), but lives on the xenheap, where it stays reachable from
every context. Move the page to the domain heap and map it at its
fixed slot in the owning vCPU's state window, so that under Address
Space Isolation it is only mapped while that vCPU's page tables are
loaded. Hardware is unaffected: the VMCS MSR-area pointers hold the
physical address, now derived from the page rather than from
virt_to_maddr() on a directmap VA.
The host MSR load list is left on the xenheap: it holds Xen's own
exit-time values, not guest data, so relocating it would serve no
isolation purpose.
The window VA is only valid in the owning vCPU's context. Almost all
accesses have v == current and keep going through vmx->msr_area
unchanged (the vmentry-path LBR fixups, the failed-vmentry dump, vPMU,
and the MSR intercepts). The remote accessors — vmx_add_msr() /
vmx_del_msr() from vCPU construction, policy updates and context
load, and single-entry reads/writes from the HVM context save/load
and get/set-reg paths, all of which already require the target vCPU
to be paused — dispatch through a transient mapping of the page,
dropped with unmap_vcpu_state_page(), which also drops this CPU's
translations of it:
* vmx_add_msr() and vmx_del_msr() map internally around their
search/insert/remove.
* vmx_read_guest_msr(), vmx_read_guest_loadonly_msr() and
vmx_write_guest_msr() move out of line and do the same, via
vmx_find_msr_in(), the existing lookup parameterised on the mapped
area. vmx_find_msr() itself is unchanged for its remaining
callers, which all run in current context and dereference the
returned pointer through the window.
vmx_destroy_vmcs() takes the page out of the window before freeing it,
with unmap_vcpu_state(): without per-vCPU page tables the window is in
the domain-wide area, which outlives a vCPU whose creation fails.
The new page pointer goes with the existing page_info pointers after
pi_desc rather than next to msr_area: struct pi_desc is 64-byte
aligned, so inserting eight bytes ahead of it would have added 56
bytes of padding, growing struct vmx_vcpu from 448 to 512 bytes and,
via the union in struct arch_vcpu, every struct vcpu by 64 bytes
(x86_64 defconfig).
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- New in this version.
---
xen/arch/x86/hvm/vmx/vmcs.c | 134 +++++++++++++++++++++---
xen/arch/x86/include/asm/hvm/vmx/vmcs.h | 63 +++--------
2 files changed, 139 insertions(+), 58 deletions(-)
diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
index 8ec7ae7403..02ad89e864 100644
--- a/xen/arch/x86/hvm/vmx/vmcs.c
+++ b/xen/arch/x86/hvm/vmx/vmcs.c
@@ -1416,8 +1416,11 @@ static struct vmx_msr_entry *locate_msr_entry(
return start;
}
-struct vmx_msr_entry *vmx_find_msr(const struct vcpu *v, uint32_t msr,
- enum vmx_msr_list_type type)
+/* As vmx_find_msr(), but reaching the guest MSR area through @area. */
+static struct vmx_msr_entry *vmx_find_msr_in(const struct vcpu *v,
+ struct vmx_msr_entry *area,
+ uint32_t msr,
+ enum vmx_msr_list_type type)
{
const struct vmx_vcpu *vmx = &v->arch.hvm.vmx;
struct vmx_msr_entry *start = NULL, *ent, *end;
@@ -1435,11 +1438,11 @@ struct vmx_msr_entry *vmx_find_msr(const struct vcpu *v, uint32_t msr,
break;
case VMX_MSR_GUEST:
- start = vmx->msr_area;
+ start = area;
break;
case VMX_MSR_GUEST_LOADONLY:
- start = vmx->msr_area;
+ start = area;
substart = subend;
subend = total;
break;
@@ -1458,6 +1461,69 @@ struct vmx_msr_entry *vmx_find_msr(const struct vcpu *v, uint32_t msr,
return ((ent < end) && (ent->index == msr)) ? ent : NULL;
}
+struct vmx_msr_entry *vmx_find_msr(const struct vcpu *v, uint32_t msr,
+ enum vmx_msr_list_type type)
+{
+ ASSERT(v == current || type == VMX_MSR_HOST);
+
+ return vmx_find_msr_in(v, v->arch.hvm.vmx.msr_area, msr, type);
+}
+
+static struct vmx_msr_entry *vmx_msr_area_map(const struct vcpu *v)
+{
+ const struct vmx_vcpu *vmx = &v->arch.hvm.vmx;
+
+ if ( v == current || !vmx->msr_area )
+ return vmx->msr_area;
+
+ return map_domain_page(page_to_mfn(vmx->msr_area_pg));
+}
+
+static void vmx_msr_area_unmap(const struct vcpu *v,
+ const struct vmx_msr_entry *area)
+{
+ if ( v != current && area )
+ unmap_vcpu_state_page(area);
+}
+
+int vmx_read_guest_msr(const struct vcpu *v, uint32_t msr, uint64_t *val)
+{
+ struct vmx_msr_entry *area = vmx_msr_area_map(v);
+ const struct vmx_msr_entry *ent =
+ area ? vmx_find_msr_in(v, area, msr, VMX_MSR_GUEST) : NULL;
+
+ *val = ent ? ent->data : 0;
+ vmx_msr_area_unmap(v, area);
+
+ return ent ? 0 : -ESRCH;
+}
+
+int vmx_read_guest_loadonly_msr(const struct vcpu *v, uint32_t msr,
+ uint64_t *val)
+{
+ struct vmx_msr_entry *area = vmx_msr_area_map(v);
+ const struct vmx_msr_entry *ent =
+ area ? vmx_find_msr_in(v, area, msr, VMX_MSR_GUEST_LOADONLY) : NULL;
+
+ *val = ent ? ent->data : 0;
+ vmx_msr_area_unmap(v, area);
+
+ return ent ? 0 : -ESRCH;
+}
+
+int vmx_write_guest_msr(struct vcpu *v, uint32_t msr, uint64_t val)
+{
+ struct vmx_msr_entry *area = vmx_msr_area_map(v);
+ struct vmx_msr_entry *ent =
+ area ? vmx_find_msr_in(v, area, msr, VMX_MSR_GUEST) : NULL;
+
+ if ( ent )
+ ent->data = val;
+ vmx_msr_area_unmap(v, area);
+
+ return ent ? 0 : -ESRCH;
+}
+
int vmx_add_msr(struct vcpu *v, uint32_t msr, uint64_t val,
enum vmx_msr_list_type type)
{
@@ -1503,13 +1569,38 @@ int vmx_add_msr(struct vcpu *v, uint32_t msr, uint64_t val,
{
paddr_t addr;
- if ( (*ptr = alloc_xenheap_page()) == NULL )
+ if ( type == VMX_MSR_HOST )
{
- rc = -ENOMEM;
- goto out;
+ if ( (*ptr = alloc_xenheap_page()) == NULL )
+ {
+ rc = -ENOMEM;
+ goto out;
+ }
+
+ addr = virt_to_maddr(*ptr);
}
+ else
+ {
+ struct page_info *pg = alloc_domheap_page(v->domain,
+ MEMF_no_owner);
+ mfn_t mfn;
+
+ if ( pg == NULL )
+ {
+ rc = -ENOMEM;
+ goto out;
+ }
- addr = virt_to_maddr(*ptr);
+ mfn = page_to_mfn(pg);
+ clear_domain_page(mfn);
+ populate_perdomain_mapping(v,
+ VCPU_STATE_ADDR(v,
+ VCPU_STATE_IDX_VMX_MSR),
+ &mfn, 1, __PAGE_HYPERVISOR_RW);
+ vmx->msr_area_pg = pg;
+ *ptr = (void *)VCPU_STATE_ADDR(v, VCPU_STATE_IDX_VMX_MSR);
+ addr = mfn_to_maddr(mfn);
+ }
switch ( type )
{
@@ -1525,7 +1616,7 @@ int vmx_add_msr(struct vcpu *v, uint32_t msr, uint64_t val,
}
}
- start = *ptr;
+ start = type == VMX_MSR_HOST ? *ptr : vmx_msr_area_map(v);
end = start + total;
ent = locate_msr_entry(start + substart, start + subend, msr);
@@ -1565,6 +1656,8 @@ int vmx_add_msr(struct vcpu *v, uint32_t msr, uint64_t val,
rc = 0;
out:
+ if ( type != VMX_MSR_HOST )
+ vmx_msr_area_unmap(v, start);
vmx_vmcs_exit(v);
return rc;
@@ -1576,6 +1669,7 @@ int vmx_del_msr(struct vcpu *v, uint32_t msr, enum vmx_msr_list_type type)
struct vmx_msr_entry *start = NULL, *ent, *end;
unsigned int substart = 0, subend = vmx->msr_save_count;
unsigned int total = vmx->msr_load_count;
+ int rc = 0;
ASSERT(v == current || !vcpu_runnable(v));
@@ -1605,11 +1699,17 @@ int vmx_del_msr(struct vcpu *v, uint32_t msr, enum vmx_msr_list_type type)
if ( !start )
return -ESRCH;
+ if ( type != VMX_MSR_HOST )
+ start = vmx_msr_area_map(v);
+
end = start + total;
ent = locate_msr_entry(start + substart, start + subend, msr);
if ( (ent == end) || (ent->index != msr) )
- return -ESRCH;
+ {
+ rc = -ESRCH;
+ goto out;
+ }
memmove(ent, ent + 1, sizeof(*ent) * (end - ent - 1));
@@ -1632,7 +1732,11 @@ int vmx_del_msr(struct vcpu *v, uint32_t msr, enum vmx_msr_list_type type)
vmx_vmcs_exit(v);
- return 0;
+ out:
+ if ( type != VMX_MSR_HOST )
+ vmx_msr_area_unmap(v, start);
+
+ return rc;
}
void vmx_set_eoi_exit_bitmap(struct vcpu *v, u8 vector)
@@ -1829,7 +1933,13 @@ void vmx_destroy_vmcs(struct vcpu *v)
vmx_free_vmcs(vmx->vmcs_pa);
free_xenheap_page(v->arch.hvm.vmx.host_msr_area);
- free_xenheap_page(v->arch.hvm.vmx.msr_area);
+ if ( v->arch.hvm.vmx.msr_area_pg )
+ {
+ unmap_vcpu_state(v, VCPU_STATE_IDX_VMX_MSR, 1);
+ free_domheap_page(v->arch.hvm.vmx.msr_area_pg);
+ v->arch.hvm.vmx.msr_area_pg = NULL;
+ v->arch.hvm.vmx.msr_area = NULL;
+ }
free_xenheap_page(v->arch.hvm.vmx.msr_bitmap);
}
diff --git a/xen/arch/x86/include/asm/hvm/vmx/vmcs.h b/xen/arch/x86/include/asm/hvm/vmx/vmcs.h
index d0716e97d2..e5730780e2 100644
--- a/xen/arch/x86/include/asm/hvm/vmx/vmcs.h
+++ b/xen/arch/x86/include/asm/hvm/vmx/vmcs.h
@@ -140,6 +140,10 @@ struct vmx_vcpu {
* context. However, the data can be modified by toolstack/migration
* actions. Remote access is only permitted for paused vcpus, and is
* protected under the domctl lock.
+ *
+ * The guest list is a domain-heap page (msr_area_pg, below) mapped at a
+ * fixed slot in the owning vCPU's state window: msr_area holds that
+ * window VA, which is only valid in the owning vCPU's context.
*/
struct vmx_msr_entry *msr_area;
struct vmx_msr_entry *host_msr_area;
@@ -185,6 +189,9 @@ struct vmx_vcpu {
struct page_info *pml_pg;
+ /* Backing page of msr_area; see the comment there. */
+ struct page_info *msr_area_pg;
+
/* Bitmask of trapped CR4 bits. */
unsigned long cr4_host_mask;
@@ -669,55 +676,19 @@ static inline int vmx_add_host_load_msr(struct vcpu *v, uint32_t msr,
return vmx_add_msr(v, msr, val, VMX_MSR_HOST);
}
+/*
+ * Look an MSR up in the guest or host list of @v. The returned pointer
+ * reaches the guest lists through the vCPU state window, so for those it
+ * is only valid (and the call only permitted) in current context; the
+ * single-entry accessors below handle paused remote vCPUs.
+ */
struct vmx_msr_entry *vmx_find_msr(const struct vcpu *v, uint32_t msr,
enum vmx_msr_list_type type);
-static inline int vmx_read_guest_msr(const struct vcpu *v, uint32_t msr,
- uint64_t *val)
-{
- const struct vmx_msr_entry *ent = vmx_find_msr(v, msr, VMX_MSR_GUEST);
-
- if ( !ent )
- {
- *val = 0;
- return -ESRCH;
- }
-
- *val = ent->data;
-
- return 0;
-}
-
-static inline int vmx_read_guest_loadonly_msr(
- const struct vcpu *v, uint32_t msr, uint64_t *val)
-{
- const struct vmx_msr_entry *ent =
- vmx_find_msr(v, msr, VMX_MSR_GUEST_LOADONLY);
-
- if ( !ent )
- {
- *val = 0;
- return -ESRCH;
- }
-
- *val = ent->data;
-
- return 0;
-}
-
-static inline int vmx_write_guest_msr(struct vcpu *v, uint32_t msr,
- uint64_t val)
-{
- struct vmx_msr_entry *ent = vmx_find_msr(v, msr, VMX_MSR_GUEST);
-
- if ( !ent )
- return -ESRCH;
-
- ent->data = val;
-
- return 0;
-}
-
+int vmx_read_guest_msr(const struct vcpu *v, uint32_t msr, uint64_t *val);
+int vmx_read_guest_loadonly_msr(const struct vcpu *v, uint32_t msr,
+ uint64_t *val);
+int vmx_write_guest_msr(struct vcpu *v, uint32_t msr, uint64_t val);
/* MSR intercept bitmap infrastructure. */
enum vmx_msr_intercept_type {
--
2.55.0
^ permalink raw reply related [flat|nested] 39+ messages in thread