* [PATCH v6 00/44] ASI: HVM through X4
@ 2026-11-03 22:13 George Dunlap
2026-11-03 22:13 ` [PATCH v6 01/44] x86/mm: allocate the per-domain L3 in arch_domain_create() George Dunlap
` (43 more replies)
0 siblings, 44 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
This is the first part of a series to implement address space
isolation.
The approach is the same as v3 to v5: Start by focusing on just
getting ASI and intra-domain isolation for HVM guests. PV support is
added where it's simple to do.
The full series can be roughly divided as below:
Included in this version, all posted before:
- VH (01-13): Give each vCPU of an HVM guest its own per-domain area
and its own mapcache.
- DV (14-23): Maintain two "views" of the directmap: A full directmap,
and a sparse directmap. The full directmap is what we have now; the
sparse directmap removes non-xenheap heap memory.
- X1 (24-30): Add a vCPU state window to the per-domain area (both for
PV and HVM). Move SVM VMCB structures and VMX guest MSR areas into
it.
- X2 (31-34): Move every vCPU's saved register frame (PV and HVM), and
the HVM emulator's operand caches, into the vCPU state window. 34
also clears the vCPU's copy of an emulated I/O request once it has
been consumed.
- X3 (35-37): Introduce map_domain_pages(), which maps several pages
in the calling vCPU's per-domain area, and use it in place of vmap()
for the shadow code's cross-page emulated writes and the HVM
emulator's multi-frame accesses.
- X4 (38-44): Move the guest XSAVE areas into the vCPU state window.
Implemented but not yet posted:
- PH1: Preparation for per-pCPU stacks, modifying rendezvous and
call-function IPIs to not use the stack
- PH2: Moving pCPU stacks into the perdomain area, allocating them
from the domheap
- O: A number of optimizations to improve performance
The functionality of v6 (through X4) is the bare minimum that
XenServer needs to make it into 4.23.
v5 can be found here:
https://lore.kernel.org/xen-devel/20261028-asi-hvm-v5-0-13b908e0df0b@xenproject.org/
v6 is based on staging 2043d22b7e92. v5's 01, "x86/PoD: map one page
at a time in p2m_pod_zero_check()", is now in staging (41ddcfe808ec),
so v5's NN is v6's NN-1.
The changes since v5 have been tested (below) and skimmed, but not
yet been over with a fine-tooth comb.
Changes in v6:
- Following Jan's suggestion on v5's 02, the per-domain L3 is now
allocated when the domain is created, instead of refusing to promote
an L4 while there is none: 01, "x86/mm: allocate the per-domain L3
in arch_domain_create()", replaces v5's "x86/mm: refuse to promote
an L4 while the domain has no per-domain L3". A vCPU-PT domain has
no domain-wide L3: each of its vCPUs gets its own in
arch_vcpu_create() (08). create_perdomain_mapping() no longer
allocates an L3.
- To go with it, 07 gives init_xen_l4_slots() two forms around a
common core: one taking the vCPU that a root page table is private
to (HVM monitor tables, the compat L4), and init_xen_l4_slots_pv(),
taking only the domain, for the root page tables a PV domain's vCPUs
share (its L4s and their shadows). promote_l4_table() therefore no
longer needs vCPU 0.
- 02, 04, 05, 20 and 25 change to match, and 13 now cites the PoD fix
by its hash; each patch's changelog has the details. Jan's
Reviewed-by on v5's 06 (now 05) has been dropped.
- In 16, sparse_dmap_init() is still called once the heap has all
memory. Since "x86: invoke xenheap_max_mfn() unconditionally"
(f93ffa1309cd), that means after the loop for RAM above 5 TiB,
rather than right after the change to SYS_STATE_boot.
Some more details:
15 is a work-around for a slow-down which causes a watchdog to fire.
(Proper performance fixes have been implemented but won't be posted
until the end of the entire series.)
34 also limits exposure of ioreq state: Completely isolating each
vCPU's I/O requests from being read by other vCPUs of the same domain
is complicated; as a mitigation, this patch clears the emulated I/O
request once it has been consumed, reducing the window available to
read the payload.
38-41 are Alejandro's patches from "x86: Address Space Isolation FPU
preparations" (v3, January 2025), revived.
State is moved from the xenheap into the per-domain area for both ASI
and non-ASI domains: VMCB, MSR List, vCPU register frame, and
XSAVE/FPU areas. Same is true for users of map_domain_pages. This
protects non-ASI domains from ASI domains.
One key thing to get feedback on is the question below 44's "---":
xstate_init() now panics on hardware whose XSAVE area needs more than
the three pages reserved for it in the vCPU state window. No current
hardware does. Should it do more, for instance size the reservation
from the size found at boot, or fall back to xenheap areas on such
hardware?
Note that at the end of this series, PV guests can be given the sparse
directmap; this isolates them from other domains, but doesn't isolate
vCPUs within a domain. With this series, a PV domain's register
frames and XSAVE areas are also out of reach of domains on the sparse
directmap, whether or not the PV domain itself is on it.
Testing of v6:
- On the series as reworked, on staging 8b0a70ddd29d, before the
rebase (so without "x86: invoke xenheap_max_mfn() unconditionally"
and without the smaller review fixes):
- Builds of xen/ at each of the 44 patches, in seven
configurations: debug and release, !HVM, !PV, the PV shim, and
shadow paging, debug and release. No warnings or errors beyond
those staging itself gives.
- On an Intel NUC: whole-tree builds (xen, tools, stubdom, docs) at
each of the 44, with tools/tests/vpci run.
- QEMU (TCG) smoke boots at 08 and 44: PV and PVH dom0s (the PVH
one on HAP and on shadow), and XTF's 32-bit PV and 32-bit HVM
selftests as dom0, without asi= and with asi=1. Each boot gets to
dom0's userspace (or the test's result) with no crash, assertion
or warning in Xen's log.
- On the NUC: boots of 08 and 44 without asi= and with asi=1. In
each, a 64-bit PV guest is live-migrated to the same host in
shadow log-dirty mode under a verified workload, with an HVM guest
beside it; with asi=1, XTF's full suite, 32-bit PV included,
passes twice (the usual four skips).
- On the rebased series: each patch's tree differs from the tested one
only by staging's changes and the review fixes in 01, 07, 08 and 16,
checked file by file. The same builds and boots are being repeated
on it; those finished so far have passed.
- The same basic code has been run through XenRT, with the issue
addressed by 15 (slow dom0 kernel decompression triggering a
watchdog on large machines) the only issue found.
A dependency graph of the above functionality can be found here:
https://xenbits.xenproject.org/people/gdunlap/asi-series-deps.html
Extending the intra-domain isolation to PV guests would involve
implementing per-vCPU root pagetables for PV guests. There are draft
implementations of this, but they are quite complicated.
Alejandro Vallejo (4):
x86/emulator: Refactor FXSAVE_AREA to use wrappers
x86/mpx: Map/unmap xsave area in read_bndcfgu()
x86/fpu: Pass explicit xsave areas to fpu_(f)xsave()
x86/fpu: Pass explicit xsave areas to fpu_(f)xrstor()
George Dunlap (30):
x86/mm: allocate the per-domain L3 in arch_domain_create()
x86/mm: have init_xen_l4_slots() take a vCPU in place of the domain
x86/domain_page: drop redundant is_pv_vcpu() checks
x86/domain_page: don't unmap a mapcache entry that is not present
x86/mm: let Xen's page-table walker operate on a given root
xen/gunzip: process pending softirqs once per deflate block
x86/mm: add hooks for a sparse view of the directmap
x86/mm: build and maintain a sparse view of the directmap
x86/domain_page: key the directmap fast paths on the loaded view
x86/domain_page: clear a mapcache slot at its last unmap
x86/mm: run guest contexts on the sparse directmap view
x86/spec-ctrl: wire the sparse directmap view into the asi= option
xen/xmalloc: return an emptied region after dropping the pool lock
x86/mm: split a sparse directmap superpage with a page being freed
x86/mm: introduce a per-vCPU state window in the perdomain area
x86/svm: introduce SVM_MAP_VMCB() and SVM_UNMAP_VMCB()
x86/svm: restore guest debug registers in svm_do_resume()
x86/svm: drop svm_vmcb_dump()'s paddr
x86/svm: allocate VMCBs from the domain heap behind the vCPU state
window
x86/vmx: move the guest MSR load/save list off always-mapped memory
x86: access a vCPU's saved register frame through map/unmap helpers
x86: move the saved register frame behind the vCPU state window
x86/hvm: move the emulator's operand caches behind the vCPU state
window
x86/hvm: clear the vCPU's copy of an emulated I/O request once
consumed
x86/domain_page: introduce map_domain_pages()
x86/shadow: use map_domain_pages() for cross-page emulated writes
x86/hvm: use map_domain_pages() for multi-frame emulated accesses
x86/xstate: classify xsave-area seam accesses by failure mode
x86/domain: restore FPU state after switching to the new page tables
x86/xstate: move guest XSAVE areas behind the vCPU state window
Roger Pau Monné (10):
x86/mm: purge unneeded destroy_perdomain_mapping()
x86/mm: prepare destroy_perdomain_mapping() for per-vCPU perdomain
areas
x86/domain_page: drop redundant create_perdomain_mapping() call
x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain
areas
x86/spec-ctrl: introduce Address Space Isolation command line option
x86/hvm: introduce per-vCPU L3 page-table
x86/domain_page: prepare the mapcache for per-vCPU accounting
x86/domain_page: move struct mapcache_vcpu to struct arch_vcpu
x86/hvm: introduce per-vCPU mapcache for vCPU-PT domains
x86/mm: introduce populate_perdomain_mapping()
docs/misc/xen-command-line.pandoc | 57 +++
xen/arch/x86/Kconfig | 1 +
xen/arch/x86/Makefile | 1 +
xen/arch/x86/domain.c | 239 ++++++++--
xen/arch/x86/domain_page.c | 461 ++++++++++++++++----
xen/arch/x86/domctl.c | 98 +++--
xen/arch/x86/hvm/domain.c | 6 +-
xen/arch/x86/hvm/emulate.c | 243 ++++++++---
xen/arch/x86/hvm/hvm.c | 122 +++---
xen/arch/x86/hvm/svm/nestedsvm.c | 12 +-
xen/arch/x86/hvm/svm/svm.c | 147 +++++--
xen/arch/x86/hvm/svm/vmcb.c | 74 +++-
xen/arch/x86/hvm/svm/vmcb.h | 29 +-
xen/arch/x86/hvm/vmx/vmcs.c | 151 ++++++-
xen/arch/x86/hvm/vmx/vmx.c | 7 +-
xen/arch/x86/hypercall.c | 4 +-
xen/arch/x86/i387.c | 60 ++-
xen/arch/x86/include/asm/config.h | 42 +-
xen/arch/x86/include/asm/domain.h | 109 ++++-
xen/arch/x86/include/asm/hvm/emulate.h | 9 +-
xen/arch/x86/include/asm/hvm/vcpu.h | 7 +
xen/arch/x86/include/asm/hvm/vmx/vmcs.h | 63 +--
xen/arch/x86/include/asm/i387.h | 1 +
xen/arch/x86/include/asm/mm.h | 32 +-
xen/arch/x86/include/asm/sparse-directmap.h | 83 ++++
xen/arch/x86/include/asm/spec_ctrl.h | 3 +
xen/arch/x86/include/asm/xstate.h | 73 ++--
xen/arch/x86/mm.c | 459 ++++++++++++++++---
xen/arch/x86/mm/hap/hap.c | 2 +-
xen/arch/x86/mm/shadow/hvm.c | 6 +-
xen/arch/x86/mm/shadow/multi.c | 5 +-
xen/arch/x86/pv/dom0_build.c | 11 +-
xen/arch/x86/pv/domain.c | 30 +-
xen/arch/x86/pv/shim.c | 4 +-
xen/arch/x86/setup.c | 4 +
xen/arch/x86/sparse-directmap.c | 455 +++++++++++++++++++
xen/arch/x86/spec_ctrl.c | 183 +++++++-
xen/arch/x86/traps.c | 20 +-
xen/arch/x86/x86_64/mm.c | 14 +-
xen/arch/x86/x86_emulate/blk.c | 36 +-
xen/arch/x86/xstate.c | 212 ++++++++-
xen/common/Kconfig | 18 +
xen/common/gzip/gunzip.c | 1 +
xen/common/gzip/inflate.c | 1 +
xen/common/page_alloc.c | 51 ++-
xen/common/xmalloc_tlsf.c | 12 +-
xen/include/xen/domain_page.h | 21 +
xen/include/xen/mm.h | 40 ++
48 files changed, 3101 insertions(+), 618 deletions(-)
create mode 100644 xen/arch/x86/include/asm/sparse-directmap.h
create mode 100644 xen/arch/x86/sparse-directmap.c
base-commit: 2043d22b7e926f508cc23391b4c75325fa7211e9
--
2.56.0
^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH v6 01/44] x86/mm: allocate the per-domain L3 in arch_domain_create()
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-04 10:46 ` Jan Beulich
2026-11-03 22:13 ` [PATCH v6 02/44] x86/mm: purge unneeded destroy_perdomain_mapping() George Dunlap
` (42 subsequent siblings)
43 siblings, 1 reply; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
init_xen_l4_slots() relies on d->arch.perdomain_l3_pg being non-NULL,
or it will fill in a garbage value for slot 260. Currently, this
is allocated only implicitly, as a side effect of calling
create_perdomain_mapping(). But depending on the specific
circumstances, this may not happen until several hypercalls after the
domain is created. This is not robust: callers need to keep track of
when exactly certain structures happen to be created to ensure that
init_xen_l4_slots() operates properly.
Since fcf91ce9de66 ("x86/pv: don't populate the GDT/LDT L3 slot at
domain creation"), a PV domain gets its L3 with its first vCPU
(pv_create_gdt_ldt_l1tab()). Only mapcache_domain_init() makes it
earlier, and release builds skip that unless memory hotplug is enabled
or the host's memory reaches past the part of the directmap PV guests
can see. So in release builds on most hosts, an L4 that is promoted
before XEN_DOMCTL_max_vcpus has created the domain's vCPUs gets a
slot 260 entry made from a NULL page (0x803e900000000063). libxl is
unaffected: it creates the vCPUs before it builds or restores a domain.
Allocate it explicitly in arch_domain_create() instead. Introduce
allocate_perdomain_l3(), which will allocate a page with the
appropriate parameters and zero it. create_perdomain_mapping() now
requires the L3 to exist, and BUG()s if it does not.
We return a pg rather than setting d->arch.perdomain_l3_pg directly,
because later we'll use the same function to initialize the per-vCPU
perdomain L3.
This removes the need to allocate it semi-implicitly (calling
create_perdomain_mapping() with nr == 0) in hvm_domain_initialise().
Fixes: fcf91ce9de66 ("x86/pv: don't populate the GDT/LDT L3 slot at domain creation")
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v6:
- New in this version, replacing "x86/mm: refuse to promote an L4 while
the domain has no per-domain L3": allocate the L3 with the domain
instead (Jan).
---
xen/arch/x86/domain.c | 6 ++++++
xen/arch/x86/hvm/hvm.c | 5 -----
xen/arch/x86/include/asm/mm.h | 1 +
xen/arch/x86/mm.c | 32 +++++++++++++++-----------------
4 files changed, 22 insertions(+), 22 deletions(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 368e4deff3..e67c336b82 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -901,6 +901,12 @@ int arch_domain_create(struct domain *d,
psr_domain_init(d);
+ if ( (d->arch.perdomain_l3_pg = allocate_perdomain_l3(d)) == NULL )
+ {
+ rc = -ENOMEM;
+ goto fail;
+ }
+
if ( is_hvm_domain(d) )
{
if ( (rc = hvm_domain_initialise(d, config)) != 0 )
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index ef9c7cd057..46d3a520d5 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -620,10 +620,6 @@ int hvm_domain_initialise(struct domain *d,
INIT_LIST_HEAD(&d->arch.hvm.mmcfg_regions);
INIT_LIST_HEAD(&d->arch.hvm.msix_tables);
- rc = create_perdomain_mapping(d, PERDOMAIN_VIRT_START, 0, NULL, NULL);
- if ( rc )
- goto fail;
-
hvm_init_cacheattr_region_list(d);
rc = paging_enable(d, PG_refcounts|PG_translate|PG_external);
@@ -731,7 +727,6 @@ int hvm_domain_initialise(struct domain *d,
fail0:
hvm_destroy_cacheattr_region_list(d);
destroy_perdomain_mapping(d, PERDOMAIN_VIRT_START, 0);
- fail:
hvm_domain_relinquish_resources(d);
XFREE(d->arch.hvm.io_handler);
XFREE(d->arch.hvm.pl_time);
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index aa515efbee..e77ac5ccdf 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -606,6 +606,7 @@ int compat_arch_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg);
#define NIL(type) ((type *)-sizeof(type))
#define IS_NIL(ptr) (!((uintptr_t)(ptr) + sizeof(*(ptr))))
+struct page_info *allocate_perdomain_l3(struct domain *d);
int create_perdomain_mapping(struct domain *d, unsigned long va,
unsigned int nr, l1_pgentry_t **pl1tab,
struct page_info **ppg);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index a1b7770db0..63e6aadcda 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -6235,6 +6235,16 @@ static bool perdomain_l1e_needs_freeing(l1_pgentry_t l1e)
(_PAGE_PRESENT | _PAGE_AVAIL0);
}
+struct page_info *allocate_perdomain_l3(struct domain *d)
+{
+ struct page_info *pg = alloc_domheap_page(d, MEMF_no_owner);
+
+ if ( pg )
+ clear_domain_page(page_to_mfn(pg));
+
+ return pg;
+}
+
int create_perdomain_mapping(struct domain *d, unsigned long va,
unsigned int nr, l1_pgentry_t **pl1tab,
struct page_info **ppg)
@@ -6248,24 +6258,12 @@ int create_perdomain_mapping(struct domain *d, unsigned long va,
ASSERT(va >= PERDOMAIN_VIRT_START &&
va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
- if ( !d->arch.perdomain_l3_pg )
- {
- pg = alloc_domheap_page(d, MEMF_no_owner);
- if ( !pg )
- return -ENOMEM;
- l3tab = __map_domain_page(pg);
- clear_page(l3tab);
- d->arch.perdomain_l3_pg = pg;
- if ( !nr )
- {
- unmap_domain_page(l3tab);
- return 0;
- }
- }
- else if ( !nr )
+ BUG_ON(!d->arch.perdomain_l3_pg);
+
+ if ( !nr )
return 0;
- else
- l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
+
+ l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
ASSERT(!l3_table_offset(va ^ (va + nr * PAGE_SIZE - 1)));
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 02/44] x86/mm: purge unneeded destroy_perdomain_mapping()
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
2026-11-03 22:13 ` [PATCH v6 01/44] x86/mm: allocate the per-domain L3 in arch_domain_create() George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 03/44] x86/mm: prepare destroy_perdomain_mapping() for per-vCPU perdomain areas George Dunlap
` (41 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
We want to change per-domain mappings to be per-vCPU mappings. In
preparation for that, we want to arrange that
destroy_perdomain_mapping() work either with a single perdomain area,
or with a per-vCPU perdomain area.
There are two calls made from domain-scoped contexts; both calls turn
out to be unnecessary:
- destroy_perdomain_mapping() is not logically the undo of
create_perdomain_mapping(), as the name and its use in
hvm_domain_initialise() suggest. destroy_ tears down mappings
without freeing the per-domain L3; and since the call here passes
nr == 0, it tears down nothing at all. The per-domain L3 page is
actually freed by free_perdomain_mappings(), which
hvm_domain_initialise()'s caller, arch_domain_create(), already
invokes on its failure path.
- The call in pv_domain_destroy() is redundant at both of its call
sites. arch_domain_destroy() unconditionally calls
free_perdomain_mappings(), which tears down the same entries and
additionally frees the page-table structures; and the other caller,
the error path of pv_domain_initialise(), returns into
arch_domain_create()'s failure path, which invokes
free_perdomain_mappings() as well.
One vCPU-scoped caller, pv_destroy_gdt_ldt_l1tab(), is covered the
same way: a vCPU is only ever destroyed together with its domain, or
on the failure path of creating it, and both routes end in
free_perdomain_mappings(); the GDT/LDT slots hold no area-owned pages,
so there is nothing else for the call to do. While we're getting rid
of unnecessary callers, drop this one too.
Note that free_compat_arg_xlat() keeps its explicit teardown:
switch_compat() calls it on the failure path. While in practice most
toolstacks will immediately destroy the domain, in theory a toolstack
could let it live on as a 64-bit guest. Release the resources
immediately rather than waiting for domain destruction.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
---
Changes in v6:
- Message: create_perdomain_mapping() no longer allocates the per-domain
L3, since "x86/mm: allocate the per-domain L3 in arch_domain_create()".
Changes in v4:
- Remove an unnecessary comment (Jan).
Changes in v3:
- Note that pv_domain_destroy()'s other call site, the error path of
pv_domain_initialise(), is covered the same way (Jan).
- Also drop pv_destroy_gdt_ldt_l1tab(), the per-vCPU teardown call
(Jan), and comment on why free_compat_arg_xlat() keeps its own. The
Reviewed-by tags given for the previous form are dropped.
Changes in v2:
- Added to the series
Changes since the previously posted version:
- Reworked the commit message to make it more clear how it fits in
with the larger series. No functional change.
---
xen/arch/x86/hvm/hvm.c | 1 -
xen/arch/x86/pv/domain.c | 10 ----------
2 files changed, 11 deletions(-)
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 46d3a520d5..71da15c0a4 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -726,7 +726,6 @@ int hvm_domain_initialise(struct domain *d,
XFREE(d->arch.hvm.irq);
fail0:
hvm_destroy_cacheattr_region_list(d);
- destroy_perdomain_mapping(d, PERDOMAIN_VIRT_START, 0);
hvm_domain_relinquish_resources(d);
XFREE(d->arch.hvm.io_handler);
XFREE(d->arch.hvm.pl_time);
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index 0c42ae58aa..28ea0742ab 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -319,12 +319,6 @@ static int pv_create_gdt_ldt_l1tab(struct vcpu *v)
NULL);
}
-static void pv_destroy_gdt_ldt_l1tab(struct vcpu *v)
-{
- destroy_perdomain_mapping(v->domain, GDT_VIRT_START(v),
- 1U << GDT_LDT_VCPU_SHIFT);
-}
-
void pv_vcpu_destroy(struct vcpu *v)
{
if ( is_pv_32bit_vcpu(v) )
@@ -333,7 +327,6 @@ void pv_vcpu_destroy(struct vcpu *v)
release_compat_l4(v);
}
- pv_destroy_gdt_ldt_l1tab(v);
XFREE(v->arch.pv.trap_ctxt);
}
@@ -381,9 +374,6 @@ void pv_domain_destroy(struct domain *d)
{
pv_l1tf_domain_destroy(d);
- destroy_perdomain_mapping(d, GDT_LDT_VIRT_START,
- GDT_LDT_MBYTES << (20 - PAGE_SHIFT));
-
XFREE(d->arch.pv.cpuidmasks);
FREE_XENHEAP_PAGE(d->arch.pv.gdt_ldt_l1tab);
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 03/44] x86/mm: prepare destroy_perdomain_mapping() for per-vCPU perdomain areas
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
2026-11-03 22:13 ` [PATCH v6 01/44] x86/mm: allocate the per-domain L3 in arch_domain_create() George Dunlap
2026-11-03 22:13 ` [PATCH v6 02/44] x86/mm: purge unneeded destroy_perdomain_mapping() George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 04/44] x86/domain_page: drop redundant create_perdomain_mapping() call George Dunlap
` (40 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Roger Pau Monné, Jan Beulich,
Andrew Cooper, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
We want to change per-domain mappings to be per-vCPU mappings. In
preparation for that, we want to arrange that
destroy_perdomain_mapping() work either with a single perdomain area,
or with a per-vCPU perdomain area.
The one remaining caller, free_compat_arg_xlat(), is already in a vCPU
context, so we just need to change the parameter from a domain pointer
to a vCPU pointer.
No functional change.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Roger Pau Monné <roger@xenproject.org>
---
Changes in v4:
- Say "No functional change." (Roger).
Changes in v3:
- Rebased onto staging without the GDT/LDT conversions (for the
HVM-only series): free_compat_arg_xlat() is the only caller left.
- Drop the linear-page-table fast path altogether (Jan): its only user
would have been guest-driven LDT teardown, and without it the
function does not depend on the linear mapping being available.
Changes in v2:
- Added to the series
Changes since the previously posted version:
- The fast path is gone (v3), and with it the previously posted
version's sync_local_execstate() call.
- Reword and retitle for clarity (was: "x86/mm: switch
destroy_perdomain_mapping() parameter from domain to vCPU").
---
xen/arch/x86/include/asm/mm.h | 2 +-
xen/arch/x86/mm.c | 3 ++-
xen/arch/x86/x86_64/mm.c | 2 +-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index e77ac5ccdf..1cdd30f7e2 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -610,7 +610,7 @@ struct page_info *allocate_perdomain_l3(struct domain *d);
int create_perdomain_mapping(struct domain *d, unsigned long va,
unsigned int nr, l1_pgentry_t **pl1tab,
struct page_info **ppg);
-void destroy_perdomain_mapping(struct domain *d, unsigned long va,
+void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr);
void free_perdomain_mappings(struct domain *d);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 63e6aadcda..c784c4f726 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -6356,10 +6356,11 @@ int create_perdomain_mapping(struct domain *d, unsigned long va,
return rc;
}
-void destroy_perdomain_mapping(struct domain *d, unsigned long va,
+void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr)
{
const l3_pgentry_t *l3tab, *pl3e;
+ const struct domain *d = v->domain;
ASSERT(va >= PERDOMAIN_VIRT_START &&
va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index 8eadab7933..f467ed426b 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -739,7 +739,7 @@ int setup_compat_arg_xlat(struct vcpu *v)
void free_compat_arg_xlat(struct vcpu *v)
{
- destroy_perdomain_mapping(v->domain, ARG_XLAT_START(v),
+ destroy_perdomain_mapping(v, ARG_XLAT_START(v),
PFN_UP(COMPAT_ARG_XLAT_SIZE));
}
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 04/44] x86/domain_page: drop redundant create_perdomain_mapping() call
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (2 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 03/44] x86/mm: prepare destroy_perdomain_mapping() for per-vCPU perdomain areas George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 05/44] x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain areas George Dunlap
` (39 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Roger Pau Monné, Jan Beulich,
Andrew Cooper, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
We want to change per-domain mappings to be per-vCPU mappings. In
preparation for that, we want to arrange that
create_perdomain_mapping() work either with a single perdomain area,
or with a per-vCPU perdomain area.
The one remaining call in a domain context turns out to be
unnecessary:
mapcache_domain_init() pre-plumbs L1 tables over the whole
inuse/garbage bitmap range (sized for the full MAPCACHE_ENTRIES
capacity), without populating any data pages. The plumbing is
redundant: mapcache_vcpu_init() installs the bitmap pages the domain
will actually use with populating calls, which allocate any missing
page-table structure on demand -- and since d->max_vcpus is fixed
before any vCPU is created, the range those calls cover never grows.
The pre-plumbed tail beyond it backs virtual addresses that are never
populated at all.
In debug builds, and on hosts with memory hotplug or with more memory
than the PV directmap covers, the call was previously what first
allocated a PV domain's per-domain L3. Since "x86/mm: allocate the
per-domain L3 in arch_domain_create()", the L3 comes with the domain.
Drop the call.
With the only fallible operation gone, mapcache_domain_init() becomes
void, and arch_domain_create()'s error handling for it goes away.
The per-domain-mapping work in mapcache_vcpu_init() stays where it is
rather than moving into mapcache_domain_init() along with this change,
ready for a later patch which introduces per-vCPU mapcaches.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Roger Pau Monné <roger@xenproject.org>
---
Changes in v6:
- Message: since "x86/mm: allocate the per-domain L3 in
arch_domain_create()", the L3 comes with the domain, and this is the
only call left in a domain context.
Changes in v3:
- Rebased onto staging without the GDT/LDT conversions (for the
HVM-only series); no functional difference.
- Say why the per-vCPU part stays in mapcache_vcpu_init() (Jan).
- Say that the call also first allocated a PV domain's per-domain L3 in
some builds, which all builds now do with the first vCPU.
Changes in v2:
- Added to the series (split out of the following patch).
Changes since the previously posted version:
- Split out of "x86/mm: switch {create,destroy}_perdomain_mapping()
domain parameter to vCPU", where the removal was folded into the
parameter switch without its own rationale.
---
xen/arch/x86/domain.c | 3 +--
xen/arch/x86/domain_page.c | 8 ++------
xen/arch/x86/include/asm/domain.h | 2 +-
3 files changed, 4 insertions(+), 9 deletions(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index e67c336b82..173daf9000 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -914,8 +914,7 @@ int arch_domain_create(struct domain *d,
}
else if ( is_pv_domain(d) )
{
- if ( (rc = mapcache_domain_init(d)) != 0 )
- goto fail;
+ mapcache_domain_init(d);
if ( (rc = pv_domain_initialise(d)) != 0 )
goto fail;
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 5744754f11..f02be1675b 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -223,7 +223,7 @@ void unmap_domain_page(const void *ptr)
local_irq_restore(flags);
}
-int mapcache_domain_init(struct domain *d)
+void mapcache_domain_init(struct domain *d)
{
struct mapcache_domain *dcache = &d->arch.pv.mapcache;
unsigned int bitmap_pages;
@@ -232,7 +232,7 @@ int mapcache_domain_init(struct domain *d)
#ifdef NDEBUG
if ( !mem_hotplug && max_page <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
- return 0;
+ return;
#endif
BUILD_BUG_ON(MAPCACHE_VIRT_END + PAGE_SIZE * (3 +
@@ -244,10 +244,6 @@ int mapcache_domain_init(struct domain *d)
(bitmap_pages + 1) * PAGE_SIZE / sizeof(long);
spin_lock_init(&dcache->lock);
-
- return create_perdomain_mapping(d, (unsigned long)dcache->inuse,
- 2 * bitmap_pages + 1,
- NIL(l1_pgentry_t *), NULL);
}
int mapcache_vcpu_init(struct vcpu *v)
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 2d0a915410..212daae1f5 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -89,7 +89,7 @@ struct mapcache_domain {
unsigned long *garbage;
};
-int mapcache_domain_init(struct domain *d);
+void mapcache_domain_init(struct domain *d);
int mapcache_vcpu_init(struct vcpu *v);
/* x86/64: toggle guest between kernel and user modes. */
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 05/44] x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain areas
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (3 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 04/44] x86/domain_page: drop redundant create_perdomain_mapping() call George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 06/44] x86/spec-ctrl: introduce Address Space Isolation command line option George Dunlap
` (38 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
We want to change per-domain mappings to be per-vCPU mappings. In
preparation for that, we want to arrange that
create_perdomain_mapping() work either with a single perdomain area,
or with a per-vCPU perdomain area.
The remaining callers are already in a vCPU context. This is no
accident: much of the perdomain area has always been laid out in
per-vCPU slices -- each vCPU has its own GDT/LDT window and its own
COMPAT_ARG_XLAT pages -- and each vCPU's slice is set up as that vCPU
is created. For these callers, we just need to change the parameter
from a domain pointer to a vCPU pointer. Once the perdomain area
itself becomes per-vCPU, the same calls will populate the owning
vCPU's own area rather than slices of a shared one.
Note that we cannot yet do a parallel movement of
free_perdomain_mappings(): the per-domain page-table hierarchy is
still a single domain-wide structure shared by all vCPUs, so tearing
it down from a per-vCPU path would pull the mappings out from under
sibling vCPUs (e.g. on a partially failed, retryable
XEN_DOMCTL_max_vcpus), and vCPU-create error paths can rely on domain
destruction to free a partially set up hierarchy. Once a domain's
area is per-vCPU, a later change gives its vCPUs a teardown of their
own; the domain-wide teardown stays for domains with a shared area, PV
ones included.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v6:
- Rebase over "x86/mm: allocate the per-domain L3 in
arch_domain_create()".
- No longer add an implicit L3 creation in arch_vcpu_create().
Changes in v5:
- Message: the domain-wide mapcache is one pool, not per-vCPU windows:
drop it from the per-vCPU slices, and say "much of" the area.
Changes in v4:
- Message: drop a sentence repeating its paragraph, and say that the
domain-wide teardown stays for domains with a shared area (Roger).
- End the comment in arch_vcpu_create() at the colon (Roger).
Changes in v3:
- Rebased onto staging without the GDT/LDT conversions (for the
HVM-only series): the capture parameters stay, as the PV GDT/LDT
stash still uses them.
Changes in v2:
- Added to the series
Changes since the previously posted version:
- Keep free_perdomain_mappings() (and hence perdomain teardown)
domain-scoped.
- Keep the idle domain without a perdomain area.
- Retitle (was: "x86/mm: switch {create,destroy}_perdomain_mapping()
domain parameter to vCPU"); destroy_perdomain_mapping() was switched
in a separate patch.
- Split the removal of mapcache_domain_init()'s redundant
create_perdomain_mapping() call into its own (preceding) patch.
---
xen/arch/x86/domain_page.c | 6 +++---
xen/arch/x86/include/asm/mm.h | 2 +-
xen/arch/x86/mm.c | 3 ++-
xen/arch/x86/pv/domain.c | 2 +-
xen/arch/x86/x86_64/mm.c | 2 +-
5 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index f02be1675b..f3e65e2d44 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -260,15 +260,15 @@ int mapcache_vcpu_init(struct vcpu *v)
if ( ents > dcache->entries )
{
/* Populate page tables. */
- int rc = create_perdomain_mapping(d, MAPCACHE_VIRT_START, ents,
+ int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START, ents,
NIL(l1_pgentry_t *), NULL);
/* Populate bit maps. */
if ( !rc )
- rc = create_perdomain_mapping(d, (unsigned long)dcache->inuse,
+ rc = create_perdomain_mapping(v, (unsigned long)dcache->inuse,
nr, NULL, NIL(struct page_info *));
if ( !rc )
- rc = create_perdomain_mapping(d, (unsigned long)dcache->garbage,
+ rc = create_perdomain_mapping(v, (unsigned long)dcache->garbage,
nr, NULL, NIL(struct page_info *));
if ( rc )
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 1cdd30f7e2..e85a6db203 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -607,7 +607,7 @@ int compat_arch_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg);
#define IS_NIL(ptr) (!((uintptr_t)(ptr) + sizeof(*(ptr))))
struct page_info *allocate_perdomain_l3(struct domain *d);
-int create_perdomain_mapping(struct domain *d, unsigned long va,
+int create_perdomain_mapping(struct vcpu *v, unsigned long va,
unsigned int nr, l1_pgentry_t **pl1tab,
struct page_info **ppg);
void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index c784c4f726..61bd4aa02a 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -6245,10 +6245,11 @@ struct page_info *allocate_perdomain_l3(struct domain *d)
return pg;
}
-int create_perdomain_mapping(struct domain *d, unsigned long va,
+int create_perdomain_mapping(struct vcpu *v, unsigned long va,
unsigned int nr, l1_pgentry_t **pl1tab,
struct page_info **ppg)
{
+ struct domain *d = v->domain;
struct page_info *pg;
l3_pgentry_t *l3tab;
l2_pgentry_t *l2tab;
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index 28ea0742ab..f3c711b85b 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -313,7 +313,7 @@ int switch_compat(struct domain *d)
static int pv_create_gdt_ldt_l1tab(struct vcpu *v)
{
- return create_perdomain_mapping(v->domain, GDT_VIRT_START(v),
+ return create_perdomain_mapping(v, GDT_VIRT_START(v),
1U << GDT_LDT_VCPU_SHIFT,
v->domain->arch.pv.gdt_ldt_l1tab,
NULL);
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index f467ed426b..7b04537066 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -732,7 +732,7 @@ void __init zap_low_mappings(void)
int setup_compat_arg_xlat(struct vcpu *v)
{
- return create_perdomain_mapping(v->domain, ARG_XLAT_START(v),
+ return create_perdomain_mapping(v, ARG_XLAT_START(v),
PFN_UP(COMPAT_ARG_XLAT_SIZE),
NULL, NIL(struct page_info *));
}
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 06/44] x86/spec-ctrl: introduce Address Space Isolation command line option
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (4 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 05/44] x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain areas George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 07/44] x86/mm: have init_xen_l4_slots() take a vCPU in place of the domain George Dunlap
` (37 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Andrew Cooper, Anthony PERARD, Michal Orzel,
Jan Beulich, Julien Grall, Roger Pau Monné,
Stefano Stabellini, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
Introduce the `asi=` command line option, and the opt_vcpu_pt_hvm and
opt_vcpu_pt_hwdom knobs plus the per-domain d->arch.asi.vcpu_pt setting
they control. The option is introduced ahead of the functionality it
enables, so that the newly added code can be keyed on it from the start;
all knobs currently default to off, and any use of the option, `asi=0`
included, adds a boot-time warning that the feature is not functional
and is meant for development only.
The mechanisms this option controls apply to HVM domains: an HVM vCPU
already runs on its own monitor table, so giving each vCPU its own
per-domain area needs no work at context switch. d->arch.asi.vcpu_pt
is therefore only ever set for HVM domains, a PVH hardware domain
included; PV domains, a PV hardware domain included, are unaffected.
Besides the plain boolean, `pv=`, `hvm=` and `dom0=` select the features
for PV domains or HVM domains other than the hardware domain, or for the
hardware domain, PV or PVH. None of the features applies to PV domains,
so `pv=` has no effect beyond a boot-log line saying it is ignored, and
`dom0=` acts on a PVH hardware domain only.
The boot log gains "ASI features for ..." lines for Dom0 and HVM
domains, so hardware-domain-only configurations remain visible.
Further per-mechanism tokens arrive with their mechanisms in later
patches.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- Rename opt_pvh_vcpu_pt_hwdom back to opt_vcpu_pt_hwdom: that it acts
on a PVH hardware domain only is a limitation of the implementation,
not of the design (Roger).
- pv= logs that it is ignored (Roger).
- parse_asi() adds the development warning, so any use of asi= gives
it and later sub-options need not extend a condition (Roger).
- print_details(): IS_ENABLED(CONFIG_HVM) rather than #ifdef (Roger).
Changes in v4:
- Docs: "domains" throughout, not "guests" (Roger).
- Group the per-domain settings as one-bit fields of d->arch.asi
(Roger).
- Rename opt_vcpu_pt_hwdom to opt_pvh_vcpu_pt_hwdom: it acts on a PVH
hardware domain only (Jan).
- parse_asi(): break in the success branch and fall through to
default:, rather than a default: label after an else (MISRA C:2012
Rule 16.2; Jan).
- Message: enabling a knob adds a boot-time warning; it does not taint
(Roger).
- Add the pv= and dom0= sub-options here, beside hvm= (Roger).
Changes in v3:
- HVM only: drop the PV knob and its interaction with XPTI. vCPU-PT
applies to HVM domains and to a PVH hardware domain; the Dom0 boot
log line says None for a PV one.
Changes in v2:
- Added to the series
Changes since the previously posted version:
- Include the hardware domain in the development warning and the boot
log summary.
- Make the opt_vcpu_pt_* knobs plain booleans preinitialised to false,
dropping the late -1 resolution.
- Documentation: mention possible protection against unmitigated
attacks, and state that hvm= does not affect the hardware domain.
- Rewrite the commit message.
---
docs/misc/xen-command-line.pandoc | 29 ++++++++++
xen/arch/x86/include/asm/domain.h | 6 ++
xen/arch/x86/include/asm/spec_ctrl.h | 2 +
xen/arch/x86/spec_ctrl.c | 87 ++++++++++++++++++++++++++++
4 files changed, 124 insertions(+)
diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
index a6f08463d1..839d06da26 100644
--- a/docs/misc/xen-command-line.pandoc
+++ b/docs/misc/xen-command-line.pandoc
@@ -202,6 +202,35 @@ to appropriate auditing by Xen. Argo is disabled by default.
This option is disabled by default, to protect domains from a DoS by a
buggy or malicious other domain spamming the ring.
+### asi (x86)
+> `= List of [ <bool>, {pv,hvm,dom0}=<bool>,
+> vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
+
+> Default: `false`
+
+Offers control over whether the hypervisor will engage in Address Space
+Isolation, by not having potentially sensitive information permanently mapped
+in the VMM page-tables. Using this option might avoid the need to apply
+mitigations for certain speculative related attacks, at the cost of mapping
+sensitive information on-demand. It might also offer some protection against
+unmitigated speculation-related attacks.
+
+The mechanisms currently implemented apply to HVM domains, including a PVH
+hardware domain. PV domains, including a PV hardware domain, are unaffected.
+
+* `pv=`, `hvm=` and `dom0=` enable the features for PV domains or HVM domains
+ other than the hardware domain, or for the hardware domain, PV or PVH. The
+ plain boolean does all three: `asi=1,dom0=0` leaves the hardware domain
+ out. The forms naming no domain type (an un-suffixed `vcpu-pt=<bool>`)
+ cover the hardware domain too.
+
+**WARNING: manual de-selection of enabled options will invalidate any
+protection offered by the feature. The fine grained options provided below
+are meant to be used for debugging purposes only.**
+
+* `vcpu-pt` gives each vCPU its own per-domain area: the per-domain slot of
+ each vCPU's monitor table maps a region private to that vCPU.
+
### asid (x86)
> `= <boolean>`
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 212daae1f5..12f79feb42 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -474,6 +474,12 @@ struct arch_domain
/* Don't unconditionally inject #GP for unhandled MSRs. */
bool msr_relaxed;
+ /* Fine grained ASI related options. */
+ struct {
+ /* Give each vCPU its own per-domain area (HVM only). */
+ bool vcpu_pt:1;
+ } asi;
+
/* Emulated devices enabled bitmap. */
uint32_t emulation_flags;
} __cacheline_aligned;
diff --git a/xen/arch/x86/include/asm/spec_ctrl.h b/xen/arch/x86/include/asm/spec_ctrl.h
index 8f82533c41..b62229ab5d 100644
--- a/xen/arch/x86/include/asm/spec_ctrl.h
+++ b/xen/arch/x86/include/asm/spec_ctrl.h
@@ -87,6 +87,8 @@ extern uint8_t default_scf;
extern int8_t opt_xpti_hwdom, opt_xpti_domu;
+extern bool opt_vcpu_pt_hwdom, opt_vcpu_pt_hvm;
+
extern bool cpu_has_bug_l1tf;
extern int8_t opt_pv_l1tf_hwdom, opt_pv_l1tf_domu;
extern bool opt_bp_spec_reduce;
diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c
index bc8538a56f..8797523a62 100644
--- a/xen/arch/x86/spec_ctrl.c
+++ b/xen/arch/x86/spec_ctrl.c
@@ -86,6 +86,10 @@ bool __ro_after_init opt_bp_spec_reduce = true;
static bool __initdata opt_ibpb_alt;
+/* Give each vCPU of an HVM domain its own per-domain area (vCPU-PT). */
+bool __ro_after_init opt_vcpu_pt_hvm;
+bool __ro_after_init opt_vcpu_pt_hwdom;
+
static int __init cf_check parse_spec_ctrl(const char *s)
{
const char *ss;
@@ -487,6 +491,80 @@ static int __init cf_check parse_pv_l1tf(const char *s)
}
custom_param("pv-l1tf", parse_pv_l1tf);
+static int __init cf_check parse_asi(const char *s)
+{
+ static bool __initdata warned;
+ const char *ss;
+ int val, rc = 0;
+
+ if ( !warned )
+ {
+ warned = true;
+ warning_add(
+ "Address Space Isolation is not functional, this option is\n"
+ "intended to be used only for development purposes.\n");
+ }
+
+ /* Interpret 'asi' alone in its positive boolean form. */
+ if ( *s == '\0' )
+ opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = true;
+
+ do {
+ ss = strchr(s, ',');
+ if ( !ss )
+ ss = strchr(s, '\0');
+
+ val = parse_bool(s, ss);
+ switch ( val )
+ {
+ case 0:
+ case 1:
+ opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
+ break;
+
+ default:
+ if ( (val = parse_boolean("pv", s, ss)) >= 0 )
+ printk(XENLOG_WARNING
+ "No ASI feature applies to PV domains - ignoring 'asi=%.*s' setting\n",
+ (int)(ss - s), s);
+ else if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
+ opt_vcpu_pt_hvm = val;
+ else if ( (val = parse_boolean("dom0", s, ss)) >= 0 )
+ opt_vcpu_pt_hwdom = val;
+ else if ( (val = parse_boolean("vcpu-pt", s, ss)) != -1 )
+ {
+ switch ( val )
+ {
+ case 1:
+ case 0:
+ opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
+ break;
+
+ case -2:
+ s += strlen("vcpu-pt=");
+ if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
+ {
+ opt_vcpu_pt_hvm = val;
+ break;
+ }
+ fallthrough;
+ default:
+ rc = -EINVAL;
+ break;
+ }
+ }
+ else if ( *s )
+ rc = -EINVAL;
+ break;
+ }
+
+ s = ss + 1;
+ } while ( *ss );
+
+ return rc;
+}
+custom_param("asi", parse_asi);
+
static void __init print_details(enum ind_thunk thunk)
{
unsigned int _7d0 = 0, _7d2 = 0, e8b = 0, e21a = 0, e21c = 0, max = 0, tmp;
@@ -680,6 +758,12 @@ static void __init print_details(enum ind_thunk thunk)
opt_pv_l1tf_hwdom ? "enabled" : "disabled",
opt_pv_l1tf_domu ? "enabled" : "disabled");
#endif
+
+ printk(" ASI features for Dom0:%s\n",
+ opt_dom0_pvh && opt_vcpu_pt_hwdom ? " vCPU-PT" : " None");
+ if ( IS_ENABLED(CONFIG_HVM) )
+ printk(" ASI features for HVM VMs:%s\n",
+ opt_vcpu_pt_hvm ? " vCPU-PT" : " None");
}
static bool __init check_smt_enabled(void)
@@ -1866,6 +1950,9 @@ void spec_ctrl_init_domain(struct domain *d)
if ( pv )
d->arch.pv.xpti = is_hardware_domain(d) ? opt_xpti_hwdom
: opt_xpti_domu;
+
+ d->arch.asi.vcpu_pt = !pv && (is_hardware_domain(d) ? opt_vcpu_pt_hwdom
+ : opt_vcpu_pt_hvm);
}
void __init init_speculation_mitigations(void)
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 07/44] x86/mm: have init_xen_l4_slots() take a vCPU in place of the domain
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (5 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 06/44] x86/spec-ctrl: introduce Address Space Isolation command line option George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 08/44] x86/hvm: introduce per-vCPU L3 page-table George Dunlap
` (36 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie,
Tim Deegan
A later change gives the vCPUs of some domains per-domain areas of
their own, and slot 260 of a table private to one vCPU must then hold
that vCPU's L3. Eventually, all guest types may be able to support
either per-domain L3s or per-vCPU L3s.
HVM domains always have one root page table per vCPU; this does not
change. So init_xen_l4_slots() needs the appropriate L3 at vCPU
creation, either the domain-wide L3 or the per-vCPU L3.
PV domains with a domain-wide L3 should behave as now: the Xen slots
of guest root pagetables need the domain-wide L3 installed. But PV
guests with per-vCPU page tables can never run directly on guest
pagetables; they will always need some separate page table, which can
be initialized with the per-vCPU L3.
So, create two versions of init_xen_l4_slots(). The first is used in
contexts where we may be initializing a non-guest root, and must have
a vCPU L3 if per-vCPU L3s are in use; this takes a vCPU as an
argument. The second is used for roots that all of a domain's vCPUs
share (a PV guest's own L4s, and their shadows), which only need an
L3 if domain-wide L3s are in use; this takes a domain as an argument.
Both wrap an internal function, which takes a domain and an L3
pointer, and leaves the slot empty if this pointer is NULL. At the
moment, both always pass on the domain-wide L3, but in the future this
will change if the guest is using per-vCPU pagetables.
Otherwise no functional change: slot 260 still gets the domain's
per-domain L3.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v6:
- Add init_xen_l4_slots_pv() form, which only takes a domain as an
argument; promote_l4_table(), the PV dom0 builder and the shadows of
PV guests' L4s use it, so promote_l4_table() no longer needs vCPU 0.
Changes in v5:
- promote_l4_table() checks for vCPU 0 itself, now that "x86/mm: refuse
to promote an L4 while the domain has no per-domain L3" tests for the
L3 instead.
Changes in v4:
- New in this version, split out of "x86/hvm: introduce per-vCPU L3
page-table": init_xen_l4_slots() takes a vCPU in place of the domain
(Roger, Jan).
- Reflow promote_l4_table()'s call to init_xen_l4_slots() (Jan).
---
xen/arch/x86/include/asm/mm.h | 4 +++-
xen/arch/x86/mm.c | 35 +++++++++++++++++++++++++++++-----
xen/arch/x86/mm/hap/hap.c | 2 +-
xen/arch/x86/mm/shadow/hvm.c | 2 +-
xen/arch/x86/mm/shadow/multi.c | 5 +++--
xen/arch/x86/pv/dom0_build.c | 4 ++--
xen/arch/x86/pv/domain.c | 2 +-
7 files changed, 41 insertions(+), 13 deletions(-)
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index e85a6db203..0d2cecf1c7 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -370,7 +370,9 @@ int devalidate_page(struct page_info *page, unsigned long type,
void init_xen_pae_l2_slots(l2_pgentry_t *l2t, const struct domain *d);
void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
- const struct domain *d, mfn_t sl4mfn, bool ro_mpt);
+ const struct vcpu *v, mfn_t sl4mfn, bool ro_mpt);
+void init_xen_l4_slots_pv(l4_pgentry_t *l4t, mfn_t l4mfn,
+ const struct domain *d, mfn_t sl4mfn, bool ro_mpt);
bool fill_ro_mpt(mfn_t mfn);
void zap_ro_mpt(mfn_t mfn);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 61bd4aa02a..10106d9f03 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -1693,8 +1693,10 @@ static int promote_l3_table(struct page_info *page)
* appropriate slots. Pagetables not shared with guests will gain the
* extended directmap.
*/
-void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
- const struct domain *d, mfn_t sl4mfn, bool ro_mpt)
+static void _init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
+ const struct domain *d,
+ const struct page_info *l3_pg,
+ mfn_t sl4mfn, bool ro_mpt)
{
/*
* PV vcpus need a shortened directmap. HVM and Idle vcpus get the full
@@ -1723,7 +1725,7 @@ void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
/* Slot 260: Per-domain mappings. */
l4t[l4_table_offset(PERDOMAIN_VIRT_START)] =
- l4e_from_page(d->arch.perdomain_l3_pg, __PAGE_HYPERVISOR_RW);
+ l3_pg ? l4e_from_page(l3_pg, __PAGE_HYPERVISOR_RW) : l4e_empty();
/* Slot 4: Per-domain mappings mirror. */
BUILD_BUG_ON(IS_ENABLED(CONFIG_PV32) &&
@@ -1770,6 +1772,29 @@ void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
}
}
+/*
+ * v is the vCPU the table is private to. For shared root pagetables, use
+ * init_xen_l4_slots_pv().
+ */
+void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
+ const struct vcpu *v, mfn_t sl4mfn, bool ro_mpt)
+{
+ const struct domain *d = v->domain;
+
+ _init_xen_l4_slots(l4t, l4mfn, d, d->arch.perdomain_l3_pg, sl4mfn, ro_mpt);
+}
+
+/*
+ * This is a version of init_xen_l4_slots for shared root pagetables;
+ * it only propagates domain-wide perdomain L3s, and so only needs a
+ * domain pointer.
+ */
+void init_xen_l4_slots_pv(l4_pgentry_t *l4t, mfn_t l4mfn,
+ const struct domain *d, mfn_t sl4mfn, bool ro_mpt)
+{
+ _init_xen_l4_slots(l4t, l4mfn, d, d->arch.perdomain_l3_pg, sl4mfn, ro_mpt);
+}
+
bool fill_ro_mpt(mfn_t mfn)
{
l4_pgentry_t *l4tab = map_domain_page(mfn);
@@ -1878,8 +1903,8 @@ static int promote_l4_table(struct page_info *page)
if ( !rc )
{
- init_xen_l4_slots(pl4e, l4mfn,
- d, INVALID_MFN, VM_ASSIST(d, m2p_strict));
+ init_xen_l4_slots_pv(pl4e, l4mfn, d, INVALID_MFN,
+ VM_ASSIST(d, m2p_strict));
atomic_inc(&d->arch.pv.nr_l4_pages);
}
unmap_domain_page(pl4e);
diff --git a/xen/arch/x86/mm/hap/hap.c b/xen/arch/x86/mm/hap/hap.c
index 0ede4181a0..aba4f77df9 100644
--- a/xen/arch/x86/mm/hap/hap.c
+++ b/xen/arch/x86/mm/hap/hap.c
@@ -407,7 +407,7 @@ static mfn_t hap_make_monitor_table(struct vcpu *v)
m4mfn = page_to_mfn(pg);
l4e = map_domain_page(m4mfn);
- init_xen_l4_slots(l4e, m4mfn, d, INVALID_MFN, false);
+ init_xen_l4_slots(l4e, m4mfn, v, INVALID_MFN, false);
unmap_domain_page(l4e);
return m4mfn;
diff --git a/xen/arch/x86/mm/shadow/hvm.c b/xen/arch/x86/mm/shadow/hvm.c
index e6fb97c4b6..0b23326214 100644
--- a/xen/arch/x86/mm/shadow/hvm.c
+++ b/xen/arch/x86/mm/shadow/hvm.c
@@ -760,7 +760,7 @@ mfn_t sh_make_monitor_table(const struct vcpu *v, unsigned int shadow_levels)
* shadow-linear mapping will either be inserted below when creating
* lower level monitor tables, or later in sh_update_cr3().
*/
- init_xen_l4_slots(l4e, m4mfn, d, INVALID_MFN, false);
+ init_xen_l4_slots(l4e, m4mfn, v, INVALID_MFN, false);
if ( shadow_levels < 4 )
{
diff --git a/xen/arch/x86/mm/shadow/multi.c b/xen/arch/x86/mm/shadow/multi.c
index 7f72b4b7b2..1831091b95 100644
--- a/xen/arch/x86/mm/shadow/multi.c
+++ b/xen/arch/x86/mm/shadow/multi.c
@@ -974,8 +974,9 @@ sh_make_shadow(struct vcpu *v, mfn_t gmfn, u32 shadow_type)
BUILD_BUG_ON(sizeof(l4_pgentry_t) != sizeof(shadow_l4e_t));
- init_xen_l4_slots(l4t, gmfn, d, smfn, (!is_pv_32bit_domain(d) &&
- VM_ASSIST(d, m2p_strict)));
+ init_xen_l4_slots_pv(l4t, gmfn, d, smfn,
+ (!is_pv_32bit_domain(d) &&
+ VM_ASSIST(d, m2p_strict)));
unmap_domain_page(l4t);
}
break;
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index 24f2fa0b7f..dd64af5616 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -724,8 +724,8 @@ static int __init dom0_construct(const struct boot_domain *bd)
maddr_to_page(mpt_alloc)->u.inuse.type_info = PGT_l4_page_table;
l4start = l4tab = __va(mpt_alloc); mpt_alloc += PAGE_SIZE;
clear_page(l4tab);
- init_xen_l4_slots(l4tab, _mfn(virt_to_mfn(l4start)),
- d, INVALID_MFN, true);
+ init_xen_l4_slots_pv(l4tab, _mfn(virt_to_mfn(l4start)),
+ d, INVALID_MFN, true);
v->arch.guest_table = pagetable_from_paddr(__pa(l4start));
}
else
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index f3c711b85b..f139d8f83c 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -127,7 +127,7 @@ static int setup_compat_l4(struct vcpu *v)
mfn = page_to_mfn(pg);
l4tab = map_domain_page(mfn);
clear_page(l4tab);
- init_xen_l4_slots(l4tab, mfn, v->domain, INVALID_MFN, false);
+ init_xen_l4_slots(l4tab, mfn, v, INVALID_MFN, false);
unmap_domain_page(l4tab);
/* This page needs to look like a pagetable so that it can be shadowed */
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 08/44] x86/hvm: introduce per-vCPU L3 page-table
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (6 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 07/44] x86/mm: have init_xen_l4_slots() take a vCPU in place of the domain George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 09/44] x86/domain_page: drop redundant is_pv_vcpu() checks George Dunlap
` (35 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
Currently all of a domain's vCPUs share its per-domain area: one L3,
referenced from every root page-table associated with the domain, so
every mapping in it is visible to every vCPU of the domain. Much of the
area is already laid out in per-vCPU slices (each vCPU's COMPAT_ARG_XLAT
pages, and for PV each vCPU's GDT/LDT window), but the visibility is
domain-wide. Meanwhile, much of the per-vCPU state Xen maintains -- the
VMCB, the VMX MSR load/save areas, FPU/XSAVE state -- lives in
always-mapped memory.
For HVM domains, allow each vCPU a per-domain area of its own instead
("vCPU-PT"). HVM monitor tables are already per-vCPU, so this needs
nothing at context switch: every monitor table carries its vCPU's own L3
from creation, and the per-vCPU mappings are isolated from other running
vCPUs at once. We will later build on this, adding a per-vCPU mapcache
and a per-vCPU state window, into which we can move the vCPU state
mentioned above.
Add pervcpu_l3_pg to the arch_vcpu struct, to correspond to the
perdomain_l3_pg in the domain struct. (We retain both so that vCPU-PT
can be chosen domain by domain; PV domains always use the domain-wide
L3.) Modify arch_domain_create() to only allocate a per-domain L3 if
vcpu_pt is not enabled for the domain; modify arch_vcpu_create() to
allocate a per-vCPU L3 if it is.
In {create,destroy}_perdomain_mapping(), if d->arch.asi.vcpu_pt, use
pervcpu_l3_pg as the per-domain L3; otherwise, use perdomain_l3_pg.
Introduce a helper, perdomain_l3(), to consistently choose the correct
one.
Introduce free_pervcpu_mappings() to free this tree, called from
arch_vcpu_destroy() on normal teardown. Since the vcpu structure holds
the only reference to pervcpu_l3_pg, arch_vcpu_create() must also call
it on its error paths: nothing else records the allocation once the
vcpu struct is torn down. The domain-wide free_perdomain_mappings()
keeps covering non-vCPU-PT domains.
init_xen_l4_slots() is given the vCPU that a table is private to.
Have it install that vCPU's perdomain_l3() in slot 260.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-fable-5-1, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v6:
- Port to new explicit L3 allocation regime.
Changes in v5:
- init_xen_l4_slots()'s BUG_ON() follows slot 260 to perdomain_l3().
- Message: all of a domain's vCPUs share its per-domain area; the
mapcache is not among the per-vCPU slices; the mode is "vCPU-PT".
The comment above init_xen_l4_slots() names perdomain_l3(v) and the
domain-wide L3.
Changes in v4:
- Rebase over earlier changes
- d->arch.vcpu_pt is now d->arch.asi.vcpu_pt.
- init_xen_l4_slots() taking a vCPU in place of the domain, and
promote_l4_table() failing while the domain has none, are now patches
of their own (Roger, Jan).
- Drop the fail_early label: every cleanup step after fail: is safe
before its setup has run (Roger).
- free_pervcpu_mappings(): clear the pointer, then free, as XFREE()
does (Roger).
- Likewise in free_perdomain_mappings().
Changes in v3:
- HVM only: PV domains keep the domain-wide area, so the PV parts go
(the single-vCPU limit, the shadow refusals, the XPTI slot install).
init_xen_l4_slots() takes the vCPU as an extra, optional argument
rather than in place of the domain, so PV guest L4s, which no vCPU
owns, are promoted exactly as before.
- Retitle (was: "x86/mm: introduce per-vCPU L3 page-table").
- free_compat_arg_xlat()'s comment names free_pervcpu_mappings() as the
teardown for a vCPU-PT domain.
Changes in v2:
- Added to the series
Changes since the previously posted version:
- Expand the commit message with the motivation and the design
rationale.
- Keep domain-wide freeing intact and introduce a vCPU-scoped
free_pervcpu_mappings() instead of re-scoping
free_perdomain_mappings(); fix the arch_vcpu_create() error-path
leaks of a partially built per-vCPU hierarchy.
- Add a perdomain_l3() helper for the root selection, rather than
open-coding the vcpu_pt choice (and testing both root pointers) at
each site.
---
xen/arch/x86/domain.c | 17 +++++++--
xen/arch/x86/include/asm/domain.h | 4 ++
xen/arch/x86/include/asm/mm.h | 2 +
xen/arch/x86/mm.c | 61 +++++++++++++++++++++++--------
4 files changed, 66 insertions(+), 18 deletions(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 173daf9000..46ca945b48 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -511,16 +511,23 @@ int arch_vcpu_create(struct vcpu *v)
v->arch.flags = TF_kernel_mode;
+ if ( d->arch.asi.vcpu_pt &&
+ (v->arch.pervcpu_l3_pg = allocate_perdomain_l3(d)) == NULL )
+ {
+ rc = -ENOMEM;
+ goto fail;
+ }
+
rc = mapcache_vcpu_init(v);
if ( rc )
- return rc;
+ goto fail;
if ( !is_idle_domain(d) )
{
paging_vcpu_init(v);
if ( (rc = vcpu_init_fpu(v)) != 0 )
- return rc;
+ goto fail;
vmce_init_vcpu(v);
@@ -568,6 +575,7 @@ int arch_vcpu_create(struct vcpu *v)
vcpu_destroy_fpu(v);
xfree(v->arch.msrs);
v->arch.msrs = NULL;
+ free_pervcpu_mappings(v);
return rc;
}
@@ -588,6 +596,8 @@ void arch_vcpu_destroy(struct vcpu *v)
pv_vcpu_destroy(v);
else
ASSERT_UNREACHABLE();
+
+ free_pervcpu_mappings(v);
}
int arch_sanitise_domain_config(const struct xen_domctl_createdomain *config)
@@ -901,7 +911,8 @@ int arch_domain_create(struct domain *d,
psr_domain_init(d);
- if ( (d->arch.perdomain_l3_pg = allocate_perdomain_l3(d)) == NULL )
+ if ( !d->arch.asi.vcpu_pt &&
+ (d->arch.perdomain_l3_pg = allocate_perdomain_l3(d)) == NULL )
{
rc = -ENOMEM;
goto fail;
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 12f79feb42..4140a92f15 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -332,6 +332,7 @@ struct monitor_write_data {
struct arch_domain
{
+ /* Domain-wide L3 for the per-domain slot; NULL in a vCPU-PT domain. */
struct page_info *perdomain_l3_pg;
/* I/O-port admin-specified access capabilities. */
@@ -688,6 +689,9 @@ struct arch_vcpu
struct vcpu_msrs *msrs;
+ /* Per-vCPU L3 for the per-domain slot; NULL in a non-vCPU-PT domain. */
+ struct page_info *pervcpu_l3_pg;
+
struct {
bool next_interrupt_enabled;
} monitor;
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 0d2cecf1c7..7b9a0f561d 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -369,6 +369,7 @@ int devalidate_page(struct page_info *page, unsigned long type,
int preemptible);
void init_xen_pae_l2_slots(l2_pgentry_t *l2t, const struct domain *d);
+struct page_info *perdomain_l3(const struct vcpu *v);
void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
const struct vcpu *v, mfn_t sl4mfn, bool ro_mpt);
void init_xen_l4_slots_pv(l4_pgentry_t *l4t, mfn_t l4mfn,
@@ -615,6 +616,7 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr);
void free_perdomain_mappings(struct domain *d);
+void free_pervcpu_mappings(struct vcpu *v);
void __iomem *ioremap_wc(paddr_t pa, size_t len);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 10106d9f03..f52c72a605 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -1682,6 +1682,18 @@ static int promote_l3_table(struct page_info *page)
}
#endif /* CONFIG_PV */
+/*
+ * The root of the per-domain area in use by @v: the vCPU's own L3 for a
+ * vCPU-PT domain, the domain-wide one otherwise.
+ */
+struct page_info *perdomain_l3(const struct vcpu *v)
+{
+ const struct domain *d = v->domain;
+
+ return d->arch.asi.vcpu_pt ? v->arch.pervcpu_l3_pg
+ : d->arch.perdomain_l3_pg;
+}
+
/*
* Fill an L4 with Xen entries.
*
@@ -1780,8 +1792,11 @@ void init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
const struct vcpu *v, mfn_t sl4mfn, bool ro_mpt)
{
const struct domain *d = v->domain;
+ struct page_info *l3_pg = perdomain_l3(v);
- _init_xen_l4_slots(l4t, l4mfn, d, d->arch.perdomain_l3_pg, sl4mfn, ro_mpt);
+ BUG_ON(!l3_pg);
+
+ _init_xen_l4_slots(l4t, l4mfn, d, l3_pg, sl4mfn, ro_mpt);
}
/*
@@ -6275,7 +6290,7 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
struct page_info **ppg)
{
struct domain *d = v->domain;
- struct page_info *pg;
+ struct page_info *pg, *l3_pg = perdomain_l3(v);
l3_pgentry_t *l3tab;
l2_pgentry_t *l2tab;
l1_pgentry_t *l1tab;
@@ -6284,12 +6299,12 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
ASSERT(va >= PERDOMAIN_VIRT_START &&
va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
- BUG_ON(!d->arch.perdomain_l3_pg);
+ BUG_ON(!l3_pg);
if ( !nr )
return 0;
- l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
+ l3tab = __map_domain_page(l3_pg);
ASSERT(!l3_table_offset(va ^ (va + nr * PAGE_SIZE - 1)));
@@ -6386,16 +6401,16 @@ void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr)
{
const l3_pgentry_t *l3tab, *pl3e;
- const struct domain *d = v->domain;
+ struct page_info *l3_pg = perdomain_l3(v);
ASSERT(va >= PERDOMAIN_VIRT_START &&
va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
ASSERT(!nr || !l3_table_offset(va ^ (va + nr * PAGE_SIZE - 1)));
- if ( !d->arch.perdomain_l3_pg )
+ if ( !l3_pg )
return;
- l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
+ l3tab = __map_domain_page(l3_pg);
pl3e = l3tab + l3_table_offset(va);
if ( l3e_get_flags(*pl3e) & _PAGE_PRESENT )
@@ -6434,16 +6449,11 @@ void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unmap_domain_page(l3tab);
}
-void free_perdomain_mappings(struct domain *d)
+static void free_perdomain_l3(struct page_info *l3pg)
{
- l3_pgentry_t *l3tab;
+ l3_pgentry_t *l3tab = __map_domain_page(l3pg);
unsigned int i;
- if ( !d->arch.perdomain_l3_pg )
- return;
-
- l3tab = __map_domain_page(d->arch.perdomain_l3_pg);
-
for ( i = 0; i < PERDOMAIN_SLOTS; ++i)
if ( l3e_get_flags(l3tab[i]) & _PAGE_PRESENT )
{
@@ -6479,8 +6489,29 @@ void free_perdomain_mappings(struct domain *d)
}
unmap_domain_page(l3tab);
- free_domheap_page(d->arch.perdomain_l3_pg);
+ free_domheap_page(l3pg);
+}
+
+void free_perdomain_mappings(struct domain *d)
+{
+ struct page_info *l3pg = d->arch.perdomain_l3_pg;
+
+ if ( !l3pg )
+ return;
+
d->arch.perdomain_l3_pg = NULL;
+ free_perdomain_l3(l3pg);
+}
+
+void free_pervcpu_mappings(struct vcpu *v)
+{
+ struct page_info *l3pg = v->arch.pervcpu_l3_pg;
+
+ if ( !l3pg )
+ return;
+
+ v->arch.pervcpu_l3_pg = NULL;
+ free_perdomain_l3(l3pg);
}
static void write_sss_token(unsigned long *ptr)
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 09/44] x86/domain_page: drop redundant is_pv_vcpu() checks
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (7 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 08/44] x86/hvm: introduce per-vCPU L3 page-table George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 10/44] x86/domain_page: prepare the mapcache for per-vCPU accounting George Dunlap
` (34 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
The is_pv_vcpu() checks after mapcache_current_vcpu() never fail: they
have been redundant since 622c9a5ba95d ("x86/mm: accurately track
which vCPU page-tables are loaded") made mapcache_current_vcpu() do
the check itself. mapcache_current_vcpu() is now guaranteed to return
a PV vCPU with a caching structure (which is what is needed for the
subsequent dcache->inuse check): either a scheduled PV vCPU, or this
CPU's idle vCPU (possibly after finishing a lazy context switch).
Drop the checks; their speculation barriers (evaluate_nospec()) go
too.
mapcache_current_vcpu() returns NULL if this CPU's pgtable_vcpu is not
yet set (early boot) or is not a PV vCPU. Otherwise it reads
pgtable_vcpu again and returns that. In between, pgtable_vcpu changes
only when a lazy context switch from a PV vCPU is completed, by
mapcache_current_vcpu() itself (sync_local_execstate()) or first by an
interrupt such as a FLUSH_VCPU_STATE IPI; completing it loads the idle
vCPU's page-tables and sets pgtable_vcpu to the idle vCPU. No other
writer of pgtable_vcpu runs in an interrupt that returns to the code it
interrupted. During an EFI runtime call pgtable_vcpu is the idle
vCPU: efi_rs_enter() completes any lazy switch, then loads EFI's
page-tables, which carry the directmap. So the vCPU returned is either
the PV vCPU checked or this CPU's idle vCPU.
The idle vCPU passes is_pv_vcpu(), as the idle domain is created
without XEN_DOMCTL_CDF_hvm, but its mapcache is never set up:
domain_create() returns for the idle domain before calling
arch_domain_create(), which is what calls mapcache_domain_init(). With
its dcache->inuse NULL, map_domain_page() uses the directmap, and
unmap_domain_page()'s ASSERT() on dcache->inuse still catches a
mapcache address unmapped with no mapcache in use. In builds without
CONFIG_PV, is_pv_vcpu() is always false and mapcache_current_vcpu()
always returns NULL.
No functional change.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- New in this version, split out of "x86/domain_page: prepare the
mapcache for per-vCPU accounting", with a justification of its own.
NB I've included Claude's complete analysis in the commit message.
I've reworded the commit message so that the middle two paragraphs can
be dropped, or reworded more briefly on commit.
---
xen/arch/x86/domain_page.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index f3e65e2d44..ed68483695 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -74,7 +74,7 @@ void *map_domain_page(mfn_t mfn)
#endif
v = mapcache_current_vcpu();
- if ( !v || !is_pv_vcpu(v) )
+ if ( !v )
return mfn_to_virt(mfn_x(mfn));
dcache = &v->domain->arch.pv.mapcache;
@@ -179,7 +179,7 @@ void unmap_domain_page(const void *ptr)
ASSERT(va >= MAPCACHE_VIRT_START && va < MAPCACHE_VIRT_END);
v = mapcache_current_vcpu();
- ASSERT(v && is_pv_vcpu(v));
+ ASSERT(v);
dcache = &v->domain->arch.pv.mapcache;
ASSERT(dcache->inuse);
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 10/44] x86/domain_page: prepare the mapcache for per-vCPU accounting
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (8 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 09/44] x86/domain_page: drop redundant is_pv_vcpu() checks George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 11/44] x86/domain_page: move struct mapcache_vcpu to struct arch_vcpu George Dunlap
` (33 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
Currently, only PV domains have a mapcache, which is a domain-wide one.
In preparation for enabling per-vCPU mapcaches for HVM domains, split
the accounting out into a struct mapcache, embedded in struct
mapcache_domain.
Introduce has_mapcache(), for whether a vCPU has a mapcache, and
vcpu_mapcache(), which returns the accounting for a vCPU's mapcache
and, in *dcache, the domain-wide mapcache whose lock and TLB epoch the
caller has to take into account. For now has_mapcache() is
is_pv_vcpu(), and vcpu_mapcache() always returns the domain's
accounting.
Move map_domain_page()'s handling of the domain-wide lock and TLB epoch
into three helpers: mapcache_enter() takes the lock and catches up with
a wrap another vCPU caused, mapcache_new_epoch() starts a new epoch
after a wrap, and mapcache_exit() drops the lock. A per-vCPU
mapcache can then skip them without map_domain_page() being
re-indented.
struct mapcache_domain keeps its size (40 bytes, x86_64 debug build),
and so do the structures containing it.
No functional change.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- Reword has_mapcache()'s comment (Roger).
- Move the comment on the domain-wide lock and TLB epoch into the
function body (Roger).
- Rename mapcache_lock() / mapcache_unlock() to mapcache_enter() /
mapcache_exit() (Roger).
- mapcache_new_epoch(): assert that the domain-wide lock is held
(Roger).
- mapcache_vcpu_init(): drop the comment on the early return for a
disabled domain-wide mapcache (Jan).
- mapcache_vcpu_init(): initialise ents and nr at their declarations,
as before (Jan).
- Dropping the redundant is_pv_vcpu() checks in map_domain_page() and
unmap_domain_page() is now a patch of its own (Jan).
Changes in v3:
- New in this version, split out of "x86/hvm: introduce per-vCPU
mapcache for vCPU-PT domains" for review.
---
xen/arch/x86/domain_page.c | 137 ++++++++++++++++++++----------
xen/arch/x86/include/asm/domain.h | 19 +++--
2 files changed, 102 insertions(+), 54 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index ed68483695..e4ea7a254f 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -18,17 +18,19 @@
#include <asm/hardirq.h>
#include <asm/setup.h>
+/* Whether the vCPU supports mapcache usage, might not be in use. */
+static bool has_mapcache(const struct vcpu *v)
+{
+ return is_pv_vcpu(v);
+}
+
static inline struct vcpu *mapcache_current_vcpu(void)
{
const struct vcpu *v = this_cpu(pgtable_vcpu);
const struct vcpu *curr = current;
- /*
- * During early boot pgtable_vcpu is not set, callers must handle NULL.
- * Non-PV domains don't have a mapcache, the directmap covers all physical
- * address space.
- */
- if ( !v || !is_pv_vcpu(v) )
+ /* During early boot pgtable_vcpu is not set, callers must handle NULL. */
+ if ( !v || !has_mapcache(v) )
return NULL;
/*
@@ -54,6 +56,52 @@ static inline struct vcpu *mapcache_current_vcpu(void)
return ACCESS_ONCE(this_cpu(pgtable_vcpu));
}
+/*
+ * The accounting for v's mapcache: the domain's, with *dcache set so the
+ * caller takes the domain-wide lock and TLB-epoch machinery into account.
+ */
+static struct mapcache *vcpu_mapcache(struct vcpu *v,
+ struct mapcache_domain **dcache)
+{
+ struct domain *d = v->domain;
+
+ *dcache = &d->arch.pv.mapcache;
+ return &(*dcache)->cache;
+}
+
+static void mapcache_enter(struct mapcache_domain *dcache,
+ struct mapcache_vcpu *vcache)
+{
+ /* The domain's vCPUs share the mapcache's slots. */
+ spin_lock(&dcache->lock);
+
+ /* Has some other CPU caused a wrap? We must flush if so. */
+ if ( unlikely(dcache->epoch != vcache->shadow_epoch) )
+ {
+ vcache->shadow_epoch = dcache->epoch;
+ if ( NEED_FLUSH(this_cpu(tlbflush_time), dcache->tlbflush_timestamp) )
+ {
+ perfc_incr(domain_page_tlb_flush);
+ flush_tlb_local();
+ }
+ }
+}
+
+/* A wrap has just flushed this CPU's TLB: start a new epoch. */
+static void mapcache_new_epoch(struct mapcache_domain *dcache,
+ struct mapcache_vcpu *vcache)
+{
+ ASSERT(spin_is_locked(&dcache->lock));
+
+ vcache->shadow_epoch = ++dcache->epoch;
+ dcache->tlbflush_timestamp = tlbflush_current_time();
+}
+
+static void mapcache_exit(struct mapcache_domain *dcache)
+{
+ spin_unlock(&dcache->lock);
+}
+
#define mapcache_l2_entry(e) ((e) >> PAGETABLE_ORDER)
#define MAPCACHE_L2_ENTRIES (mapcache_l2_entry(MAPCACHE_ENTRIES - 1) + 1)
#define MAPCACHE_L1ENT(idx) \
@@ -66,6 +114,7 @@ void *map_domain_page(mfn_t mfn)
struct vcpu *v;
struct mapcache_domain *dcache;
struct mapcache_vcpu *vcache;
+ struct mapcache *cache;
struct vcpu_maphash_entry *hashent;
#ifdef NDEBUG
@@ -77,9 +126,9 @@ void *map_domain_page(mfn_t mfn)
if ( !v )
return mfn_to_virt(mfn_x(mfn));
- dcache = &v->domain->arch.pv.mapcache;
+ cache = vcpu_mapcache(v, &dcache);
vcache = &v->arch.pv.mapcache;
- if ( !dcache->inuse )
+ if ( !cache->inuse )
return mfn_to_virt(mfn_x(mfn));
perfc_incr(map_domain_page_count);
@@ -90,41 +139,30 @@ void *map_domain_page(mfn_t mfn)
if ( hashent->mfn == mfn_x(mfn) )
{
idx = hashent->idx;
- ASSERT(idx < dcache->entries);
+ ASSERT(idx < cache->entries);
hashent->refcnt++;
ASSERT(hashent->refcnt);
ASSERT(mfn_eq(l1e_get_mfn(MAPCACHE_L1ENT(idx)), mfn));
goto out;
}
- spin_lock(&dcache->lock);
+ mapcache_enter(dcache, vcache);
- /* Has some other CPU caused a wrap? We must flush if so. */
- if ( unlikely(dcache->epoch != vcache->shadow_epoch) )
- {
- vcache->shadow_epoch = dcache->epoch;
- if ( NEED_FLUSH(this_cpu(tlbflush_time), dcache->tlbflush_timestamp) )
- {
- perfc_incr(domain_page_tlb_flush);
- flush_tlb_local();
- }
- }
-
- idx = find_next_zero_bit(dcache->inuse, dcache->entries, dcache->cursor);
- if ( unlikely(idx >= dcache->entries) )
+ idx = find_next_zero_bit(cache->inuse, cache->entries, cache->cursor);
+ if ( unlikely(idx >= cache->entries) )
{
unsigned long accum = 0, prev = 0;
/* /First/, clean the garbage map and update the inuse list. */
- for ( i = 0; i < BITS_TO_LONGS(dcache->entries); i++ )
+ for ( i = 0; i < BITS_TO_LONGS(cache->entries); i++ )
{
accum |= prev;
- dcache->inuse[i] &= ~xchg(&dcache->garbage[i], 0);
- prev = ~dcache->inuse[i];
+ cache->inuse[i] &= ~xchg(&cache->garbage[i], 0);
+ prev = ~cache->inuse[i];
}
- if ( accum | (prev & BITMAP_LAST_WORD_MASK(dcache->entries)) )
- idx = find_first_zero_bit(dcache->inuse, dcache->entries);
+ if ( accum | (prev & BITMAP_LAST_WORD_MASK(cache->entries)) )
+ idx = find_first_zero_bit(cache->inuse, cache->entries);
else
{
/* Replace a hash entry instead. */
@@ -144,19 +182,18 @@ void *map_domain_page(mfn_t mfn)
i = 0;
} while ( i != MAPHASH_HASHFN(mfn_x(mfn)) );
}
- BUG_ON(idx >= dcache->entries);
+ BUG_ON(idx >= cache->entries);
/* /Second/, flush TLBs. */
perfc_incr(domain_page_tlb_flush);
flush_tlb_local();
- vcache->shadow_epoch = ++dcache->epoch;
- dcache->tlbflush_timestamp = tlbflush_current_time();
+ mapcache_new_epoch(dcache, vcache);
}
- set_bit(idx, dcache->inuse);
- dcache->cursor = idx + 1;
+ set_bit(idx, cache->inuse);
+ cache->cursor = idx + 1;
- spin_unlock(&dcache->lock);
+ mapcache_exit(dcache);
l1e_write(&MAPCACHE_L1ENT(idx), l1e_from_mfn(mfn, __PAGE_HYPERVISOR_RW));
@@ -170,6 +207,7 @@ void unmap_domain_page(const void *ptr)
unsigned int idx;
struct vcpu *v;
struct mapcache_domain *dcache;
+ struct mapcache *cache;
unsigned long va = (unsigned long)ptr, mfn, flags;
struct vcpu_maphash_entry *hashent;
@@ -181,8 +219,8 @@ void unmap_domain_page(const void *ptr)
v = mapcache_current_vcpu();
ASSERT(v);
- dcache = &v->domain->arch.pv.mapcache;
- ASSERT(dcache->inuse);
+ cache = vcpu_mapcache(v, &dcache);
+ ASSERT(cache->inuse);
idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
mfn = l1e_get_pfn(MAPCACHE_L1ENT(idx));
@@ -205,7 +243,7 @@ void unmap_domain_page(const void *ptr)
hashent->mfn);
l1e_write(&MAPCACHE_L1ENT(hashent->idx), l1e_empty());
/* /Second/, mark as garbage. */
- set_bit(hashent->idx, dcache->garbage);
+ set_bit(hashent->idx, cache->garbage);
}
/* Add newly-freed mapping to the maphash. */
@@ -217,7 +255,7 @@ void unmap_domain_page(const void *ptr)
/* /First/, zap the PTE. */
l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
/* /Second/, mark as garbage. */
- set_bit(idx, dcache->garbage);
+ set_bit(idx, cache->garbage);
}
local_irq_restore(flags);
@@ -239,9 +277,9 @@ void mapcache_domain_init(struct domain *d)
2 * PFN_UP(BITS_TO_LONGS(MAPCACHE_ENTRIES) * sizeof(long))) >
MAPCACHE_VIRT_START + (PERDOMAIN_SLOT_MBYTES << 20));
bitmap_pages = PFN_UP(BITS_TO_LONGS(MAPCACHE_ENTRIES) * sizeof(long));
- dcache->inuse = (void *)MAPCACHE_VIRT_END + PAGE_SIZE;
- dcache->garbage = dcache->inuse +
- (bitmap_pages + 1) * PAGE_SIZE / sizeof(long);
+ dcache->cache.inuse = (void *)MAPCACHE_VIRT_END + PAGE_SIZE;
+ dcache->cache.garbage = dcache->cache.inuse +
+ (bitmap_pages + 1) * PAGE_SIZE / sizeof(long);
spin_lock_init(&dcache->lock);
}
@@ -249,15 +287,20 @@ void mapcache_domain_init(struct domain *d)
int mapcache_vcpu_init(struct vcpu *v)
{
struct domain *d = v->domain;
- struct mapcache_domain *dcache = &d->arch.pv.mapcache;
+ struct mapcache_domain *dcache;
+ struct mapcache *cache;
unsigned long i;
unsigned int ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
unsigned int nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
- if ( !is_pv_vcpu(v) || !dcache->inuse )
+ if ( !has_mapcache(v) )
+ return 0;
+
+ cache = vcpu_mapcache(v, &dcache);
+ if ( !cache->inuse )
return 0;
- if ( ents > dcache->entries )
+ if ( ents > cache->entries )
{
/* Populate page tables. */
int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START, ents,
@@ -265,16 +308,16 @@ int mapcache_vcpu_init(struct vcpu *v)
/* Populate bit maps. */
if ( !rc )
- rc = create_perdomain_mapping(v, (unsigned long)dcache->inuse,
+ rc = create_perdomain_mapping(v, (unsigned long)cache->inuse,
nr, NULL, NIL(struct page_info *));
if ( !rc )
- rc = create_perdomain_mapping(v, (unsigned long)dcache->garbage,
+ rc = create_perdomain_mapping(v, (unsigned long)cache->garbage,
nr, NULL, NIL(struct page_info *));
if ( rc )
return rc;
- dcache->entries = ents;
+ cache->entries = ents;
}
/* Mark all maphash entries as not in use. */
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 4140a92f15..542c6b5eca 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -57,6 +57,16 @@ struct trap_bounce {
unsigned long eip;
};
+struct mapcache {
+ /* The number of array entries, and a cursor into the array. */
+ unsigned int entries;
+ unsigned int cursor;
+
+ /* Which mappings are in use, and which are garbage to reap next epoch? */
+ unsigned long *inuse;
+ unsigned long *garbage;
+};
+
#define MAPHASH_ENTRIES 8
#define MAPHASH_HASHFN(pfn) ((pfn) & (MAPHASH_ENTRIES-1))
#define MAPHASHENT_NOTINUSE ((u32)~0U)
@@ -73,10 +83,6 @@ struct mapcache_vcpu {
};
struct mapcache_domain {
- /* The number of array entries, and a cursor into the array. */
- unsigned int entries;
- unsigned int cursor;
-
/* Protects map_domain_page(). */
spinlock_t lock;
@@ -84,9 +90,8 @@ struct mapcache_domain {
unsigned int epoch;
u32 tlbflush_timestamp;
- /* Which mappings are in use, and which are garbage to reap next epoch? */
- unsigned long *inuse;
- unsigned long *garbage;
+ /* Accounting of the domain-wide mapcache. */
+ struct mapcache cache;
};
void mapcache_domain_init(struct domain *d);
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 11/44] x86/domain_page: move struct mapcache_vcpu to struct arch_vcpu
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (9 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 10/44] x86/domain_page: prepare the mapcache for per-vCPU accounting George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 12/44] x86/domain_page: don't unmap a mapcache entry that is not present George Dunlap
` (32 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Roger Pau Monné, Jan Beulich,
Andrew Cooper, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
struct mapcache_vcpu, the per-vCPU part of the mapcache (the maphash
and the vCPU's shadow of the domain's TLB epoch), lives in struct
pv_vcpu, as only PV vCPUs have a mapcache. In preparation for
implementing mapcaches for HVM domains, move it to struct arch_vcpu:
v->arch.pv.mapcache becomes v->arch.mapcache.
struct pv_vcpu shrinks from 512 to 376 bytes, and struct vcpu grows
from 2880 to 3008 bytes (x86_64, debug build), within its page: from
here on HVM vCPUs carry the 136-byte struct mapcache_vcpu too, ahead of
its use by those of vCPU-PT domains.
No functional change.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Roger Pau Monné <roger@xenproject.org>
---
Changes in v3:
- New in this version, split out of "x86/hvm: introduce per-vCPU
mapcache for vCPU-PT domains" for review.
---
xen/arch/x86/domain_page.c | 6 +++---
xen/arch/x86/include/asm/domain.h | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index e4ea7a254f..205d7b832a 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -127,7 +127,7 @@ void *map_domain_page(mfn_t mfn)
return mfn_to_virt(mfn_x(mfn));
cache = vcpu_mapcache(v, &dcache);
- vcache = &v->arch.pv.mapcache;
+ vcache = &v->arch.mapcache;
if ( !cache->inuse )
return mfn_to_virt(mfn_x(mfn));
@@ -224,7 +224,7 @@ void unmap_domain_page(const void *ptr)
idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
mfn = l1e_get_pfn(MAPCACHE_L1ENT(idx));
- hashent = &v->arch.pv.mapcache.hash[MAPHASH_HASHFN(mfn)];
+ hashent = &v->arch.mapcache.hash[MAPHASH_HASHFN(mfn)];
local_irq_save(flags);
@@ -324,7 +324,7 @@ int mapcache_vcpu_init(struct vcpu *v)
BUILD_BUG_ON(MAPHASHENT_NOTINUSE < MAPCACHE_ENTRIES);
for ( i = 0; i < MAPHASH_ENTRIES; i++ )
{
- struct vcpu_maphash_entry *hashent = &v->arch.pv.mapcache.hash[i];
+ struct vcpu_maphash_entry *hashent = &v->arch.mapcache.hash[i];
hashent->mfn = ~0UL; /* never valid to map */
hashent->idx = MAPHASHENT_NOTINUSE;
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 542c6b5eca..45d308407e 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -545,9 +545,6 @@ struct arch_domain
struct pv_vcpu
{
- /* map_domain_page() mapping cache. */
- struct mapcache_vcpu mapcache;
-
unsigned int vgc_flags;
struct trap_info *trap_ctxt;
@@ -643,6 +640,9 @@ struct arch_vcpu
#define async_exception_state(t) async_exception_state[(t)-1]
uint8_t async_exception_mask;
+ /* map_domain_page() mapping cache. */
+ struct mapcache_vcpu mapcache;
+
/* Virtual Machine Extensions */
union {
struct pv_vcpu pv;
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 12/44] x86/domain_page: don't unmap a mapcache entry that is not present
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (10 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 11/44] x86/domain_page: move struct mapcache_vcpu to struct arch_vcpu George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 13/44] x86/hvm: introduce per-vCPU mapcache for vCPU-PT domains George Dunlap
` (31 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper, Teddy Astie
Mappings must be torn down by the vCPU that created them, and only
once. Have unmap_domain_page() check that the entry it is handed is
present and bail out otherwise (ASSERT_UNREACHABLE()), rather than read
an MFN out of an empty PTE and act on it, as a second unmap of an entry
the first one zapped would. (A second unmap of an entry the maphash
kept is caught, as before, only by the refcount ASSERT() in debug
builds.)
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Roger Pau Monné <roger@xenproject.org>
---
Changes in v4:
- Reword the comment on the new check (Roger).
Changes in v3:
- New in this version, split out of "x86/hvm: introduce per-vCPU
mapcache for vCPU-PT domains" for review.
---
xen/arch/x86/domain_page.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 205d7b832a..47ccefa4e9 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -223,6 +223,15 @@ void unmap_domain_page(const void *ptr)
ASSERT(cache->inuse);
idx = PFN_DOWN(va - MAPCACHE_VIRT_START);
+ /*
+ * Sanity check: an entry that isn't present means a double or stray
+ * unmap. Play it safe: don't act on the MFN in an empty PTE.
+ */
+ if ( !(l1e_get_flags(MAPCACHE_L1ENT(idx)) & _PAGE_PRESENT) )
+ {
+ ASSERT_UNREACHABLE();
+ return;
+ }
mfn = l1e_get_pfn(MAPCACHE_L1ENT(idx));
hashent = &v->arch.mapcache.hash[MAPHASH_HASHFN(mfn)];
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 13/44] x86/hvm: introduce per-vCPU mapcache for vCPU-PT domains
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (11 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 12/44] x86/domain_page: don't unmap a mapcache entry that is not present George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 14/44] x86/mm: let Xen's page-table walker operate on a given root George Dunlap
` (30 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
Add a struct mapcache to struct mapcache_vcpu for the per-vCPU
accounting, with MAPCACHE_PERVCPU_ENTRIES (64) slots. Its inuse and
garbage bitmaps are then an unsigned long each: declare them inline
in struct mapcache_vcpu, rather than mapping them in the per-domain
area as the domain-wide bitmaps are.
A per-vCPU mapcache is only ever used by its own vCPU, and so by one
pCPU at a time. We can therefore skip both the lock and the TLB-epoch
tracking the domain-wide mapcache needs; a wrap still flushes the
local TLB.
struct mapcache_vcpu grows from 136 to 176 bytes, and struct vcpu from
3008 to 3072 bytes (x86_64, debug build), within its page.
Unlike a domain-wide mapcache, whose one pool all the domain's vCPUs
draw on, a per-vCPU one cannot, by design, use more than its own
slots: MAPCACHE_PERVCPU_ENTRIES (64) is the most single-page
mappings such a vCPU can hold at once, interrupt handlers' included,
and one more hits the BUG_ON() in map_domain_page(). The PoD
zero-page sweep held up to 16 on top of its callers' until
41ddcfe808ec ("x86/PoD: map one page at a time in
p2m_pod_zero_check()"). A domain-wide
mapcache is sized at MAPCACHE_VCPU_ENTRIES (16) slots per vCPU, in
one shared pool; a per-vCPU one gets 64, for a wider margin. The
extra slots take no memory: the area is a whole L1 page whatever the
number of slots, and each bitmap is still one unsigned long. Wraps,
each of which flushes the local TLB, also come less often.
mapcache_current_vcpu()'s lazy-context-switch handling
(sync_local_execstate() plus a pgtable_vcpu re-read) now applies to
any vCPU with a mapcache: when the idle vCPU runs lazily on an HVM
vCPU's monitor table, the full switch happens before the idle context
uses a mapcache.
In release builds map_domain_page() still tries the directmap first,
for any vCPU; debug builds exercise the new mapcache.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v6:
- Message: "x86/PoD: map one page at a time in p2m_pod_zero_check()" is
in staging now, so cite it by its hash.
Changes in v5:
- Message: a domain-wide mapcache is one pool that all the domain's
vCPUs draw on, not per-vCPU slots another vCPU could borrow.
Changes in v4:
- d->arch.vcpu_pt is now d->arch.asi.vcpu_pt.
- mapcache_lock() / mapcache_unlock() are now mapcache_enter() /
mapcache_exit(). has_mapcache()'s comment no longer changes, and the
early return in mapcache_enter() has a one-line comment instead of a
sentence added to the block (Roger).
- mapcache_vcpu_init(): ents and nr become assignments here, ents now
depending on the kind of mapcache (Jan).
- A per-vCPU mapcache gets 64 slots (MAPCACHE_PERVCPU_ENTRIES) in
place of MAPCACHE_VCPU_ENTRIES (16), which still sizes the
domain-wide mapcache; its bitmaps, now full, are described as an
unsigned long each (Roger).
- Move the comment on why a per-vCPU mapcache needs only a local flush
from map_domain_page()'s TLB flush to mapcache_new_epoch()'s early
return, as one line (Roger).
Changes in v3:
- HVM only, and per-vCPU from the start: the HVM enablement of "x86:
Initialize the mapcache for PV, HVM, and idle domains" (from the
directmap removal series) is folded in, without the idle domain and
without a domain-wide HVM mapcache.
- Retitle (was: "x86/mm: introduce a per-vCPU mapcache when using ASI").
- Say which second unmaps the not-present check in unmap_domain_page()
catches: those of an entry the first unmap zapped.
- State the budget of single-page mappings a vCPU can hold at once.
- The mapcache accounting refactor, the move of struct mapcache_vcpu
to struct arch_vcpu and the not-present check in unmap_domain_page()
are split out, for review, into the three patches before this one.
---
xen/arch/x86/domain_page.c | 61 +++++++++++++++++++++++--------
xen/arch/x86/include/asm/config.h | 2 +
xen/arch/x86/include/asm/domain.h | 6 +++
3 files changed, 53 insertions(+), 16 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 47ccefa4e9..770fee80a4 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -21,7 +21,7 @@
/* Whether the vCPU supports mapcache usage, might not be in use. */
static bool has_mapcache(const struct vcpu *v)
{
- return is_pv_vcpu(v);
+ return is_pv_vcpu(v) || v->domain->arch.asi.vcpu_pt;
}
static inline struct vcpu *mapcache_current_vcpu(void)
@@ -34,10 +34,10 @@ static inline struct vcpu *mapcache_current_vcpu(void)
return NULL;
/*
- * If we are in a lazy context-switch state from a PV vCPU do a full switch
- * to the idle vCPU now, otherwise an incoming FLUSH_VCPU_STATE IPI would
- * change the page tables under our feet an invalidate any in-use mapcache
- * entries.
+ * If we are in a lazy context-switch state from a vCPU with a mapcache do
+ * a full switch to the idle vCPU now, otherwise an incoming
+ * FLUSH_VCPU_STATE IPI would change the page tables under our feet and
+ * invalidate any in-use mapcache entries.
*/
if ( unlikely(this_cpu(curr_vcpu) != curr) )
{
@@ -57,14 +57,21 @@ static inline struct vcpu *mapcache_current_vcpu(void)
}
/*
- * The accounting for v's mapcache: the domain's, with *dcache set so the
- * caller takes the domain-wide lock and TLB-epoch machinery into account.
+ * The accounting for v's mapcache: its own for a vCPU-PT domain, the domain's
+ * otherwise, in which case *dcache is set so the caller takes the domain-wide
+ * lock and TLB-epoch machinery into account.
*/
static struct mapcache *vcpu_mapcache(struct vcpu *v,
struct mapcache_domain **dcache)
{
struct domain *d = v->domain;
+ if ( d->arch.asi.vcpu_pt )
+ {
+ *dcache = NULL;
+ return &v->arch.mapcache.cache;
+ }
+
*dcache = &d->arch.pv.mapcache;
return &(*dcache)->cache;
}
@@ -72,6 +79,10 @@ static struct mapcache *vcpu_mapcache(struct vcpu *v,
static void mapcache_enter(struct mapcache_domain *dcache,
struct mapcache_vcpu *vcache)
{
+ /* Only its own vCPU uses a per-vCPU mapcache: no lock or epoch needed. */
+ if ( !dcache )
+ return;
+
/* The domain's vCPUs share the mapcache's slots. */
spin_lock(&dcache->lock);
@@ -91,6 +102,10 @@ static void mapcache_enter(struct mapcache_domain *dcache,
static void mapcache_new_epoch(struct mapcache_domain *dcache,
struct mapcache_vcpu *vcache)
{
+ /* Per-vCPU: no epoch, as page-table loads flush its non-global entries. */
+ if ( !dcache )
+ return;
+
ASSERT(spin_is_locked(&dcache->lock));
vcache->shadow_epoch = ++dcache->epoch;
@@ -99,7 +114,8 @@ static void mapcache_new_epoch(struct mapcache_domain *dcache,
static void mapcache_exit(struct mapcache_domain *dcache)
{
- spin_unlock(&dcache->lock);
+ if ( dcache )
+ spin_unlock(&dcache->lock);
}
#define mapcache_l2_entry(e) ((e) >> PAGETABLE_ORDER)
@@ -299,15 +315,25 @@ int mapcache_vcpu_init(struct vcpu *v)
struct mapcache_domain *dcache;
struct mapcache *cache;
unsigned long i;
- unsigned int ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
- unsigned int nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
+ unsigned int ents, nr;
if ( !has_mapcache(v) )
return 0;
cache = vcpu_mapcache(v, &dcache);
- if ( !cache->inuse )
+ if ( !dcache )
+ {
+ /* Per-vCPU bitmaps are an unsigned long each, in the vCPU itself. */
+ cache->inuse = v->arch.mapcache.inuse;
+ cache->garbage = v->arch.mapcache.garbage;
+ ents = MAPCACHE_PERVCPU_ENTRIES;
+ }
+ else if ( !cache->inuse )
return 0;
+ else
+ ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
+
+ nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
if ( ents > cache->entries )
{
@@ -315,13 +341,16 @@ int mapcache_vcpu_init(struct vcpu *v)
int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START, ents,
NIL(l1_pgentry_t *), NULL);
- /* Populate bit maps. */
- if ( !rc )
+ /* Populate the domain-wide bit maps (kept in the per-domain area). */
+ if ( !rc && dcache )
+ {
rc = create_perdomain_mapping(v, (unsigned long)cache->inuse,
nr, NULL, NIL(struct page_info *));
- if ( !rc )
- rc = create_perdomain_mapping(v, (unsigned long)cache->garbage,
- nr, NULL, NIL(struct page_info *));
+ if ( !rc )
+ rc = create_perdomain_mapping(v, (unsigned long)cache->garbage,
+ nr, NULL,
+ NIL(struct page_info *));
+ }
if ( rc )
return rc;
diff --git a/xen/arch/x86/include/asm/config.h b/xen/arch/x86/include/asm/config.h
index aebd3e8f45..ee84c57432 100644
--- a/xen/arch/x86/include/asm/config.h
+++ b/xen/arch/x86/include/asm/config.h
@@ -279,6 +279,8 @@ extern unsigned long xen_phys_start;
/* map_domain_page() map cache. The second per-domain-mapping sub-area. */
#define MAPCACHE_VCPU_ENTRIES (CONFIG_PAGING_LEVELS * CONFIG_PAGING_LEVELS)
+/* Single-page slots of a per-vCPU mapcache (vCPU-PT domains). */
+#define MAPCACHE_PERVCPU_ENTRIES 64
#define MAPCACHE_ENTRIES (MAX_VIRT_CPUS * MAPCACHE_VCPU_ENTRIES)
#define MAPCACHE_VIRT_START PERDOMAIN_VIRT_SLOT(1)
#define MAPCACHE_VIRT_END (MAPCACHE_VIRT_START + \
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 45d308407e..7076e38848 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -80,6 +80,12 @@ struct mapcache_vcpu {
uint32_t idx;
uint32_t refcnt;
} hash[MAPHASH_ENTRIES];
+
+ /* Accounting of a per-vCPU mapcache (vCPU-PT domains). */
+ struct mapcache cache;
+ /* Backing for cache.{inuse,garbage}. */
+ DECLARE_BITMAP(inuse, MAPCACHE_PERVCPU_ENTRIES);
+ DECLARE_BITMAP(garbage, MAPCACHE_PERVCPU_ENTRIES);
};
struct mapcache_domain {
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 14/44] x86/mm: let Xen's page-table walker operate on a given root
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (12 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 13/44] x86/hvm: introduce per-vCPU mapcache for vCPU-PT domains George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 15/44] xen/gunzip: process pending softirqs once per deflate block George Dunlap
` (29 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
map_pages_to_xen() and modify_xen_mappings() maintain Xen's own
mappings, walking from idle_pg_table, whose Xen slots every other root
page-table copies. A later patch introduces a second, sparse directmap
hierarchy: page tables for the directmap virtual address range that some
guest contexts use in place of idle_pg_table's. It needs the same
maintenance at the same virtual addresses.
Give virt_to_xen_l{3,2,1}e() the L4 to walk from, and move the bodies of
map_pages_to_xen() and modify_xen_mappings() into map_pages_in() and
modify_mappings_in(), which take the root as well. The public functions
pass idle_pg_table. The EFI runtime-services page-tables mirror
idle_pg_table's L4 entries only, so a newly allocated L3 is propagated
to them only when the root is idle_pg_table.
map_pages_in() and modify_mappings_in() free page-table pages through
free_pagetable_in(), which takes the root too and for now only calls
free_xen_pagetable().
TLB flushing is unaffected: flushes are by virtual address and go to
every online CPU, which covers both hierarchies.
No functional change.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- Free page-table pages through free_pagetable_in(), for now a plain
wrapper, split out of "x86/mm: build and maintain a sparse view of the
directmap".
Changes in v4:
- Message: name the second hierarchy as sparse; flushes reach every CPU
(Roger).
Changes in v3:
- New in this version.
---
xen/arch/x86/mm.c | 78 ++++++++++++++++++++++++++++++++++-------------
1 file changed, 56 insertions(+), 22 deletions(-)
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index f52c72a605..39c6fa3d56 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -5247,13 +5247,14 @@ static DEFINE_SPINLOCK(map_pgdir_lock);
/*
* For virt_to_xen_lXe() functions, they take a linear address and return a
- * pointer to Xen's LX entry. Caller needs to unmap the pointer.
+ * pointer to Xen's LX entry in the page-table hierarchy rooted at @root
+ * (idle_pg_table for Xen's own mappings). Caller needs to unmap the pointer.
*/
-static l3_pgentry_t *virt_to_xen_l3e(unsigned long v)
+static l3_pgentry_t *virt_to_xen_l3e(l4_pgentry_t *root, unsigned long v)
{
l4_pgentry_t *pl4e;
- pl4e = &idle_pg_table[l4_table_offset(v)];
+ pl4e = &root[l4_table_offset(v)];
if ( !(l4e_get_flags(*pl4e) & _PAGE_PRESENT) )
{
bool locking = system_state > SYS_STATE_boot;
@@ -5269,7 +5270,8 @@ static l3_pgentry_t *virt_to_xen_l3e(unsigned long v)
l4_pgentry_t l4e = l4e_from_mfn(l3mfn, __PAGE_HYPERVISOR);
l4e_write(pl4e, l4e);
- efi_update_l4_pgtable(l4_table_offset(v), l4e);
+ if ( root == idle_pg_table )
+ efi_update_l4_pgtable(l4_table_offset(v), l4e);
l3mfn = INVALID_MFN;
}
if ( locking )
@@ -5280,11 +5282,11 @@ static l3_pgentry_t *virt_to_xen_l3e(unsigned long v)
return map_l3t_from_l4e(*pl4e) + l3_table_offset(v);
}
-static l2_pgentry_t *virt_to_xen_l2e(unsigned long v)
+static l2_pgentry_t *virt_to_xen_l2e(l4_pgentry_t *root, unsigned long v)
{
l3_pgentry_t *pl3e, l3e;
- pl3e = virt_to_xen_l3e(v);
+ pl3e = virt_to_xen_l3e(root, v);
if ( !pl3e )
return NULL;
@@ -5318,11 +5320,11 @@ static l2_pgentry_t *virt_to_xen_l2e(unsigned long v)
return map_l2t_from_l3e(l3e) + l2_table_offset(v);
}
-static l1_pgentry_t *virt_to_xen_l1e(unsigned long v)
+static l1_pgentry_t *virt_to_xen_l1e(l4_pgentry_t *root, unsigned long v)
{
l2_pgentry_t *pl2e, l2e;
- pl2e = virt_to_xen_l2e(v);
+ pl2e = virt_to_xen_l2e(root, v);
if ( !pl2e )
return NULL;
@@ -5398,7 +5400,7 @@ mfn_t xen_map_to_mfn(unsigned long va)
bool locking = system_state > SYS_STATE_boot;
unsigned int l2_offset = l2_table_offset(va);
unsigned int l1_offset = l1_table_offset(va);
- const l3_pgentry_t *pl3e = virt_to_xen_l3e(va);
+ const l3_pgentry_t *pl3e = virt_to_xen_l3e(idle_pg_table, va);
const l2_pgentry_t *pl2e = NULL;
const l1_pgentry_t *pl1e = NULL;
struct page_info *l3page;
@@ -5438,7 +5440,21 @@ mfn_t xen_map_to_mfn(unsigned long va)
return ret;
}
-int map_pages_to_xen(
+/*
+ * Free a page-table page that the hierarchy rooted at @root has stopped
+ * using.
+ */
+static void free_pagetable_in(const l4_pgentry_t *root, mfn_t mfn)
+{
+ free_xen_pagetable(mfn);
+}
+
+/*
+ * map_pages_to_xen() in the hierarchy rooted at @root: idle_pg_table's, or
+ * another hierarchy sharing its virtual addresses.
+ */
+static int map_pages_in(
+ l4_pgentry_t *root,
unsigned long virt,
mfn_t mfn,
unsigned long nr_mfns,
@@ -5489,7 +5505,7 @@ int map_pages_to_xen(
UNMAP_DOMAIN_PAGE(pl3e);
UNMAP_DOMAIN_PAGE(pl2e);
- pl3e = virt_to_xen_l3e(virt);
+ pl3e = virt_to_xen_l3e(root, virt);
if ( !pl3e )
goto out;
@@ -5542,10 +5558,10 @@ int map_pages_to_xen(
ol2e = l2t[i];
if ( (l2e_get_flags(ol2e) & _PAGE_PRESENT) &&
!(l2e_get_flags(ol2e) & _PAGE_PSE) )
- free_xen_pagetable(l2e_get_mfn(ol2e));
+ free_pagetable_in(root, l2e_get_mfn(ol2e));
}
unmap_domain_page(l2t);
- free_xen_pagetable(l3e_get_mfn(ol3e));
+ free_pagetable_in(root, l3e_get_mfn(ol3e));
}
}
@@ -5613,7 +5629,7 @@ int map_pages_to_xen(
free_xen_pagetable(l2mfn);
}
- pl2e = virt_to_xen_l2e(virt);
+ pl2e = virt_to_xen_l2e(root, virt);
if ( !pl2e )
goto out;
@@ -5643,7 +5659,7 @@ int map_pages_to_xen(
flush_flags(l1e_get_flags(l1t[i]));
flush_area(virt, flush_flags);
unmap_domain_page(l1t);
- free_xen_pagetable(l2e_get_mfn(ol2e));
+ free_pagetable_in(root, l2e_get_mfn(ol2e));
}
}
@@ -5658,7 +5674,7 @@ int map_pages_to_xen(
/* Normal page mapping. */
if ( !(l2e_get_flags(*pl2e) & _PAGE_PRESENT) )
{
- pl1e = virt_to_xen_l1e(virt);
+ pl1e = virt_to_xen_l1e(root, virt);
if ( pl1e == NULL )
goto out;
}
@@ -5775,7 +5791,7 @@ int map_pages_to_xen(
flush_area(virt - PAGE_SIZE,
FLUSH_TLB_GLOBAL |
FLUSH_ORDER(PAGETABLE_ORDER));
- free_xen_pagetable(l2e_get_mfn(ol2e));
+ free_pagetable_in(root, l2e_get_mfn(ol2e));
}
else if ( locking )
spin_unlock(&map_pgdir_lock);
@@ -5822,7 +5838,7 @@ int map_pages_to_xen(
flush_area(virt - PAGE_SIZE,
FLUSH_TLB_GLOBAL |
FLUSH_ORDER(2*PAGETABLE_ORDER));
- free_xen_pagetable(l3e_get_mfn(ol3e));
+ free_pagetable_in(root, l3e_get_mfn(ol3e));
}
else if ( locking )
spin_unlock(&map_pgdir_lock);
@@ -5843,6 +5859,15 @@ int map_pages_to_xen(
return rc;
}
+int map_pages_to_xen(
+ unsigned long virt,
+ mfn_t mfn,
+ unsigned long nr_mfns,
+ pte_attr_t flags)
+{
+ return map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
+}
+
int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
{
return map_pages_to_xen(virt, INVALID_MFN, nr_mfns, MAP_SMALL_PAGES);
@@ -5859,8 +5884,12 @@ int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
* flags are ignored.
*
* It is an error to call with present flags over an unpopulated range.
+ *
+ * modify_mappings_in() does so in the hierarchy rooted at @root, as
+ * map_pages_in() does.
*/
-int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
+static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
+ unsigned long e, pte_attr_t nf)
{
bool locking = system_state > SYS_STATE_boot;
l3_pgentry_t *pl3e = NULL;
@@ -5887,7 +5916,7 @@ int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
UNMAP_DOMAIN_PAGE(pl2e);
UNMAP_DOMAIN_PAGE(pl3e);
- pl3e = virt_to_xen_l3e(v);
+ pl3e = virt_to_xen_l3e(root, v);
if ( !pl3e )
goto out;
@@ -6070,7 +6099,7 @@ int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
if ( locking )
spin_unlock(&map_pgdir_lock);
flush_area(NULL, FLUSH_TLB_GLOBAL); /* flush before free */
- free_xen_pagetable(l1mfn);
+ free_pagetable_in(root, l1mfn);
}
else if ( locking )
spin_unlock(&map_pgdir_lock);
@@ -6114,7 +6143,7 @@ int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
if ( locking )
spin_unlock(&map_pgdir_lock);
flush_area(NULL, FLUSH_TLB_GLOBAL); /* flush before free */
- free_xen_pagetable(l2mfn);
+ free_pagetable_in(root, l2mfn);
}
else if ( locking )
spin_unlock(&map_pgdir_lock);
@@ -6138,6 +6167,11 @@ int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
#undef flush_area
+int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
+{
+ return modify_mappings_in(idle_pg_table, s, e, nf);
+}
+
int destroy_xen_mappings(unsigned long s, unsigned long e)
{
return modify_xen_mappings(s, e, _PAGE_NONE);
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 15/44] xen/gunzip: process pending softirqs once per deflate block
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (13 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 14/44] x86/mm: let Xen's page-table walker operate on a given root George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-18 11:56 ` Jan Beulich
2026-11-03 22:13 ` [PATCH v6 16/44] x86/mm: add hooks for a sparse view of the directmap George Dunlap
` (28 subsequent siblings)
43 siblings, 1 reply; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Andrew Cooper, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné, Stefano Stabellini
Nothing in the gzip decompressor lets the CPU run softirqs. The NMI
watchdog treats a CPU as stuck once its timer softirq has not run for
the watchdog's timeout, 5s by default, so a decompression that runs
that long panics the host.
Decompression is normally quick, but it need not be. Every compressed
deflate block builds its Huffman tables with xmalloc() and frees them
with xfree(), and those calls reach the xenheap whenever a request is a
page or more, or a free empties an xmalloc pool page. A subsequent
change can make every xenheap free end in a TLB flush on all online
CPUs. With it in effect, decompressing the PV dom0 kernel and reading
its ELF headers on a 480-CPU host took 5.2s instead of 0.5s; with
cpufreq=xen:performance max_cstate=1,0 dom0_nodes=0,relaxed added to
the command line as well, the watchdog fired before decompression had
finished, on each of eight boots.
Process pending softirqs after each deflate block, as libelf does for
each ELF note since c5089fae0f6f ("x86: avoid tripping watchdog when
constructing dom0"), and as the PV and PVH dom0 builders do in their
long loops. Every caller of perform_gunzip() runs during domain
construction at boot, on the boot CPU with interrupts enabled and no
lock held: x86's PV and PVH dom0 builders, which call
process_pending_softirqs() themselves, and kernel_decompress(), which
Arm uses for dom0 and for dom0less domains.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New in this version.
---
xen/common/gzip/gunzip.c | 1 +
xen/common/gzip/inflate.c | 1 +
2 files changed, 2 insertions(+)
diff --git a/xen/common/gzip/gunzip.c b/xen/common/gzip/gunzip.c
index 89f45d4050..9551d470aa 100644
--- a/xen/common/gzip/gunzip.c
+++ b/xen/common/gzip/gunzip.c
@@ -3,6 +3,7 @@
#include <xen/init.h>
#include <xen/lib.h>
#include <xen/mm.h>
+#include <xen/softirq.h>
#define WSIZE 0x80000000U
diff --git a/xen/common/gzip/inflate.c b/xen/common/gzip/inflate.c
index cb146555c8..737aff505e 100644
--- a/xen/common/gzip/inflate.c
+++ b/xen/common/gzip/inflate.c
@@ -1018,6 +1018,7 @@ static int __init inflate(struct gunzip_state *s)
r = inflate_block(s, &e);
if (r)
return r;
+ process_pending_softirqs();
} while (!e);
/* Undo too much lookahead. The next read will be byte aligned so we
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 16/44] x86/mm: add hooks for a sparse view of the directmap
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (14 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 15/44] xen/gunzip: process pending softirqs once per deflate block George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 17/44] x86/mm: build and maintain " George Dunlap
` (27 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie,
Anthony PERARD, Michal Orzel, Julien Grall, Stefano Stabellini
The directmap maps all of RAM, and every root page-table carries it,
so while any guest context runs, all guest memory is reachable from
Xen's address space. For ASI, guest contexts are to run on a sparse
view of the directmap which maps only what Xen must reach through it
at all times, leaving guest memory to transient mappings
(map_domain_page()).
A later patch builds such a sparse view alongside the full directmap,
once the boot allocator has handed its memory to the heap. It is then
kept in step:
- xenheap allocations are added, and taken out again before the pages
are freed. If that fails the pages stay mapped, and are leaked with
a warning rather than reused;
- memory handed to the heap later (boot modules, .init) is taken out,
and is not handed over if that fails;
- a PV dom0's initrd, assigned to dom0 in place, is taken out;
- page-tables from the boot allocator, which the sparse view maps with
every other boot-time allocation, are taken out when idle_pg_table's
hierarchy frees them, and are not freed if that fails, so that the
sparse view never keeps a page the heap may hand out;
- changes to the full directmap through map_pages_to_xen() and
modify_xen_mappings() are mirrored. After boot, map_pages_to_xen()
maps into the directmap only memory Xen keeps (stack guard and shadow
stack pages, firmware tables); memory hot-add maps the new range into
the full directmap only, so that the sparse view never needs
page-table pages for it.
Add a hook at each of these points, and one where the view is to be
built; x86's are stubs in a new asm/sparse-directmap.h. map_pages_in()
and modify_mappings_in() are no longer static: memory hot-add calls
map_pages_in() on idle_pg_table, and the sparse view will use both.
Common code gains four hooks for this, arch_heap_pages_init(),
arch_xenheap_needs_scrub() and arch_xenheap_pages_{alloc,free}(), no-ops
until an architecture implements them. A failed
arch_xenheap_pages_alloc() may have mapped part of the pages:
alloc_xenheap_pages() then takes them out through
arch_xenheap_pages_free(), as a free would, before freeing them, and
leaks them if that fails too.
alloc_xenheap_pages() accepts pages still awaiting scrubbing
(MEMF_no_scrub). That is harmless while the directmap maps all memory
anyway, but the sparse view would make such a page's contents, a dying
domain's data for instance, reachable from every context on it until
overwritten. With a sparse view configured, arch_xenheap_needs_scrub()
makes xenheap allocations scrub those pages first, as domain heap
allocations do; clean pages need nothing extra.
arch_mfns_in_directmap() says whether an MFN range is reachable through
the directmap; it now answers for every context, so false once a sparse
view is configured. Its one caller, init_node_heap(), then keeps a NUMA
node's heap metadata in the xenheap instead of carving it out of the
node's memory through the directmap: carved out of a range handed to the
heap, it would be left out of the sparse view, which every heap
allocation needs it in.
The hooks for now are just stubs. No functional change.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New in this version, split out of "x86/mm: build and maintain a sparse
view of the directmap".
---
xen/arch/x86/include/asm/mm.h | 12 +++++-
xen/arch/x86/include/asm/sparse-directmap.h | 30 +++++++++++++++
xen/arch/x86/mm.c | 30 ++++++++++++---
xen/arch/x86/pv/dom0_build.c | 4 ++
xen/arch/x86/setup.c | 4 ++
xen/arch/x86/x86_64/mm.c | 10 ++++-
xen/common/page_alloc.c | 41 ++++++++++++++++++++-
xen/include/xen/mm.h | 21 +++++++++++
8 files changed, 142 insertions(+), 10 deletions(-)
create mode 100644 xen/arch/x86/include/asm/sparse-directmap.h
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 7b9a0f561d..61dc317dd4 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -7,6 +7,7 @@
#include <xen/rwlock.h>
#include <asm/io.h>
#include <asm/page.h>
+#include <asm/sparse-directmap.h>
#include <asm/uaccess.h>
/*
@@ -599,6 +600,11 @@ mfn_t alloc_xen_pagetable(void);
void free_xen_pagetable(mfn_t mfn);
void *alloc_mapped_pagetable(mfn_t *pmfn);
+int map_pages_in(l4_pgentry_t *root, unsigned long virt, mfn_t mfn,
+ unsigned long nr_mfns, pte_attr_t flags);
+int modify_mappings_in(l4_pgentry_t *root, unsigned long s, unsigned long e,
+ pte_attr_t nf);
+
int __sync_local_execstate(void);
/* Arch-specific portion of memory_op hypercall. */
@@ -660,12 +666,16 @@ void write_32bit_pse_identmap(uint32_t *l2);
/*
* x86 maps part of physical memory via the directmap region.
- * Return whether the range of MFN falls in the directmap region.
+ * Return whether the range of MFN falls in the directmap region in every
+ * context: never, with a sparse directmap view configured.
*/
static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)
{
unsigned long eva = min(DIRECTMAP_VIRT_END, HYPERVISOR_VIRT_END);
+ if ( sparse_dmap_configured() )
+ return false;
+
return (mfn + nr) <= (virt_to_mfn(eva - 1) + 1);
}
diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
new file mode 100644
index 0000000000..53747a98ad
--- /dev/null
+++ b/xen/arch/x86/include/asm/sparse-directmap.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef X86_SPARSE_DIRECTMAP_H
+#define X86_SPARSE_DIRECTMAP_H
+
+#include <xen/mm-frame.h>
+#include <xen/mm-types.h>
+#include <asm/page.h>
+
+static inline bool sparse_dmap_configured(void) { return false; }
+static inline void sparse_dmap_init(void) {}
+static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
+ unsigned long nr, pte_attr_t flags)
+{
+ return 0;
+}
+static inline int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
+ pte_attr_t nf)
+{
+ return 0;
+}
+static inline int sparse_dmap_remove(mfn_t mfn, unsigned long nr)
+{
+ return 0;
+}
+static inline int sparse_dmap_drop_pagetable(mfn_t mfn)
+{
+ return 0;
+}
+
+#endif /* X86_SPARSE_DIRECTMAP_H */
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 39c6fa3d56..904ed7a258 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -127,6 +127,7 @@
#include <asm/setup.h>
#include <asm/shadow.h>
#include <asm/shared.h>
+#include <asm/sparse-directmap.h>
#include <asm/trampoline.h>
#include <asm/traps.h>
#include <asm/x86_emulate.h>
@@ -5442,10 +5443,17 @@ mfn_t xen_map_to_mfn(unsigned long va)
/*
* Free a page-table page that the hierarchy rooted at @root has stopped
- * using.
+ * using. One of idle_pg_table's may be in the sparse directmap view: it is
+ * taken out first, under the sparse view's L3 lock inside an L3 lock of
+ * idle_pg_table's hierarchy (the only order in which the two nest), and
+ * kept if that fails. The sparse view's own page-tables are not in it, and
+ * are freed directly: the caller may hold the sparse view's L3 lock.
*/
static void free_pagetable_in(const l4_pgentry_t *root, mfn_t mfn)
{
+ if ( root == idle_pg_table && sparse_dmap_drop_pagetable(mfn) )
+ return;
+
free_xen_pagetable(mfn);
}
@@ -5453,7 +5461,7 @@ static void free_pagetable_in(const l4_pgentry_t *root, mfn_t mfn)
* map_pages_to_xen() in the hierarchy rooted at @root: idle_pg_table's, or
* another hierarchy sharing its virtual addresses.
*/
-static int map_pages_in(
+int map_pages_in(
l4_pgentry_t *root,
unsigned long virt,
mfn_t mfn,
@@ -5865,7 +5873,12 @@ int map_pages_to_xen(
unsigned long nr_mfns,
pte_attr_t flags)
{
- return map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
+ int rc = map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
+
+ if ( !rc )
+ rc = sparse_dmap_mirror_map(virt, mfn, nr_mfns, flags);
+
+ return rc;
}
int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
@@ -5888,8 +5901,8 @@ int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
* modify_mappings_in() does so in the hierarchy rooted at @root, as
* map_pages_in() does.
*/
-static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
- unsigned long e, pte_attr_t nf)
+int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
+ unsigned long e, pte_attr_t nf)
{
bool locking = system_state > SYS_STATE_boot;
l3_pgentry_t *pl3e = NULL;
@@ -6169,7 +6182,12 @@ static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
{
- return modify_mappings_in(idle_pg_table, s, e, nf);
+ int rc = modify_mappings_in(idle_pg_table, s, e, nf);
+
+ if ( !rc )
+ rc = sparse_dmap_mirror_modify(s, e, nf);
+
+ return rc;
}
int destroy_xen_mappings(unsigned long s, unsigned long e)
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index dd64af5616..829eafd67d 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -22,6 +22,7 @@
#include <asm/pv/mm.h>
#include <asm/setup.h>
#include <asm/shadow.h>
+#include <asm/sparse-directmap.h>
/* Allow ring-3 access in long mode as guest cannot use ring 1 ... */
#define BASE_PROT (_PAGE_PRESENT|_PAGE_RW|_PAGE_ACCESSED|_PAGE_USER)
@@ -650,6 +651,9 @@ static int __init dom0_construct(const struct boot_domain *bd)
{
if ( assign_pages(mfn_to_page(_mfn(initrd_mfn)), count, d, 0) )
BUG();
+ /* The initrd is dom0's now: out of the sparse directmap. */
+ if ( sparse_dmap_remove(_mfn(initrd_mfn), PFN_UP(initrd_len)) )
+ panic("Cannot unmap dom0's initrd\n");
/*
* We have mapped the initrd directly into dom0, and assigned the
* pages. Tell the boot_module handling that we've freed it, so the
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index df7cfcc563..e23ded0c86 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -54,6 +54,7 @@
#include <asm/spec_ctrl.h>
#include <asm/stubs.h>
#include <asm/tboot.h>
+#include <asm/sparse-directmap.h>
#include <asm/trampoline.h>
#include <asm/traps.h>
@@ -1946,6 +1947,9 @@ void asmlinkage __init noreturn __start_xen(void)
}
}
+ /* The heap now has all memory, and page-tables come from it. */
+ sparse_dmap_init();
+
bsp_stack = cpu_alloc_stack(0); /* Needs to know IDT vs FRED */
if ( !bsp_stack )
panic("No memory for BSP stack\n");
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index 7b04537066..c5d9dd64a3 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -1231,8 +1231,14 @@ int memory_add(unsigned long spfn, unsigned long epfn, unsigned int pxm)
i = virt_to_mfn(HYPERVISOR_VIRT_END - 1) + 1;
if ( spfn < i )
{
- ret = map_pages_to_xen((unsigned long)mfn_to_virt(spfn), _mfn(spfn),
- min(epfn, i) - spfn, PAGE_HYPERVISOR);
+ /*
+ * Into the full directmap only, not a sparse directmap view: the
+ * frame table and M2P carved from the start of the range are
+ * reached through their own mappings, and the rest goes to the
+ * heap, which the sparse view leaves out.
+ */
+ ret = map_pages_in(idle_pg_table, (unsigned long)mfn_to_virt(spfn),
+ _mfn(spfn), min(epfn, i) - spfn, PAGE_HYPERVISOR);
if ( ret )
goto destroy_directmap;
}
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 5243dba29f..2150dcbd5e 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -1971,6 +1971,9 @@ static void init_heap_pages(
unsigned long i;
bool need_scrub = scrub_debug;
+ if ( !arch_heap_pages_init(page_to_mfn(pg), nr_pages) )
+ return;
+
/*
* Keep MFN 0 away from the buddy allocator to avoid crossing zone
* boundary when merging two buddies.
@@ -2540,10 +2543,33 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags)
if ( !(memflags >> _MEMF_bits) )
memflags |= MEMF_bits(xenheap_bits);
- pg = alloc_domheap_pages(NULL, order, memflags | MEMF_no_scrub);
+ /*
+ * Pages still awaiting scrubbing may do as xenheap pages: the directmap
+ * maps them anyway. Not so if an arch's sparse directmap view makes them
+ * reachable where the rest of the heap's memory is not.
+ */
+ if ( !arch_xenheap_needs_scrub() )
+ memflags |= MEMF_no_scrub;
+
+ pg = alloc_domheap_pages(NULL, order, memflags);
if ( unlikely(pg == NULL) )
return NULL;
+ if ( arch_xenheap_pages_alloc(page_to_mfn(pg), order) )
+ {
+ /*
+ * The hook may have mapped part of the pages before failing: take
+ * them out again as a free would, and leak them if that fails too.
+ */
+ if ( arch_xenheap_pages_free(page_to_mfn(pg), order) )
+ printk(XENLOG_WARNING
+ "Leaking xenheap pages %"PRI_mfn"+%u: still mapped\n",
+ mfn_x(page_to_mfn(pg)), 1U << order);
+ else
+ free_domheap_pages(pg, order);
+ return NULL;
+ }
+
for ( i = 0; i < (1u << order); i++ )
pg[i].count_info |= PGC_xen_heap;
@@ -2562,6 +2588,19 @@ void free_xenheap_pages(void *v, unsigned int order)
pg = virt_to_page(v);
+ /*
+ * If the pages cannot be taken out of an arch's sparse directmap view,
+ * they may remain reachable through it: leak them rather than hand them
+ * out.
+ */
+ if ( arch_xenheap_pages_free(page_to_mfn(pg), order) )
+ {
+ printk(XENLOG_WARNING
+ "Leaking xenheap pages %"PRI_mfn"+%u: still mapped\n",
+ mfn_x(page_to_mfn(pg)), 1U << order);
+ return;
+ }
+
for ( i = 0; i < (1u << order); i++ )
pg[i].count_info &= ~PGC_xen_heap;
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index 153c57f5dd..e8f9993286 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -94,6 +94,27 @@ bool scrub_free_pages(void);
#define alloc_xenheap_page() (alloc_xenheap_pages(0,0))
#define free_xenheap_page(v) (free_xenheap_pages(v,0))
+/*
+ * Hooks for an architecture keeping a second, sparse view of its directmap,
+ * which maps the xenheap but not the rest of the heap's memory.
+ */
+static inline bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
+{
+ return true;
+}
+static inline bool arch_xenheap_needs_scrub(void)
+{
+ return false;
+}
+static inline int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
+{
+ return 0;
+}
+static inline int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
+{
+ return 0;
+}
+
/* Free an allocation, and zero the pointer to it. */
#define FREE_XENHEAP_PAGES(p, o) do { \
void *_ptr_ = (p); \
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 17/44] x86/mm: build and maintain a sparse view of the directmap
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (15 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 16/44] x86/mm: add hooks for a sparse view of the directmap George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 18/44] x86/domain_page: key the directmap fast paths on the loaded view George Dunlap
` (26 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie,
Anthony PERARD, Michal Orzel, Julien Grall, Stefano Stabellini
The directmap maps all of RAM, and every root page-table carries it, so
while any guest context runs, all guest memory is reachable from Xen's
address space. For ASI, guest contexts are to run on a view of the
directmap which maps only what Xen must reach through it at all times,
leaving guest memory to transient mappings (map_domain_page()).
Rather than remove the directmap for everyone, keep the full directmap
as it is, for the idle domain and for guest contexts not using ASI, and
build a second, sparse view: page-tables for the same virtual addresses
(the part of the directmap PV guests can see, where everything Xen
reaches through the directmap lives), mapping a subset of what the full
directmap maps, each entry a copy of the full directmap's. Boot and the
idle domain keep the full directmap, so nothing in them needs
converting.
The sparse view maps the xenheap, and memory never handed to the heap
allocator (Xen's image, boot-time allocations, firmware tables). It is
built once the boot allocator has handed its memory to the heap, from
the full directmap minus the ranges the heap was given, plus the xenheap
allocations made meanwhile. The page-tables then come from the heap,
and no other CPU is up. The sparse view has an L3 for each of its L4
slots from the start, so its L4 entries never change and root
page-tables can take copies of them. It is then kept in step through
the hooks added by "x86/mm: add hooks for a sparse view of the
directmap", which this patch implements for CONFIG_SPARSE_DIRECTMAP.
modify_xen_mappings_lite() needs no change: it only changes the mappings
of Xen's image, in L4 slot 261, which every root page-table shares with
idle_pg_table, outside the range the sparse view replaces.
xen_mapping_lookup(), built only with
CONFIG_SPARSE_DIRECTMAP, reads an entry of either hierarchy under the
L3 table lock.
The allocator may still have to scrub a page before returning it; this
will be mapped with map_domain_page(). This is fine for HVM domains
on the sparse view, as the mapcache is per-vCPU, so no other vCPU can
read unscrubbed data. PV domains' mapcache is still domain-wide, so
for PV domains on the sparse view, this opens up a window where other
vCPUs could potentially speculatively access the unscrubbed data.
This would be fixed by having per-vCPU mapcaches for PV domains, or
synchronously scrubbing pages on free. Neither are in the scope of the
current work.
The sparse view is built only when some domain type is to use it, which
nothing can request yet: this patch changes nothing on its own.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- Split: free_pagetable_in() moves to "x86/mm: let Xen's page-table
walker operate on a given root", and the hooks with their stubs and
callers to the new "x86/mm: add hooks for a sparse view of the
directmap".
Changes in v4:
- Message: "domain type", not "guest type".
- Wrap a line that reached 80 columns.
- Memory hot-add maps the new range into the full directmap only.
- Define opt_sparse_dmap_hwdom here, beside the PV and HVM knobs
(Roger).
- add_to_sparse_view(): assert that the lookup finds the MFN expected
(Roger).
- cmp_mfn_range(): explicit return values, as in cmp_node() (Roger).
- Comments and variable name changes in sparse_dmap_init()'s loop over
the heap's ranges (Roger).
- Message: drop "(NUMA node heap metadata)" (Roger).
- Message: drop the reference to the directmap removal series (Roger).
- Message: modify_xen_mappings_lite() needs no change (Roger).
- Message: note the limit of scrubbing at allocation for PV domains
(Roger).
- Message: only boot and the idle domain keep the full directmap.
- Set the NX bit in this patch, rather than adding it in a later patch.
Changes in v3:
- New in this version.
---
xen/arch/x86/Kconfig | 1 +
xen/arch/x86/Makefile | 1 +
xen/arch/x86/include/asm/mm.h | 4 +
xen/arch/x86/include/asm/sparse-directmap.h | 37 ++
xen/arch/x86/mm.c | 74 ++++
xen/arch/x86/sparse-directmap.c | 431 ++++++++++++++++++++
xen/common/Kconfig | 18 +
xen/include/xen/mm.h | 14 +
8 files changed, 580 insertions(+)
create mode 100644 xen/arch/x86/sparse-directmap.c
diff --git a/xen/arch/x86/Kconfig b/xen/arch/x86/Kconfig
index e5535ac484..cd72102ccc 100644
--- a/xen/arch/x86/Kconfig
+++ b/xen/arch/x86/Kconfig
@@ -31,6 +31,7 @@ config X86
select HAS_SCHED_GRANULARITY
select HAS_SHARED_INFO
imply HAS_SOFT_RESET
+ select HAS_SPARSE_DIRECTMAP
select HAS_UBSAN
select HAS_VMAP
select HAS_VPCI if HVM
diff --git a/xen/arch/x86/Makefile b/xen/arch/x86/Makefile
index 2be90ea098..86adf042e9 100644
--- a/xen/arch/x86/Makefile
+++ b/xen/arch/x86/Makefile
@@ -63,6 +63,7 @@ obj-y += shutdown.o
obj-y += smp.o
obj-y += smpboot.o
obj-y += spec_ctrl.o
+obj-$(CONFIG_SPARSE_DIRECTMAP) += sparse-directmap.o
obj-y += srat.o
obj-y += string.o
obj-$(CONFIG_SYSCTL) += sysctl.o
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 61dc317dd4..113ade0bbd 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -604,6 +604,10 @@ int map_pages_in(l4_pgentry_t *root, unsigned long virt, mfn_t mfn,
unsigned long nr_mfns, pte_attr_t flags);
int modify_mappings_in(l4_pgentry_t *root, unsigned long s, unsigned long e,
pte_attr_t nf);
+#ifdef CONFIG_SPARSE_DIRECTMAP
+mfn_t xen_mapping_lookup(l4_pgentry_t *root, unsigned long va,
+ pte_attr_t *flags, unsigned long *nr);
+#endif
int __sync_local_execstate(void);
diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
index 53747a98ad..ea14d44334 100644
--- a/xen/arch/x86/include/asm/sparse-directmap.h
+++ b/xen/arch/x86/include/asm/sparse-directmap.h
@@ -6,6 +6,41 @@
#include <xen/mm-types.h>
#include <asm/page.h>
+#ifdef CONFIG_SPARSE_DIRECTMAP
+
+/*
+ * The part of the directmap the sparse view covers: the part PV guests can
+ * see, which is where everything Xen reaches through the directmap lives.
+ * SPARSE_DMAP_END_MFN is the first MFN past it.
+ */
+#define SPARSE_DMAP_START DIRECTMAP_VIRT_START
+#define SPARSE_DMAP_END HYPERVISOR_VIRT_END
+#define SPARSE_DMAP_END_MFN (virt_to_mfn(SPARSE_DMAP_END - 1) + 1)
+
+/*
+ * Which domains use the sparse view: PV domains and HVM domains other than
+ * the hardware domain, and the hardware domain, PV or PVH.
+ */
+extern bool opt_sparse_dmap_pv, opt_sparse_dmap_hvm, opt_sparse_dmap_hwdom;
+
+/* Root of the sparse view, once built (NULL before, or if not configured). */
+extern l4_pgentry_t *sparse_dmap_root;
+
+static inline bool sparse_dmap_configured(void)
+{
+ return opt_sparse_dmap_pv || opt_sparse_dmap_hvm || opt_sparse_dmap_hwdom;
+}
+
+void sparse_dmap_init(void);
+int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
+ pte_attr_t flags);
+int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
+ pte_attr_t nf);
+int sparse_dmap_remove(mfn_t mfn, unsigned long nr);
+int sparse_dmap_drop_pagetable(mfn_t mfn);
+
+#else /* !CONFIG_SPARSE_DIRECTMAP */
+
static inline bool sparse_dmap_configured(void) { return false; }
static inline void sparse_dmap_init(void) {}
static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
@@ -27,4 +62,6 @@ static inline int sparse_dmap_drop_pagetable(mfn_t mfn)
return 0;
}
+#endif /* CONFIG_SPARSE_DIRECTMAP */
+
#endif /* X86_SPARSE_DIRECTMAP_H */
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 904ed7a258..0d475432ff 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -5441,6 +5441,80 @@ mfn_t xen_map_to_mfn(unsigned long va)
return ret;
}
+#ifdef CONFIG_SPARSE_DIRECTMAP
+/*
+ * Look @va up in the hierarchy rooted at @root, without allocating anything.
+ * Returns the MFN mapped at @va, or INVALID_MFN, with the mapping's flags in
+ * L1 form in *flags, and in *nr the number of pages from @va to the end of
+ * the entry that maps @va (or that finds it unmapped).
+ */
+mfn_t xen_mapping_lookup(l4_pgentry_t *root, unsigned long va,
+ pte_attr_t *flags, unsigned long *nr)
+{
+ bool locking = system_state > SYS_STATE_boot;
+ l4_pgentry_t l4e = root[l4_table_offset(va)];
+ const l3_pgentry_t *pl3e = NULL;
+ const l2_pgentry_t *pl2e = NULL;
+ const l1_pgentry_t *pl1e = NULL;
+ struct page_info *l3page;
+ unsigned int shift = L4_PAGETABLE_SHIFT;
+ mfn_t ret = INVALID_MFN;
+
+ L3T_INIT(l3page);
+ *flags = 0;
+
+ if ( !(l4e_get_flags(l4e) & _PAGE_PRESENT) )
+ goto out;
+
+ pl3e = map_l3t_from_l4e(l4e) + l3_table_offset(va);
+ l3page = l4e_get_page(l4e);
+ L3T_LOCK(l3page);
+
+ shift = L3_PAGETABLE_SHIFT;
+ if ( !(l3e_get_flags(*pl3e) & _PAGE_PRESENT) )
+ goto out;
+ if ( l3e_get_flags(*pl3e) & _PAGE_PSE )
+ {
+ /* The superpage PAT bit sits among the address bits. */
+ *flags = lNf_to_l1f(l3e_get_flags(*pl3e)) |
+ (l3e_get_intpte(*pl3e) & _PAGE_PSE_PAT ? _PAGE_PAT : 0);
+ ret = _mfn((l3e_get_pfn(*pl3e) &
+ ~((1UL << (2 * PAGETABLE_ORDER)) - 1)) +
+ (PFN_DOWN(va) & ((1UL << (2 * PAGETABLE_ORDER)) - 1)));
+ goto out;
+ }
+
+ pl2e = map_l2t_from_l3e(*pl3e) + l2_table_offset(va);
+ shift = L2_PAGETABLE_SHIFT;
+ if ( !(l2e_get_flags(*pl2e) & _PAGE_PRESENT) )
+ goto out;
+ if ( l2e_get_flags(*pl2e) & _PAGE_PSE )
+ {
+ *flags = lNf_to_l1f(l2e_get_flags(*pl2e)) |
+ (l2e_get_intpte(*pl2e) & _PAGE_PSE_PAT ? _PAGE_PAT : 0);
+ ret = _mfn((l2e_get_pfn(*pl2e) & ~((1UL << PAGETABLE_ORDER) - 1)) +
+ l1_table_offset(va));
+ goto out;
+ }
+
+ pl1e = map_l1t_from_l2e(*pl2e) + l1_table_offset(va);
+ shift = PAGE_SHIFT;
+ if ( l1e_get_flags(*pl1e) & _PAGE_PRESENT )
+ {
+ *flags = l1e_get_flags(*pl1e);
+ ret = l1e_get_mfn(*pl1e);
+ }
+
+ out:
+ L3T_UNLOCK(l3page);
+ unmap_domain_page(pl1e);
+ unmap_domain_page(pl2e);
+ unmap_domain_page(pl3e);
+ *nr = ((((va >> shift) + 1) << shift) - va) >> PAGE_SHIFT;
+ return ret;
+}
+#endif /* CONFIG_SPARSE_DIRECTMAP */
+
/*
* Free a page-table page that the hierarchy rooted at @root has stopped
* using. One of idle_pg_table's may be in the sparse directmap view: it is
diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
new file mode 100644
index 0000000000..d35a8d6979
--- /dev/null
+++ b/xen/arch/x86/sparse-directmap.c
@@ -0,0 +1,431 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * A sparse view of the directmap.
+ *
+ * Xen maps all of RAM at DIRECTMAP_VIRT_START, and every root page-table
+ * carries those mappings. The sparse view is a second set of page-tables
+ * for the same virtual addresses (the part of the directmap PV guests can
+ * see), mapping only what Xen needs to reach through the directmap at all
+ * times:
+ *
+ * - Memory never handed to the heap: Xen's image, boot-time allocations,
+ * firmware tables. Not a PV dom0's initrd, used in place, nor the frame
+ * table and M2P pages carved from hot-added memory.
+ * - Memory handed to the heap, then allocated to the xenheap.
+ *
+ * The rest of the heap's memory, where guest memory comes from, is not
+ * mapped. Every mapping in the sparse view is either identical to the
+ * full directmap's at the same address, or empty.
+ *
+ * Guest contexts selected on the command line run on root page-tables
+ * carrying the sparse view's L3s in the directmap slots.
+ */
+
+#include <xen/domain_page.h>
+#include <xen/init.h>
+#include <xen/lib.h>
+#include <xen/mm.h>
+#include <xen/numa.h>
+#include <xen/sort.h>
+
+#include <asm/page.h>
+#include <asm/sparse-directmap.h>
+
+bool __ro_after_init opt_sparse_dmap_pv;
+bool __ro_after_init opt_sparse_dmap_hvm;
+bool __ro_after_init opt_sparse_dmap_hwdom;
+
+/* Root of the sparse view: only ever walked and copied from, never loaded. */
+static l4_pgentry_t __aligned(PAGE_SIZE) sparse_l4[L4_PAGETABLE_ENTRIES];
+l4_pgentry_t *__ro_after_init sparse_dmap_root;
+
+/*
+ * Before the sparse view is built: the MFN ranges handed to the heap, which
+ * it leaves out, and the xenheap allocations made since, which it puts back.
+ */
+struct mfn_range {
+ unsigned long s, e;
+};
+static struct mfn_range early_heap[256];
+static unsigned int nr_early_heap;
+static struct mfn_range early_xenheap[2 * MAX_NUMNODES];
+static unsigned int nr_early_xenheap;
+static bool early_overflow;
+
+/*
+ * Clip [*mfn, *mfn + *nr) to what the sparse view covers; false if nothing is
+ * left.
+ */
+static bool clip_to_sparse_view(unsigned long *mfn, unsigned long *nr)
+{
+ unsigned long end = SPARSE_DMAP_END_MFN;
+
+ if ( *mfn >= end )
+ return false;
+ if ( *nr > end - *mfn )
+ *nr = end - *mfn;
+
+ return *nr;
+}
+
+static void early_range_add(struct mfn_range *r, unsigned int *nr,
+ unsigned int max, unsigned long s, unsigned long e)
+{
+ if ( *nr && r[*nr - 1].e == s )
+ r[*nr - 1].e = e;
+ else if ( *nr < max )
+ r[(*nr)++] = (struct mfn_range){ s, e };
+ else
+ early_overflow = true;
+}
+
+static void early_range_del(struct mfn_range *r, unsigned int *nr,
+ unsigned long s, unsigned long e)
+{
+ unsigned int i;
+
+ for ( i = 0; i < *nr; i++ )
+ if ( r[i].s == s && r[i].e == e )
+ {
+ r[i] = r[--*nr];
+ return;
+ }
+}
+
+/*
+ * Copy the full directmap's mappings of MFNs [mfn, mfn + nr) into the sparse
+ * view.
+ */
+static int add_to_sparse_view(l4_pgentry_t *root, unsigned long mfn,
+ unsigned long nr)
+{
+ unsigned long start = (unsigned long)mfn_to_virt(mfn);
+ unsigned long va = start, end = start + (nr << PAGE_SHIFT);
+
+ while ( va < end )
+ {
+ pte_attr_t flags;
+ unsigned long n;
+ mfn_t va_mfn = xen_mapping_lookup(idle_pg_table, va, &flags, &n);
+ int rc;
+
+ ASSERT(mfn_eq(va_mfn, INVALID_MFN) ||
+ mfn_x(va_mfn) == mfn + ((va - start) >> PAGE_SHIFT));
+ n = min(n, (end - va) >> PAGE_SHIFT);
+ if ( !mfn_eq(va_mfn, INVALID_MFN) &&
+ (rc = map_pages_in(root, va, va_mfn, n, flags)) )
+ return rc;
+ va += n << PAGE_SHIFT;
+ }
+
+ return 0;
+}
+
+static int remove_from_sparse_view(unsigned long mfn, unsigned long nr)
+{
+ unsigned long va = (unsigned long)mfn_to_virt(mfn);
+
+ return modify_mappings_in(sparse_dmap_root, va, va + (nr << PAGE_SHIFT),
+ _PAGE_NONE);
+}
+
+/* Memory is about to be handed to the heap allocator. */
+bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
+{
+ unsigned long s = mfn_x(mfn);
+
+ if ( !sparse_dmap_configured() || !clip_to_sparse_view(&s, &nr) )
+ return true;
+
+ if ( !sparse_dmap_root )
+ {
+ early_range_add(early_heap, &nr_early_heap, ARRAY_SIZE(early_heap),
+ s, s + nr);
+ return true;
+ }
+
+ if ( remove_from_sparse_view(s, nr) )
+ {
+ printk(XENLOG_ERR
+ "sparse directmap: cannot unmap MFNs %#lx-%#lx, not freeing\n",
+ s, s + nr - 1);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Xenheap pages enter the sparse view, which guest contexts run on: those
+ * still holding a previous owner's data must be scrubbed first.
+ */
+bool arch_xenheap_needs_scrub(void)
+{
+ return sparse_dmap_configured();
+}
+
+/* Pages have been allocated from the heap as xenheap pages. */
+int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
+{
+ unsigned long s = mfn_x(mfn), nr = 1UL << order;
+
+ if ( !sparse_dmap_configured() )
+ return 0;
+
+ /* Xenheap pages must be reachable through the sparse view. */
+ BUG_ON(s + nr > SPARSE_DMAP_END_MFN);
+
+ if ( !sparse_dmap_root )
+ {
+ early_range_add(early_xenheap, &nr_early_xenheap,
+ ARRAY_SIZE(early_xenheap), s, s + nr);
+ return 0;
+ }
+
+ return add_to_sparse_view(sparse_dmap_root, s, nr);
+}
+
+/*
+ * Xenheap pages are about to be freed. A failure means the pages may still
+ * be reachable through the sparse view, and must not be reused.
+ */
+int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
+{
+ unsigned long s = mfn_x(mfn), nr = 1UL << order;
+
+ if ( !sparse_dmap_configured() )
+ return 0;
+
+ BUG_ON(s + nr > SPARSE_DMAP_END_MFN);
+
+ if ( !sparse_dmap_root )
+ {
+ early_range_del(early_xenheap, &nr_early_xenheap, s, s + nr);
+ return 0;
+ }
+
+ return remove_from_sparse_view(s, nr);
+}
+
+/* Memory leaves Xen's hands without passing through the heap. */
+int sparse_dmap_remove(mfn_t mfn, unsigned long nr)
+{
+ return arch_heap_pages_init(mfn, nr) ? 0 : -ENOMEM;
+}
+
+/*
+ * A page-table page of idle_pg_table's hierarchy is about to be freed. One
+ * the boot allocator provided is in the sparse view, as every boot-time
+ * allocation is: take it out before the heap may hand it on. One allocated
+ * since came from the heap and is not in the sparse view, which a lookup
+ * finds without changing, or flushing, anything. Non-zero: the page may
+ * still be mapped, and must not be freed.
+ */
+int sparse_dmap_drop_pagetable(mfn_t mfn)
+{
+ unsigned long s = mfn_x(mfn), nr = 1, n;
+ pte_attr_t flags;
+ int rc;
+
+ if ( !sparse_dmap_root || !clip_to_sparse_view(&s, &nr) ||
+ mfn_eq(xen_mapping_lookup(sparse_dmap_root,
+ (unsigned long)mfn_to_virt(s), &flags, &n),
+ INVALID_MFN) )
+ return 0;
+
+ rc = remove_from_sparse_view(s, 1);
+ if ( rc )
+ printk(XENLOG_WARNING
+ "Leaking page-table page %"PRI_mfn": still in the sparse directmap\n",
+ mfn_x(mfn));
+
+ return rc;
+}
+
+int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
+ pte_attr_t flags)
+{
+ if ( !sparse_dmap_root || virt < SPARSE_DMAP_START ||
+ virt >= SPARSE_DMAP_END )
+ return 0;
+
+ nr = min(nr, (SPARSE_DMAP_END - virt) >> PAGE_SHIFT);
+
+ return map_pages_in(sparse_dmap_root, virt, mfn, nr, flags);
+}
+
+int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
+ pte_attr_t nf)
+{
+ unsigned long va;
+
+ if ( !sparse_dmap_root || e <= SPARSE_DMAP_START ||
+ s >= SPARSE_DMAP_END )
+ return 0;
+
+ s = max(s, SPARSE_DMAP_START);
+ e = min(e, SPARSE_DMAP_END);
+
+ if ( !(nf & _PAGE_PRESENT) )
+ return modify_mappings_in(sparse_dmap_root, s, e, nf);
+
+ /*
+ * A permission change applies where the sparse view maps anything: it
+ * never creates mappings, and the sparse view need not map all of [s, e).
+ */
+ for ( va = s; va < e; )
+ {
+ pte_attr_t flags;
+ unsigned long n;
+ mfn_t mfn = xen_mapping_lookup(sparse_dmap_root, va, &flags, &n);
+ int rc;
+
+ n = min(n, (e - va) >> PAGE_SHIFT);
+ if ( !mfn_eq(mfn, INVALID_MFN) &&
+ (rc = modify_mappings_in(sparse_dmap_root, va,
+ va + (n << PAGE_SHIFT), nf)) )
+ return rc;
+ va += n << PAGE_SHIFT;
+ }
+
+ return 0;
+}
+
+static int __init cf_check cmp_mfn_range(const void *a, const void *b)
+{
+ const struct mfn_range *l = a, *r = b;
+
+ if ( l->s > r->s )
+ return 1;
+ if ( l->s < r->s )
+ return -1;
+
+ return 0;
+}
+
+static void __init cf_check swap_mfn_range(void *a, void *b)
+{
+ SWAP(*(struct mfn_range *)a, *(struct mfn_range *)b);
+}
+
+/*
+ * Build the sparse view: the full directmap, minus the ranges handed to the
+ * heap so far, plus the xenheap allocations made from them. Called once the
+ * heap has taken over from the boot allocator, so that the page-tables come
+ * from the heap, and before any other CPU is up.
+ */
+void __init sparse_dmap_init(void)
+{
+ unsigned long va, mapped = 0;
+ unsigned int i, heap_idx;
+
+ if ( !sparse_dmap_configured() )
+ return;
+
+ if ( early_overflow )
+ {
+ printk(XENLOG_WARNING
+ "Sparse directmap: too many early heap ranges, disabled\n");
+ opt_sparse_dmap_pv = opt_sparse_dmap_hvm = opt_sparse_dmap_hwdom =
+ false;
+ return;
+ }
+
+ /* An L3 for every slot, so that the L4 entries never change. */
+ for ( va = SPARSE_DMAP_START; va < SPARSE_DMAP_END;
+ va += 1UL << L4_PAGETABLE_SHIFT )
+ {
+ mfn_t l3mfn;
+ void *l3t = alloc_mapped_pagetable(&l3mfn);
+
+ if ( !l3t )
+ panic("Sparse directmap: out of memory\n");
+ unmap_domain_page(l3t);
+ sparse_l4[l4_table_offset(va)] =
+ l4e_from_mfn(l3mfn, __PAGE_HYPERVISOR);
+ }
+
+ sort(early_heap, nr_early_heap, sizeof(*early_heap), cmp_mfn_range,
+ swap_mfn_range);
+
+ /*
+ * The full directmap, leaf by leaf, minus the ranges added to the
+ * heap. Leaves come in address order, which is MFN order in the
+ * directmap, and the heap ranges are sorted and disjoint, so
+ * heap_idx only moves forward.
+ */
+ for ( va = SPARSE_DMAP_START, heap_idx = 0; va < SPARSE_DMAP_END; )
+ {
+ pte_attr_t flags;
+ unsigned long n, s, e;
+
+ /* Get information about the leaf PTE at va. */
+ mfn_t mfn = xen_mapping_lookup(idle_pg_table, va, &flags, &n);
+
+ /* Clip the va range we're considering to SPARSE_DMAP_END. */
+ n = min(n, (SPARSE_DMAP_END - va) >> PAGE_SHIFT);
+ if ( mfn_eq(mfn, INVALID_MFN) )
+ {
+ va += n << PAGE_SHIFT;
+ continue;
+ }
+
+ /*
+ * Walk through the range [mfn, mfn + n) looking for ranges not
+ * handed to the heap.
+ */
+ for ( s = mfn_x(mfn), e = s + n; s < e; )
+ {
+ unsigned long upto = e;
+
+ /* Step past heap ranges that end at or before s. */
+ while ( heap_idx < nr_early_heap && early_heap[heap_idx].e <= s )
+ heap_idx++;
+
+ /* s lies in a heap range: skip to its end. */
+ if ( heap_idx < nr_early_heap && early_heap[heap_idx].s <= s )
+ {
+ s = min(e, early_heap[heap_idx].e);
+ continue;
+ }
+
+ /* Map up to the next heap range, or to the end of the leaf. */
+ if ( heap_idx < nr_early_heap )
+ upto = min(e, early_heap[heap_idx].s);
+
+ if ( map_pages_in(sparse_l4, va + ((s - mfn_x(mfn)) << PAGE_SHIFT),
+ _mfn(s), upto - s, flags) )
+ panic("Sparse directmap: out of memory\n");
+ mapped += upto - s;
+ s = upto;
+ }
+
+ va += n << PAGE_SHIFT;
+ }
+
+ /*
+ * Now add back in ranges from the heap which were allocated to
+ * the xenheap.
+ */
+ for ( i = 0; i < nr_early_xenheap; i++ )
+ {
+ unsigned long nr = early_xenheap[i].e - early_xenheap[i].s;
+
+ if ( add_to_sparse_view(sparse_l4, early_xenheap[i].s, nr) )
+ panic("Sparse directmap: out of memory\n");
+ mapped += nr;
+ }
+
+ /*
+ * Nothing in the directmap is executed: its slots are NX, as the full
+ * directmap's are.
+ */
+ if ( cpu_has_nx )
+ for ( i = l4_table_offset(SPARSE_DMAP_START);
+ i < l4_table_offset(SPARSE_DMAP_END); i++ )
+ l4e_add_flags(sparse_l4[i], _PAGE_NX_BIT);
+
+ sparse_dmap_root = sparse_l4;
+
+ printk(XENLOG_INFO "Sparse directmap: %lu pages mapped\n", mapped);
+}
diff --git a/xen/common/Kconfig b/xen/common/Kconfig
index 5b289e444f..8f2f605e45 100644
--- a/xen/common/Kconfig
+++ b/xen/common/Kconfig
@@ -167,6 +167,9 @@ config HAS_STATIC_MEMORY
config HAS_SOFT_RESET
bool
+config HAS_SPARSE_DIRECTMAP
+ bool
+
config HAS_STACK_PROTECTOR
bool
@@ -299,6 +302,21 @@ config SPECULATIVE_HARDEN_LOCK
This option is disabled by default at run time, and needs to be
enabled on the command line.
+
+config SPARSE_DIRECTMAP
+ bool "Sparse directmap view"
+ depends on HAS_SPARSE_DIRECTMAP
+ help
+ Build support for a sparse view of the directmap, alongside the
+ full one: it maps only the memory Xen needs to reach at all times
+ (the Xen heap, Xen's own image, firmware tables), while guest
+ memory is reached through transient mappings. Guest contexts
+ selected with the `asi=` command line option run on the sparse
+ directmap view, narrowing the memory reachable from Xen's address
+ space while they run, and hence the memory disclosure surface of
+ speculative side channels.
+
+ If unsure, say N.
endmenu
menu "Other hardening"
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index e8f9993286..2bcc45cf58 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -98,6 +98,19 @@ bool scrub_free_pages(void);
* Hooks for an architecture keeping a second, sparse view of its directmap,
* which maps the xenheap but not the rest of the heap's memory.
*/
+#ifdef CONFIG_SPARSE_DIRECTMAP
+/* Memory is about to be handed to the heap. False: don't, keep it out. */
+bool arch_heap_pages_init(mfn_t mfn, unsigned long nr);
+/* Xenheap pages are to be allocated. True: scrub those needing it first. */
+bool arch_xenheap_needs_scrub(void);
+/*
+ * Xenheap pages were allocated. Non-zero: failed, possibly part-way; the
+ * pages go through arch_xenheap_pages_free() before being freed again.
+ */
+int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order);
+/* Xenheap pages are to be freed. Non-zero: failed, don't reuse them. */
+int arch_xenheap_pages_free(mfn_t mfn, unsigned int order);
+#else
static inline bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
{
return true;
@@ -114,6 +127,7 @@ static inline int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
{
return 0;
}
+#endif
/* Free an allocation, and zero the pointer to it. */
#define FREE_XENHEAP_PAGES(p, o) do { \
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 18/44] x86/domain_page: key the directmap fast paths on the loaded view
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (16 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 17/44] x86/mm: build and maintain " George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 19/44] x86/domain_page: clear a mapcache slot at its last unmap George Dunlap
` (25 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
map_domain_page() returns directmap addresses whenever it can: for any
MFN the directmap covers in release builds, and for vCPUs without a
mapcache. Once guest contexts may run on the sparse view of the
directmap, modify this to take the fast path returning the directmap
address only when the full directmap is loaded.
The release-build shortcut ahead of the vCPU lookup looks at the vCPU
whose page-tables are loaded, without completing a lazy context
switch: if that vCPU is not on the sparse view, its page-tables map
the page through the directmap, and completing the switch would only
move the CPU to the idle vCPU's page-tables, which map it too.
A vCPU on the sparse view always has a mapcache: HVM ones through
per-vCPU page-tables, PV ones through the domain-wide mapcache, which
mapcache_domain_init() now sets up for a domain using the sparse view in
release builds too, rather than only when the directmap does not cover
all memory.
map_domain_page_global() mappings must be valid in every context: use
vmap() for all of them once a sparse view is configured, as
arch_mfns_in_directmap() then says.
Add d->arch.asi.sparse_dmap for this; nothing sets it yet.
No functional change.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- d->arch.sparse_dmap is now a one-bit field of d->arch.asi, beside
vcpu_pt.
- Wrap a line that reached 80 columns.
- Drop sparse_dmap_active: d->arch.asi.sparse_dmap alone says whether a
domain's root page-tables carry the sparse view (Roger). Builds
without CONFIG_SPARSE_DIRECTMAP compile the test out, as before.
- Drop the second directmap check after the vCPU lookup: the
!cache->inuse case covers it.
Changes in v3:
- New in this version.
---
xen/arch/x86/domain_page.c | 34 ++++++++++++++++++++++++++++---
xen/arch/x86/include/asm/domain.h | 3 +++
2 files changed, 34 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 770fee80a4..6b662706fb 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -56,6 +56,13 @@ static inline struct vcpu *mapcache_current_vcpu(void)
return ACCESS_ONCE(this_cpu(pgtable_vcpu));
}
+/* Whether @v's page-tables carry the sparse view of the directmap. */
+static bool on_sparse_dmap(const struct vcpu *v)
+{
+ return IS_ENABLED(CONFIG_SPARSE_DIRECTMAP) &&
+ v->domain->arch.asi.sparse_dmap;
+}
+
/*
* The accounting for v's mapcache: its own for a vCPU-PT domain, the domain's
* otherwise, in which case *dcache is set so the caller takes the domain-wide
@@ -135,9 +142,19 @@ void *map_domain_page(mfn_t mfn)
#ifdef NDEBUG
if ( mfn_x(mfn) <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
- return mfn_to_virt(mfn_x(mfn));
+ {
+ const struct vcpu *pt_vcpu = this_cpu(pgtable_vcpu);
+
+ /*
+ * The page-tables loaded map @mfn through the directmap, unless they
+ * carry the sparse view.
+ */
+ if ( !pt_vcpu || !on_sparse_dmap(pt_vcpu) )
+ return mfn_to_virt(mfn_x(mfn));
+ }
#endif
+ /* No vCPU (early boot), or no mapcache: the full directmap is loaded. */
v = mapcache_current_vcpu();
if ( !v )
return mfn_to_virt(mfn_x(mfn));
@@ -145,7 +162,10 @@ void *map_domain_page(mfn_t mfn)
cache = vcpu_mapcache(v, &dcache);
vcache = &v->arch.mapcache;
if ( !cache->inuse )
+ {
+ ASSERT(!on_sparse_dmap(v));
return mfn_to_virt(mfn_x(mfn));
+ }
perfc_incr(map_domain_page_count);
@@ -294,7 +314,11 @@ void mapcache_domain_init(struct domain *d)
ASSERT(is_pv_domain(d));
#ifdef NDEBUG
- if ( !mem_hotplug && max_page <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
+ /*
+ * Not needed when the directmap loaded for the domain covers everything.
+ */
+ if ( !d->arch.asi.sparse_dmap && !mem_hotplug &&
+ max_page <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
return;
#endif
@@ -378,8 +402,12 @@ void *map_domain_page_global(mfn_t mfn)
system_state < SYS_STATE_active) ||
local_irq_is_enabled()));
+ /*
+ * Global mappings must hold in every context: a sparse view of the
+ * directmap rules the directmap out.
+ */
#ifdef NDEBUG
- if ( mfn_x(mfn) <= PFN_DOWN(__pa(HYPERVISOR_VIRT_END - 1)) )
+ if ( arch_mfns_in_directmap(mfn_x(mfn), 1) )
return mfn_to_virt(mfn_x(mfn));
#endif
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 7076e38848..f84ac15759 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -490,6 +490,9 @@ struct arch_domain
struct {
/* Give each vCPU its own per-domain area (HVM only). */
bool vcpu_pt:1;
+
+ /* The domain's root page-tables carry the sparse directmap view. */
+ bool sparse_dmap:1;
} asi;
/* Emulated devices enabled bitmap. */
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 19/44] x86/domain_page: clear a mapcache slot at its last unmap
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (17 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 18/44] x86/domain_page: key the directmap fast paths on the loaded view George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 20/44] x86/mm: run guest contexts on the sparse directmap view George Dunlap
` (24 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
unmap_domain_page() keeps a recently unmapped slot in the vCPU's
maphash (MAPHASH_ENTRIES, 8 per vCPU) for a quick remap of the same
MFN, and leaves its PTE in place until the entry is displaced by a later
unmap or reclaimed by a full cache. Until then the page stays mapped,
although nothing refers to the mapping and the mapcache holds no
reference to the page. Where the mapcache clears a slot's PTE, it
defers the flush: the slot's translation can stay in the TLB of the CPU
which used it until that CPU next flushes.
While the full directmap is loaded, that reaches nothing it does not
map anyway. On the sparse view of the directmap, Xen reaches
domain-heap memory only through mappings made for the purpose. A
mapping or a translation left behind keeps the page reachable after its
last unmap: while Xen serves another of the guest's contexts on the
same vCPU, and, once the page is freed and handed to another domain,
from a context which has nothing to do with it.
So clear a slot's PTE when its last reference goes, maphash slots
included: the hash entry keeps the slot, not the mapping, and a later
hit maps the page again in the same slot. The slot has mapped nothing
else in between, so that needs no flush. Reusing the slot for another
page still goes through the garbage map, or through the replacement of
a hash entry when the cache is full, both of which flush first.
In a per-vCPU mapcache, as vCPU-PT domains have, also invalidate the
slot's translation on the local CPU, right after clearing the PTE. The
slot has a second address, in the mirror of the per-domain area that
every domain but a 64-bit PV one gets for the compat argument
translation area, and its translation is invalidated too: two INVLPGs.
That leaves no translation on any CPU, as the vCPU's page-tables are
loaded on one CPU at a time, and a CPU loading other page-tables
(switch_cr3_cr4()) flushes non-global translations, the mapcache's
included.
A PV domain's domain-wide mapcache has its PTEs cleared but no
INVLPG, as CPUs running the domain's other vCPUs may hold the slot's
translation too, which a local INVLPG would not reach. For it, as
before, builds with PV support flush a page freed by its owner from
TLBs before the page is handed out again.
The cost is one PTE write per unmap into the maphash and per remap from
it, and in a per-vCPU mapcache two INVLPGs per slot's last unmap. A
second unmap of an entry in the maphash now finds its PTE not present
and is caught by the check unmap_domain_page() has for entries the
first unmap zapped, rather than by the reference count's ASSERT() in
debug builds only.
The change does not depend on asi= or on a sparse view of the directmap
being configured: PV vCPUs use the domain-wide mapcache on the default
command line too. For them it adds the PTE writes above, and leaves up
to MAPHASH_ENTRIES fewer pages per vCPU mapped in the
domain-wide mapcache.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v4:
- Merge v3's 15/18 ("x86/mm: flush owned pages before reuse with the
sparse directmap view") and 16/18 ("x86/domain_page: keep the
maphash's slots, not their mappings"). Rather than flush pages freed
by their owner before reuse wherever a sparse view is configured,
invalidate a per-vCPU mapcache slot's translations on the local CPU
right after its PTE is cleared (Roger).
- Message: don't restate when PV vCPUs use the mapcache (Roger).
Changes in v3:
- New in this version.
For reviewers: with a slot cleared and invalidated at its last unmap,
a per-vCPU mapcache's maphash saves only the search for a free slot,
and a second slot for a page mapped twice at once. One page in two
slots is fine, and already happens when it is not in the maphash. So
per-vCPU mapcaches could drop the maphash, and with it their garbage
map and wrap-time flush; that is left for a follow-up.
---
xen/arch/x86/domain_page.c | 53 +++++++++++++++++++++++++------
xen/arch/x86/include/asm/domain.h | 5 ++-
2 files changed, 48 insertions(+), 10 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 6b662706fb..4198ee7c18 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -130,6 +130,26 @@ static void mapcache_exit(struct mapcache_domain *dcache)
#define MAPCACHE_L1ENT(idx) \
__linear_l1_table[l1_linear_offset(MAPCACHE_VIRT_START + pfn_to_paddr(idx))]
+/*
+ * Clear a slot's PTE. For a per-vCPU mapcache (HVM domains only), then also
+ * drop the slot's translations from this CPU's TLB, at its address and at its
+ * alias in the per-domain mirror, which every HVM domain has. No other CPU
+ * can hold them: the vCPU's page-tables are loaded on one CPU at a time, and
+ * loading other page-tables flushes non-global translations.
+ */
+static void mapcache_zap(const struct mapcache_domain *dcache,
+ unsigned int idx)
+{
+ unsigned long va = MAPCACHE_VIRT_START + pfn_to_paddr(idx);
+
+ l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
+ if ( dcache )
+ return;
+
+ flush_tlb_one_local(va);
+ flush_tlb_one_local(va - PERDOMAIN_VIRT_START + PERDOMAIN_ALT_VIRT_START);
+}
+
void *map_domain_page(mfn_t mfn)
{
unsigned long flags;
@@ -176,6 +196,14 @@ void *map_domain_page(mfn_t mfn)
{
idx = hashent->idx;
ASSERT(idx < cache->entries);
+ /*
+ * An entry the maphash kept holds on to its slot, not to its mapping
+ * (see unmap_domain_page()): map the page again. No flush is needed,
+ * as the slot has mapped nothing else since.
+ */
+ if ( !hashent->refcnt )
+ l1e_write(&MAPCACHE_L1ENT(idx),
+ l1e_from_mfn(mfn, __PAGE_HYPERVISOR_RW));
hashent->refcnt++;
ASSERT(hashent->refcnt);
ASSERT(mfn_eq(l1e_get_mfn(MAPCACHE_L1ENT(idx)), mfn));
@@ -208,8 +236,8 @@ void *map_domain_page(mfn_t mfn)
if ( hashent->idx != MAPHASHENT_NOTINUSE && !hashent->refcnt )
{
idx = hashent->idx;
- ASSERT(l1e_get_pfn(MAPCACHE_L1ENT(idx)) == hashent->mfn);
- l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
+ ASSERT(!(l1e_get_flags(MAPCACHE_L1ENT(idx)) &
+ _PAGE_PRESENT));
hashent->idx = MAPHASHENT_NOTINUSE;
hashent->mfn = ~0UL;
break;
@@ -273,32 +301,39 @@ void unmap_domain_page(const void *ptr)
local_irq_save(flags);
+ /*
+ * The maphash keeps a freed slot, not its mapping: the page may change
+ * hands meanwhile. A domain-wide mapcache's translation left in a TLB
+ * is covered by the flush the allocator makes before reusing an owned
+ * page.
+ */
if ( hashent->idx == idx )
{
ASSERT(hashent->mfn == mfn);
ASSERT(hashent->refcnt);
hashent->refcnt--;
+ if ( !hashent->refcnt )
+ mapcache_zap(dcache, idx);
}
else if ( !hashent->refcnt )
{
+ /* The entry replaced has no mapping left: mark its slot as garbage. */
if ( hashent->idx != MAPHASHENT_NOTINUSE )
{
- /* /First/, zap the PTE. */
- ASSERT(l1e_get_pfn(MAPCACHE_L1ENT(hashent->idx)) ==
- hashent->mfn);
- l1e_write(&MAPCACHE_L1ENT(hashent->idx), l1e_empty());
- /* /Second/, mark as garbage. */
+ ASSERT(!(l1e_get_flags(MAPCACHE_L1ENT(hashent->idx)) &
+ _PAGE_PRESENT));
set_bit(hashent->idx, cache->garbage);
}
- /* Add newly-freed mapping to the maphash. */
+ /* Add the newly-freed slot to the maphash, without its mapping. */
+ mapcache_zap(dcache, idx);
hashent->mfn = mfn;
hashent->idx = idx;
}
else
{
/* /First/, zap the PTE. */
- l1e_write(&MAPCACHE_L1ENT(idx), l1e_empty());
+ mapcache_zap(dcache, idx);
/* /Second/, mark as garbage. */
set_bit(idx, cache->garbage);
}
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index f84ac15759..0e405157b4 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -74,7 +74,10 @@ struct mapcache_vcpu {
/* Shadow of mapcache_domain.epoch. */
unsigned int shadow_epoch;
- /* Lock-free per-VCPU hash of recently-used mappings. */
+ /*
+ * Lock-free per-VCPU hash of recently-used mappings. An entry without
+ * references keeps its slot but not its mapping (see unmap_domain_page()).
+ */
struct vcpu_maphash_entry {
unsigned long mfn;
uint32_t idx;
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 20/44] x86/mm: run guest contexts on the sparse directmap view
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (18 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 19/44] x86/domain_page: clear a mapcache slot at its last unmap George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 21/44] x86/spec-ctrl: wire the sparse directmap view into the asi= option George Dunlap
` (23 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
In _init_xen_l4_slots(), if d->arch.asi.sparse_dmap is set, point the
directmap slots at the sparse view, copying from idle_pg_table only
the Xen slots below them. A root page-table not shared with a PV
guest gets no extended directmap, which the sparse view does not
cover.
d->arch.asi.sparse_dmap is decided at domain creation: PV domains all
have a mapcache, HVM domains need per-vCPU page-tables for one. No
option asks for the sparse view yet.
The sparse view is built before any domain is created, so the root
page-tables of every domain using it carry it from their creation,
dom0's included. Dom0's builders reach dom0's memory only through guest
virtual addresses and map_domain_page(). In PV shim mode,
replace_va_mapping() reads the guest's L4 through the directmap; map it
with map_domain_page() instead, as the lower levels already are. Shadow
L4s, dom0=shadow's included, get the sparse view from
init_xen_l4_slots_pv().
The two directmaps differ where the sparse view leaves memory out, so no
translation of the full directmap may remain usable on page-tables
carrying the sparse view. Directmap mappings stay global nevertheless:
every CR3 write which can move a CPU onto the sparse view goes through
switch_cr3_cr4(), which flushes the old page-tables' translations,
global entries included, before it returns. The CR3 writes which keep
entries -- a PV vCPU's switch between its kernel and user page-tables,
XPTI's entry and exit -- stay within one vCPU's page-tables, which carry
the same directmap, or go to XPTI's per-CPU root, which maps only Xen's
own per-CPU pages there. Removals from the sparse view are flushed on
every CPU, global entries included. PV guests that run with global
pages, among them a PV dom0 not using the sparse view, therefore keep
their directmap TLB entries across switches between guest kernel and
user mode.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v6:
- Port onto new dual-init_xen_l4_slots system.
Changes in v4:
- d->arch.vcpu_pt and d->arch.sparse_dmap are now d->arch.asi.vcpu_pt
and d->arch.asi.sparse_dmap.
- Drop sparse_dmap_active and sparse_dmap_activate(): root page-tables
carry the sparse view from their creation, dom0's included (Roger).
- The sparse view's slots are now marked NX in "x86/mm: build and
maintain a sparse view of the directmap".
- PV shim: replace_va_mapping() maps the guest's L4 with
map_domain_page().
- Give sparse_dmap_wanted() its hardware-domain case (Roger).
- init_xen_l4_slots() takes the directmap slots from the sparse view,
rather than copying the full directmap's and overwriting them;
sparse_dmap_install() goes (Roger).
- Message: drop the sentence on NX (Roger).
Changes in v3:
- New in this version.
---
xen/arch/x86/include/asm/sparse-directmap.h | 9 +++++++
xen/arch/x86/mm.c | 28 +++++++++++++++++++--
xen/arch/x86/pv/shim.c | 4 ++-
xen/arch/x86/spec_ctrl.c | 7 ++++++
4 files changed, 45 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
index ea14d44334..2f8b94f4c1 100644
--- a/xen/arch/x86/include/asm/sparse-directmap.h
+++ b/xen/arch/x86/include/asm/sparse-directmap.h
@@ -31,6 +31,14 @@ static inline bool sparse_dmap_configured(void)
return opt_sparse_dmap_pv || opt_sparse_dmap_hvm || opt_sparse_dmap_hwdom;
}
+/* Whether a new domain of the given type is to use the sparse view. */
+static inline bool sparse_dmap_wanted(bool pv, bool hwdom)
+{
+ return sparse_dmap_root &&
+ (hwdom ? opt_sparse_dmap_hwdom
+ : pv ? opt_sparse_dmap_pv : opt_sparse_dmap_hvm);
+}
+
void sparse_dmap_init(void);
int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
pte_attr_t flags);
@@ -42,6 +50,7 @@ int sparse_dmap_drop_pagetable(mfn_t mfn);
#else /* !CONFIG_SPARSE_DIRECTMAP */
static inline bool sparse_dmap_configured(void) { return false; }
+static inline bool sparse_dmap_wanted(bool pv, bool hwdom) { return false; }
static inline void sparse_dmap_init(void) {}
static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
unsigned long nr, pte_attr_t flags)
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 0d475432ff..36a241a41c 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -1704,7 +1704,8 @@ struct page_info *perdomain_l3(const struct vcpu *v)
* l4t, l4mfn, and d are mandatory, but l4mfn doesn't need to be the mfn under
* *l4t. All other parameters are optional and will either fill or zero the
* appropriate slots. Pagetables not shared with guests will gain the
- * extended directmap.
+ * extended directmap, unless the domain runs on the sparse view of the
+ * directmap, which replaces the directmap slots.
*/
static void _init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
const struct domain *d,
@@ -1713,7 +1714,7 @@ static void _init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
{
/*
* PV vcpus need a shortened directmap. HVM and Idle vcpus get the full
- * directmap.
+ * directmap, unless the domain runs on the sparse view (below).
*/
bool short_directmap = !paging_mode_external(d);
@@ -1748,6 +1749,29 @@ static void _init_xen_l4_slots(l4_pgentry_t *l4t, mfn_t l4mfn,
l4t[l4_table_offset(PERDOMAIN_VIRT_START)];
/* Slot 261-: text/data/bss, RW M2P, vmap, frametable, directmap. */
+#ifdef CONFIG_SPARSE_DIRECTMAP
+ if ( d->arch.asi.sparse_dmap )
+ {
+ unsigned int i = l4_table_offset(SPARSE_DMAP_START);
+
+ memcpy(&l4t[l4_table_offset(XEN_VIRT_START)],
+ &idle_pg_table[l4_table_offset(XEN_VIRT_START)],
+ (i - l4_table_offset(XEN_VIRT_START)) * sizeof(*l4t));
+
+ /*
+ * Every slot the sparse view covers gets its L3, whatever the full
+ * directmap has there, so that memory added later is covered too. A
+ * PV root has guest slots above; any other root loses the rest of the
+ * directmap, which the sparse view does not cover.
+ */
+ for ( ; i < l4_table_offset(SPARSE_DMAP_END); i++ )
+ l4t[i] = sparse_dmap_root[i];
+ if ( !short_directmap )
+ for ( ; i < l4_table_offset(DIRECTMAP_VIRT_END); i++ )
+ l4t[i] = l4e_empty();
+ }
+ else
+#endif
#ifndef NDEBUG
if ( short_directmap &&
unlikely(root_pgt_pv_xen_slots < ROOT_PAGETABLE_PV_XEN_SLOTS) )
diff --git a/xen/arch/x86/pv/shim.c b/xen/arch/x86/pv/shim.c
index 41000b0523..2a47dbc56c 100644
--- a/xen/arch/x86/pv/shim.c
+++ b/xen/arch/x86/pv/shim.c
@@ -148,12 +148,14 @@ const struct platform_bad_page *__init pv_shim_reserved_pages(unsigned int *size
static void __init replace_va_mapping(struct domain *d, l4_pgentry_t *l4start,
unsigned long va, mfn_t mfn)
{
- l4_pgentry_t l4e = l4start[l4_table_offset(va)];
+ const l4_pgentry_t *l4t = map_domain_page(virt_to_mfn(l4start));
+ l4_pgentry_t l4e = l4t[l4_table_offset(va)];
l3_pgentry_t l3e = l3e_from_l4e(l4e, l3_table_offset(va));
l2_pgentry_t l2e = l2e_from_l3e(l3e, l2_table_offset(va));
l1_pgentry_t *pl1e = map_l1t_from_l2e(l2e) + l1_table_offset(va);
struct page_info *page = mfn_to_page(l1e_get_mfn(*pl1e));
+ unmap_domain_page(l4t);
put_page_and_type(page);
*pl1e = l1e_from_mfn(mfn, (!is_pv_32bit_domain(d) ? L1_PROT
diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c
index 8797523a62..bcdcdf0476 100644
--- a/xen/arch/x86/spec_ctrl.c
+++ b/xen/arch/x86/spec_ctrl.c
@@ -1953,6 +1953,13 @@ void spec_ctrl_init_domain(struct domain *d)
d->arch.asi.vcpu_pt = !pv && (is_hardware_domain(d) ? opt_vcpu_pt_hwdom
: opt_vcpu_pt_hvm);
+
+ /*
+ * The sparse directmap view needs a mapcache: PV domains all have one,
+ * HVM domains through per-vCPU page-tables.
+ */
+ d->arch.asi.sparse_dmap = (pv || d->arch.asi.vcpu_pt) &&
+ sparse_dmap_wanted(pv, is_hardware_domain(d));
}
void __init init_speculation_mitigations(void)
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 21/44] x86/spec-ctrl: wire the sparse directmap view into the asi= option
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (19 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 20/44] x86/mm: run guest contexts on the sparse directmap view George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 22/44] xen/xmalloc: return an emptied region after dropping the pool lock George Dunlap
` (22 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Andrew Cooper, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné, Stefano Stabellini,
Teddy Astie
Let asi= choose which domains run on the sparse view of the directmap,
when the hypervisor is built with CONFIG_SPARSE_DIRECTMAP:
* `asi` and `asi=<bool>` enable or disable the sparse view, along with
vCPU-PT, for every domain including the hardware domain, PV or PVH.
HVM domains, a PVH hardware domain included, use the sparse view
together with vCPU-PT, which gives them the mapcache it needs; PV
domains all have one. A PV domain on the sparse view keeps its
domain-wide mapcache and per-domain mappings: the sparse view
separates it from other domains, not its vCPUs from each other.
* `asi=hvm=<bool>`, `asi=pv=<bool>` and `asi=dom0=<bool>` do the same
for HVM or PV domains other than the hardware domain, or for the
hardware domain, whatever its type: `asi=1,dom0=0` leaves the
hardware domain out of ASI altogether.
* A new `asi=directmap=<bool>` selects the sparse view for all domains,
as `asi=vcpu-pt=<bool>` selects vCPU-PT; `vcpu-pt=` alone leaves the
sparse view off. On a hypervisor built without
CONFIG_SPARSE_DIRECTMAP the forms enabling all features leave the
sparse view off, and `directmap=` is rejected as invalid, as
compiled-out `spec-ctrl=` sub-options are. A domain type the build
lacks is ignored, so that the sparse view is not built for nothing.
* For now, a PV domain uses one of XPTI and the sparse view, not both.
Enabled explicitly, for dom0, domUs or both, XPTI takes precedence:
the PV domains it covers keep the full directmap, and the boot log
says so. Left to its default, XPTI yields: it is disabled for the PV
domains using the sparse view, with a boot-log warning where it would
otherwise be enabled, as the sparse view does not replace it yet (it
still maps the xenheap and Xen's stacks). Each domain's case is
settled at its creation, and again if it becomes 32-bit, which XPTI
does not cover: switch_compat() then puts a domain that XPTI kept off
the sparse view on it, with the domain-wide mapcache, which release
builds may not have set up for it.
The boot log gains a line with the directmap mode, "sparse" once a
sparse view is configured and "full" otherwise, and the "ASI features
for ..." lines, one for PV domUs now among them, say which domains use
the sparse view.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- Follow "x86/spec-ctrl: introduce Address Space Isolation command line
option": opt_vcpu_pt_hwdom's name, the development warning in
parse_asi(), IS_ENABLED() in the boot log; pv= now replaces the line
saying it is ignored.
Changes in v4:
- Docs, comments and message: "domains" throughout, not "guests"
(Roger).
- Follow the rename of opt_vcpu_pt_hwdom to opt_pvh_vcpu_pt_hwdom and
the move of the per-domain settings into d->arch.asi.
- pv= and dom0= move to the patch introducing asi=, beside hvm=;
opt_sparse_dmap_hwdom and sparse_dmap_wanted()'s hardware-domain case
to the patches that add the other knobs and sparse_dmap_wanted()
(Roger).
- directmap= on a build without CONFIG_SPARSE_DIRECTMAP is rejected as
invalid, as compiled-out spec-ctrl= sub-options are, rather than
ignored with a warning (Roger).
- A PV domain that XPTI kept off the sparse view goes on it when it
becomes 32-bit, which ends XPTI; it used to end up with neither.
Changes in v3:
- New in this version.
---
docs/misc/xen-command-line.pandoc | 30 +++++-
xen/arch/x86/include/asm/sparse-directmap.h | 7 ++
xen/arch/x86/pv/domain.c | 16 +++
xen/arch/x86/spec_ctrl.c | 102 +++++++++++++++++---
4 files changed, 138 insertions(+), 17 deletions(-)
diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
index 839d06da26..092c858f34 100644
--- a/docs/misc/xen-command-line.pandoc
+++ b/docs/misc/xen-command-line.pandoc
@@ -203,7 +203,7 @@ to appropriate auditing by Xen. Argo is disabled by default.
buggy or malicious other domain spamming the ring.
### asi (x86)
-> `= List of [ <bool>, {pv,hvm,dom0}=<bool>,
+> `= List of [ <bool>, {pv,hvm,dom0}=<bool>, directmap=<bool>,
> vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
> Default: `false`
@@ -215,14 +215,16 @@ mitigations for certain speculative related attacks, at the cost of mapping
sensitive information on-demand. It might also offer some protection against
unmitigated speculation-related attacks.
-The mechanisms currently implemented apply to HVM domains, including a PVH
-hardware domain. PV domains, including a PV hardware domain, are unaffected.
+The mechanisms currently implemented are `vcpu-pt`, for HVM domains including
+a PVH hardware domain, and `directmap`, for all domains including the hardware
+domain, PV or PVH. The plain boolean enables or disables both, wherever they
+apply.
* `pv=`, `hvm=` and `dom0=` enable the features for PV domains or HVM domains
other than the hardware domain, or for the hardware domain, PV or PVH. The
plain boolean does all three: `asi=1,dom0=0` leaves the hardware domain
- out. The forms naming no domain type (an un-suffixed `vcpu-pt=<bool>`)
- cover the hardware domain too.
+ out. The forms naming no domain type (an un-suffixed `vcpu-pt=<bool>`, and
+ `directmap=<bool>`) cover the hardware domain too.
**WARNING: manual de-selection of enabled options will invalidate any
protection offered by the feature. The fine grained options provided below
@@ -231,6 +233,18 @@ are meant to be used for debugging purposes only.**
* `vcpu-pt` gives each vCPU its own per-domain area: the per-domain slot of
each vCPU's monitor table maps a region private to that vCPU.
+* `directmap` (only available when Xen is built with
+ `CONFIG_SPARSE_DIRECTMAP`) runs guest contexts on a sparse view of the
+ directmap, which maps the memory Xen needs to reach at all times (the Xen
+ heap, Xen's own image, firmware tables), while guest memory is mapped
+ transiently. This selection is system-wide; `pv=`, `hvm=` and `dom0=`
+ make it for one domain type, along with the other features. HVM domains use
+ the sparse view only together with `vcpu-pt`, and PV domains only without
+ XPTI, which yields to the sparse view unless enabled explicitly (see
+ `xpti`). The whole-feature forms (`asi`, `asi=<bool>`, and `pv=`, `hvm=`
+ and `dom0=`) set this too, on builds that support it; on other builds they
+ leave it off.
+
### asid (x86)
> `= <boolean>`
@@ -3132,6 +3146,12 @@ Meltdown for all domains.
With `dom0` and `domu` it is possible to control page table isolation
for dom0 or guest domains only.
+With `asi`, XPTI and the sparse directmap view (`asi=directmap`) exclude each
+other for a PV domain. Enabled explicitly, as a whole or with `dom0` or
+`domu`, XPTI keeps the PV domains it covers off the sparse view, and the boot
+log says so. Left to its default, XPTI is disabled for the PV domains using
+the sparse view, with a boot-log warning where it would otherwise be enabled.
+
### xsave (x86)
> `= <boolean>`
diff --git a/xen/arch/x86/include/asm/sparse-directmap.h b/xen/arch/x86/include/asm/sparse-directmap.h
index 2f8b94f4c1..db4df82215 100644
--- a/xen/arch/x86/include/asm/sparse-directmap.h
+++ b/xen/arch/x86/include/asm/sparse-directmap.h
@@ -31,6 +31,10 @@ static inline bool sparse_dmap_configured(void)
return opt_sparse_dmap_pv || opt_sparse_dmap_hvm || opt_sparse_dmap_hwdom;
}
+static inline bool sparse_dmap_pv(void) { return opt_sparse_dmap_pv; }
+static inline bool sparse_dmap_hvm(void) { return opt_sparse_dmap_hvm; }
+static inline bool sparse_dmap_hwdom(void) { return opt_sparse_dmap_hwdom; }
+
/* Whether a new domain of the given type is to use the sparse view. */
static inline bool sparse_dmap_wanted(bool pv, bool hwdom)
{
@@ -51,6 +55,9 @@ int sparse_dmap_drop_pagetable(mfn_t mfn);
static inline bool sparse_dmap_configured(void) { return false; }
static inline bool sparse_dmap_wanted(bool pv, bool hwdom) { return false; }
+static inline bool sparse_dmap_pv(void) { return false; }
+static inline bool sparse_dmap_hvm(void) { return false; }
+static inline bool sparse_dmap_hwdom(void) { return false; }
static inline void sparse_dmap_init(void) {}
static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
unsigned long nr, pte_attr_t flags)
diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c
index f139d8f83c..579d6133f5 100644
--- a/xen/arch/x86/pv/domain.c
+++ b/xen/arch/x86/pv/domain.c
@@ -248,6 +248,20 @@ int switch_compat(struct domain *d)
d->arch.has_32bit_shinfo = 1;
d->arch.pv.is_32bit = true;
+ /*
+ * XPTI, cleared below, is for 64-bit domains. A domain it kept off the
+ * sparse directmap view goes on it now if wanted, and the domain-wide
+ * mapcache the view needs is set up for the domain and its vCPUs.
+ */
+ if ( d->arch.pv.xpti && sparse_dmap_wanted(true, is_hardware_domain(d)) )
+ {
+ d->arch.asi.sparse_dmap = true;
+ mapcache_domain_init(d);
+ for_each_vcpu ( d, v )
+ if ( (rc = mapcache_vcpu_init(v)) )
+ goto undo_and_fail;
+ }
+
/*
* For 32bit PV guests the shared_info machine address must fit in a 32-bit
* field within the guest's start_info structure. We might need to free
@@ -301,6 +315,8 @@ int switch_compat(struct domain *d)
undo_and_fail:
d->arch.pv.is_32bit = d->arch.has_32bit_shinfo = false;
+ if ( d->arch.pv.xpti )
+ d->arch.asi.sparse_dmap = false;
for_each_vcpu( d, v )
{
free_compat_arg_xlat(v);
diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c
index bcdcdf0476..c86f5ff886 100644
--- a/xen/arch/x86/spec_ctrl.c
+++ b/xen/arch/x86/spec_ctrl.c
@@ -387,6 +387,19 @@ int8_t __ro_after_init opt_xpti_domu = -1;
static __init void xpti_init_default(void)
{
+ /*
+ * XPTI and the sparse directmap view exclude each other for a PV domain
+ * (see spec_ctrl_init_domain()). Enabled explicitly, XPTI takes
+ * precedence: the PV domains it covers stay off the sparse view. Left to
+ * its default, it yields: it is off where the sparse view is asked for.
+ */
+ if ( !opt_dom0_pvh && opt_xpti_hwdom == 1 && sparse_dmap_hwdom() )
+ printk(XENLOG_ERR
+ "XPTI enabled, disabling Dom0 sparse-directmap\n");
+ if ( opt_xpti_domu == 1 && sparse_dmap_pv() )
+ printk(XENLOG_ERR
+ "XPTI enabled, disabling PV DomU sparse-directmap\n");
+
if ( (boot_cpu_data.vendor & (X86_VENDOR_AMD | X86_VENDOR_HYGON)) ||
cpu_has_rdcl_no )
{
@@ -397,10 +410,21 @@ static __init void xpti_init_default(void)
}
else
{
+ bool yield_hwdom = opt_xpti_hwdom < 0 && !opt_dom0_pvh &&
+ sparse_dmap_hwdom();
+ bool yield_domu = opt_xpti_domu < 0 && sparse_dmap_pv();
+
+ /* The default would be on here: say where it yields. */
+ if ( yield_hwdom || yield_domu )
+ printk(XENLOG_WARNING
+ "XPTI defaults off for PV %s on the sparse directmap\n",
+ yield_hwdom ? (yield_domu ? "Dom0 and DomU" : "Dom0")
+ : "DomU");
+
if ( opt_xpti_hwdom < 0 )
- opt_xpti_hwdom = 1;
+ opt_xpti_hwdom = !sparse_dmap_hwdom();
if ( opt_xpti_domu < 0 )
- opt_xpti_domu = 1;
+ opt_xpti_domu = !sparse_dmap_pv();
}
}
@@ -491,6 +515,23 @@ static int __init cf_check parse_pv_l1tf(const char *s)
}
custom_param("pv-l1tf", parse_pv_l1tf);
+/*
+ * Set the sparse directmap view's knob for each domain type given. A type
+ * the build lacks stays off, so that no view is built for it, and without
+ * CONFIG_SPARSE_DIRECTMAP all stay off.
+ */
+static void __init set_sparse_dmap(bool pv, bool hvm, bool hwdom, bool val)
+{
+#ifdef CONFIG_SPARSE_DIRECTMAP
+ if ( pv && IS_ENABLED(CONFIG_PV) )
+ opt_sparse_dmap_pv = val;
+ if ( hvm && IS_ENABLED(CONFIG_HVM) )
+ opt_sparse_dmap_hvm = val;
+ if ( hwdom )
+ opt_sparse_dmap_hwdom = val;
+#endif
+}
+
static int __init cf_check parse_asi(const char *s)
{
static bool __initdata warned;
@@ -507,7 +548,10 @@ static int __init cf_check parse_asi(const char *s)
/* Interpret 'asi' alone in its positive boolean form. */
if ( *s == '\0' )
+ {
opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = true;
+ set_sparse_dmap(true, true, true, true);
+ }
do {
ss = strchr(s, ',');
@@ -520,17 +564,33 @@ static int __init cf_check parse_asi(const char *s)
case 0:
case 1:
opt_vcpu_pt_hwdom = opt_vcpu_pt_hvm = val;
+ set_sparse_dmap(true, true, true, val);
break;
default:
if ( (val = parse_boolean("pv", s, ss)) >= 0 )
- printk(XENLOG_WARNING
- "No ASI feature applies to PV domains - ignoring 'asi=%.*s' setting\n",
- (int)(ss - s), s);
+ set_sparse_dmap(true, false, false, val);
else if ( (val = parse_boolean("hvm", s, ss)) >= 0 )
+ {
opt_vcpu_pt_hvm = val;
+ set_sparse_dmap(false, true, false, val);
+ }
else if ( (val = parse_boolean("dom0", s, ss)) >= 0 )
+ {
opt_vcpu_pt_hwdom = val;
+ set_sparse_dmap(false, false, true, val);
+ }
+ else if ( (val = parse_boolean("directmap", s, ss)) >= 0 )
+ {
+ /* System-wide: every domain type, the hardware domain too. */
+ if ( IS_ENABLED(CONFIG_SPARSE_DIRECTMAP) )
+ set_sparse_dmap(true, true, true, val);
+ else
+ {
+ no_config_param("SPARSE_DIRECTMAP", "asi", s, ss);
+ rc = -EINVAL;
+ }
+ }
else if ( (val = parse_boolean("vcpu-pt", s, ss)) != -1 )
{
switch ( val )
@@ -759,11 +819,28 @@ static void __init print_details(enum ind_thunk thunk)
opt_pv_l1tf_domu ? "enabled" : "disabled");
#endif
- printk(" ASI features for Dom0:%s\n",
- opt_dom0_pvh && opt_vcpu_pt_hwdom ? " vCPU-PT" : " None");
- if ( IS_ENABLED(CONFIG_HVM) )
- printk(" ASI features for HVM VMs:%s\n",
- opt_vcpu_pt_hvm ? " vCPU-PT" : " None");
+ {
+ bool hwdom_pt = opt_dom0_pvh && opt_vcpu_pt_hwdom;
+ bool hwdom_sd = sparse_dmap_hwdom() &&
+ (opt_dom0_pvh ? hwdom_pt : !opt_xpti_hwdom);
+
+ printk(" ASI directmap mode: %s\n",
+ sparse_dmap_configured() ? "sparse" : "full");
+ printk(" ASI features for Dom0:%s%s%s\n",
+ hwdom_pt || hwdom_sd ? "" : " None",
+ hwdom_pt ? " vCPU-PT" : "",
+ hwdom_sd ? " sparse-directmap" : "");
+ if ( IS_ENABLED(CONFIG_HVM) )
+ printk(" ASI features for HVM VMs:%s%s%s\n",
+ opt_vcpu_pt_hvm ? "" : " None",
+ opt_vcpu_pt_hvm ? " vCPU-PT" : "",
+ opt_vcpu_pt_hvm && sparse_dmap_hvm()
+ ? " sparse-directmap" : "");
+ if ( IS_ENABLED(CONFIG_PV) )
+ printk(" ASI features for PV VMs:%s\n",
+ sparse_dmap_pv() && !opt_xpti_domu
+ ? " sparse-directmap" : " None");
+ }
}
static bool __init check_smt_enabled(void)
@@ -1956,9 +2033,10 @@ void spec_ctrl_init_domain(struct domain *d)
/*
* The sparse directmap view needs a mapcache: PV domains all have one,
- * HVM domains through per-vCPU page-tables.
+ * HVM domains through per-vCPU page-tables. A PV domain using XPTI
+ * stays on the full directmap.
*/
- d->arch.asi.sparse_dmap = (pv || d->arch.asi.vcpu_pt) &&
+ d->arch.asi.sparse_dmap = (pv ? !d->arch.pv.xpti : d->arch.asi.vcpu_pt) &&
sparse_dmap_wanted(pv, is_hardware_domain(d));
}
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 22/44] xen/xmalloc: return an emptied region after dropping the pool lock
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (20 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 21/44] x86/spec-ctrl: wire the sparse directmap view into the asi= option George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-18 12:05 ` Jan Beulich
2026-11-03 22:13 ` [PATCH v6 23/44] x86/mm: split a sparse directmap superpage with a page being freed George Dunlap
` (21 subsequent siblings)
43 siblings, 1 reply; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Andrew Cooper, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné, Stefano Stabellini
When a free leaves a whole region of an xmalloc pool empty,
xmem_pool_free() hands the region back through pool->put_mem() with
pool->lock still held. For the xenheap pool that is
free_xenheap_pages(), which takes the heap lock inside the pool lock
and runs arch_xenheap_pages_free(). On x86 with a sparse directmap
view configured, the latter takes the page out of the sparse view,
at the cost of one to three rounds of TLB-flush IPIs to every CPU,
while every other CPU's sub-page xmalloc() and xfree() waits on the
pool lock.
By then the region is on no free list and no longer counted in the
pool, so nothing else can reach it. Drop the lock first and return the
region afterwards, as xmem_pool_alloc() already does when it calls
pool->get_mem(), and say at both callbacks that they are called without
the pool lock, so that they keep not relying on it.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
xen/common/xmalloc_tlsf.c | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/xen/common/xmalloc_tlsf.c b/xen/common/xmalloc_tlsf.c
index b43219b4af..caf33f1a04 100644
--- a/xen/common/xmalloc_tlsf.c
+++ b/xen/common/xmalloc_tlsf.c
@@ -106,7 +106,10 @@ struct xmem_pool {
unsigned long used_size;
unsigned long num_regions;
- /* User provided functions for expanding/shrinking pool */
+ /*
+ * User provided functions for expanding/shrinking pool. They are
+ * called without the pool lock held, and must not rely on it.
+ */
xmem_pool_get_memory *get_mem;
xmem_pool_put_memory *put_mem;
@@ -488,17 +491,18 @@ void xmem_pool_free(void *ptr, struct xmem_pool *pool)
if ( (b->prev_hdr == NULL) && ((tmp_b->size & BLOCK_SIZE_MASK) == 0) )
{
- pool->put_mem(b);
pool->num_regions--;
pool->used_size -= BHDR_OVERHEAD; /* sentinel block header */
- goto out;
+ spin_unlock(&pool->lock);
+ pool->put_mem(b);
+ return;
}
INSERT_BLOCK(b, pool, fl, sl);
tmp_b->size |= PREV_FREE;
tmp_b->prev_hdr = b;
- out:
+
spin_unlock(&pool->lock);
}
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 23/44] x86/mm: split a sparse directmap superpage with a page being freed
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (21 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 22/44] xen/xmalloc: return an emptied region after dropping the pool lock George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 24/44] x86/mm: introduce populate_perdomain_mapping() George Dunlap
` (20 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie,
Anthony PERARD, Michal Orzel, Julien Grall, Stefano Stabellini
A xenheap free takes its pages out of the sparse directmap view. Where
a superpage covers them, the superpage has to be split, and
modify_mappings_in() allocates the page-table page for that. The sparse
view has such superpages wherever a xenheap block of 2MiB or more was
mapped whole, and wherever map_pages_in() merged the entries of a 2MiB
region into one, as it does in the full directmap, once all 512 pages
were mapped. The allocation can fail once the hardware domain exists,
and the free then keeps its pages out of the heap for good, with a
warning.
Use one of the pages being freed instead. The sparse view's free hook
makes the first of them what alloc_xen_pagetable() would return, and
passes it to modify_mappings_in(), which uses it, should a split need a
page, before allocating one. The new L1 maps the rest of the superpage,
and the page's own entry is cleared along with the others being freed.
The hook tells free_xenheap_pages() whether it kept the page, and
free_xenheap_pages() then frees the rest of the block as blocks of
orders 0 to order - 1. The page goes back to the heap once the L1 has
emptied, as any page-table page of the sparse view does.
One page is enough if the sparse view has no 1GiB pages: a block smaller
than 2MiB lies within one 2MiB page and needs one L1, and a larger block
covers whole 2MiB pages and needs none, where a block within a 1GiB page
could need an L2 as well. So map_pages_in() makes 1GiB pages, by
mapping or by merging, only in idle_pg_table's hierarchy; the sparse
view keeps its 2MiB pages, mapped and merged. A xenheap free then
allocates nothing in the sparse view.
The costs: a 2MiB region split this way can merge again only once it has
emptied, as its L1 is one of its own pages; where the sparse view maps a
whole 1GiB region, which takes memory never handed to the heap and
xenheap memory filling it, contexts on it use 2MiB pages instead of one
1GiB page; and arch_xenheap_pages_free() gains a parameter. The other
removals from the sparse view (memory handed to the heap,
idle_pg_table's boot-time page-tables, and the undo of a failed xenheap
allocation) still allocate when they split a superpage.
The sparse view is correct without this, as a free that cannot split its
superpage fails safe. It is kept apart from "x86/mm: build and maintain
a sparse view of the directmap" so that it can be weighed on its own.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New patch
This is a fairly complicated patch to handle a very niche corner case:
- All the 4k pages in a superpage happen to have been allocated into
the xenheap (and the page table automatically merged into a superpage)
- The system is completely out of singleton pages to allocate as pagetables
- A free_xenheap_page() is called on a subset of those pages
Without this patch, the freed page is leaked; with this patch, one of
the freed pages is used as a pagetable page instead.
Happy to drop this patch, or move it further to the end of the series,
if it helps the main bit of functionality get in.
---
xen/arch/x86/include/asm/mm.h | 2 +-
xen/arch/x86/mm.c | 34 ++++++++++++++++++++++------
xen/arch/x86/sparse-directmap.c | 40 ++++++++++++++++++++++++++-------
xen/common/page_alloc.c | 14 ++++++++----
xen/include/xen/mm.h | 11 ++++++---
5 files changed, 78 insertions(+), 23 deletions(-)
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 113ade0bbd..50ace61ce7 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -603,7 +603,7 @@ void *alloc_mapped_pagetable(mfn_t *pmfn);
int map_pages_in(l4_pgentry_t *root, unsigned long virt, mfn_t mfn,
unsigned long nr_mfns, pte_attr_t flags);
int modify_mappings_in(l4_pgentry_t *root, unsigned long s, unsigned long e,
- pte_attr_t nf);
+ pte_attr_t nf, mfn_t *spare);
#ifdef CONFIG_SPARSE_DIRECTMAP
mfn_t xen_mapping_lookup(l4_pgentry_t *root, unsigned long va,
pte_attr_t *flags, unsigned long *nr);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 36a241a41c..e1a80e9a5c 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -5558,6 +5558,10 @@ static void free_pagetable_in(const l4_pgentry_t *root, mfn_t mfn)
/*
* map_pages_to_xen() in the hierarchy rooted at @root: idle_pg_table's, or
* another hierarchy sharing its virtual addresses.
+ *
+ * Only idle_pg_table's hierarchy gets 1GiB pages: a xenheap free splits a
+ * superpage of the sparse directmap view with one of the pages it frees,
+ * which is enough for a 2MiB page only.
*/
int map_pages_in(
l4_pgentry_t *root,
@@ -5567,6 +5571,7 @@ int map_pages_in(
pte_attr_t flags)
{
bool locking = system_state > SYS_STATE_boot;
+ bool page1gb = cpu_has_page1gb && root == idle_pg_table;
l3_pgentry_t *pl3e = NULL, ol3e;
l2_pgentry_t *pl2e = NULL, ol2e;
l1_pgentry_t *pl1e, ol1e;
@@ -5619,7 +5624,7 @@ int map_pages_in(
L3T_LOCK(current_l3page);
ol3e = *pl3e;
- if ( cpu_has_page1gb &&
+ if ( page1gb &&
IS_L3E_ALIGNED(virt, flags & _PAGE_PRESENT ? mfn : _mfn(0)) &&
nr_mfns >= (1UL << (L3_PAGETABLE_SHIFT - PAGE_SHIFT)) &&
!(flags & (_PAGE_PAT | MAP_SMALL_PAGES)) )
@@ -5905,7 +5910,7 @@ int map_pages_in(
}
check_l3:
- if ( cpu_has_page1gb &&
+ if ( page1gb &&
(flags == PAGE_HYPERVISOR) &&
((nr_mfns == 0) || IS_L3E_ALIGNED(virt, mfn)) )
{
@@ -5984,6 +5989,18 @@ int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
return map_pages_to_xen(virt, INVALID_MFN, nr_mfns, MAP_SMALL_PAGES);
}
+static mfn_t take_or_alloc_pagetable(mfn_t *spare)
+{
+ mfn_t mfn = spare ? *spare : INVALID_MFN;
+
+ if ( mfn_eq(mfn, INVALID_MFN) )
+ return alloc_xen_pagetable();
+
+ *spare = INVALID_MFN;
+
+ return mfn;
+}
+
/*
* Alter the permissions of a range of Xen virtual address space.
*
@@ -5997,10 +6014,13 @@ int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
* It is an error to call with present flags over an unpopulated range.
*
* modify_mappings_in() does so in the hierarchy rooted at @root, as
- * map_pages_in() does.
+ * map_pages_in() does. @spare, if not NULL, may hold a page the caller
+ * gives up for a page-table page, which shattering a superpage then uses
+ * before allocating one. *@spare reads INVALID_MFN once it has been used,
+ * and the page is then no longer the caller's.
*/
int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
- unsigned long e, pte_attr_t nf)
+ unsigned long e, pte_attr_t nf, mfn_t *spare)
{
bool locking = system_state > SYS_STATE_boot;
l3_pgentry_t *pl3e = NULL;
@@ -6064,7 +6084,7 @@ int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
}
/* PAGE1GB: shatter the superpage and fall through. */
- l2mfn = alloc_xen_pagetable();
+ l2mfn = take_or_alloc_pagetable(spare);
if ( mfn_eq(l2mfn, INVALID_MFN) )
goto out;
@@ -6122,7 +6142,7 @@ int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
{
l1_pgentry_t *l1t;
/* PSE: shatter the superpage and try again. */
- mfn_t l1mfn = alloc_xen_pagetable();
+ mfn_t l1mfn = take_or_alloc_pagetable(spare);
if ( mfn_eq(l1mfn, INVALID_MFN) )
goto out;
@@ -6280,7 +6300,7 @@ int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
{
- int rc = modify_mappings_in(idle_pg_table, s, e, nf);
+ int rc = modify_mappings_in(idle_pg_table, s, e, nf, NULL);
if ( !rc )
rc = sparse_dmap_mirror_modify(s, e, nf);
diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
index d35a8d6979..adec474f72 100644
--- a/xen/arch/x86/sparse-directmap.c
+++ b/xen/arch/x86/sparse-directmap.c
@@ -121,12 +121,13 @@ static int add_to_sparse_view(l4_pgentry_t *root, unsigned long mfn,
return 0;
}
-static int remove_from_sparse_view(unsigned long mfn, unsigned long nr)
+static int remove_from_sparse_view(unsigned long mfn, unsigned long nr,
+ mfn_t *spare)
{
unsigned long va = (unsigned long)mfn_to_virt(mfn);
return modify_mappings_in(sparse_dmap_root, va, va + (nr << PAGE_SHIFT),
- _PAGE_NONE);
+ _PAGE_NONE, spare);
}
/* Memory is about to be handed to the heap allocator. */
@@ -144,7 +145,7 @@ bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
return true;
}
- if ( remove_from_sparse_view(s, nr) )
+ if ( remove_from_sparse_view(s, nr, NULL) )
{
printk(XENLOG_ERR
"sparse directmap: cannot unmap MFNs %#lx-%#lx, not freeing\n",
@@ -188,10 +189,18 @@ int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
/*
* Xenheap pages are about to be freed. A failure means the pages may still
* be reachable through the sparse view, and must not be reused.
+ *
+ * If the caller allows, the first page serves as the page-table page that
+ * splitting a superpage of the sparse view needs: the new L1 maps the rest
+ * of the superpage, and the page leaves the sparse view with the others.
+ * One page is enough: the sparse view has no 1GiB pages, a block below 2MiB
+ * lies within one 2MiB page, and a larger one covers whole ones.
*/
-int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
+int arch_xenheap_pages_free(mfn_t mfn, unsigned int order, bool *first_kept)
{
unsigned long s = mfn_x(mfn), nr = 1UL << order;
+ mfn_t spare = INVALID_MFN;
+ int rc;
if ( !sparse_dmap_configured() )
return 0;
@@ -204,7 +213,22 @@ int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
return 0;
}
- return remove_from_sparse_view(s, nr);
+ if ( first_kept )
+ {
+ struct page_info *pg = mfn_to_page(mfn);
+
+ /* Made what alloc_xen_pagetable() would return, in case it is used. */
+ pg->count_info &= ~PGC_xen_heap;
+ page_set_owner(pg, NULL);
+ spare = mfn;
+ }
+
+ rc = remove_from_sparse_view(s, nr, &spare);
+
+ if ( first_kept && mfn_eq(spare, INVALID_MFN) )
+ *first_kept = true;
+
+ return rc;
}
/* Memory leaves Xen's hands without passing through the heap. */
@@ -233,7 +257,7 @@ int sparse_dmap_drop_pagetable(mfn_t mfn)
INVALID_MFN) )
return 0;
- rc = remove_from_sparse_view(s, 1);
+ rc = remove_from_sparse_view(s, 1, NULL);
if ( rc )
printk(XENLOG_WARNING
"Leaking page-table page %"PRI_mfn": still in the sparse directmap\n",
@@ -267,7 +291,7 @@ int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
e = min(e, SPARSE_DMAP_END);
if ( !(nf & _PAGE_PRESENT) )
- return modify_mappings_in(sparse_dmap_root, s, e, nf);
+ return modify_mappings_in(sparse_dmap_root, s, e, nf, NULL);
/*
* A permission change applies where the sparse view maps anything: it
@@ -283,7 +307,7 @@ int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
n = min(n, (e - va) >> PAGE_SHIFT);
if ( !mfn_eq(mfn, INVALID_MFN) &&
(rc = modify_mappings_in(sparse_dmap_root, va,
- va + (n << PAGE_SHIFT), nf)) )
+ va + (n << PAGE_SHIFT), nf, NULL)) )
return rc;
va += n << PAGE_SHIFT;
}
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 2150dcbd5e..349de3ddf3 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -2561,7 +2561,7 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags)
* The hook may have mapped part of the pages before failing: take
* them out again as a free would, and leak them if that fails too.
*/
- if ( arch_xenheap_pages_free(page_to_mfn(pg), order) )
+ if ( arch_xenheap_pages_free(page_to_mfn(pg), order, NULL) )
printk(XENLOG_WARNING
"Leaking xenheap pages %"PRI_mfn"+%u: still mapped\n",
mfn_x(page_to_mfn(pg)), 1U << order);
@@ -2580,6 +2580,7 @@ void free_xenheap_pages(void *v, unsigned int order)
{
struct page_info *pg;
unsigned int i;
+ bool first_kept = false;
ASSERT_ALLOC_CONTEXT();
@@ -2593,7 +2594,7 @@ void free_xenheap_pages(void *v, unsigned int order)
* they may remain reachable through it: leak them rather than hand them
* out.
*/
- if ( arch_xenheap_pages_free(page_to_mfn(pg), order) )
+ if ( arch_xenheap_pages_free(page_to_mfn(pg), order, &first_kept) )
{
printk(XENLOG_WARNING
"Leaking xenheap pages %"PRI_mfn"+%u: still mapped\n",
@@ -2601,10 +2602,15 @@ void free_xenheap_pages(void *v, unsigned int order)
return;
}
- for ( i = 0; i < (1u << order); i++ )
+ for ( i = first_kept ? 1 : 0; i < (1u << order); i++ )
pg[i].count_info &= ~PGC_xen_heap;
- free_heap_pages(pg, order, true);
+ if ( !first_kept )
+ free_heap_pages(pg, order, true);
+ else
+ /* The rest of the block, as the blocks of orders 0 to order - 1. */
+ for ( i = 0; i < order; i++ )
+ free_heap_pages(pg + (1u << i), i, true);
}
#endif /* CONFIG_SEPARATE_XENHEAP */
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index 2bcc45cf58..debd511caf 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -108,8 +108,12 @@ bool arch_xenheap_needs_scrub(void);
* pages go through arch_xenheap_pages_free() before being freed again.
*/
int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order);
-/* Xenheap pages are to be freed. Non-zero: failed, don't reuse them. */
-int arch_xenheap_pages_free(mfn_t mfn, unsigned int order);
+/*
+ * Xenheap pages are to be freed. Non-zero: failed, don't reuse them. With
+ * @first_kept not NULL, the arch may keep the first page for itself, and
+ * then sets *@first_kept to true: that page is not the caller's to free.
+ */
+int arch_xenheap_pages_free(mfn_t mfn, unsigned int order, bool *first_kept);
#else
static inline bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
{
@@ -123,7 +127,8 @@ static inline int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
{
return 0;
}
-static inline int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
+static inline int arch_xenheap_pages_free(mfn_t mfn, unsigned int order,
+ bool *first_kept)
{
return 0;
}
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 24/44] x86/mm: introduce populate_perdomain_mapping()
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (22 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 23/44] x86/mm: split a sparse directmap superpage with a page being freed George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 25/44] x86/mm: introduce a per-vCPU state window in the perdomain area George Dunlap
` (19 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Roger Pau Monné, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Roger Pau Monné <roger.pau@citrix.com>
The per-domain area has central machinery for building its page-tables
and for managing the backing pages it owns itself:
create_perdomain_mapping() / destroy_perdomain_mapping(), used by the
mapcache and the compat argument-translation area. What the interface
lacks is a way to install a caller's own pages at a chosen address,
which a per-vCPU state window, introduced by a later patch, needs: its
users allocate their pages themselves and map them there.
Introduce populate_perdomain_mapping(v, va, mfn, nr, flags) for this.
It writes the given MFNs, with the given page-table flags, into v's view
of the per-domain area, walking the per-domain page-table structures
with transient mappings, so v need not be current. Callers don't need
to know where the page-tables live, how the area is structured, or
whether v has a per-domain area of its own or shares its domain's.
We require the range to already have been populated down to the L1
tables by create_perdomain_mapping(). TLB flushing is left to the
caller. A present entry not owned by the area (!_PAGE_AVAIL0) is
replaced. A present entry owned by the area (_PAGE_AVAIL0, installed
by create_perdomain_mapping() itself) is never expected: such a page is
referenced only by the mapping, and no caller replaces area-owned
backing. Should one ever do so, the page is leaked with a message
rather than freed.
The function is not for contexts with interrupts off: its users map
their pages when allocating them.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- Comment above populate_perdomain_mapping(): the range lies within one
sub-area (one PERDOMAIN_VIRT_SLOT()), not one "per-domain slot".
- Message: "whether v has a per-domain area of its own or shares its
domain's".
Changes in v4:
- Re-introduced into the series.
- HVM only: the users are the state window's, all of which map their
pages at allocation time, so drop the linear-mapping fast path and the
IRQs-off walk that context-switch callers needed, along with the
map_domain_page_irqoff() variants.
---
xen/arch/x86/include/asm/mm.h | 3 ++
xen/arch/x86/mm.c | 62 +++++++++++++++++++++++++++++++++++
2 files changed, 65 insertions(+)
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 50ace61ce7..d04a935aff 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -623,6 +623,9 @@ struct page_info *allocate_perdomain_l3(struct domain *d);
int create_perdomain_mapping(struct vcpu *v, unsigned long va,
unsigned int nr, l1_pgentry_t **pl1tab,
struct page_info **ppg);
+void populate_perdomain_mapping(const struct vcpu *v, unsigned long va,
+ const mfn_t *mfn, unsigned int nr,
+ unsigned int flags);
void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr);
void free_perdomain_mappings(struct domain *d);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index e1a80e9a5c..0edf01f363 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -6567,6 +6567,68 @@ int create_perdomain_mapping(struct vcpu *v, unsigned long va,
return rc;
}
+/*
+ * Map @nr pages, @mfn[0..nr-1], at consecutive pages from @va in v's view of
+ * the per-domain area, with page-table @flags. The range must lie within a
+ * single sub-area (one PERDOMAIN_VIRT_SLOT()), and must already have been
+ * plumbed down to the L1 tables by create_perdomain_mapping(). Present
+ * entries are replaced, and no TLB flushing is done.
+ *
+ * The walk maps the per-domain page-table pages transiently, so v need not
+ * be current. It is not for contexts with interrupts disabled.
+ */
+void populate_perdomain_mapping(const struct vcpu *v, unsigned long va,
+ const mfn_t *mfn, unsigned int nr,
+ unsigned int flags)
+{
+ struct page_info *l3_pg = perdomain_l3(v);
+ const l3_pgentry_t *l3tab;
+ const l2_pgentry_t *l2tab;
+ l1_pgentry_t *l1tab = NULL;
+ unsigned int i;
+
+ ASSERT(va >= PERDOMAIN_VIRT_START &&
+ va < PERDOMAIN_VIRT_SLOT(PERDOMAIN_SLOTS));
+ ASSERT(!nr || !l3_table_offset(va ^ (va + nr * PAGE_SIZE - 1)));
+ ASSERT(local_irq_is_enabled());
+
+ if ( !nr )
+ return;
+
+ BUG_ON(!l3_pg);
+ l3tab = __map_domain_page(l3_pg);
+ BUG_ON(!(l3e_get_flags(l3tab[l3_table_offset(va)]) & _PAGE_PRESENT));
+ l2tab = map_l2t_from_l3e(l3tab[l3_table_offset(va)]);
+ unmap_domain_page(l3tab);
+
+ for ( i = 0; i < nr; i++, va += PAGE_SIZE )
+ {
+ l1_pgentry_t *pl1e;
+
+ if ( !l1tab || !l1_table_offset(va) )
+ {
+ const l2_pgentry_t *pl2e = &l2tab[l2_table_offset(va)];
+
+ unmap_domain_page(l1tab);
+ BUG_ON(!(l2e_get_flags(*pl2e) & _PAGE_PRESENT));
+ l1tab = map_l1t_from_l2e(*pl2e);
+ }
+
+ pl1e = &l1tab[l1_table_offset(va)];
+ if ( perdomain_l1e_needs_freeing(*pl1e) )
+ {
+ printk(XENLOG_ERR
+ "%pv: leaking area-owned page %"PRI_mfn" at %#lx\n",
+ v, mfn_x(l1e_get_mfn(*pl1e)), va);
+ ASSERT_UNREACHABLE();
+ }
+ l1e_write(pl1e, l1e_from_mfn(mfn[i], flags));
+ }
+
+ unmap_domain_page(l1tab);
+ unmap_domain_page(l2tab);
+}
+
void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr)
{
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 25/44] x86/mm: introduce a per-vCPU state window in the perdomain area
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (23 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 24/44] x86/mm: introduce populate_perdomain_mapping() George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-18 11:51 ` Jan Beulich
2026-11-03 22:13 ` [PATCH v6 26/44] x86/svm: introduce SVM_MAP_VMCB() and SVM_UNMAP_VMCB() George Dunlap
` (18 subsequent siblings)
43 siblings, 1 reply; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
Under Address Space Isolation, guest-owned register state that Xen
keeps in always-mapped memory moves to the domain heap, mapped only in
the contexts that use it. The long-lived per-vCPU structures involved
(the SVM VMCB, the VMX guest MSR load/save list, the guest XSAVE area,
and later the aggregated small per-vCPU allocations) all want the same
shape: one or more domain-heap pages, mapped once at fixed per-vCPU
virtual addresses for the life of the vCPU.
Add a fourth per-domain-mapping sub-area providing that destination: a
small per-vCPU stride of page slots, with a fixed index assigned to
each relocated structure and the page tables created at vCPU creation
time. Consumers allocate their own pages and install them with
populate_perdomain_mapping(); the backing pages remain caller-owned,
so their lifetime stays with the consumer, while the page tables are
torn down with the rest of the per-domain area.
Before freeing their pages, consumers take them out of the window with
unmap_vcpu_state(), on failure paths as at teardown: vCPU creation can
fail once a consumer has mapped its pages, and the domain lives on.
It flushes the domain's dirty CPUs, which include every CPU whose
page-tables carry the window, in the vCPU's own per-domain area or in
the one its domain's vCPUs share; so far that mask is empty on every
call, as the vCPU has not run or the domain is paused. A consumer
reaching a page from another context, through a transient
map_domain_page() mapping, drops the mapping with
unmap_vcpu_state_page(), which also flushes this CPU's translations of
it, there and at its alias in the per-domain mirror: a domain-wide
mapcache defers that flush, and the translations would otherwise
outlive the mapping.
For a domain running without per-vCPU page tables the window has the
same domain-wide visibility as the other per-domain sub-areas: the
virtual addresses are keyed on vcpu_id, so the same VA arithmetic
serves both modes. With vcpu-pt enabled, a vCPU's window is simply
absent from its siblings' address spaces.
This is infrastructure for moving the SVM VMCB, the VMX MSR load/save
lists, the saved register frames, the HVM emulator's caches and the
guest XSAVE areas off always-mapped memory. Nothing occupies the area
yet; the only behavioural change is the page-table allocation itself and
the failure path that comes with it: vCPU creation can now fail with
-ENOMEM. Both apply to every guest vCPU, with or without asi=, PV ones
included. A domain with per-vCPU page tables gets an L2 and an L1 per
vCPU; any other domain, every PV domain among them, one L2 plus an L1
per 64 vCPUs: 2 pages for up to 64 vCPUs. A PV domain's window stays
empty until the register frames and XSAVE areas move there.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v6:
- Port over change to L3 allocation.
Changes in v5:
- Say that the PV register frames move into the window too, not only
the XSAVE areas.
- Message and comments: "per-domain area", and which vCPUs share it.
Changes in v4:
- New in this version.
---
xen/arch/x86/domain.c | 12 ++++++++
xen/arch/x86/include/asm/config.h | 21 +++++++++++++-
xen/arch/x86/include/asm/mm.h | 2 ++
xen/arch/x86/mm.c | 47 +++++++++++++++++++++++++++++++
4 files changed, 81 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 46ca945b48..633619b400 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -524,6 +524,18 @@ int arch_vcpu_create(struct vcpu *v)
if ( !is_idle_domain(d) )
{
+ BUILD_BUG_ON((MAX_VIRT_CPUS << VCPU_STATE_VA_SHIFT) >
+ (PERDOMAIN_SLOT_MBYTES << 20));
+ BUILD_BUG_ON(VCPU_STATE_IDX_VMCB >= VCPU_STATE_NR_PAGES);
+ BUILD_BUG_ON(VCPU_STATE_IDX_N2VMCB >= VCPU_STATE_NR_PAGES);
+ BUILD_BUG_ON(VCPU_STATE_IDX_VMX_MSR >= VCPU_STATE_NR_PAGES);
+ /* Page tables only; torn down with the rest of the per-domain area. */
+ rc = create_perdomain_mapping(v, VCPU_STATE_START(v),
+ VCPU_STATE_NR_PAGES,
+ NIL(l1_pgentry_t *), NULL);
+ if ( rc != 0 )
+ goto fail;
+
paging_vcpu_init(v);
if ( (rc = vcpu_init_fpu(v)) != 0 )
diff --git a/xen/arch/x86/include/asm/config.h b/xen/arch/x86/include/asm/config.h
index ee84c57432..8a265e25c8 100644
--- a/xen/arch/x86/include/asm/config.h
+++ b/xen/arch/x86/include/asm/config.h
@@ -158,7 +158,7 @@
/* Slot 260: per-domain mappings (including map cache). */
#define PERDOMAIN_VIRT_START (PML4_ADDR(260))
#define PERDOMAIN_SLOT_MBYTES (PML4_ENTRY_BYTES >> (20 + PAGETABLE_ORDER))
-#define PERDOMAIN_SLOTS 3
+#define PERDOMAIN_SLOTS 4
#define PERDOMAIN_VIRT_SLOT(s) (PERDOMAIN_VIRT_START + (s) * \
(PERDOMAIN_SLOT_MBYTES << 20))
/* Slot 4: mirror of per-domain mappings (for compat xlat area accesses). */
@@ -293,6 +293,25 @@ extern unsigned long xen_phys_start;
#define ARG_XLAT_START(v) \
(ARG_XLAT_VIRT_START + ((v)->vcpu_id << ARG_XLAT_VA_SHIFT))
+/*
+ * Per-vCPU state window. The fourth per-domain-mapping sub-area: a small
+ * per-vCPU stride of page slots holding long-lived per-vCPU structures kept
+ * out of always-mapped memory, each mapped once for the life of the vCPU at
+ * a fixed index within the stride.
+ */
+#define VCPU_STATE_SLOT 3
+#define VCPU_STATE_VIRT_START PERDOMAIN_VIRT_SLOT(VCPU_STATE_SLOT)
+#define VCPU_STATE_ORDER 3
+#define VCPU_STATE_NR_PAGES (1U << VCPU_STATE_ORDER)
+#define VCPU_STATE_VA_SHIFT (VCPU_STATE_ORDER + PAGE_SHIFT)
+#define VCPU_STATE_START(v) \
+ (VCPU_STATE_VIRT_START + ((v)->vcpu_id << VCPU_STATE_VA_SHIFT))
+#define VCPU_STATE_ADDR(v, idx) \
+ (VCPU_STATE_START(v) + (idx) * PAGE_SIZE)
+#define VCPU_STATE_IDX_VMCB 0
+#define VCPU_STATE_IDX_N2VMCB 1
+#define VCPU_STATE_IDX_VMX_MSR 2
+
#define ELFSIZE 64
#define ARCH_CRASH_SAVE_VMCOREINFO
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index d04a935aff..b253f4a807 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -628,6 +628,8 @@ void populate_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int flags);
void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unsigned int nr);
+void unmap_vcpu_state(const struct vcpu *v, unsigned int idx, unsigned int nr);
+void unmap_vcpu_state_page(const void *va);
void free_perdomain_mappings(struct domain *d);
void free_pervcpu_mappings(struct vcpu *v);
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 0edf01f363..768270ec36 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -6681,6 +6681,53 @@ void destroy_perdomain_mapping(const struct vcpu *v, unsigned long va,
unmap_domain_page(l3tab);
}
+/*
+ * Take @nr slots of @v's state window, from @idx, out of the per-domain
+ * area, before the caller frees the pages behind them: the pages are the
+ * caller's own, and must not stay mapped once freed. The window's
+ * translations are usable only on CPUs whose page-tables carry it, all in
+ * the domain's dirty mask: with per-vCPU page-tables, the CPU running v or
+ * holding it lazily; otherwise, those of any of the domain's vCPUs.
+ */
+void unmap_vcpu_state(const struct vcpu *v, unsigned int idx, unsigned int nr)
+{
+ const struct domain *d = v->domain;
+
+ /*
+ * Beyond the stride is the next vCPU's window, where the domain's vCPUs
+ * share a per-domain area.
+ */
+ BUG_ON(idx >= VCPU_STATE_NR_PAGES || nr > VCPU_STATE_NR_PAGES - idx);
+
+ destroy_perdomain_mapping(v, VCPU_STATE_ADDR(v, idx), nr);
+
+ flush_tlb_mask(d->dirty_cpumask);
+}
+
+/*
+ * Drop a transient mapping of a page behind a vCPU's state window, and this
+ * CPU's translations of it: at its address, and at its alias in the
+ * per-domain mirror, which every domain but a 64-bit PV one has (flushing
+ * that address there is harmless). A per-vCPU mapcache flushes both
+ * already, with the slot's last mapping; a domain-wide one defers the
+ * flush, and CPUs running other vCPUs of its domain may have cached the
+ * slot too. A directmap address, where the caller has no mapcache, needs
+ * nothing.
+ */
+void unmap_vcpu_state_page(const void *va)
+{
+ unsigned long addr = (unsigned long)va;
+
+ unmap_domain_page(va);
+
+ if ( addr < DIRECTMAP_VIRT_START )
+ {
+ flush_tlb_one_local(addr);
+ flush_tlb_one_local(addr - PERDOMAIN_VIRT_START +
+ PERDOMAIN_ALT_VIRT_START);
+ }
+}
+
static void free_perdomain_l3(struct page_info *l3pg)
{
l3_pgentry_t *l3tab = __map_domain_page(l3pg);
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 26/44] x86/svm: introduce SVM_MAP_VMCB() and SVM_UNMAP_VMCB()
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (24 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 25/44] x86/mm: introduce a per-vCPU state window in the perdomain area George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 27/44] x86/svm: restore guest debug registers in svm_do_resume() George Dunlap
` (17 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Ross Lagerwall, Jan Beulich, Andrew Cooper, Roger Pau Monné,
Jason Andryuk, Teddy Astie
Some accessors of a vCPU's VMCB can run with that vCPU paused, from
another vCPU's context: context save/load, the segment and CPL getters,
PAT/TSC/intercept updates reachable from toolstack operations, vm_event,
the 'v' debug key, and construct_vmcb() itself, which runs before the
vCPU ever entered guest context. svm_nested_features_on_efer_update()
is among them: it is reachable remotely via hvm_load_cpu_ctxt() ->
svm_update_guest_efer().
Route them through the new SVM_MAP_VMCB() and SVM_UNMAP_VMCB();
construct_vmcb() takes the VMCB from its caller. svm_set_tsc_offset()
needs both VMCBs rather than just the active one, so it maps them
individually via SVM_MAP_VMCB_ONE(). A vCPU stays in nested guest mode
across a deschedule or a pause, so this path can run remotely with the
L2 VMCB active; guest mode does not imply v == current. Hot vmexit-path
accesses all have v == current and keep dereferencing svm->vmcb
directly.
SVM_MAP_VMCB(), SVM_MAP_VMCB_ONE() and SVM_UNMAP_VMCB() are macros over
svm_vmcb_map(), svm_vmcb_map_one() and svm_vmcb_unmap(), so that
SVM_UNMAP_VMCB() can clear the caller's pointer, as
VCPU_UNMAP_XSAVE_AREA() does.
No functional change: the helpers return the VMCB's existing address.
They prepare for moving VMCBs off always-mapped memory, where a VMCB's
mapping is valid only in its own vCPU's context.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
CC: Ross Lagerwall <ross.lagerwall@citrix.com>
Changes in v5:
- Map and unmap through SVM_MAP_VMCB(), SVM_MAP_VMCB_ONE() and
SVM_UNMAP_VMCB(), which clears the pointer (George).
- Retitle (was: "x86/svm: introduce svm_vmcb_map() and
svm_vmcb_unmap()").
Changes in v4:
- New in this version.
---
xen/arch/x86/hvm/svm/nestedsvm.c | 4 +-
xen/arch/x86/hvm/svm/svm.c | 133 +++++++++++++++++++++++--------
xen/arch/x86/hvm/svm/vmcb.c | 29 ++++++-
xen/arch/x86/hvm/svm/vmcb.h | 18 +++++
4 files changed, 145 insertions(+), 39 deletions(-)
diff --git a/xen/arch/x86/hvm/svm/nestedsvm.c b/xen/arch/x86/hvm/svm/nestedsvm.c
index 128008321d..f5ce3e485c 100644
--- a/xen/arch/x86/hvm/svm/nestedsvm.c
+++ b/xen/arch/x86/hvm/svm/nestedsvm.c
@@ -1528,7 +1528,7 @@ void svm_vmexit_do_clgi(struct cpu_user_regs *regs, struct vcpu *v)
*/
void svm_nested_features_on_efer_update(struct vcpu *v)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
struct nestedsvm *svm = &vcpu_nestedsvm(v);
u32 general2_intercepts;
vintr_t vintr;
@@ -1585,6 +1585,8 @@ void svm_nested_features_on_efer_update(struct vcpu *v)
vmcb_set_general2_intercepts(vmcb, general2_intercepts);
}
}
+
+ SVM_UNMAP_VMCB(v, vmcb);
}
void __init start_nested_svm(struct hvm_function_table *hvm_function_table)
diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index 1628c5e0ae..b539406f0c 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -107,7 +107,7 @@ static void cf_check svm_cpu_down(void)
static void cf_check svm_update_guest_cr(
struct vcpu *v, unsigned int cr, unsigned int flags)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
uint64_t value;
switch ( cr )
@@ -173,11 +173,13 @@ static void cf_check svm_update_guest_cr(
default:
BUG();
}
+
+ SVM_UNMAP_VMCB(v, vmcb);
}
static void cf_check svm_update_guest_efer(struct vcpu *v)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
unsigned long guest_efer = v->arch.hvm.guest_efer,
xen_efer = read_efer();
@@ -204,6 +206,7 @@ static void cf_check svm_update_guest_efer(struct vcpu *v)
guest_efer |= EFER_SVME;
vmcb_set_efer(vmcb, guest_efer);
+ SVM_UNMAP_VMCB(v, vmcb);
ASSERT(nestedhvm_enabled(v->domain) ||
!nsvm_efer_svm_enabled(v));
@@ -352,7 +355,7 @@ static void svm_restore_dr(struct vcpu *v)
static int svm_vmcb_save(struct vcpu *v, struct hvm_hw_cpu *c)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
if ( vmcb->event_inj.v &&
hvm_event_needs_reinjection(vmcb->event_inj.type,
@@ -362,13 +365,15 @@ static int svm_vmcb_save(struct vcpu *v, struct hvm_hw_cpu *c)
c->error_code = vmcb->event_inj.ec;
}
+ SVM_UNMAP_VMCB(v, vmcb);
+
return 1;
}
static int svm_vmcb_restore(struct vcpu *v, struct hvm_hw_cpu *c)
{
struct page_info *page = NULL;
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb;
struct p2m_domain *p2m = p2m_get_hostp2m(v->domain);
if ( c->pending_valid )
@@ -417,6 +422,8 @@ static int svm_vmcb_restore(struct vcpu *v, struct hvm_hw_cpu *c)
svm_update_guest_cr(v, 0, 0);
svm_update_guest_cr(v, 4, 0);
+ vmcb = SVM_MAP_VMCB(v);
+
if ( !paging_mode_shadow(v->domain) )
{
vmcb_set_np(vmcb, true);
@@ -436,6 +443,8 @@ static int svm_vmcb_restore(struct vcpu *v, struct hvm_hw_cpu *c)
vmcb->event_inj.raw = 0;
vmcb->cleanbits.raw = 0;
+ SVM_UNMAP_VMCB(v, vmcb);
+
paging_update_paging_modes(v);
return 0;
@@ -443,7 +452,7 @@ static int svm_vmcb_restore(struct vcpu *v, struct hvm_hw_cpu *c)
static void svm_save_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ const struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
data->sysenter_cs = vmcb->sysenter_cs;
data->sysenter_esp = vmcb->sysenter_esp;
@@ -453,11 +462,13 @@ static void svm_save_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
data->msr_star = vmcb->star;
data->msr_cstar = vmcb->cstar;
data->msr_syscall_mask = vmcb->sfmask;
+
+ SVM_UNMAP_VMCB(v, vmcb);
}
static void svm_load_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
vmcb->lstar = data->msr_lstar;
vmcb->star = data->msr_star;
@@ -467,6 +478,9 @@ static void svm_load_cpu_state(struct vcpu *v, struct hvm_hw_cpu *data)
vmcb->sysenter_cs = data->sysenter_cs;
vmcb->sysenter_esp = data->sysenter_esp;
vmcb->sysenter_eip = data->sysenter_eip;
+
+ SVM_UNMAP_VMCB(v, vmcb);
+
v->arch.hvm.guest_efer = data->msr_efer;
svm_update_guest_efer(v);
}
@@ -492,7 +506,7 @@ static int cf_check svm_load_vmcb_ctxt(struct vcpu *v, struct hvm_hw_cpu *ctxt)
static unsigned cf_check int svm_get_interrupt_shadow(struct vcpu *v)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
unsigned int intr_shadow = 0;
if ( vmcb->int_stat.intr_shadow )
@@ -503,13 +517,15 @@ static unsigned cf_check int svm_get_interrupt_shadow(struct vcpu *v)
: (vmcb_get_general1_intercepts(vmcb) & GENERAL1_INTERCEPT_IRET) )
intr_shadow |= HVM_INTR_SHADOW_NMI;
+ SVM_UNMAP_VMCB(v, vmcb);
+
return intr_shadow;
}
static void cf_check svm_set_interrupt_shadow(
struct vcpu *v, unsigned int intr_shadow)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
bool block_nmi = intr_shadow & HVM_INTR_SHADOW_NMI;
vmcb->int_stat.intr_shadow =
@@ -527,10 +543,13 @@ static void cf_check svm_set_interrupt_shadow(
vmcb_set_general1_intercepts(vmcb, gen1);
}
+
+ SVM_UNMAP_VMCB(v, vmcb);
}
static int cf_check svm_guest_x86_mode(struct vcpu *v)
{
+ /* Current only (see guest_cpu_user_regs() below): no mapping needed. */
struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
if ( unlikely(!(v->arch.hvm.guest_cr[0] & X86_CR0_PE)) )
@@ -544,8 +563,7 @@ static int cf_check svm_guest_x86_mode(struct vcpu *v)
static void cf_check svm_cpuid_policy_changed(struct vcpu *v)
{
- struct svm_vcpu *svm = &v->arch.hvm.svm;
- struct vmcb_struct *vmcb = svm->vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
const struct cpu_policy *cp = v->domain->arch.cpu_policy;
u32 bitmap = vmcb_get_exception_intercepts(vmcb);
@@ -555,6 +573,7 @@ static void cf_check svm_cpuid_policy_changed(struct vcpu *v)
bitmap &= ~(1U << X86_EXC_UD);
vmcb_set_exception_intercepts(vmcb, bitmap);
+ SVM_UNMAP_VMCB(v, vmcb);
/* Give access to MSR_SPEC_CTRL if the guest has been told about it. */
svm_intercept_msr(v, MSR_SPEC_CTRL,
@@ -598,13 +617,18 @@ void svm_sync_vmcb(struct vcpu *v, enum vmcb_sync_state new_state)
static unsigned int cf_check svm_get_cpl(struct vcpu *v)
{
- return vmcb_get_cpl(v->arch.hvm.svm.vmcb);
+ const struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
+ unsigned int cpl = vmcb_get_cpl(vmcb);
+
+ SVM_UNMAP_VMCB(v, vmcb);
+
+ return cpl;
}
static void cf_check svm_get_segment_register(
struct vcpu *v, enum x86_segment seg, struct segment_register *reg)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
ASSERT((v == current) || !vcpu_runnable(v));
@@ -645,12 +669,14 @@ static void cf_check svm_get_segment_register(
*reg = (struct segment_register){};
break;
}
+
+ SVM_UNMAP_VMCB(v, vmcb);
}
static void cf_check svm_set_segment_register(
struct vcpu *v, enum x86_segment seg, struct segment_register *reg)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
ASSERT((v == current) || !vcpu_runnable(v));
@@ -679,6 +705,7 @@ static void cf_check svm_set_segment_register(
default:
ASSERT_UNREACHABLE();
domain_crash(v->domain);
+ SVM_UNMAP_VMCB(v, vmcb);
return;
}
@@ -716,33 +743,41 @@ static void cf_check svm_set_segment_register(
ASSERT_UNREACHABLE();
break;
}
+
+ SVM_UNMAP_VMCB(v, vmcb);
}
static int cf_check svm_set_guest_pat(struct vcpu *v, u64 gpat)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb;
if ( paging_mode_shadow(v->domain) )
return 0;
+ vmcb = SVM_MAP_VMCB(v);
vmcb_set_g_pat(vmcb, gpat);
+ SVM_UNMAP_VMCB(v, vmcb);
+
return 1;
}
static int cf_check svm_get_guest_pat(struct vcpu *v, u64 *gpat)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ const struct vmcb_struct *vmcb;
if ( paging_mode_shadow(v->domain) )
return 0;
+ vmcb = SVM_MAP_VMCB(v);
*gpat = vmcb_get_g_pat(vmcb);
+ SVM_UNMAP_VMCB(v, vmcb);
+
return 1;
}
static void cf_check svm_set_tsc_offset(struct vcpu *v, uint64_t offset)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
struct vmcb_struct *n1vmcb, *n2vmcb;
uint64_t n2_tsc_offset = 0;
struct domain *d = v->domain;
@@ -750,25 +785,36 @@ static void cf_check svm_set_tsc_offset(struct vcpu *v, uint64_t offset)
if ( !nestedhvm_enabled(d) )
{
vmcb_set_tsc_offset(vmcb, offset);
+ SVM_UNMAP_VMCB(v, vmcb);
return;
}
- n1vmcb = vcpu_nestedhvm(v).nv_n1vmcx;
- n2vmcb = vcpu_nestedhvm(v).nv_n2vmcx;
-
+ /*
+ * A vCPU stays in nested guest mode across a deschedule or a pause, so
+ * this can run remotely with the L2 VMCB active: map both explicitly.
+ */
if ( nestedhvm_vcpu_in_guestmode(v) )
{
+ const struct nestedvcpu *nv = &vcpu_nestedhvm(v);
+
+ n1vmcb = SVM_MAP_VMCB_ONE(v, nv->nv_n1vmcx, nv->nv_n1vmcx_pa);
+ n2vmcb = SVM_MAP_VMCB_ONE(v, nv->nv_n2vmcx, nv->nv_n2vmcx_pa);
+
n2_tsc_offset = vmcb_get_tsc_offset(n2vmcb) -
vmcb_get_tsc_offset(n1vmcb);
vmcb_set_tsc_offset(n1vmcb, offset);
+
+ SVM_UNMAP_VMCB(v, n2vmcb);
+ SVM_UNMAP_VMCB(v, n1vmcb);
}
vmcb_set_tsc_offset(vmcb, offset + n2_tsc_offset);
+ SVM_UNMAP_VMCB(v, vmcb);
}
static void cf_check svm_set_rdtsc_exiting(struct vcpu *v, bool enable)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
u32 general1_intercepts = vmcb_get_general1_intercepts(vmcb);
u32 general2_intercepts = vmcb_get_general2_intercepts(vmcb);
@@ -783,13 +829,14 @@ static void cf_check svm_set_rdtsc_exiting(struct vcpu *v, bool enable)
vmcb_set_general1_intercepts(vmcb, general1_intercepts);
vmcb_set_general2_intercepts(vmcb, general2_intercepts);
+ SVM_UNMAP_VMCB(v, vmcb);
}
#ifdef CONFIG_VM_EVENT
static void cf_check svm_set_descriptor_access_exiting(
struct vcpu *v, bool enable)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
u32 general1_intercepts = vmcb_get_general1_intercepts(vmcb);
u32 mask = GENERAL1_INTERCEPT_IDTR_READ | GENERAL1_INTERCEPT_GDTR_READ
| GENERAL1_INTERCEPT_LDTR_READ | GENERAL1_INTERCEPT_TR_READ
@@ -802,6 +849,7 @@ static void cf_check svm_set_descriptor_access_exiting(
general1_intercepts &= ~mask;
vmcb_set_general1_intercepts(vmcb, general1_intercepts);
+ SVM_UNMAP_VMCB(v, vmcb);
}
#endif /* CONFIG_VM_EVENT */
@@ -1360,7 +1408,12 @@ static void cf_check svm_inject_event(const struct x86_event *event)
static bool cf_check svm_event_pending(const struct vcpu *v)
{
- return v->arch.hvm.svm.vmcb->event_inj.v;
+ const struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
+ bool pending = vmcb->event_inj.v;
+
+ SVM_UNMAP_VMCB(v, vmcb);
+
+ return pending;
}
static void cf_check svm_cpu_dead(unsigned int cpu)
@@ -2318,45 +2371,55 @@ static void cf_check svm_invlpg(struct vcpu *v, unsigned long linear)
static bool cf_check svm_get_pending_event(
struct vcpu *v, struct x86_event *info)
{
- const struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ const struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
+ bool pending = vmcb->event_inj.v;
- if ( !vmcb->event_inj.v )
- return false;
+ if ( pending )
+ {
+ info->vector = vmcb->event_inj.vector;
+ info->type = vmcb->event_inj.type;
+ info->error_code = vmcb->event_inj.ec;
+ }
- info->vector = vmcb->event_inj.vector;
- info->type = vmcb->event_inj.type;
- info->error_code = vmcb->event_inj.ec;
+ SVM_UNMAP_VMCB(v, vmcb);
- return true;
+ return pending;
}
static uint64_t cf_check svm_get_reg(struct vcpu *v, unsigned int reg)
{
struct vcpu *curr = current;
- const struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ const struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
struct domain *d = v->domain;
+ uint64_t val = 0;
switch ( reg )
{
case MSR_SPEC_CTRL:
- return vmcb->spec_ctrl;
+ val = vmcb->spec_ctrl;
+ break;
case MSR_SHADOW_GS_BASE:
if ( v == curr )
svm_sync_vmcb(v, vmcb_in_sync);
- return vmcb->kerngsbase;
+ val = vmcb->kerngsbase;
+ break;
default:
printk(XENLOG_G_ERR "%s(%pv, 0x%08x) Bad register\n",
__func__, v, reg);
domain_crash(d);
- return 0;
+ break;
}
+
+ SVM_UNMAP_VMCB(v, vmcb);
+
+ return val;
}
static void cf_check svm_set_reg(struct vcpu *v, unsigned int reg, uint64_t val)
{
- struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
+ struct vmcb_struct *vmcb = SVM_MAP_VMCB(v);
struct domain *d = v->domain;
switch ( reg )
@@ -2371,6 +2434,8 @@ static void cf_check svm_set_reg(struct vcpu *v, unsigned int reg, uint64_t val)
domain_crash(d);
break;
}
+
+ SVM_UNMAP_VMCB(v, vmcb);
}
static struct hvm_function_table __initdata_cf_clobber svm_function_table = {
diff --git a/xen/arch/x86/hvm/svm/vmcb.c b/xen/arch/x86/hvm/svm/vmcb.c
index a6f09672a7..5e193be9cd 100644
--- a/xen/arch/x86/hvm/svm/vmcb.c
+++ b/xen/arch/x86/hvm/svm/vmcb.c
@@ -43,11 +43,25 @@ void free_vmcb(struct vmcb_struct *vmcb)
free_xenheap_page(vmcb);
}
+struct vmcb_struct *svm_vmcb_map_one(const struct vcpu *v,
+ struct vmcb_struct *va, paddr_t pa)
+{
+ return va;
+}
+
+struct vmcb_struct *svm_vmcb_map(const struct vcpu *v)
+{
+ return svm_vmcb_map_one(v, v->arch.hvm.svm.vmcb, v->arch.hvm.svm.vmcb_pa);
+}
+
+void svm_vmcb_unmap(const struct vcpu *v, const struct vmcb_struct *vmcb)
+{
+}
+
/* This function can directly access fields which are covered by clean bits. */
-static int construct_vmcb(struct vcpu *v)
+static int construct_vmcb(struct vcpu *v, struct vmcb_struct *vmcb)
{
struct svm_vcpu *svm = &v->arch.hvm.svm;
- struct vmcb_struct *vmcb = svm->vmcb;
vmcb->_general1_intercepts =
GENERAL1_INTERCEPT_INTR | GENERAL1_INTERCEPT_NMI |
@@ -201,6 +215,7 @@ int svm_create_vmcb(struct vcpu *v)
{
struct nestedvcpu *nv = &vcpu_nestedhvm(v);
struct svm_vcpu *svm = &v->arch.hvm.svm;
+ struct vmcb_struct *vmcb;
int rc;
if ( (nv->nv_n1vmcx == NULL) &&
@@ -211,7 +226,10 @@ int svm_create_vmcb(struct vcpu *v)
}
svm->vmcb = nv->nv_n1vmcx;
- rc = construct_vmcb(v);
+
+ vmcb = SVM_MAP_VMCB(v);
+ rc = construct_vmcb(v, vmcb);
+ SVM_UNMAP_VMCB(v, vmcb);
if ( rc != 0 )
{
free_vmcb(nv->nv_n1vmcx);
@@ -407,6 +425,7 @@ static void cf_check vmcb_dump(unsigned char ch)
{
struct domain *d;
struct vcpu *v;
+ struct vmcb_struct *vmcb;
printk("*********** VMCB Areas **************\n");
@@ -425,7 +444,9 @@ static void cf_check vmcb_dump(unsigned char ch)
continue;
}
printk("\tVCPU %d\n", v->vcpu_id);
- svm_vmcb_dump("key_handler", v->arch.hvm.svm.vmcb);
+ vmcb = SVM_MAP_VMCB(v);
+ svm_vmcb_dump("key_handler", vmcb);
+ SVM_UNMAP_VMCB(v, vmcb);
process_pending_softirqs();
}
diff --git a/xen/arch/x86/hvm/svm/vmcb.h b/xen/arch/x86/hvm/svm/vmcb.h
index 28dae325ce..d49c910d46 100644
--- a/xen/arch/x86/hvm/svm/vmcb.h
+++ b/xen/arch/x86/hvm/svm/vmcb.h
@@ -559,6 +559,24 @@ struct vmcb_struct {
struct vmcb_struct *alloc_vmcb(void);
void free_vmcb(struct vmcb_struct *vmcb);
+/*
+ * Access a VMCB of a vCPU that may not be current. SVM_MAP_VMCB() takes
+ * the active VMCB, SVM_MAP_VMCB_ONE() a specific one.
+ */
+struct vmcb_struct *svm_vmcb_map_one(const struct vcpu *v,
+ struct vmcb_struct *va, paddr_t pa);
+struct vmcb_struct *svm_vmcb_map(const struct vcpu *v);
+void svm_vmcb_unmap(const struct vcpu *v, const struct vmcb_struct *vmcb);
+
+#define SVM_MAP_VMCB(v) svm_vmcb_map(v)
+#define SVM_MAP_VMCB_ONE(v, va, pa) svm_vmcb_map_one(v, va, pa)
+
+/* Drops the mapping of a VMCB and nullifies its pointer. */
+#define SVM_UNMAP_VMCB(v, vmcb) do { \
+ svm_vmcb_unmap(v, vmcb); \
+ (vmcb) = NULL; \
+} while ( false )
+
int svm_create_vmcb(struct vcpu *v);
void svm_destroy_vmcb(struct vcpu *v);
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 27/44] x86/svm: restore guest debug registers in svm_do_resume()
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (25 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 26/44] x86/svm: introduce SVM_MAP_VMCB() and SVM_UNMAP_VMCB() George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 28/44] x86/svm: drop svm_vmcb_dump()'s paddr George Dunlap
` (16 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Ross Lagerwall, Jan Beulich, Andrew Cooper, Roger Pau Monné,
Jason Andryuk, Teddy Astie
svm_ctxt_switch_to() restores the guest's debug registers when its DR7
enables any of them, which also updates the VMCB. Do it in
svm_do_resume() instead, which follows it whenever the vCPU is
scheduled in.
svm_do_resume() also runs when a vCPU keeps running on its pCPU
(continue_running()), without svm_ctxt_switch_to(). Where the guest's
debug registers are already loaded, the restore does nothing:
flag_dr_dirty records that they are, and only svm_save_dr() clears it,
from svm_ctxt_switch_from().
This prepares for moving VMCBs off always-mapped memory:
svm_ctxt_switch_to() runs before write_ptbase(), where the incoming
vCPU's VMCB will not be mapped.
Not yet run with a guest using debug registers.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
CC: Ross Lagerwall <ross.lagerwall@citrix.com>
Changes in v4:
- New in this version.
---
xen/arch/x86/hvm/svm/svm.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index b539406f0c..df56245311 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -953,8 +953,6 @@ static void cf_check svm_ctxt_switch_to(struct vcpu *v)
*/
disable_each_ist(per_cpu(idt, cpu));
- svm_restore_dr(v);
-
svm_tsc_ratio_load(v);
if ( cpu_has_msr_tsc_aux )
@@ -978,6 +976,8 @@ static void noreturn cf_check svm_do_resume(void)
bool vcpu_guestmode = false;
struct vlapic *vlapic = vcpu_vlapic(v);
+ svm_restore_dr(v);
+
if ( nestedhvm_enabled(v->domain) && nestedhvm_vcpu_in_guestmode(v) )
vcpu_guestmode = 1;
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 28/44] x86/svm: drop svm_vmcb_dump()'s paddr
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (26 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 27/44] x86/svm: restore guest debug registers in svm_do_resume() George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-18 11:54 ` Jan Beulich
2026-11-03 22:13 ` [PATCH v6 29/44] x86/svm: allocate VMCBs from the domain heap behind the vCPU state window George Dunlap
` (15 subsequent siblings)
43 siblings, 1 reply; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Ross Lagerwall, Jan Beulich, Andrew Cooper, Roger Pau Monné,
Jason Andryuk, Teddy Astie
svm_vmcb_dump() prints virt_to_maddr() of the VMCB, which is right only
while VMCBs are reached through the directmap. Print the virtual
address alone.
This prepares for moving VMCBs off always-mapped memory.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
CC: Ross Lagerwall <ross.lagerwall@citrix.com>
Changes in v5:
- Keep VCPU_svm_vmcb: svm_asm_do_resume() now uses it, since
df8d101a3854 ("x86/svm: Work around VMRUN's corruption of
VMCB.INTR_SHADOW") in staging.
- Retitle (was: "x86/svm: drop svm_vmcb_dump()'s paddr and the unused
VCPU_svm_vmcb").
Changes in v4:
- New in this version.
---
xen/arch/x86/hvm/svm/vmcb.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/hvm/svm/vmcb.c b/xen/arch/x86/hvm/svm/vmcb.c
index 5e193be9cd..21005dc57f 100644
--- a/xen/arch/x86/hvm/svm/vmcb.c
+++ b/xen/arch/x86/hvm/svm/vmcb.c
@@ -280,8 +280,8 @@ void svm_vmcb_dump(const char *from, const struct vmcb_struct *vmcb)
svm_sync_vmcb(curr, vmcb_in_sync);
printk("Dumping guest's current state at %s...\n", from);
- printk("Size of VMCB = %zu, paddr = %"PRIpaddr", vaddr = %p\n",
- sizeof(struct vmcb_struct), virt_to_maddr(vmcb), vmcb);
+ printk("Size of VMCB = %zu, vaddr = %p\n",
+ sizeof(struct vmcb_struct), vmcb);
printk("cr_intercepts = %#x dr_intercepts = %#x "
"exception_intercepts = %#x\n",
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 29/44] x86/svm: allocate VMCBs from the domain heap behind the vCPU state window
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (27 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 28/44] x86/svm: drop svm_vmcb_dump()'s paddr George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 30/44] x86/vmx: move the guest MSR load/save list off always-mapped memory George Dunlap
` (14 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Ross Lagerwall, Jan Beulich, Andrew Cooper, Roger Pau Monné,
Jason Andryuk, Teddy Astie
The VMCB holds the full guest register save area, but lives on the
xenheap, where it stays reachable from every context. Move it to the
domain heap and map it at a fixed slot in the owning vCPU's state
window, so that under Address Space Isolation it is only mapped while
that vCPU's page tables are loaded. Hardware is unaffected: VMRUN,
VMLOAD and VMSAVE address the VMCB physically, and vmcb_pa now derives
from the page rather than from virt_to_maddr() on a directmap VA.
A vCPU can have two VMCBs: the primary (nv_n1vmcx) and the nested L2
one (nv_n2vmcx). Both move, taking the first two slots in the window
stride. The nested code's swapping of svm->vmcb / svm->vmcb_pa
between the two works unchanged, now swapping between two window VAs.
free_vmcb() takes a VMCB out of the window with unmap_vcpu_state()
before freeing the page: without per-vCPU page tables the window is in
the per-domain area that all its domain's vCPUs share, which outlives
a vCPU whose creation fails.
The window VA is only valid in the owning vCPU's context.
SVM_MAP_VMCB() and SVM_MAP_VMCB_ONE() return it for v == current, and
a transient map_domain_page() mapping otherwise, which SVM_UNMAP_VMCB()
drops with unmap_vcpu_state_page(), along with this CPU's translations
of it. svm_ctxt_switch_to() no longer touches the VMCB, as it runs
before write_ptbase(); svm_ctxt_switch_from() keeps dereferencing the
window VA, as the outgoing page tables are still loaded at that point.
svm_asm_do_resume() reads the VMCB's interrupt shadow through the
window VA just before VMRUN, with the vCPU's own page tables loaded.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
CC: Ross Lagerwall <ross.lagerwall@citrix.com>
Changes in v5:
- Describe the mapping in terms of SVM_MAP_VMCB(), SVM_MAP_VMCB_ONE()
and SVM_UNMAP_VMCB(), which clears the pointer (George).
- v4's 26/30, "x86/svm: clear the VMCB clean bits in
svm_ctxt_switch_from()", is dropped. Ross's "x86/svm: Don't clear
clean bits on every context switch", now in staging as 69633efbca3c,
removes the clearing from svm_ctxt_switch_to() instead.
- svm_asm_do_resume() now reads the VMCB through svm.vmcb (staging's
"x86/svm: Work around VMRUN's corruption of VMCB.INTR_SHADOW"); it
runs in the vCPU's own context, where the window VA is valid.
Changes in v4:
- New in this version.
---
xen/arch/x86/hvm/svm/nestedsvm.c | 8 +++---
xen/arch/x86/hvm/svm/svm.c | 10 ++++++++
xen/arch/x86/hvm/svm/vmcb.c | 43 ++++++++++++++++++++++----------
xen/arch/x86/hvm/svm/vmcb.h | 15 ++++++++---
4 files changed, 56 insertions(+), 20 deletions(-)
diff --git a/xen/arch/x86/hvm/svm/nestedsvm.c b/xen/arch/x86/hvm/svm/nestedsvm.c
index f5ce3e485c..9a308cf628 100644
--- a/xen/arch/x86/hvm/svm/nestedsvm.c
+++ b/xen/arch/x86/hvm/svm/nestedsvm.c
@@ -71,10 +71,9 @@ int cf_check nsvm_vcpu_initialise(struct vcpu *v)
goto err;
memset(msrpm, 0x0, MSRPM_SIZE);
- nv->nv_n2vmcx = alloc_vmcb();
+ nv->nv_n2vmcx = alloc_vmcb(v, VCPU_STATE_IDX_N2VMCB, &nv->nv_n2vmcx_pa);
if ( nv->nv_n2vmcx == NULL )
goto err;
- nv->nv_n2vmcx_pa = virt_to_maddr(nv->nv_n2vmcx);
return 0;
@@ -95,7 +94,10 @@ void cf_check nsvm_vcpu_destroy(struct vcpu *v)
* of l1 vmcb page.
*/
if ( nv->nv_n1vmcx )
+ {
v->arch.hvm.svm.vmcb = nv->nv_n1vmcx;
+ v->arch.hvm.svm.vmcb_pa = nv->nv_n1vmcx_pa;
+ }
if ( svm->ns_cached_msrpm )
{
@@ -113,7 +115,7 @@ void cf_check nsvm_vcpu_destroy(struct vcpu *v)
if ( nv->nv_n2vmcx )
{
- free_vmcb(nv->nv_n2vmcx);
+ free_vmcb(v, VCPU_STATE_IDX_N2VMCB, nv->nv_n2vmcx_pa);
nv->nv_n2vmcx = NULL;
nv->nv_n2vmcx_pa = INVALID_PADDR;
}
diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c
index df56245311..8b593b1b39 100644
--- a/xen/arch/x86/hvm/svm/svm.c
+++ b/xen/arch/x86/hvm/svm/svm.c
@@ -265,6 +265,10 @@ static void cf_check svm_enable_msr_interception(struct domain *d, uint32_t msr)
static void svm_save_dr(struct vcpu *v)
{
+ /*
+ * @v is the outgoing vCPU, whose page tables are still loaded: the
+ * VMCB's window VA serves, although current has moved on.
+ */
struct vmcb_struct *vmcb = v->arch.hvm.svm.vmcb;
unsigned int flag_dr_dirty = v->arch.hvm.flag_dr_dirty;
@@ -940,6 +944,12 @@ static void cf_check svm_ctxt_switch_to(struct vcpu *v)
{
int cpu = smp_processor_id();
+ /*
+ * NB we cannot access the VMCB here: this runs before
+ * write_ptbase(), so the incoming vCPU's state window is not
+ * reachable yet.
+ */
+
/*
* This is required, because VMRUN does consistency check and some of the
* DOM0 selectors are pointing to invalid GDT locations, and cause AMD
diff --git a/xen/arch/x86/hvm/svm/vmcb.c b/xen/arch/x86/hvm/svm/vmcb.c
index 21005dc57f..6588ec87e6 100644
--- a/xen/arch/x86/hvm/svm/vmcb.c
+++ b/xen/arch/x86/hvm/svm/vmcb.c
@@ -6,6 +6,7 @@
*
*/
+#include <xen/domain_page.h>
#include <xen/init.h>
#include <xen/keyhandler.h>
#include <xen/lib.h>
@@ -23,30 +24,40 @@
#include "svm.h"
#include "vmcb.h"
-struct vmcb_struct *alloc_vmcb(void)
+struct vmcb_struct *alloc_vmcb(struct vcpu *v, unsigned int idx, paddr_t *pa)
{
- struct vmcb_struct *vmcb;
+ struct page_info *pg = alloc_domheap_page(v->domain, MEMF_no_owner);
+ mfn_t mfn;
+
+ ASSERT(idx < VCPU_STATE_NR_PAGES);
- vmcb = alloc_xenheap_page();
- if ( vmcb == NULL )
+ if ( pg == NULL )
{
printk(XENLOG_WARNING "Warning: failed to allocate vmcb.\n");
return NULL;
}
- clear_page(vmcb);
- return vmcb;
+ mfn = page_to_mfn(pg);
+ clear_domain_page(mfn);
+ populate_perdomain_mapping(v, VCPU_STATE_ADDR(v, idx), &mfn, 1,
+ __PAGE_HYPERVISOR_RW);
+ *pa = mfn_to_maddr(mfn);
+
+ return (struct vmcb_struct *)VCPU_STATE_ADDR(v, idx);
}
-void free_vmcb(struct vmcb_struct *vmcb)
+void free_vmcb(struct vcpu *v, unsigned int idx, paddr_t pa)
{
- free_xenheap_page(vmcb);
+ ASSERT(idx < VCPU_STATE_NR_PAGES);
+
+ unmap_vcpu_state(v, idx, 1);
+ free_domheap_page(maddr_to_page(pa));
}
struct vmcb_struct *svm_vmcb_map_one(const struct vcpu *v,
struct vmcb_struct *va, paddr_t pa)
{
- return va;
+ return v == current ? va : map_domain_page(maddr_to_mfn(pa));
}
struct vmcb_struct *svm_vmcb_map(const struct vcpu *v)
@@ -56,6 +67,8 @@ struct vmcb_struct *svm_vmcb_map(const struct vcpu *v)
void svm_vmcb_unmap(const struct vcpu *v, const struct vmcb_struct *vmcb)
{
+ if ( v != current )
+ unmap_vcpu_state_page(vmcb);
}
/* This function can directly access fields which are covered by clean bits. */
@@ -219,26 +232,29 @@ int svm_create_vmcb(struct vcpu *v)
int rc;
if ( (nv->nv_n1vmcx == NULL) &&
- (nv->nv_n1vmcx = alloc_vmcb()) == NULL )
+ (nv->nv_n1vmcx = alloc_vmcb(v, VCPU_STATE_IDX_VMCB,
+ &nv->nv_n1vmcx_pa)) == NULL )
{
printk("Failed to create a new VMCB\n");
return -ENOMEM;
}
svm->vmcb = nv->nv_n1vmcx;
+ svm->vmcb_pa = nv->nv_n1vmcx_pa;
vmcb = SVM_MAP_VMCB(v);
rc = construct_vmcb(v, vmcb);
SVM_UNMAP_VMCB(v, vmcb);
if ( rc != 0 )
{
- free_vmcb(nv->nv_n1vmcx);
+ free_vmcb(v, VCPU_STATE_IDX_VMCB, nv->nv_n1vmcx_pa);
nv->nv_n1vmcx = NULL;
+ nv->nv_n1vmcx_pa = INVALID_PADDR;
svm->vmcb = NULL;
+ svm->vmcb_pa = INVALID_PADDR;
return rc;
}
- svm->vmcb_pa = nv->nv_n1vmcx_pa = virt_to_maddr(svm->vmcb);
return 0;
}
@@ -248,7 +264,7 @@ void svm_destroy_vmcb(struct vcpu *v)
struct svm_vcpu *svm = &v->arch.hvm.svm;
if ( nv->nv_n1vmcx != NULL )
- free_vmcb(nv->nv_n1vmcx);
+ free_vmcb(v, VCPU_STATE_IDX_VMCB, nv->nv_n1vmcx_pa);
if ( svm->msrpm != NULL )
{
@@ -260,6 +276,7 @@ void svm_destroy_vmcb(struct vcpu *v)
nv->nv_n1vmcx = NULL;
nv->nv_n1vmcx_pa = INVALID_PADDR;
svm->vmcb = NULL;
+ svm->vmcb_pa = INVALID_PADDR;
}
static void svm_dump_sel(const char *name, const struct segment_register *s)
diff --git a/xen/arch/x86/hvm/svm/vmcb.h b/xen/arch/x86/hvm/svm/vmcb.h
index d49c910d46..9e1e9ec1cd 100644
--- a/xen/arch/x86/hvm/svm/vmcb.h
+++ b/xen/arch/x86/hvm/svm/vmcb.h
@@ -556,12 +556,19 @@ struct vmcb_struct {
uint64_t res18[291];
};
-struct vmcb_struct *alloc_vmcb(void);
-void free_vmcb(struct vmcb_struct *vmcb);
+/*
+ * VMCBs are domain-heap pages: hardware addresses them physically, and
+ * Xen maps each one at a fixed slot in the owning vCPU's state window,
+ * so the returned virtual address is valid only in that vCPU's context.
+ * @idx is the VCPU_STATE_IDX_* slot to place the mapping in.
+ */
+struct vmcb_struct *alloc_vmcb(struct vcpu *v, unsigned int idx, paddr_t *pa);
+void free_vmcb(struct vcpu *v, unsigned int idx, paddr_t pa);
/*
- * Access a VMCB of a vCPU that may not be current. SVM_MAP_VMCB() takes
- * the active VMCB, SVM_MAP_VMCB_ONE() a specific one.
+ * Access a VMCB of a vCPU that may not be current: the window VA for
+ * current, a transient mapping otherwise. SVM_MAP_VMCB() takes the active
+ * VMCB, SVM_MAP_VMCB_ONE() a specific one.
*/
struct vmcb_struct *svm_vmcb_map_one(const struct vcpu *v,
struct vmcb_struct *va, paddr_t pa);
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 30/44] x86/vmx: move the guest MSR load/save list off always-mapped memory
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (28 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 29/44] x86/svm: allocate VMCBs from the domain heap behind the vCPU state window George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 31/44] x86: access a vCPU's saved register frame through map/unmap helpers George Dunlap
` (13 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
The guest MSR load/save list holds guest MSR values (SPEC_CTRL, LBR,
PT, vPMU global control, and EFER on hardware without dedicated
controls), but lives on the xenheap, where it stays reachable from
every context. Move the page to the domain heap and map it at its
fixed slot in the owning vCPU's state window, so that under Address
Space Isolation it is only mapped while that vCPU's page tables are
loaded. Hardware is unaffected: the VMCS MSR-area pointers hold the
physical address, now derived from the page rather than from
virt_to_maddr() on a directmap VA.
The host MSR load list is left on the xenheap: it holds Xen's own
exit-time values, not guest data, so relocating it would serve no
isolation purpose.
The window VA is only valid in the owning vCPU's context. Almost all
accesses have v == current and keep going through vmx->msr_area
unchanged (the vmentry-path LBR fixups, the failed-vmentry dump, vPMU,
and the MSR intercepts). The remote accessors -- vmx_add_msr() /
vmx_del_msr() from vCPU construction, policy updates and context
load, and single-entry reads/writes from the HVM context save/load
and get/set-reg paths, all of which already require the target vCPU
to be paused -- dispatch through a transient mapping of the page,
dropped with unmap_vcpu_state_page(), which also drops this CPU's
translations of it:
* vmx_add_msr() and vmx_del_msr() map internally around their
search/insert/remove.
* vmx_read_guest_msr(), vmx_read_guest_loadonly_msr() and
vmx_write_guest_msr() move out of line and do the same, via
vmx_find_msr_in(), the existing lookup parameterised on the mapped
area. vmx_find_msr() itself is unchanged for its remaining
callers, which all run in current context and dereference the
returned pointer through the window.
vmx_destroy_vmcs() takes the page out of the window before freeing it,
with unmap_vcpu_state(): without per-vCPU page tables the window is in
the per-domain area that all its domain's vCPUs share, which outlives
a vCPU whose creation fails.
The new page pointer goes with the existing page_info pointers after
pi_desc rather than next to msr_area: struct pi_desc is 64-byte
aligned, so inserting eight bytes ahead of it would have added 56
bytes of padding, growing struct vmx_vcpu from 448 to 512 bytes and,
via the union in struct arch_vcpu, every struct vcpu by 64 bytes
(x86_64 defconfig).
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- Map and unmap through VMX_MAP_MSR_AREA() and VMX_UNMAP_MSR_AREA(),
which clears the pointer (George).
Changes in v4:
- New in this version.
---
xen/arch/x86/hvm/vmx/vmcs.c | 142 ++++++++++++++++++++++--
xen/arch/x86/include/asm/hvm/vmx/vmcs.h | 63 +++--------
2 files changed, 147 insertions(+), 58 deletions(-)
diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
index 8ec7ae7403..592722f1ca 100644
--- a/xen/arch/x86/hvm/vmx/vmcs.c
+++ b/xen/arch/x86/hvm/vmx/vmcs.c
@@ -1416,8 +1416,11 @@ static struct vmx_msr_entry *locate_msr_entry(
return start;
}
-struct vmx_msr_entry *vmx_find_msr(const struct vcpu *v, uint32_t msr,
- enum vmx_msr_list_type type)
+/* As vmx_find_msr(), but reaching the guest MSR area through @area. */
+static struct vmx_msr_entry *vmx_find_msr_in(const struct vcpu *v,
+ struct vmx_msr_entry *area,
+ uint32_t msr,
+ enum vmx_msr_list_type type)
{
const struct vmx_vcpu *vmx = &v->arch.hvm.vmx;
struct vmx_msr_entry *start = NULL, *ent, *end;
@@ -1435,11 +1438,11 @@ struct vmx_msr_entry *vmx_find_msr(const struct vcpu *v, uint32_t msr,
break;
case VMX_MSR_GUEST:
- start = vmx->msr_area;
+ start = area;
break;
case VMX_MSR_GUEST_LOADONLY:
- start = vmx->msr_area;
+ start = area;
substart = subend;
subend = total;
break;
@@ -1458,6 +1461,77 @@ struct vmx_msr_entry *vmx_find_msr(const struct vcpu *v, uint32_t msr,
return ((ent < end) && (ent->index == msr)) ? ent : NULL;
}
+struct vmx_msr_entry *vmx_find_msr(const struct vcpu *v, uint32_t msr,
+ enum vmx_msr_list_type type)
+{
+ ASSERT(v == current || type == VMX_MSR_HOST);
+
+ return vmx_find_msr_in(v, v->arch.hvm.vmx.msr_area, msr, type);
+}
+
+static struct vmx_msr_entry *vmx_msr_area_map(const struct vcpu *v)
+{
+ const struct vmx_vcpu *vmx = &v->arch.hvm.vmx;
+
+ if ( v == current || !vmx->msr_area )
+ return vmx->msr_area;
+
+ return map_domain_page(page_to_mfn(vmx->msr_area_pg));
+}
+
+static void vmx_msr_area_unmap(const struct vcpu *v,
+ const struct vmx_msr_entry *area)
+{
+ if ( v != current && area )
+ unmap_vcpu_state_page(area);
+}
+
+#define VMX_MAP_MSR_AREA(v) vmx_msr_area_map(v)
+
+/* Drops the mapping of a guest MSR area and nullifies its pointer. */
+#define VMX_UNMAP_MSR_AREA(v, area) do { \
+ vmx_msr_area_unmap(v, area); \
+ (area) = NULL; \
+} while ( false )
+
+int vmx_read_guest_msr(const struct vcpu *v, uint32_t msr, uint64_t *val)
+{
+ struct vmx_msr_entry *area = VMX_MAP_MSR_AREA(v);
+ const struct vmx_msr_entry *ent =
+ area ? vmx_find_msr_in(v, area, msr, VMX_MSR_GUEST) : NULL;
+
+ *val = ent ? ent->data : 0;
+ VMX_UNMAP_MSR_AREA(v, area);
+
+ return ent ? 0 : -ESRCH;
+}
+
+int vmx_read_guest_loadonly_msr(const struct vcpu *v, uint32_t msr,
+ uint64_t *val)
+{
+ struct vmx_msr_entry *area = VMX_MAP_MSR_AREA(v);
+ const struct vmx_msr_entry *ent =
+ area ? vmx_find_msr_in(v, area, msr, VMX_MSR_GUEST_LOADONLY) : NULL;
+
+ *val = ent ? ent->data : 0;
+ VMX_UNMAP_MSR_AREA(v, area);
+
+ return ent ? 0 : -ESRCH;
+}
+
+int vmx_write_guest_msr(struct vcpu *v, uint32_t msr, uint64_t val)
+{
+ struct vmx_msr_entry *area = VMX_MAP_MSR_AREA(v);
+ struct vmx_msr_entry *ent =
+ area ? vmx_find_msr_in(v, area, msr, VMX_MSR_GUEST) : NULL;
+
+ if ( ent )
+ ent->data = val;
+ VMX_UNMAP_MSR_AREA(v, area);
+
+ return ent ? 0 : -ESRCH;
+}
+
int vmx_add_msr(struct vcpu *v, uint32_t msr, uint64_t val,
enum vmx_msr_list_type type)
{
@@ -1503,13 +1577,38 @@ int vmx_add_msr(struct vcpu *v, uint32_t msr, uint64_t val,
{
paddr_t addr;
- if ( (*ptr = alloc_xenheap_page()) == NULL )
+ if ( type == VMX_MSR_HOST )
{
- rc = -ENOMEM;
- goto out;
+ if ( (*ptr = alloc_xenheap_page()) == NULL )
+ {
+ rc = -ENOMEM;
+ goto out;
+ }
+
+ addr = virt_to_maddr(*ptr);
}
+ else
+ {
+ struct page_info *pg = alloc_domheap_page(v->domain,
+ MEMF_no_owner);
+ mfn_t mfn;
- addr = virt_to_maddr(*ptr);
+ if ( pg == NULL )
+ {
+ rc = -ENOMEM;
+ goto out;
+ }
+
+ mfn = page_to_mfn(pg);
+ clear_domain_page(mfn);
+ populate_perdomain_mapping(v,
+ VCPU_STATE_ADDR(v,
+ VCPU_STATE_IDX_VMX_MSR),
+ &mfn, 1, __PAGE_HYPERVISOR_RW);
+ vmx->msr_area_pg = pg;
+ *ptr = (void *)VCPU_STATE_ADDR(v, VCPU_STATE_IDX_VMX_MSR);
+ addr = mfn_to_maddr(mfn);
+ }
switch ( type )
{
@@ -1525,7 +1624,7 @@ int vmx_add_msr(struct vcpu *v, uint32_t msr, uint64_t val,
}
}
- start = *ptr;
+ start = type == VMX_MSR_HOST ? *ptr : VMX_MAP_MSR_AREA(v);
end = start + total;
ent = locate_msr_entry(start + substart, start + subend, msr);
@@ -1565,6 +1664,8 @@ int vmx_add_msr(struct vcpu *v, uint32_t msr, uint64_t val,
rc = 0;
out:
+ if ( type != VMX_MSR_HOST )
+ VMX_UNMAP_MSR_AREA(v, start);
vmx_vmcs_exit(v);
return rc;
@@ -1576,6 +1677,7 @@ int vmx_del_msr(struct vcpu *v, uint32_t msr, enum vmx_msr_list_type type)
struct vmx_msr_entry *start = NULL, *ent, *end;
unsigned int substart = 0, subend = vmx->msr_save_count;
unsigned int total = vmx->msr_load_count;
+ int rc = 0;
ASSERT(v == current || !vcpu_runnable(v));
@@ -1605,11 +1707,17 @@ int vmx_del_msr(struct vcpu *v, uint32_t msr, enum vmx_msr_list_type type)
if ( !start )
return -ESRCH;
+ if ( type != VMX_MSR_HOST )
+ start = VMX_MAP_MSR_AREA(v);
+
end = start + total;
ent = locate_msr_entry(start + substart, start + subend, msr);
if ( (ent == end) || (ent->index != msr) )
- return -ESRCH;
+ {
+ rc = -ESRCH;
+ goto out;
+ }
memmove(ent, ent + 1, sizeof(*ent) * (end - ent - 1));
@@ -1632,7 +1740,11 @@ int vmx_del_msr(struct vcpu *v, uint32_t msr, enum vmx_msr_list_type type)
vmx_vmcs_exit(v);
- return 0;
+ out:
+ if ( type != VMX_MSR_HOST )
+ VMX_UNMAP_MSR_AREA(v, start);
+
+ return rc;
}
void vmx_set_eoi_exit_bitmap(struct vcpu *v, u8 vector)
@@ -1829,7 +1941,13 @@ void vmx_destroy_vmcs(struct vcpu *v)
vmx_free_vmcs(vmx->vmcs_pa);
free_xenheap_page(v->arch.hvm.vmx.host_msr_area);
- free_xenheap_page(v->arch.hvm.vmx.msr_area);
+ if ( v->arch.hvm.vmx.msr_area_pg )
+ {
+ unmap_vcpu_state(v, VCPU_STATE_IDX_VMX_MSR, 1);
+ free_domheap_page(v->arch.hvm.vmx.msr_area_pg);
+ v->arch.hvm.vmx.msr_area_pg = NULL;
+ v->arch.hvm.vmx.msr_area = NULL;
+ }
free_xenheap_page(v->arch.hvm.vmx.msr_bitmap);
}
diff --git a/xen/arch/x86/include/asm/hvm/vmx/vmcs.h b/xen/arch/x86/include/asm/hvm/vmx/vmcs.h
index d0716e97d2..e5730780e2 100644
--- a/xen/arch/x86/include/asm/hvm/vmx/vmcs.h
+++ b/xen/arch/x86/include/asm/hvm/vmx/vmcs.h
@@ -140,6 +140,10 @@ struct vmx_vcpu {
* context. However, the data can be modified by toolstack/migration
* actions. Remote access is only permitted for paused vcpus, and is
* protected under the domctl lock.
+ *
+ * The guest list is a domain-heap page (msr_area_pg, below) mapped at a
+ * fixed slot in the owning vCPU's state window: msr_area holds that
+ * window VA, which is only valid in the owning vCPU's context.
*/
struct vmx_msr_entry *msr_area;
struct vmx_msr_entry *host_msr_area;
@@ -185,6 +189,9 @@ struct vmx_vcpu {
struct page_info *pml_pg;
+ /* Backing page of msr_area; see the comment there. */
+ struct page_info *msr_area_pg;
+
/* Bitmask of trapped CR4 bits. */
unsigned long cr4_host_mask;
@@ -669,55 +676,19 @@ static inline int vmx_add_host_load_msr(struct vcpu *v, uint32_t msr,
return vmx_add_msr(v, msr, val, VMX_MSR_HOST);
}
+/*
+ * Look an MSR up in the guest or host list of @v. The returned pointer
+ * reaches the guest lists through the vCPU state window, so for those it
+ * is only valid (and the call only permitted) in current context; the
+ * single-entry accessors below handle paused remote vCPUs.
+ */
struct vmx_msr_entry *vmx_find_msr(const struct vcpu *v, uint32_t msr,
enum vmx_msr_list_type type);
-static inline int vmx_read_guest_msr(const struct vcpu *v, uint32_t msr,
- uint64_t *val)
-{
- const struct vmx_msr_entry *ent = vmx_find_msr(v, msr, VMX_MSR_GUEST);
-
- if ( !ent )
- {
- *val = 0;
- return -ESRCH;
- }
-
- *val = ent->data;
-
- return 0;
-}
-
-static inline int vmx_read_guest_loadonly_msr(
- const struct vcpu *v, uint32_t msr, uint64_t *val)
-{
- const struct vmx_msr_entry *ent =
- vmx_find_msr(v, msr, VMX_MSR_GUEST_LOADONLY);
-
- if ( !ent )
- {
- *val = 0;
- return -ESRCH;
- }
-
- *val = ent->data;
-
- return 0;
-}
-
-static inline int vmx_write_guest_msr(struct vcpu *v, uint32_t msr,
- uint64_t val)
-{
- struct vmx_msr_entry *ent = vmx_find_msr(v, msr, VMX_MSR_GUEST);
-
- if ( !ent )
- return -ESRCH;
-
- ent->data = val;
-
- return 0;
-}
-
+int vmx_read_guest_msr(const struct vcpu *v, uint32_t msr, uint64_t *val);
+int vmx_read_guest_loadonly_msr(const struct vcpu *v, uint32_t msr,
+ uint64_t *val);
+int vmx_write_guest_msr(struct vcpu *v, uint32_t msr, uint64_t val);
/* MSR intercept bitmap infrastructure. */
enum vmx_msr_intercept_type {
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 31/44] x86: access a vCPU's saved register frame through map/unmap helpers
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (29 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 30/44] x86/vmx: move the guest MSR load/save list off always-mapped memory George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 32/44] x86: move the saved register frame behind the vCPU state window George Dunlap
` (12 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
When a vCPU is switched out, __context_switch() copies its register
frame from the stack into struct vcpu, and back when the vCPU is
switched in. Code working on a vCPU other than the one running uses
that copy: vCPU creation, XEN_DOMCTL_{get,set}vcpucontext, saving and
loading the HVM CPU record, VCPUOP_initialise, INIT/SIPI and S3 resets,
the result of a hypercall continued on another CPU, PV dom0
construction, and the debug-key dumps.
Under Address Space Isolation the frame is to move out of struct vcpu,
which is mapped in every context, into memory which other domains'
contexts do not map. Its users then need a transient mapping whenever
they run in another context. Prepare for that:
introduce VCPU_MAP_REGS() and VCPU_UNMAP_REGS(), after
VCPU_MAP_XSAVE_AREA(), and convert every user but the context switch,
which moves the frame between the stack and its storage. The field
becomes _user_regs, so that an access bypassing the helpers fails to
build: once the frame can live elsewhere, such an access would silently
use a stale copy.
vcpu_show_registers() now prints from a copy of the frame for PV vCPUs
as well as HVM ones, so that it holds no mapping while printing, and
vmx_set_info_guest() reads EFLAGS.TF before engaging the VMCS.
No functional change.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New in this version.
---
xen/arch/x86/domain.c | 94 ++++++++++++++++++-------------
xen/arch/x86/domctl.c | 72 ++++++++++++-----------
xen/arch/x86/hvm/domain.c | 6 +-
xen/arch/x86/hvm/hvm.c | 89 ++++++++++++++++-------------
xen/arch/x86/hvm/vmx/vmcs.c | 9 +--
xen/arch/x86/hvm/vmx/vmx.c | 7 ++-
xen/arch/x86/hypercall.c | 4 +-
xen/arch/x86/include/asm/domain.h | 28 ++++++++-
xen/arch/x86/pv/dom0_build.c | 3 +-
xen/arch/x86/traps.c | 20 ++++---
10 files changed, 204 insertions(+), 128 deletions(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 633619b400..143f76babf 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -496,8 +496,12 @@ unsigned int arch_domain_struct_memflags(void)
/* Initialise various registers to their architectural INIT/RESET state. */
void arch_vcpu_regs_init(struct vcpu *v)
{
- memset(&v->arch.user_regs, 0, sizeof(v->arch.user_regs));
- v->arch.user_regs.eflags = X86_EFLAGS_MBS;
+ struct cpu_user_regs *regs = VCPU_MAP_REGS(v);
+
+ memset(regs, 0, sizeof(*regs));
+ regs->eflags = X86_EFLAGS_MBS;
+
+ VCPU_UNMAP_REGS(v, regs);
memset(v->arch.dr, 0, sizeof(v->arch.dr));
v->arch.dr6 = X86_DR6_DEFAULT;
@@ -1152,6 +1156,7 @@ int arch_set_info_guest(
{
struct domain *d = v->domain;
const struct cpu_policy *p = d->arch.cpu_policy;
+ struct cpu_user_regs *regs;
unsigned int i;
unsigned long flags;
bool compat;
@@ -1279,30 +1284,32 @@ int arch_set_info_guest(
else
vcpu_reset_fpu(v);
- memset(&v->arch.user_regs, 0, sizeof(v->arch.user_regs));
+ regs = VCPU_MAP_REGS(v);
+ memset(regs, 0, sizeof(*regs));
if ( !compat )
{
- v->arch.user_regs.r15 = c.nat->user_regs.r15;
- v->arch.user_regs.r14 = c.nat->user_regs.r14;
- v->arch.user_regs.r13 = c.nat->user_regs.r13;
- v->arch.user_regs.r12 = c.nat->user_regs.r12;
- v->arch.user_regs.rbp = c.nat->user_regs.rbp;
- v->arch.user_regs.rbx = c.nat->user_regs.rbx;
- v->arch.user_regs.r11 = c.nat->user_regs.r11;
- v->arch.user_regs.r10 = c.nat->user_regs.r10;
- v->arch.user_regs.r9 = c.nat->user_regs.r9;
- v->arch.user_regs.r8 = c.nat->user_regs.r8;
- v->arch.user_regs.rax = c.nat->user_regs.rax;
- v->arch.user_regs.rcx = c.nat->user_regs.rcx;
- v->arch.user_regs.rdx = c.nat->user_regs.rdx;
- v->arch.user_regs.rsi = c.nat->user_regs.rsi;
- v->arch.user_regs.rdi = c.nat->user_regs.rdi;
- v->arch.user_regs.rip = c.nat->user_regs.rip;
- v->arch.user_regs.cs = c.nat->user_regs.cs;
- v->arch.user_regs.rflags = (c.nat->user_regs.rflags & X86_EFLAGS_ALL) | X86_EFLAGS_MBS;
- v->arch.user_regs.rsp = c.nat->user_regs.rsp;
- v->arch.user_regs.ss = c.nat->user_regs.ss;
+ regs->r15 = c.nat->user_regs.r15;
+ regs->r14 = c.nat->user_regs.r14;
+ regs->r13 = c.nat->user_regs.r13;
+ regs->r12 = c.nat->user_regs.r12;
+ regs->rbp = c.nat->user_regs.rbp;
+ regs->rbx = c.nat->user_regs.rbx;
+ regs->r11 = c.nat->user_regs.r11;
+ regs->r10 = c.nat->user_regs.r10;
+ regs->r9 = c.nat->user_regs.r9;
+ regs->r8 = c.nat->user_regs.r8;
+ regs->rax = c.nat->user_regs.rax;
+ regs->rcx = c.nat->user_regs.rcx;
+ regs->rdx = c.nat->user_regs.rdx;
+ regs->rsi = c.nat->user_regs.rsi;
+ regs->rdi = c.nat->user_regs.rdi;
+ regs->rip = c.nat->user_regs.rip;
+ regs->cs = c.nat->user_regs.cs;
+ regs->rflags =
+ (c.nat->user_regs.rflags & X86_EFLAGS_ALL) | X86_EFLAGS_MBS;
+ regs->rsp = c.nat->user_regs.rsp;
+ regs->ss = c.nat->user_regs.ss;
v->arch.pv.es = c.nat->user_regs.es;
v->arch.pv.ds = c.nat->user_regs.ds;
v->arch.pv.fs = c.nat->user_regs.fs;
@@ -1315,18 +1322,19 @@ int arch_set_info_guest(
#ifdef CONFIG_COMPAT
else
{
- v->arch.user_regs.ebx = c.cmp->user_regs.ebx;
- v->arch.user_regs.ecx = c.cmp->user_regs.ecx;
- v->arch.user_regs.edx = c.cmp->user_regs.edx;
- v->arch.user_regs.esi = c.cmp->user_regs.esi;
- v->arch.user_regs.edi = c.cmp->user_regs.edi;
- v->arch.user_regs.ebp = c.cmp->user_regs.ebp;
- v->arch.user_regs.eax = c.cmp->user_regs.eax;
- v->arch.user_regs.eip = c.cmp->user_regs.eip;
- v->arch.user_regs.cs = c.cmp->user_regs.cs;
- v->arch.user_regs.eflags = (c.cmp->user_regs.eflags & X86_EFLAGS_ALL) | X86_EFLAGS_MBS;
- v->arch.user_regs.esp = c.cmp->user_regs.esp;
- v->arch.user_regs.ss = c.cmp->user_regs.ss;
+ regs->ebx = c.cmp->user_regs.ebx;
+ regs->ecx = c.cmp->user_regs.ecx;
+ regs->edx = c.cmp->user_regs.edx;
+ regs->esi = c.cmp->user_regs.esi;
+ regs->edi = c.cmp->user_regs.edi;
+ regs->ebp = c.cmp->user_regs.ebp;
+ regs->eax = c.cmp->user_regs.eax;
+ regs->eip = c.cmp->user_regs.eip;
+ regs->cs = c.cmp->user_regs.cs;
+ regs->eflags =
+ (c.cmp->user_regs.eflags & X86_EFLAGS_ALL) | X86_EFLAGS_MBS;
+ regs->esp = c.cmp->user_regs.esp;
+ regs->ss = c.cmp->user_regs.ss;
v->arch.pv.es = c.cmp->user_regs.es;
v->arch.pv.ds = c.cmp->user_regs.ds;
v->arch.pv.fs = c.cmp->user_regs.fs;
@@ -1341,6 +1349,8 @@ int arch_set_info_guest(
}
#endif
+ VCPU_UNMAP_REGS(v, regs);
+
if ( v->vcpu_id == 0 && (c(vm_assist) & ~arch_vm_assist_valid_mask(d)) )
return -EINVAL;
@@ -1359,12 +1369,16 @@ int arch_set_info_guest(
}
#ifdef CONFIG_PV
+ regs = VCPU_MAP_REGS(v);
+
/* IOPL privileges are virtualised. */
- v->arch.pv.iopl = v->arch.user_regs.eflags & X86_EFLAGS_IOPL;
- v->arch.user_regs.eflags &= ~X86_EFLAGS_IOPL;
+ v->arch.pv.iopl = regs->eflags & X86_EFLAGS_IOPL;
+ regs->eflags &= ~X86_EFLAGS_IOPL;
/* Ensure real hardware interrupts are enabled. */
- v->arch.user_regs.eflags |= X86_EFLAGS_IF;
+ regs->eflags |= X86_EFLAGS_IF;
+
+ VCPU_UNMAP_REGS(v, regs);
nr_gdt_frames = DIV_ROUND_UP(c(gdt_ents), 512);
if ( nr_gdt_frames > ARRAY_SIZE(v->arch.pv.gdt_frames) )
@@ -2135,7 +2149,7 @@ static void __context_switch(void)
if ( !is_idle_domain(pd) )
{
ASSERT(read_atomic(&p->dirty_cpu) == cpu);
- memcpy(&p->arch.user_regs, stack_regs, CTXT_SWITCH_STACK_BYTES);
+ memcpy(&p->arch._user_regs, stack_regs, CTXT_SWITCH_STACK_BYTES);
vcpu_save_fpu(p);
pd->arch.ctxt_switch->from(p);
}
@@ -2151,7 +2165,7 @@ static void __context_switch(void)
if ( !is_idle_domain(nd) )
{
- memcpy(stack_regs, &n->arch.user_regs, CTXT_SWITCH_STACK_BYTES);
+ memcpy(stack_regs, &n->arch._user_regs, CTXT_SWITCH_STACK_BYTES);
if ( cpu_has_xsave )
{
if ( !set_xcr0(n->arch.xcr0 ?: XSTATE_FP_SSE) )
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 1e00510702..04bcde5580 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1461,6 +1461,7 @@ void arch_get_info_guest(struct vcpu *v, vcpu_guest_context_u c)
const struct domain *d = v->domain;
bool compat = is_pv_32bit_domain(d);
const struct xsave_struct *xsave_area;
+ const struct cpu_user_regs *regs;
#ifdef CONFIG_COMPAT
#define c(fld) (!compat ? (c.nat->fld) : (c.cmp->fld))
#else
@@ -1479,29 +1480,32 @@ void arch_get_info_guest(struct vcpu *v, vcpu_guest_context_u c)
c(flags |= VGCF_i387_valid);
if ( !(v->pause_flags & VPF_down) )
c(flags |= VGCF_online);
+
+ regs = VCPU_MAP_REGS(v);
+
if ( !compat )
{
/* Backing memory is pre-zeroed. */
- c.nat->user_regs.r15 = v->arch.user_regs.r15;
- c.nat->user_regs.r14 = v->arch.user_regs.r14;
- c.nat->user_regs.r13 = v->arch.user_regs.r13;
- c.nat->user_regs.r12 = v->arch.user_regs.r12;
- c.nat->user_regs.rbp = v->arch.user_regs.rbp;
- c.nat->user_regs.rbx = v->arch.user_regs.rbx;
- c.nat->user_regs.r11 = v->arch.user_regs.r11;
- c.nat->user_regs.r10 = v->arch.user_regs.r10;
- c.nat->user_regs.r9 = v->arch.user_regs.r9;
- c.nat->user_regs.r8 = v->arch.user_regs.r8;
- c.nat->user_regs.rax = v->arch.user_regs.rax;
- c.nat->user_regs.rcx = v->arch.user_regs.rcx;
- c.nat->user_regs.rdx = v->arch.user_regs.rdx;
- c.nat->user_regs.rsi = v->arch.user_regs.rsi;
- c.nat->user_regs.rdi = v->arch.user_regs.rdi;
- c.nat->user_regs.rip = v->arch.user_regs.rip;
- c.nat->user_regs.cs = v->arch.user_regs.cs;
- c.nat->user_regs.rflags = v->arch.user_regs.rflags;
- c.nat->user_regs.rsp = v->arch.user_regs.rsp;
- c.nat->user_regs.ss = v->arch.user_regs.ss;
+ c.nat->user_regs.r15 = regs->r15;
+ c.nat->user_regs.r14 = regs->r14;
+ c.nat->user_regs.r13 = regs->r13;
+ c.nat->user_regs.r12 = regs->r12;
+ c.nat->user_regs.rbp = regs->rbp;
+ c.nat->user_regs.rbx = regs->rbx;
+ c.nat->user_regs.r11 = regs->r11;
+ c.nat->user_regs.r10 = regs->r10;
+ c.nat->user_regs.r9 = regs->r9;
+ c.nat->user_regs.r8 = regs->r8;
+ c.nat->user_regs.rax = regs->rax;
+ c.nat->user_regs.rcx = regs->rcx;
+ c.nat->user_regs.rdx = regs->rdx;
+ c.nat->user_regs.rsi = regs->rsi;
+ c.nat->user_regs.rdi = regs->rdi;
+ c.nat->user_regs.rip = regs->rip;
+ c.nat->user_regs.cs = regs->cs;
+ c.nat->user_regs.rflags = regs->rflags;
+ c.nat->user_regs.rsp = regs->rsp;
+ c.nat->user_regs.ss = regs->ss;
c.nat->user_regs.es = v->arch.pv.es;
c.nat->user_regs.ds = v->arch.pv.ds;
c.nat->user_regs.fs = v->arch.pv.fs;
@@ -1515,18 +1519,18 @@ void arch_get_info_guest(struct vcpu *v, vcpu_guest_context_u c)
else
{
/* Backing memory is pre-zeroed. */
- c.cmp->user_regs.ebp = v->arch.user_regs.ebp;
- c.cmp->user_regs.ebx = v->arch.user_regs.ebx;
- c.cmp->user_regs.eax = v->arch.user_regs.eax;
- c.cmp->user_regs.ecx = v->arch.user_regs.ecx;
- c.cmp->user_regs.edx = v->arch.user_regs.edx;
- c.cmp->user_regs.esi = v->arch.user_regs.esi;
- c.cmp->user_regs.edi = v->arch.user_regs.edi;
- c.cmp->user_regs.eip = v->arch.user_regs.eip;
- c.cmp->user_regs.cs = v->arch.user_regs.cs;
- c.cmp->user_regs.eflags = v->arch.user_regs.eflags;
- c.cmp->user_regs.esp = v->arch.user_regs.esp;
- c.cmp->user_regs.ss = v->arch.user_regs.ss;
+ c.cmp->user_regs.ebp = regs->ebp;
+ c.cmp->user_regs.ebx = regs->ebx;
+ c.cmp->user_regs.eax = regs->eax;
+ c.cmp->user_regs.ecx = regs->ecx;
+ c.cmp->user_regs.edx = regs->edx;
+ c.cmp->user_regs.esi = regs->esi;
+ c.cmp->user_regs.edi = regs->edi;
+ c.cmp->user_regs.eip = regs->eip;
+ c.cmp->user_regs.cs = regs->cs;
+ c.cmp->user_regs.eflags = regs->eflags;
+ c.cmp->user_regs.esp = regs->esp;
+ c.cmp->user_regs.ss = regs->ss;
c.cmp->user_regs.es = v->arch.pv.es;
c.cmp->user_regs.ds = v->arch.pv.ds;
c.cmp->user_regs.fs = v->arch.pv.fs;
@@ -1638,10 +1642,12 @@ void arch_get_info_guest(struct vcpu *v, vcpu_guest_context_u c)
}
#endif
- if ( guest_kernel_mode(v, &v->arch.user_regs) )
+ if ( guest_kernel_mode(v, regs) )
c(flags |= VGCF_in_kernel);
}
+ VCPU_UNMAP_REGS(v, regs);
+
c(vm_assist = d->vm_assist);
#undef c
}
diff --git a/xen/arch/x86/hvm/domain.c b/xen/arch/x86/hvm/domain.c
index 978d8b2222..71e16e25ba 100644
--- a/xen/arch/x86/hvm/domain.c
+++ b/xen/arch/x86/hvm/domain.c
@@ -103,7 +103,7 @@ static int check_segment(struct segment_register *reg, enum x86_segment seg)
int arch_set_info_hvm_guest(struct vcpu *v, const struct vcpu_hvm_context *ctx)
{
const struct domain *d = v->domain;
- struct cpu_user_regs *uregs = &v->arch.user_regs;
+ struct cpu_user_regs *uregs;
struct segment_register cs, ds, ss, es, tr;
const char *errstr;
int rc;
@@ -188,6 +188,7 @@ int arch_set_info_hvm_guest(struct vcpu *v, const struct vcpu_hvm_context *ctx)
return -EINVAL;
}
+ uregs = VCPU_MAP_REGS(v);
uregs->rax = regs->eax;
uregs->rcx = regs->ecx;
uregs->rdx = regs->edx;
@@ -198,6 +199,7 @@ int arch_set_info_hvm_guest(struct vcpu *v, const struct vcpu_hvm_context *ctx)
uregs->rdi = regs->edi;
uregs->rip = regs->eip;
uregs->rflags = (regs->eflags & X86_EFLAGS_ALL) | X86_EFLAGS_MBS;
+ VCPU_UNMAP_REGS(v, uregs);
v->arch.hvm.guest_cr[0] = regs->cr0;
v->arch.hvm.guest_cr[3] = regs->cr3;
@@ -239,6 +241,7 @@ int arch_set_info_hvm_guest(struct vcpu *v, const struct vcpu_hvm_context *ctx)
return -EINVAL;
}
+ uregs = VCPU_MAP_REGS(v);
uregs->rax = regs->rax;
uregs->rcx = regs->rcx;
uregs->rdx = regs->rdx;
@@ -249,6 +252,7 @@ int arch_set_info_hvm_guest(struct vcpu *v, const struct vcpu_hvm_context *ctx)
uregs->rdi = regs->rdi;
uregs->rip = regs->rip;
uregs->rflags = (regs->rflags & X86_EFLAGS_ALL) | X86_EFLAGS_MBS;
+ VCPU_UNMAP_REGS(v, uregs);
v->arch.hvm.guest_cr[0] = regs->cr0;
v->arch.hvm.guest_cr[3] = regs->cr3;
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 71da15c0a4..eadccf3d79 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -827,28 +827,29 @@ HVM_REGISTER_SAVE_RESTORE(TSC_ADJUST, hvm_save_tsc_adjust, NULL,
static int cf_check hvm_save_cpu_ctxt(struct vcpu *v, hvm_domain_context_t *h)
{
+ const struct cpu_user_regs *regs = VCPU_MAP_REGS(v);
struct segment_register seg;
struct hvm_hw_cpu ctxt = {
.tsc = hvm_get_guest_tsc_fixed(v, v->domain->arch.hvm.sync_tsc),
.msr_tsc_aux = v->arch.msrs->tsc_aux,
- .rax = v->arch.user_regs.rax,
- .rbx = v->arch.user_regs.rbx,
- .rcx = v->arch.user_regs.rcx,
- .rdx = v->arch.user_regs.rdx,
- .rbp = v->arch.user_regs.rbp,
- .rsi = v->arch.user_regs.rsi,
- .rdi = v->arch.user_regs.rdi,
- .rsp = v->arch.user_regs.rsp,
- .rip = v->arch.user_regs.rip,
- .rflags = v->arch.user_regs.rflags,
- .r8 = v->arch.user_regs.r8,
- .r9 = v->arch.user_regs.r9,
- .r10 = v->arch.user_regs.r10,
- .r11 = v->arch.user_regs.r11,
- .r12 = v->arch.user_regs.r12,
- .r13 = v->arch.user_regs.r13,
- .r14 = v->arch.user_regs.r14,
- .r15 = v->arch.user_regs.r15,
+ .rax = regs->rax,
+ .rbx = regs->rbx,
+ .rcx = regs->rcx,
+ .rdx = regs->rdx,
+ .rbp = regs->rbp,
+ .rsi = regs->rsi,
+ .rdi = regs->rdi,
+ .rsp = regs->rsp,
+ .rip = regs->rip,
+ .rflags = regs->rflags,
+ .r8 = regs->r8,
+ .r9 = regs->r9,
+ .r10 = regs->r10,
+ .r11 = regs->r11,
+ .r12 = regs->r12,
+ .r13 = regs->r13,
+ .r14 = regs->r14,
+ .r15 = regs->r15,
.cr0 = v->arch.hvm.guest_cr[0],
.cr2 = v->arch.hvm.guest_cr[2],
.cr3 = v->arch.hvm.guest_cr[3],
@@ -863,6 +864,8 @@ static int cf_check hvm_save_cpu_ctxt(struct vcpu *v, hvm_domain_context_t *h)
};
const struct xsave_struct *xsave_area;
+ VCPU_UNMAP_REGS(v, regs);
+
/*
* We don't need to save state for a vcpu that is down; the restore
* code will leave it down if there is nothing saved.
@@ -1020,6 +1023,7 @@ static int cf_check hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
const struct cpu_policy *p = d->arch.cpu_policy;
unsigned int vcpuid = hvm_load_instance(h);
struct vcpu *v;
+ struct cpu_user_regs *regs;
struct hvm_hw_cpu ctxt;
struct segment_register seg;
const char *errstr;
@@ -1182,24 +1186,26 @@ static int cf_check hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
else
vcpu_reset_fpu(v);
- v->arch.user_regs.rax = ctxt.rax;
- v->arch.user_regs.rbx = ctxt.rbx;
- v->arch.user_regs.rcx = ctxt.rcx;
- v->arch.user_regs.rdx = ctxt.rdx;
- v->arch.user_regs.rbp = ctxt.rbp;
- v->arch.user_regs.rsi = ctxt.rsi;
- v->arch.user_regs.rdi = ctxt.rdi;
- v->arch.user_regs.rsp = ctxt.rsp;
- v->arch.user_regs.rip = ctxt.rip;
- v->arch.user_regs.rflags = ctxt.rflags | X86_EFLAGS_MBS;
- v->arch.user_regs.r8 = ctxt.r8;
- v->arch.user_regs.r9 = ctxt.r9;
- v->arch.user_regs.r10 = ctxt.r10;
- v->arch.user_regs.r11 = ctxt.r11;
- v->arch.user_regs.r12 = ctxt.r12;
- v->arch.user_regs.r13 = ctxt.r13;
- v->arch.user_regs.r14 = ctxt.r14;
- v->arch.user_regs.r15 = ctxt.r15;
+ regs = VCPU_MAP_REGS(v);
+ regs->rax = ctxt.rax;
+ regs->rbx = ctxt.rbx;
+ regs->rcx = ctxt.rcx;
+ regs->rdx = ctxt.rdx;
+ regs->rbp = ctxt.rbp;
+ regs->rsi = ctxt.rsi;
+ regs->rdi = ctxt.rdi;
+ regs->rsp = ctxt.rsp;
+ regs->rip = ctxt.rip;
+ regs->rflags = ctxt.rflags | X86_EFLAGS_MBS;
+ regs->r8 = ctxt.r8;
+ regs->r9 = ctxt.r9;
+ regs->r10 = ctxt.r10;
+ regs->r11 = ctxt.r11;
+ regs->r12 = ctxt.r12;
+ regs->r13 = ctxt.r13;
+ regs->r14 = ctxt.r14;
+ regs->r15 = ctxt.r15;
+ VCPU_UNMAP_REGS(v, regs);
v->arch.dr[0] = ctxt.dr0;
v->arch.dr[1] = ctxt.dr1;
v->arch.dr[2] = ctxt.dr2;
@@ -2563,7 +2569,7 @@ bool hvm_vcpu_virtual_to_linear(
{
unsigned long addr = offset, last_byte;
const struct cpu_user_regs *regs = v == current ? guest_cpu_user_regs()
- : &v->arch.user_regs;
+ : VCPU_MAP_REGS(v);
bool okay = false;
ASSERT(v == current || !vcpu_runnable(v));
@@ -2688,6 +2694,10 @@ bool hvm_vcpu_virtual_to_linear(
* failed. The permissions failure is not relevant to some callers.
*/
*linear_addr = addr;
+
+ if ( v != current )
+ VCPU_UNMAP_REGS(v, regs);
+
return okay;
}
@@ -3956,6 +3966,7 @@ void hvm_init_hypercall_page(struct domain *d, void *ptr)
void hvm_vcpu_reset_state(struct vcpu *v, uint16_t cs, uint16_t ip)
{
struct domain *d = v->domain;
+ struct cpu_user_regs *regs;
struct segment_register reg;
domain_lock(d);
@@ -3973,7 +3984,9 @@ void hvm_vcpu_reset_state(struct vcpu *v, uint16_t cs, uint16_t ip)
vcpu_reset_fpu(v);
arch_vcpu_regs_init(v);
- v->arch.user_regs.rip = ip;
+ regs = VCPU_MAP_REGS(v);
+ regs->rip = ip;
+ VCPU_UNMAP_REGS(v, regs);
v->arch.hvm.guest_cr[0] = X86_CR0_ET;
hvm_update_guest_cr(v, 0);
diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
index 592722f1ca..78b0af29f2 100644
--- a/xen/arch/x86/hvm/vmx/vmcs.c
+++ b/xen/arch/x86/hvm/vmx/vmcs.c
@@ -2083,15 +2083,13 @@ static void vmx_dump_sel2(const char *name, uint32_t lim)
void vmcs_dump_vcpu(struct vcpu *v)
{
- struct cpu_user_regs *regs = &v->arch.user_regs;
+ const struct cpu_user_regs *regs = v == current ? guest_cpu_user_regs()
+ : VCPU_MAP_REGS(v);
uint32_t vmentry_ctl, vmexit_ctl;
unsigned long cr4;
uint64_t efer;
unsigned int i, n;
- if ( v == current )
- regs = guest_cpu_user_regs();
-
vmx_vmcs_enter(v);
vmentry_ctl = vmr32(VM_ENTRY_CONTROLS),
@@ -2236,6 +2234,9 @@ void vmcs_dump_vcpu(struct vcpu *v)
vmr16(VIRTUAL_PROCESSOR_ID), vmr(VM_FUNCTION_CONTROL));
vmx_vmcs_exit(v);
+
+ if ( v != current )
+ VCPU_UNMAP_REGS(v, regs);
}
static void cf_check vmcs_dump(unsigned char ch)
diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c
index 3c7374a9ff..bdbcd0ad4d 100644
--- a/xen/arch/x86/hvm/vmx/vmx.c
+++ b/xen/arch/x86/hvm/vmx/vmx.c
@@ -2159,8 +2159,12 @@ static bool cf_check vmx_event_pending(const struct vcpu *v)
static void cf_check vmx_set_info_guest(struct vcpu *v)
{
+ const struct cpu_user_regs *regs = VCPU_MAP_REGS(v);
+ bool single_step = regs->eflags & X86_EFLAGS_TF;
unsigned long intr_shadow;
+ VCPU_UNMAP_REGS(v, regs);
+
vmx_vmcs_enter(v);
__vmwrite(GUEST_DR7, v->arch.dr7);
@@ -2174,8 +2178,7 @@ static void cf_check vmx_set_info_guest(struct vcpu *v)
* immediately vmexit and hence make no progress.
*/
__vmread(GUEST_INTERRUPTIBILITY_INFO, &intr_shadow);
- if ( v->domain->debugger_attached &&
- (v->arch.user_regs.eflags & X86_EFLAGS_TF) &&
+ if ( v->domain->debugger_attached && single_step &&
(intr_shadow & VMX_INTR_SHADOW_STI) )
{
intr_shadow &= ~VMX_INTR_SHADOW_STI;
diff --git a/xen/arch/x86/hypercall.c b/xen/arch/x86/hypercall.c
index dc0a90ca09..bfdc462874 100644
--- a/xen/arch/x86/hypercall.c
+++ b/xen/arch/x86/hypercall.c
@@ -107,9 +107,11 @@ unsigned long hypercall_create_continuation(
void arch_hypercall_tasklet_result(struct vcpu *v, long res)
{
- struct cpu_user_regs *regs = &v->arch.user_regs;
+ struct cpu_user_regs *regs = VCPU_MAP_REGS(v);
regs->rax = res;
+
+ VCPU_UNMAP_REGS(v, regs);
}
int hypercall_xlat_continuation(unsigned int *id, unsigned int nr,
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 0e405157b4..457eb69ace 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -632,7 +632,12 @@ struct pv_vcpu
struct arch_vcpu
{
- struct cpu_user_regs user_regs;
+ /*
+ * The vCPU's register frame while it is not running: __context_switch()
+ * copies it here from the stack when the vCPU is switched out, and back
+ * when it is switched in. Everything else goes through VCPU_MAP_REGS().
+ */
+ struct cpu_user_regs _user_regs;
/* Debug registers. */
unsigned long dr[4];
@@ -741,6 +746,27 @@ static inline void free_vcpu_guest_context(struct vcpu_guest_context *vgc)
vfree(vgc);
}
+/*
+ * Fetch a pointer to the register frame a vCPU was last switched out with,
+ * or which a toolstack, debugger or reset has put there since: the vCPU's
+ * registers for code working on a vCPU other than the one running (whose
+ * registers are on the stack, see guest_cpu_user_regs()).
+ *
+ * The pointer is valid until VCPU_UNMAP_REGS(), which clears it. A mapping
+ * must not be held across a scheduling point.
+ *
+ * @param v Owner of the register frame
+ */
+#define VCPU_MAP_REGS(v) (&(v)->arch._user_regs)
+
+/*
+ * Drops the mapping of a vCPU's register frame and nullifies its pointer.
+ *
+ * @param v Owner of the register frame
+ * @param r Register frame of v, from VCPU_MAP_REGS()
+ */
+#define VCPU_UNMAP_REGS(v, r) do { (void)(v); (r) = NULL; } while ( false )
+
void arch_vcpu_regs_init(struct vcpu *v);
struct vcpu_hvm_context;
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index 829eafd67d..a08362df16 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -1007,7 +1007,7 @@ static int __init dom0_construct(const struct boot_domain *bd)
* rSI = start_info
* [rAX,rBX,rCX,rDX,rDI,rBP,R8-R15 are zero]
*/
- regs = &v->arch.user_regs;
+ regs = VCPU_MAP_REGS(v);
v->arch.pv.ds = (compat ? FLAT_COMPAT_KERNEL_DS : FLAT_KERNEL_DS);
v->arch.pv.es = (compat ? FLAT_COMPAT_KERNEL_DS : FLAT_KERNEL_DS);
v->arch.pv.fs = (compat ? FLAT_COMPAT_KERNEL_DS : FLAT_KERNEL_DS);
@@ -1018,6 +1018,7 @@ static int __init dom0_construct(const struct boot_domain *bd)
regs->rsp = vstack_end;
regs->rsi = vstartinfo_start;
regs->eflags = X86_EFLAGS_IF | X86_EFLAGS_MBS;
+ VCPU_UNMAP_REGS(v, regs);
/*
* We don't call arch_set_info_guest(), so some initialisation needs doing
diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
index ac01ba0872..3b4a01ba52 100644
--- a/xen/arch/x86/traps.c
+++ b/xen/arch/x86/traps.c
@@ -273,16 +273,17 @@ void show_registers(const struct cpu_user_regs *regs)
void vcpu_show_registers(struct vcpu *v)
{
- const struct cpu_user_regs *regs = &v->arch.user_regs;
- struct cpu_user_regs aux_regs;
+ const struct cpu_user_regs *regs = VCPU_MAP_REGS(v);
+ struct cpu_user_regs aux_regs = *regs;
struct extra_state state;
enum context context;
+ VCPU_UNMAP_REGS(v, regs);
+ regs = &aux_regs;
+
if ( is_hvm_vcpu(v) )
{
- aux_regs = *regs;
get_hvm_registers(v, &aux_regs, &state);
- regs = &aux_regs;
context = CTXT_hvm_guest;
}
else
@@ -916,6 +917,7 @@ void cf_check show_execution_state(const struct cpu_user_regs *regs)
void vcpu_show_execution_state(struct vcpu *v)
{
+ const struct cpu_user_regs *regs;
unsigned long flags = 0;
if ( test_bit(_VPF_down, &v->pause_flags) )
@@ -961,12 +963,16 @@ void vcpu_show_execution_state(struct vcpu *v)
*/
console_unlock_recursive_irqrestore(flags);
- show_hvm_stack(v, &v->arch.user_regs);
+ regs = VCPU_MAP_REGS(v);
+ show_hvm_stack(v, regs);
+ VCPU_UNMAP_REGS(v, regs);
}
else
{
- if ( guest_kernel_mode(v, &v->arch.user_regs) )
- show_guest_stack(v, &v->arch.user_regs);
+ regs = VCPU_MAP_REGS(v);
+ if ( guest_kernel_mode(v, regs) )
+ show_guest_stack(v, regs);
+ VCPU_UNMAP_REGS(v, regs);
console_unlock_recursive_irqrestore(flags);
}
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 32/44] x86: move the saved register frame behind the vCPU state window
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (30 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 31/44] x86: access a vCPU's saved register frame through map/unmap helpers George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 33/44] x86/hvm: move the emulator's operand caches " George Dunlap
` (11 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
While a vCPU is not running, its register frame is saved in struct
vcpu: a xenheap page, mapped in every context, the sparse directmap view
included. Under Address Space Isolation that leaves the general-purpose
registers of every descheduled vCPU reachable from other domains'
contexts, whether or not the vCPU's own domain is on the sparse view.
Keep the frame instead in a domain-heap page of the vCPU's own, mapped
for the life of the vCPU at a new slot (index 6) of its state window,
like the SVM VMCB and the VMX MSR list; indexes 3-5 are left for the
guest XSAVE area, which takes up to three pages. Like the VMCB and MSR
list moves, this does not depend on asi=, and it applies to every guest
vCPU, PV and HVM: struct vcpu no longer holds a copy. The page is
allocated in arch_vcpu_create() ahead of arch_vcpu_regs_init(), which
initialises the frame, and unmapped before it is freed, on the failure
path as at teardown.
An HVM vCPU using per-vCPU page tables has its window in its own page
tables. For any other vCPU, PV ones included, the window is in the
per-domain area that all its domain's vCPUs share, which only that
domain's page tables map: a context on the sparse view reaches no other
domain's frames. That includes a domain left off the sparse view, such
as dom0 under asi=1,dom0=0, whose vCPUs' frames are no longer mapped in
the contexts of the domUs on the sparse view.
__context_switch() saves the outgoing vCPU's frame while its page tables
are still loaded, and now restores the incoming vCPU's only after
write_ptbase(), once its window is mapped. Nothing in between reads the
frame: set_xcr0(), the FPU restore, the ctxt_switch->to() hooks, and the
PSR and GDT updates. Both copies go through the window address, with
interrupts off; a transient mapping would complete the lazy context
switch and re-enter __context_switch().
The save is skipped if the outgoing vCPU's page tables are no longer
loaded. Only the shadow-mode paths which abandon a PV vCPU's page
tables early (update_cr3() failure in new_guest_cr3() and
_toggle_guest_pt()) leave them so, loading the idle page tables once
they have crashed the domain; anything else reaching that point is a
page-table tracking bug, fatal in every build. After such a crash the
vCPU's saved frame is the one it was last switched in with, not its
registers at the crash, and that is what XEN_DOMCTL_getvcpucontext and
the debug keys report for it.
VCPU_MAP_REGS() returns the window address when the vCPU is running
here on its own page tables (v == current and this_cpu(pgtable_vcpu) ==
v), and otherwise a transient mapping of the page, which, for a single
page, cannot fail. Its users all work on a vCPU other than the one
running here, normally paused or not yet run: vCPU creation, the
toolstack's vCPU context, the HVM CPU record, VCPUOP_initialise,
INIT/SIPI and S3 resets, a hypercall continued on another CPU, PV dom0
construction, and the debug-key dumps. The vCPU's own page tables
being loaded is not enough: while a CPU holds a vCPU's state lazily
they are loaded there, but an IPI syncing that state for another CPU
can switch them away at any time. VCPU_UNMAP_REGS() drops a transient
mapping with unmap_vcpu_state_page(), which also flushes this CPU's
translation.
The live frame on the pCPU's stack, which holds a running vCPU's
registers, is not affected.
Cost: a page per guest vCPU, and a transient mapping on each of the
cold paths above. On every switch into a guest vCPU, the restore
reaches the frame through a 4KiB, non-global PTE in the per-domain
area, just after switch_cr3_cr4() has flushed the TLB, so it takes a
page walk, where struct vcpu was reached through a superpage of the
full directmap. struct vcpu shrinks from 3072 to 2880 bytes (x86_64
defconfig).
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New in this version.
---
xen/arch/x86/domain.c | 98 ++++++++++++++++++++++++++++++-
xen/arch/x86/include/asm/config.h | 1 +
xen/arch/x86/include/asm/domain.h | 26 +++++---
3 files changed, 116 insertions(+), 9 deletions(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 143f76babf..03f6572086 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -17,6 +17,7 @@
#include <xen/cpu.h>
#include <xen/delay.h>
#include <xen/domain.h>
+#include <xen/domain_page.h>
#include <xen/errno.h>
#include <xen/event.h>
#include <xen/grant_table.h>
@@ -493,6 +494,66 @@ unsigned int arch_domain_struct_memflags(void)
return MEMF_bits(bits);
}
+/*
+ * A vCPU keeps its register frame, while it is not running, in a domain-heap
+ * page of its own, mapped for the life of the vCPU at a fixed slot of its
+ * state window.
+ */
+static struct cpu_user_regs *regs_window(const struct vcpu *v)
+{
+ return (void *)VCPU_STATE_ADDR(v, VCPU_STATE_IDX_REGS);
+}
+
+static int vcpu_regs_alloc(struct vcpu *v)
+{
+ struct page_info *pg = alloc_domheap_page(v->domain, MEMF_no_owner);
+ mfn_t mfn;
+
+ if ( !pg )
+ return -ENOMEM;
+
+ mfn = page_to_mfn(pg);
+ clear_domain_page(mfn);
+ populate_perdomain_mapping(v, VCPU_STATE_ADDR(v, VCPU_STATE_IDX_REGS),
+ &mfn, 1, __PAGE_HYPERVISOR_RW);
+ v->arch.regs_pg = pg;
+
+ return 0;
+}
+
+static void vcpu_regs_free(struct vcpu *v)
+{
+ struct page_info *pg = v->arch.regs_pg;
+
+ if ( !pg )
+ return;
+
+ unmap_vcpu_state(v, VCPU_STATE_IDX_REGS, 1);
+ v->arch.regs_pg = NULL;
+ free_domheap_page(pg);
+}
+
+struct cpu_user_regs *vcpu_map_regs(const struct vcpu *v)
+{
+ /*
+ * v's page tables being loaded is not enough: while this CPU holds v's
+ * state lazily, an IPI syncing that state for another CPU can switch
+ * them away at any time.
+ */
+ if ( v == current && this_cpu(pgtable_vcpu) == v )
+ return regs_window(v);
+
+ return __map_domain_page(v->arch.regs_pg);
+}
+
+void vcpu_unmap_regs(const struct vcpu *v, const struct cpu_user_regs *regs)
+{
+ if ( !regs || regs == regs_window(v) )
+ return;
+
+ unmap_vcpu_state_page(regs);
+}
+
/* Initialise various registers to their architectural INIT/RESET state. */
void arch_vcpu_regs_init(struct vcpu *v)
{
@@ -533,6 +594,7 @@ int arch_vcpu_create(struct vcpu *v)
BUILD_BUG_ON(VCPU_STATE_IDX_VMCB >= VCPU_STATE_NR_PAGES);
BUILD_BUG_ON(VCPU_STATE_IDX_N2VMCB >= VCPU_STATE_NR_PAGES);
BUILD_BUG_ON(VCPU_STATE_IDX_VMX_MSR >= VCPU_STATE_NR_PAGES);
+ BUILD_BUG_ON(VCPU_STATE_IDX_REGS >= VCPU_STATE_NR_PAGES);
/* Page tables only; torn down with the rest of the per-domain area. */
rc = create_perdomain_mapping(v, VCPU_STATE_START(v),
VCPU_STATE_NR_PAGES,
@@ -547,6 +609,10 @@ int arch_vcpu_create(struct vcpu *v)
vmce_init_vcpu(v);
+ /* The frame's storage, ahead of its initialisation. */
+ if ( (rc = vcpu_regs_alloc(v)) != 0 )
+ goto fail;
+
arch_vcpu_regs_init(v);
if ( (rc = init_vcpu_msr_policy(v)) )
@@ -589,6 +655,7 @@ int arch_vcpu_create(struct vcpu *v)
fail:
paging_vcpu_teardown(v);
vcpu_destroy_fpu(v);
+ vcpu_regs_free(v);
xfree(v->arch.msrs);
v->arch.msrs = NULL;
free_pervcpu_mappings(v);
@@ -613,6 +680,8 @@ void arch_vcpu_destroy(struct vcpu *v)
else
ASSERT_UNREACHABLE();
+ vcpu_regs_free(v);
+
free_pervcpu_mappings(v);
}
@@ -2135,6 +2204,19 @@ static void load_default_gdt(unsigned int cpu)
per_cpu(full_gdt_loaded, cpu) = false;
}
+/*
+ * The register frame of p or n, for __context_switch(), which runs with
+ * interrupts off and with the vCPU's own page tables loaded: the window
+ * address serves. A transient mapping is not an option from here: it would
+ * complete the lazy context switch and re-enter __context_switch().
+ */
+static struct cpu_user_regs *vcpu_user_regs(const struct vcpu *v)
+{
+ BUG_ON(this_cpu(pgtable_vcpu) != v);
+
+ return regs_window(v);
+}
+
static void __context_switch(void)
{
struct cpu_user_regs *stack_regs = guest_cpu_user_regs();
@@ -2149,7 +2231,17 @@ static void __context_switch(void)
if ( !is_idle_domain(pd) )
{
ASSERT(read_atomic(&p->dirty_cpu) == cpu);
- memcpy(&p->arch._user_regs, stack_regs, CTXT_SWITCH_STACK_BYTES);
+ /*
+ * p's frame is reached through its state window, mapped while p's
+ * own page tables are loaded, as they normally still are here. Not
+ * so after the shadow-mode paths which abandon a vCPU's tables early
+ * (update_cr3() failure), which have crashed the domain: the frame
+ * is left unsaved. A transient mapping would re-enter this function.
+ */
+ if ( likely(this_cpu(pgtable_vcpu) == p) )
+ memcpy(vcpu_user_regs(p), stack_regs, CTXT_SWITCH_STACK_BYTES);
+ else
+ BUG_ON(!domain_shutting_down(pd));
vcpu_save_fpu(p);
pd->arch.ctxt_switch->from(p);
}
@@ -2165,7 +2257,6 @@ static void __context_switch(void)
if ( !is_idle_domain(nd) )
{
- memcpy(stack_regs, &n->arch._user_regs, CTXT_SWITCH_STACK_BYTES);
if ( cpu_has_xsave )
{
if ( !set_xcr0(n->arch.xcr0 ?: XSTATE_FP_SSE) )
@@ -2189,6 +2280,9 @@ static void __context_switch(void)
write_ptbase(n);
+ if ( !is_idle_domain(nd) )
+ memcpy(stack_regs, vcpu_user_regs(n), CTXT_SWITCH_STACK_BYTES);
+
#ifdef CONFIG_PV
/* Prefetch the VMCB if we expect to use it later in the context switch */
if ( using_svm() && is_pv_64bit_domain(nd) && !is_idle_domain(nd) )
diff --git a/xen/arch/x86/include/asm/config.h b/xen/arch/x86/include/asm/config.h
index 8a265e25c8..c8d4a8ddc7 100644
--- a/xen/arch/x86/include/asm/config.h
+++ b/xen/arch/x86/include/asm/config.h
@@ -311,6 +311,7 @@ extern unsigned long xen_phys_start;
#define VCPU_STATE_IDX_VMCB 0
#define VCPU_STATE_IDX_N2VMCB 1
#define VCPU_STATE_IDX_VMX_MSR 2
+#define VCPU_STATE_IDX_REGS 6
#define ELFSIZE 64
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 457eb69ace..6b27af9eb5 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -633,11 +633,12 @@ struct pv_vcpu
struct arch_vcpu
{
/*
- * The vCPU's register frame while it is not running: __context_switch()
- * copies it here from the stack when the vCPU is switched out, and back
- * when it is switched in. Everything else goes through VCPU_MAP_REGS().
+ * The page holding the vCPU's register frame while it is not running,
+ * mapped in its state window: __context_switch() copies the frame there
+ * from the stack when the vCPU is switched out, and back when it is
+ * switched in. Everything else goes through VCPU_MAP_REGS().
*/
- struct cpu_user_regs _user_regs;
+ struct page_info *regs_pg;
/* Debug registers. */
unsigned long dr[4];
@@ -752,12 +753,20 @@ static inline void free_vcpu_guest_context(struct vcpu_guest_context *vgc)
* registers for code working on a vCPU other than the one running (whose
* registers are on the stack, see guest_cpu_user_regs()).
*
+ * The frame is in the vCPU's state window: the window address serves while
+ * the vCPU runs here on its own page tables, and the page is mapped
+ * transiently otherwise, which cannot fail.
+ *
* The pointer is valid until VCPU_UNMAP_REGS(), which clears it. A mapping
- * must not be held across a scheduling point.
+ * must not be held across a scheduling point: a transient one is only valid
+ * on the pCPU which made it.
*
* @param v Owner of the register frame
*/
-#define VCPU_MAP_REGS(v) (&(v)->arch._user_regs)
+struct cpu_user_regs *vcpu_map_regs(const struct vcpu *v);
+void vcpu_unmap_regs(const struct vcpu *v, const struct cpu_user_regs *regs);
+
+#define VCPU_MAP_REGS(v) vcpu_map_regs(v)
/*
* Drops the mapping of a vCPU's register frame and nullifies its pointer.
@@ -765,7 +774,10 @@ static inline void free_vcpu_guest_context(struct vcpu_guest_context *vgc)
* @param v Owner of the register frame
* @param r Register frame of v, from VCPU_MAP_REGS()
*/
-#define VCPU_UNMAP_REGS(v, r) do { (void)(v); (r) = NULL; } while ( false )
+#define VCPU_UNMAP_REGS(v, r) do { \
+ vcpu_unmap_regs(v, r); \
+ (r) = NULL; \
+} while ( false )
void arch_vcpu_regs_init(struct vcpu *v);
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 33/44] x86/hvm: move the emulator's operand caches behind the vCPU state window
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (31 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 32/44] x86: move the saved register frame behind the vCPU state window George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 34/44] x86/hvm: clear the vCPU's copy of an emulated I/O request once consumed George Dunlap
` (10 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
The HVM emulator keeps two per-vCPU caches of guest data: the read cache
(struct hvmemul_cache), which records every guest-memory read made while
an instruction is emulated -- guest page-table entries, instruction
bytes and memory operands -- so that a re-execution observes the same
values, and the three MMIO caches, which hold the operand bytes of an
instruction's MMIO accesses. Both are xvmalloc() allocations, from the
xmalloc pool on the xenheap, and so mapped in every context, the sparse
directmap view included. Disabling the read cache when emulation
completes leaves its entries in place.
Put both in a domain-heap page of the vCPU's own, mapped for the life of
the vCPU at a new slot (index 7) of its state window: the read cache,
counts included, then the MMIO caches, 4032 bytes of the page. As
"x86: move the saved register frame behind the vCPU state window" does
for the register frame, this applies to every HVM vCPU, whatever asi=
says; for a vCPU without per-vCPU page tables the window is in the
per-domain area that all its domain's vCPUs share, which only that
domain's page tables map. The pointers in struct hvm_vcpu_io hold
window addresses. hvmemul_cache_init() sets the page up through a
transient mapping, as the vCPU is still being built, and
hvmemul_cache_destroy(), now out of line, unmaps it before freeing it.
The emulator only ever works on the vCPU running, on its own page
tables, and reaches the caches through the window address;
hvmemul_find_mmio_cache() now asserts as much. The read cache also has
users in other contexts: a debugger's guest page walk on a paused vCPU
(gdbsx, and the guest stack dump of the debug keys), which reads and
extends it if the vCPU was paused mid-emulation, and hvmemul_cancel(),
which disables it, on toolstack, VCPUOP_initialise and INIT/SIPI paths.
Its accessors therefore map the page transiently unless the vCPU is
running on its own page tables, as VCPU_MAP_REGS() does: a single page,
so no new failure paths. They drop the mapping with
unmap_vcpu_state_page(), which also flushes this CPU's translation.
The read and write paths are split into an outer function, which checks
and maps the cache once, and a recursive inner one.
Cost: a page per HVM vCPU, where the caches took about 4KiB of the
xmalloc pool, and the emulator reaches them through a 4KiB mapping in
the per-domain area instead of the directmap.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New in this version.
Notes for reviewers: the sizes before and after, on x86-64
(CONFIG_PAGING_LEVELS 4, MAX_INST_LEN 15).
before after
read cache ents 16*2*(4+1) + 15*(4+1) (16*2 + 15) * (4+1)
= 235 (nents) = 235 (HVMEMUL_CACHE_ENTS)
read cache bytes 8 + 235*16 = 3768 the same offsetof(), at 0
MMIO buffer bytes 64 (max_bytes) 64 (MMIO_CACHE_BYTES)
MMIO cache bytes 24 + 64 = 88, three 88 (MMIO_CACHE_SIZE), three
The page:
offset bytes contents
0 8 num_ents, max_ents
8 3760 ents[235], 16 bytes each
3768 88 mmio_cache[0]: 24-byte header, buffer[64]
3856 88 mmio_cache[1]
3944 88 mmio_cache[2]
4032 64 unused
The four old allocations together asked for 4032 bytes, the span the
page now uses. The limits the code checks, max_ents (235) and space
(64), are unchanged. The BUILD_BUG_ON() in hvmemul_cache_init() keeps
the end within the page.
---
xen/arch/x86/domain.c | 2 +
xen/arch/x86/hvm/emulate.c | 196 +++++++++++++++++++------
xen/arch/x86/include/asm/config.h | 1 +
xen/arch/x86/include/asm/hvm/emulate.h | 9 +-
xen/arch/x86/include/asm/hvm/vcpu.h | 7 +
5 files changed, 165 insertions(+), 50 deletions(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 03f6572086..66d71ec41b 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -595,6 +595,8 @@ int arch_vcpu_create(struct vcpu *v)
BUILD_BUG_ON(VCPU_STATE_IDX_N2VMCB >= VCPU_STATE_NR_PAGES);
BUILD_BUG_ON(VCPU_STATE_IDX_VMX_MSR >= VCPU_STATE_NR_PAGES);
BUILD_BUG_ON(VCPU_STATE_IDX_REGS >= VCPU_STATE_NR_PAGES);
+ BUILD_BUG_ON(VCPU_STATE_IDX_EMUL >= VCPU_STATE_NR_PAGES);
+ BUILD_BUG_ON(VCPU_STATE_IDX_EMUL == VCPU_STATE_IDX_REGS);
/* Page tables only; torn down with the rest of the per-domain area. */
rc = create_perdomain_mapping(v, VCPU_STATE_START(v),
VCPU_STATE_NR_PAGES,
diff --git a/xen/arch/x86/hvm/emulate.c b/xen/arch/x86/hvm/emulate.c
index cd2831d34b..e2cf982c25 100644
--- a/xen/arch/x86/hvm/emulate.c
+++ b/xen/arch/x86/hvm/emulate.c
@@ -9,6 +9,7 @@
* Keir Fraser <keir@xen.org>
*/
+#include <xen/domain_page.h>
#include <xen/init.h>
#include <xen/iocap.h>
#include <xen/ioreq.h>
@@ -56,6 +57,26 @@ struct hvmemul_cache
} ents[];
};
+/*
+ * AVX512F scatter/gather insns can access up to 16 independent linear
+ * addresses, up to 8 bytes size. Each such linear range can span a page
+ * boundary, i.e. may require two page walks. Account for each insn byte
+ * individually, both for simplicity and to leave some slack space.
+ */
+#define HVMEMUL_CACHE_ENTS \
+ ((16 * 2 + MAX_INST_LEN) * (CONFIG_PAGING_LEVELS + 1))
+
+/* Operand bytes per MMIO cache: up to m512 or a tile row. */
+#define MMIO_CACHE_BYTES 64
+
+/* Layout of the caches' page: the read cache, then the MMIO caches. */
+#define MMIO_CACHE_SIZE \
+ ROUNDUP(offsetof(struct hvm_mmio_cache, buffer[MMIO_CACHE_BYTES]), \
+ __alignof__(struct hvm_mmio_cache))
+#define EMUL_PAGE_MMIO_CACHE(i) \
+ (ROUNDUP(offsetof(struct hvmemul_cache, ents[HVMEMUL_CACHE_ENTS]), \
+ __alignof__(struct hvm_mmio_cache)) + (i) * MMIO_CACHE_SIZE)
+
static void hvmtrace_io_assist(const ioreq_t *p)
{
unsigned int size, event;
@@ -1130,6 +1151,9 @@ static struct hvm_mmio_cache *hvmemul_find_mmio_cache(
unsigned int i;
struct hvm_mmio_cache *cache;
+ ASSERT(hvio == ¤t->arch.hvm.hvm_io);
+ ASSERT(this_cpu(pgtable_vcpu) == current);
+
for ( i = 0; i < hvio->mmio_cache_count; i ++ )
{
cache = hvio->mmio_cache[i];
@@ -3057,77 +3081,133 @@ void hvm_dump_emulation_state(const char *loglvl, const char *prefix,
int hvmemul_cache_init(struct vcpu *v)
{
- /*
- * AVX512F scatter/gather insns can access up to 16 independent linear
- * addresses, up to 8 bytes size. Each such linear range can span a page
- * boundary, i.e. may require two page walks.
- */
- unsigned int nents = 16 * 2 * (CONFIG_PAGING_LEVELS + 1);
- unsigned int i, max_bytes = 64;
+ struct hvm_vcpu_io *hvio = &v->arch.hvm.hvm_io;
+ struct page_info *pg = alloc_domheap_page(v->domain, MEMF_no_owner);
+ void *va = (void *)VCPU_STATE_ADDR(v, VCPU_STATE_IDX_EMUL), *p;
struct hvmemul_cache *cache;
+ unsigned int i;
+ mfn_t mfn;
- /*
- * Account for each insn byte individually, both for simplicity and to
- * leave some slack space.
- */
- nents += MAX_INST_LEN * (CONFIG_PAGING_LEVELS + 1);
+ BUILD_BUG_ON(EMUL_PAGE_MMIO_CACHE(ARRAY_SIZE(hvio->mmio_cache)) >
+ PAGE_SIZE);
- cache = xvmalloc_flex_struct(struct hvmemul_cache, ents, nents);
- if ( !cache )
+ if ( !pg )
return -ENOMEM;
- /* Cache is disabled initially. */
- cache->num_ents = nents + 1;
- cache->max_ents = nents;
+ mfn = page_to_mfn(pg);
+ clear_domain_page(mfn);
- v->arch.hvm.hvm_io.cache = cache;
+ /* v != current, so we can't use the vCPU state window. */
+ p = map_domain_page(mfn);
- for ( i = 0; i < ARRAY_SIZE(v->arch.hvm.hvm_io.mmio_cache); ++i )
+ /* Cache is disabled initially. */
+ cache = p;
+ cache->num_ents = HVMEMUL_CACHE_ENTS + 1;
+ cache->max_ents = HVMEMUL_CACHE_ENTS;
+
+ for ( i = 0; i < ARRAY_SIZE(hvio->mmio_cache); ++i )
{
- v->arch.hvm.hvm_io.mmio_cache[i] =
- xvmalloc_flex_struct(struct hvm_mmio_cache, buffer, max_bytes);
- if ( !v->arch.hvm.hvm_io.mmio_cache[i] )
- return -ENOMEM;
- v->arch.hvm.hvm_io.mmio_cache[i]->space = max_bytes;
+ struct hvm_mmio_cache *mmio = p + EMUL_PAGE_MMIO_CACHE(i);
+
+ mmio->space = MMIO_CACHE_BYTES;
}
+ unmap_domain_page(p);
+
+ populate_perdomain_mapping(v, (unsigned long)va, &mfn, 1,
+ __PAGE_HYPERVISOR_RW);
+
+ hvio->emul_pg = pg;
+ hvio->cache = va;
+ for ( i = 0; i < ARRAY_SIZE(hvio->mmio_cache); ++i )
+ hvio->mmio_cache[i] = va + EMUL_PAGE_MMIO_CACHE(i);
+
return 0;
}
+void hvmemul_cache_destroy(struct vcpu *v)
+{
+ struct hvm_vcpu_io *hvio = &v->arch.hvm.hvm_io;
+ unsigned int i;
+
+ if ( !hvio->emul_pg )
+ return;
+
+ unmap_vcpu_state(v, VCPU_STATE_IDX_EMUL, 1);
+ free_domheap_page(hvio->emul_pg);
+ hvio->emul_pg = NULL;
+ hvio->cache = NULL;
+ for ( i = 0; i < ARRAY_SIZE(hvio->mmio_cache); ++i )
+ hvio->mmio_cache[i] = NULL;
+}
+
+/*
+ * The read cache of @v, for access from any context: the window address in
+ * struct hvm_vcpu_io where v runs here on its own page tables, a transient
+ * mapping of the page otherwise, and the NULL left there once the cache is
+ * destroyed. v's page tables being loaded is not enough: while this CPU
+ * holds v's state lazily, an IPI syncing that state for another CPU can
+ * switch them away at any time.
+ */
+static struct hvmemul_cache *map_cache(const struct vcpu *v)
+{
+ const struct hvm_vcpu_io *hvio = &v->arch.hvm.hvm_io;
+
+ if ( !hvio->emul_pg || (v == current && this_cpu(pgtable_vcpu) == v) )
+ return hvio->cache;
+
+ /* The read cache is at the start of the page. */
+ return __map_domain_page(hvio->emul_pg);
+}
+
+static void unmap_cache(const struct vcpu *v,
+ const struct hvmemul_cache *cache)
+{
+ if ( cache != v->arch.hvm.hvm_io.cache )
+ unmap_vcpu_state_page(cache);
+}
+
+#define HVMEMUL_MAP_CACHE(v) map_cache(v)
+
+/* Drops the mapping of a vCPU's read cache and nullifies its pointer. */
+#define HVMEMUL_UNMAP_CACHE(v, cache) do { \
+ unmap_cache(v, cache); \
+ (cache) = NULL; \
+} while ( false )
+
unsigned int hvmemul_cache_disable(struct vcpu *v)
{
- struct hvmemul_cache *cache = v->arch.hvm.hvm_io.cache;
+ struct hvmemul_cache *cache = HVMEMUL_MAP_CACHE(v);
unsigned int token = cache->num_ents;
cache->num_ents = cache->max_ents + 1;
+ HVMEMUL_UNMAP_CACHE(v, cache);
+
return token;
}
void hvmemul_cache_restore(struct vcpu *v, unsigned int token)
{
- struct hvmemul_cache *cache = v->arch.hvm.hvm_io.cache;
+ struct hvmemul_cache *cache = HVMEMUL_MAP_CACHE(v);
ASSERT(cache->num_ents > cache->max_ents);
cache->num_ents = token;
+
+ HVMEMUL_UNMAP_CACHE(v, cache);
}
-bool hvmemul_read_cache(const struct vcpu *v, paddr_t gpa,
- void *buffer, unsigned int size)
+static bool read_cache(const struct hvmemul_cache *cache, paddr_t gpa,
+ void *buffer, unsigned int size)
{
- const struct hvmemul_cache *cache = v->arch.hvm.hvm_io.cache;
unsigned int i;
- /* Cache unavailable? */
- if ( !is_hvm_vcpu(v) || cache->num_ents > cache->max_ents )
- return false;
-
while ( size > sizeof(cache->ents->data) )
{
i = gpa & (sizeof(cache->ents->data) - 1)
? -gpa & (sizeof(cache->ents->data) - 1)
: sizeof(cache->ents->data);
- if ( !hvmemul_read_cache(v, gpa, buffer, i) )
+ if ( !read_cache(cache, gpa, buffer, i) )
return false;
gpa += i;
buffer += i;
@@ -3144,22 +3224,37 @@ bool hvmemul_read_cache(const struct vcpu *v, paddr_t gpa,
return false;
}
-void hvmemul_write_cache(const struct vcpu *v, paddr_t gpa,
- const void *buffer, unsigned int size)
+bool hvmemul_read_cache(const struct vcpu *v, paddr_t gpa,
+ void *buffer, unsigned int size)
{
- struct hvmemul_cache *cache = v->arch.hvm.hvm_io.cache;
- unsigned int i;
+ const struct hvmemul_cache *cache;
+ bool hit = false;
- /* Cache unavailable? */
- if ( !is_hvm_vcpu(v) || cache->num_ents > cache->max_ents )
- return;
+ if ( !is_hvm_vcpu(v) )
+ return false;
+
+ cache = HVMEMUL_MAP_CACHE(v);
+
+ /* Cache available? */
+ if ( cache->num_ents <= cache->max_ents )
+ hit = read_cache(cache, gpa, buffer, size);
+
+ HVMEMUL_UNMAP_CACHE(v, cache);
+
+ return hit;
+}
+
+static void write_cache(const struct vcpu *v, struct hvmemul_cache *cache,
+ paddr_t gpa, const void *buffer, unsigned int size)
+{
+ unsigned int i;
while ( size > sizeof(cache->ents->data) )
{
i = gpa & (sizeof(cache->ents->data) - 1)
? -gpa & (sizeof(cache->ents->data) - 1)
: sizeof(cache->ents->data);
- hvmemul_write_cache(v, gpa, buffer, i);
+ write_cache(v, cache, gpa, buffer, i);
gpa += i;
buffer += i;
size -= i;
@@ -3186,6 +3281,23 @@ void hvmemul_write_cache(const struct vcpu *v, paddr_t gpa,
cache->num_ents = i + 1;
}
+void hvmemul_write_cache(const struct vcpu *v, paddr_t gpa,
+ const void *buffer, unsigned int size)
+{
+ struct hvmemul_cache *cache;
+
+ if ( !is_hvm_vcpu(v) )
+ return;
+
+ cache = HVMEMUL_MAP_CACHE(v);
+
+ /* Cache available? */
+ if ( cache->num_ents <= cache->max_ents )
+ write_cache(v, cache, gpa, buffer, size);
+
+ HVMEMUL_UNMAP_CACHE(v, cache);
+}
+
/*
* Local variables:
* mode: C
diff --git a/xen/arch/x86/include/asm/config.h b/xen/arch/x86/include/asm/config.h
index c8d4a8ddc7..5bd9970fce 100644
--- a/xen/arch/x86/include/asm/config.h
+++ b/xen/arch/x86/include/asm/config.h
@@ -312,6 +312,7 @@ extern unsigned long xen_phys_start;
#define VCPU_STATE_IDX_N2VMCB 1
#define VCPU_STATE_IDX_VMX_MSR 2
#define VCPU_STATE_IDX_REGS 6
+#define VCPU_STATE_IDX_EMUL 7
#define ELFSIZE 64
diff --git a/xen/arch/x86/include/asm/hvm/emulate.h b/xen/arch/x86/include/asm/hvm/emulate.h
index 178ac32e15..16491a92b1 100644
--- a/xen/arch/x86/include/asm/hvm/emulate.h
+++ b/xen/arch/x86/include/asm/hvm/emulate.h
@@ -117,14 +117,7 @@ int hvmemul_do_pio_buffer(uint16_t port,
* further re-execution.
*/
int __must_check hvmemul_cache_init(struct vcpu *v);
-static inline void hvmemul_cache_destroy(struct vcpu *v)
-{
- unsigned int i;
-
- for ( i = 0; i < ARRAY_SIZE(v->arch.hvm.hvm_io.mmio_cache); ++i )
- XVFREE(v->arch.hvm.hvm_io.mmio_cache[i]);
- XVFREE(v->arch.hvm.hvm_io.cache);
-}
+void hvmemul_cache_destroy(struct vcpu *v);
bool hvmemul_read_cache(const struct vcpu *v, paddr_t gpa,
void *buffer, unsigned int size);
void hvmemul_write_cache(const struct vcpu *v, paddr_t gpa,
diff --git a/xen/arch/x86/include/asm/hvm/vcpu.h b/xen/arch/x86/include/asm/hvm/vcpu.h
index 876c230784..ee7bc8be00 100644
--- a/xen/arch/x86/include/asm/hvm/vcpu.h
+++ b/xen/arch/x86/include/asm/hvm/vcpu.h
@@ -46,6 +46,13 @@ struct hvm_vcpu_io {
unsigned char mmio_insn[16];
struct hvmemul_cache *cache;
+ /*
+ * The page holding the caches above, mapped in the vCPU's state window:
+ * mmio_cache[] and cache hold addresses in the window, for use while the
+ * vCPU's own page tables are loaded.
+ */
+ struct page_info *emul_pg;
+
/*
* For string instruction emulation we need to be able to signal a
* necessary retry through other than function return codes.
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 34/44] x86/hvm: clear the vCPU's copy of an emulated I/O request once consumed
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (32 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 33/44] x86/hvm: move the emulator's operand caches " George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 35/44] x86/domain_page: introduce map_domain_pages() George Dunlap
` (9 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
hvmemul_do_io() keeps the request it is emulating in the vCPU's struct
vcpu_io: the data written, or the response once a device model has
supplied it, with the address and repeat count. Once the access is done
only the state is set back to STATE_IOREQ_NONE, so the operand stays in
struct vcpu, a xenheap page mapped in every context, until the next
emulated I/O overwrites it.
Drop the copy wherever it is done with: when the response has been taken
into hvmemul_do_io()'s local copy on the instruction's re-execution,
when Xen's own handlers have completed the access, when the request is
retried, suspended or not sent, and in hvmemul_cancel(). A request sent
to a device model stays while it is in flight: the re-execution checks
it, and the port-I/O completion reads its address and size before
re-issuing it. One which needs no completion -- a port write, or a
request whose data is a guest buffer's address -- loses its address,
data and count as soon as it is sent: the device model has its own copy,
and the completion path reads only the state, type, direction and
data_is_ptr. Nothing reads the copy after STATE_IOREQ_NONE.
Only the x86 emulator changes. struct vcpu_io is common code, and Arm's
MMIO completion reads the response from it. Moving the in-flight copy
out of struct vcpu belongs with narrowing the ioreq server pages, which
carry the same operand while a device model handles it.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New in this version.
---
xen/arch/x86/hvm/emulate.c | 36 ++++++++++++++++++++++++++++--------
1 file changed, 28 insertions(+), 8 deletions(-)
diff --git a/xen/arch/x86/hvm/emulate.c b/xen/arch/x86/hvm/emulate.c
index e2cf982c25..da7c08b5d9 100644
--- a/xen/arch/x86/hvm/emulate.c
+++ b/xen/arch/x86/hvm/emulate.c
@@ -167,6 +167,16 @@ static const struct hvm_io_handler ioreq_server_handler = {
.ops = &ioreq_server_ops
};
+/*
+ * The vCPU's copy of an emulated I/O request is done with: nothing reads it
+ * before the next request overwrites it. Clear it all, so that the operand
+ * and the address do not stay in struct vcpu.
+ */
+static void ioreq_done(struct vcpu_io *vio)
+{
+ vio->req = (ioreq_t){ .state = STATE_IOREQ_NONE };
+}
+
/*
* Drop all records of in-flight emulation. This is needed whenever a vCPU's
* register state may have changed behind the emulator's back.
@@ -175,7 +185,7 @@ void hvmemul_cancel(struct vcpu *v)
{
struct hvm_vcpu_io *hvio = &v->arch.hvm.hvm_io;
- v->io.req.state = STATE_IOREQ_NONE;
+ ioreq_done(&v->io);
v->io.completion = VIO_no_completion;
hvio->mmio_cache_count = 0;
hvio->mmio_insn_bytes = 0;
@@ -265,8 +275,9 @@ static int hvmemul_do_io(
case STATE_IOREQ_NONE:
break;
case STATE_IORESP_READY:
- vio->req.state = STATE_IOREQ_NONE;
+ /* The local copy carries the response from here. */
p = vio->req;
+ ioreq_done(vio);
/* Verify the emulation request has been correctly re-issued */
if ( (p.type != (is_mmio ? IOREQ_TYPE_COPY : IOREQ_TYPE_PIO)) ||
@@ -330,7 +341,7 @@ static int hvmemul_do_io(
{
case X86EMUL_OKAY:
case X86EMUL_RETRY:
- vio->req.state = STATE_IOREQ_NONE;
+ ioreq_done(vio);
break;
case X86EMUL_UNHANDLEABLE:
{
@@ -383,7 +394,7 @@ static int hvmemul_do_io(
if ( s == NULL )
{
rc = X86EMUL_RETRY;
- vio->req.state = STATE_IOREQ_NONE;
+ ioreq_done(vio);
break;
}
@@ -394,7 +405,7 @@ static int hvmemul_do_io(
if ( dir == IOREQ_READ )
{
rc = hvm_process_io_intercept(&ioreq_server_handler, &p);
- vio->req.state = STATE_IOREQ_NONE;
+ ioreq_done(vio);
break;
}
}
@@ -440,7 +451,7 @@ static int hvmemul_do_io(
paddr_to_pfn(addr));
rc = X86EMUL_RETRY;
- vio->req.state = STATE_IOREQ_NONE;
+ ioreq_done(vio);
break;
}
@@ -455,15 +466,24 @@ static int hvmemul_do_io(
dir ? "read from" : "write to", addr, size);
}
rc = hvm_process_io_intercept(&null_handler, &p);
- vio->req.state = STATE_IOREQ_NONE;
+ ioreq_done(vio);
}
else
{
rc = ioreq_send(s, &p, 0);
if ( rc != X86EMUL_RETRY || vio->suspended )
- vio->req.state = STATE_IOREQ_NONE;
+ ioreq_done(vio);
else if ( !ioreq_needs_completion(&vio->req) )
+ {
+ /*
+ * Nothing comes back, and the device model has its own copy
+ * of the request: keep only what completion checks.
+ */
+ vio->req.addr = 0;
+ vio->req.data = 0;
+ vio->req.count = 0;
rc = X86EMUL_OKAY;
+ }
}
break;
}
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 35/44] x86/domain_page: introduce map_domain_pages()
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (33 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 34/44] x86/hvm: clear the vCPU's copy of an emulated I/O request once consumed George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 36/44] x86/shadow: use map_domain_pages() for cross-page emulated writes George Dunlap
` (8 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Andrew Cooper, Anthony PERARD, Michal Orzel, Jan Beulich,
Julien Grall, Roger Pau Monné, Stefano Stabellini,
Teddy Astie
map_domain_page() maps a single page, so callers needing a virtually
contiguous mapping of several (not necessarily physically contiguous)
frames have had to fall back to vmap(), which maps whatever the frames
hold in every context, those of domains on the sparse directmap view
included, for the lifetime of the mapping. Provide a multipage
counterpart which maps them in the calling vCPU's per-domain area
instead, which only its domain's page tables carry:
map_domain_pages(mfn, nr) / unmap_domain_pages(ptr, nr)
Each vCPU has MAPCACHE_MULTIPAGE_ENTRIES slots of its own for such
mappings. With per-vCPU page tables, they are just above the vCPU's
single-page mapcache. Where a domain's vCPUs share one per-domain
area, they are at the end of the mapcache sub-area, where each vCPU
has a fixed block of its own. No single-page mapcache is needed: the
multipage slots are in the same VA range whether or not the domain has
one. The mappings are order-aligned and tracked in a per-vCPU bitmap,
and bypass the maphash and the deferred garbage/epoch machinery.
At unmap a mapping's PTEs are cleared, and a local flush done. Since
each vCPU has its own slots, even with a domain-wide L3, the local
flush suffices for correctness: even if a stray TLB entry is
speculatively loaded on another processor, no other vCPU will use that
mapping, and it will be flushed before the vCPU which owns the slots
runs on that CPU.
With regard to exposure: in a domain whose vCPUs share one per-domain
area, other vCPUs may be able to reach the mapping speculatively while
it exists, and afterwards through a stale TLB entry until their CPU
next flushes. For guests on the full directmap, this doesn't expose
anything new, as the page is already mapped in the directmap anyway
(unless it's a PV domain and the page is past the short directmap).
For PV guests on the sparse directmap view, it's still an improvement
over the status quo, as exposure is limited to the domain, rather than
the whole system (as in the case of vmap).
An unmap will BUG() when given a pointer or a @nr which indexes
outside the vCPU's slots, unless that pointer is in the vmap area (see
below).
These slots are not available before the first page-table switch in
boot, nor with the idle vCPU's page tables loaded. Currently no calls
are made during boot, and the only known way to reach
map_domain_pages() with the idle page tables loaded is on an error path
that will already ultimately crash the host. Nonetheless, for
robustness, if called from a context where no mapping is available, it
will fall back to using vmap(); and unmap recognizes such an address
and vunmap()'s it. This briefly exposes contents to every context,
and requires a system-wide TLB shootdown on unmap, but allows
execution to continue. The alternative would be to BUG() in such a
context. For those who want stricter isolation guarantees, at the
risk that more situations might come up in the future, a new
sub-option, asi=vmap-strict, off by default, makes the fallback a
BUG() instead. The first fallback is logged.
The handling of a lazy context switch in mapcache_current_vcpu() moves
into pgtable_current_vcpu(), which map_domain_pages() uses whatever the
domain: an IPI can switch away page tables held lazily at any time.
Eight slots hold the largest mapping a known caller needs, a
three-page XSAVE area (four slots, order 2), with room for a two-page
mapping beside it. Where a domain's vCPUs share one per-domain area,
the multipage area takes 256MiB of address space for MAX_VIRT_CPUS,
and one L1 table per 64 vCPUs.
Later patches convert the two known in-tree consumers. Moving the
XSAVE areas off always-mapped memory needs the primitive too: the
XSAVE area exceeds one page on AMX-class hardware, and XSAVE/XRSTOR
require it to be virtually contiguous.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New in this version.
---
docs/misc/xen-command-line.pandoc | 10 +-
xen/arch/x86/domain_page.c | 171 +++++++++++++++++++++++++--
xen/arch/x86/include/asm/config.h | 14 +++
xen/arch/x86/include/asm/domain.h | 3 +
xen/arch/x86/include/asm/spec_ctrl.h | 1 +
xen/arch/x86/spec_ctrl.c | 7 ++
xen/include/xen/domain_page.h | 21 ++++
7 files changed, 216 insertions(+), 11 deletions(-)
diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc
index 092c858f34..52e10eb393 100644
--- a/docs/misc/xen-command-line.pandoc
+++ b/docs/misc/xen-command-line.pandoc
@@ -203,7 +203,7 @@ to appropriate auditing by Xen. Argo is disabled by default.
buggy or malicious other domain spamming the ring.
### asi (x86)
-> `= List of [ <bool>, {pv,hvm,dom0}=<bool>, directmap=<bool>,
+> `= List of [ <bool>, {pv,hvm,dom0}=<bool>, {directmap,vmap-strict}=<bool>,
> vcpu-pt=<bool> | vcpu-pt=hvm=<bool> ]`
> Default: `false`
@@ -245,6 +245,14 @@ are meant to be used for debugging purposes only.**
and `dom0=`) set this too, on builds that support it; on other builds they
leave it off.
+* `vmap-strict` crashes the host when a multipage transient mapping
+ (`map_domain_pages()`) is requested where the loaded page-tables
+ have no slots for one: in early boot, or on the idle vCPU's
+ page-tables. By default such a mapping falls back to `vmap()`,
+ which maps the pages in every context. The first such mapping is
+ logged. The option is for hosts where a crash is preferable to such
+ a mapping; none is known to be made in normal operation.
+
### asid (x86)
> `= <boolean>`
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 4198ee7c18..8e5ef4a34a 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -6,6 +6,7 @@
* Copyright (c) 2003-2006, Keir Fraser <keir@xensource.com>
*/
+#include <xen/bitmap.h>
#include <xen/domain_page.h>
#include <xen/efi.h>
#include <xen/mm.h>
@@ -17,6 +18,7 @@
#include <asm/flushtlb.h>
#include <asm/hardirq.h>
#include <asm/setup.h>
+#include <asm/spec_ctrl.h>
/* Whether the vCPU supports mapcache usage, might not be in use. */
static bool has_mapcache(const struct vcpu *v)
@@ -24,20 +26,22 @@ static bool has_mapcache(const struct vcpu *v)
return is_pv_vcpu(v) || v->domain->arch.asi.vcpu_pt;
}
-static inline struct vcpu *mapcache_current_vcpu(void)
+/*
+ * The vCPU whose page tables are loaded here, and stay loaded until the
+ * caller is done with what it maps in them.
+ */
+static struct vcpu *pgtable_current_vcpu(void)
{
- const struct vcpu *v = this_cpu(pgtable_vcpu);
const struct vcpu *curr = current;
/* During early boot pgtable_vcpu is not set, callers must handle NULL. */
- if ( !v || !has_mapcache(v) )
+ if ( !this_cpu(pgtable_vcpu) )
return NULL;
/*
- * If we are in a lazy context-switch state from a vCPU with a mapcache do
- * a full switch to the idle vCPU now, otherwise an incoming
- * FLUSH_VCPU_STATE IPI would change the page tables under our feet and
- * invalidate any in-use mapcache entries.
+ * If we are in a lazy context-switch state do a full switch to the idle
+ * vCPU now, otherwise an incoming FLUSH_VCPU_STATE IPI would change the
+ * page tables under our feet and invalidate any in-use mappings in them.
*/
if ( unlikely(this_cpu(curr_vcpu) != curr) )
{
@@ -56,6 +60,17 @@ static inline struct vcpu *mapcache_current_vcpu(void)
return ACCESS_ONCE(this_cpu(pgtable_vcpu));
}
+static inline struct vcpu *mapcache_current_vcpu(void)
+{
+ const struct vcpu *v = this_cpu(pgtable_vcpu);
+
+ /* During early boot pgtable_vcpu is not set, callers must handle NULL. */
+ if ( !v || !has_mapcache(v) )
+ return NULL;
+
+ return pgtable_current_vcpu();
+}
+
/* Whether @v's page-tables carry the sparse view of the directmap. */
static bool on_sparse_dmap(const struct vcpu *v)
{
@@ -341,6 +356,124 @@ void unmap_domain_page(const void *ptr)
local_irq_restore(flags);
}
+/*
+ * The mapcache index of @v's first map_domain_pages() slot: above the
+ * single-page slots of its own mapcache with per-vCPU page tables, otherwise
+ * at its stride of the domain-wide multipage area.
+ */
+static unsigned int multipage_first_slot(const struct vcpu *v)
+{
+ if ( v->domain->arch.asi.vcpu_pt )
+ return MAPCACHE_PERVCPU_ENTRIES;
+
+ return PFN_DOWN(MAPCACHE_MULTIPAGE_START(v) - MAPCACHE_VIRT_START);
+}
+
+/*
+ * Multipage mappings come from slots of the vCPU whose page tables are
+ * loaded, which no other vCPU uses, and bypass the maphash and the
+ * garbage/epoch machinery: at unmap a mapping's PTEs are zapped, and their
+ * translations dropped from this CPU's TLB, at their addresses and at their
+ * aliases in the per-domain mirror.
+ */
+void *map_domain_pages(const mfn_t *mfn, unsigned int nr)
+{
+ unsigned long flags;
+ unsigned int i, first;
+ struct vcpu *v;
+ int idx;
+
+ if ( !nr || nr > MAPCACHE_MULTIPAGE_ENTRIES )
+ {
+ ASSERT_UNREACHABLE();
+ return NULL;
+ }
+
+ if ( nr == 1 )
+ return map_domain_page(mfn[0]);
+
+ /* Early boot, or the idle vCPU's page tables: no multipage slots. */
+ v = pgtable_current_vcpu();
+ if ( !v || is_idle_vcpu(v) )
+ {
+ BUG_ON(opt_vmap_strict);
+ printk_once(XENLOG_WARNING
+ "%pS: map_domain_pages() falling back to vmap()\n",
+ __builtin_return_address(0));
+ return vmap(mfn, nr);
+ }
+
+ first = multipage_first_slot(v);
+
+ local_irq_save(flags);
+
+ idx = bitmap_find_free_region(&v->arch.mapcache.multipage_inuse,
+ MAPCACHE_MULTIPAGE_ENTRIES,
+ get_order_from_pages(nr));
+ if ( idx >= 0 )
+ for ( i = 0; i < nr; i++ )
+ l1e_write(&MAPCACHE_L1ENT(first + idx + i),
+ l1e_from_mfn(mfn[i], __PAGE_HYPERVISOR_RW));
+
+ local_irq_restore(flags);
+
+ if ( idx < 0 )
+ return NULL;
+
+ return (void *)MAPCACHE_VIRT_START + pfn_to_paddr(first + idx);
+}
+
+void unmap_domain_pages(const void *ptr, unsigned int nr)
+{
+ unsigned long va = (unsigned long)ptr & PAGE_MASK;
+ unsigned long flags, slot;
+ unsigned int idx, first, i;
+ struct vcpu *v;
+
+ BUG_ON(!nr);
+
+ if ( nr == 1 )
+ {
+ unmap_domain_page(ptr);
+ return;
+ }
+
+ if ( va >= VMAP_VIRT_START && va < VMAP_VIRT_END )
+ {
+ vunmap((void *)va);
+ return;
+ }
+
+ v = pgtable_current_vcpu();
+ BUG_ON(!v || is_idle_vcpu(v) || va < MAPCACHE_VIRT_START);
+
+ first = multipage_first_slot(v);
+ slot = PFN_DOWN(va - MAPCACHE_VIRT_START);
+ BUG_ON(slot < first || slot - first >= MAPCACHE_MULTIPAGE_ENTRIES);
+
+ idx = slot - first;
+ BUG_ON(nr > MAPCACHE_MULTIPAGE_ENTRIES - idx);
+
+ local_irq_save(flags);
+
+ for ( i = 0; i < nr; i++ )
+ {
+ unsigned long addr = va + i * PAGE_SIZE;
+
+ ASSERT(l1e_get_flags(MAPCACHE_L1ENT(slot + i)) & _PAGE_PRESENT);
+ l1e_write(&MAPCACHE_L1ENT(slot + i), l1e_empty());
+ flush_tlb_one_local(addr);
+ if ( !is_pv_64bit_vcpu(v) )
+ flush_tlb_one_local(addr - PERDOMAIN_VIRT_START +
+ PERDOMAIN_ALT_VIRT_START);
+ }
+
+ bitmap_release_region(&v->arch.mapcache.multipage_inuse, idx,
+ get_order_from_pages(nr));
+
+ local_irq_restore(flags);
+}
+
void mapcache_domain_init(struct domain *d)
{
struct mapcache_domain *dcache = &d->arch.pv.mapcache;
@@ -359,7 +492,7 @@ void mapcache_domain_init(struct domain *d)
BUILD_BUG_ON(MAPCACHE_VIRT_END + PAGE_SIZE * (3 +
2 * PFN_UP(BITS_TO_LONGS(MAPCACHE_ENTRIES) * sizeof(long))) >
- MAPCACHE_VIRT_START + (PERDOMAIN_SLOT_MBYTES << 20));
+ MAPCACHE_MULTIPAGE_VIRT_START);
bitmap_pages = PFN_UP(BITS_TO_LONGS(MAPCACHE_ENTRIES) * sizeof(long));
dcache->cache.inuse = (void *)MAPCACHE_VIRT_END + PAGE_SIZE;
dcache->cache.garbage = dcache->cache.inuse +
@@ -376,12 +509,28 @@ int mapcache_vcpu_init(struct vcpu *v)
unsigned long i;
unsigned int ents, nr;
+ BUILD_BUG_ON(MAPCACHE_MULTIPAGE_ENTRIES > BITS_PER_LONG);
+
+ /* Without per-vCPU page tables, the vCPU's stride of the multipage area. */
+ if ( !is_idle_domain(d) && !d->arch.asi.vcpu_pt )
+ {
+ int rc = create_perdomain_mapping(v, MAPCACHE_MULTIPAGE_START(v),
+ MAPCACHE_MULTIPAGE_ENTRIES,
+ NIL(l1_pgentry_t *), NULL);
+
+ if ( rc )
+ return rc;
+ }
+
if ( !has_mapcache(v) )
return 0;
cache = vcpu_mapcache(v, &dcache);
if ( !dcache )
{
+ BUILD_BUG_ON(MAPCACHE_PERVCPU_ENTRIES + MAPCACHE_MULTIPAGE_ENTRIES >
+ MAPCACHE_ENTRIES);
+
/* Per-vCPU bitmaps are an unsigned long each, in the vCPU itself. */
cache->inuse = v->arch.mapcache.inuse;
cache->garbage = v->arch.mapcache.garbage;
@@ -396,8 +545,10 @@ int mapcache_vcpu_init(struct vcpu *v)
if ( ents > cache->entries )
{
- /* Populate page tables. */
- int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START, ents,
+ /* Populate page tables, a per-vCPU cache's multipage slots included. */
+ int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START,
+ ents + (dcache ? 0
+ : MAPCACHE_MULTIPAGE_ENTRIES),
NIL(l1_pgentry_t *), NULL);
/* Populate the domain-wide bit maps (kept in the per-domain area). */
diff --git a/xen/arch/x86/include/asm/config.h b/xen/arch/x86/include/asm/config.h
index 5bd9970fce..52d1ebc0e7 100644
--- a/xen/arch/x86/include/asm/config.h
+++ b/xen/arch/x86/include/asm/config.h
@@ -281,10 +281,24 @@ extern unsigned long xen_phys_start;
#define MAPCACHE_VCPU_ENTRIES (CONFIG_PAGING_LEVELS * CONFIG_PAGING_LEVELS)
/* Single-page slots of a per-vCPU mapcache (vCPU-PT domains). */
#define MAPCACHE_PERVCPU_ENTRIES 64
+/*
+ * Multipage slots of each vCPU: above the single-page ones of a per-vCPU
+ * mapcache, otherwise at the vCPU's stride of the multipage area.
+ */
+#define MAPCACHE_MULTIPAGE_ORDER 3
+#define MAPCACHE_MULTIPAGE_ENTRIES (1U << MAPCACHE_MULTIPAGE_ORDER)
#define MAPCACHE_ENTRIES (MAX_VIRT_CPUS * MAPCACHE_VCPU_ENTRIES)
#define MAPCACHE_VIRT_START PERDOMAIN_VIRT_SLOT(1)
#define MAPCACHE_VIRT_END (MAPCACHE_VIRT_START + \
MAPCACHE_ENTRIES * PAGE_SIZE)
+/* The multipage area, at the end of the sub-area. */
+#define MAPCACHE_MULTIPAGE_VA_SHIFT (MAPCACHE_MULTIPAGE_ORDER + PAGE_SHIFT)
+#define MAPCACHE_MULTIPAGE_VIRT_START \
+ (MAPCACHE_VIRT_START + (PERDOMAIN_SLOT_MBYTES << 20) - \
+ (MAX_VIRT_CPUS << MAPCACHE_MULTIPAGE_VA_SHIFT))
+#define MAPCACHE_MULTIPAGE_START(v) \
+ (MAPCACHE_MULTIPAGE_VIRT_START + \
+ ((v)->vcpu_id << MAPCACHE_MULTIPAGE_VA_SHIFT))
/* Argument translation area. The third per-domain-mapping sub-area. */
#define ARG_XLAT_VIRT_START PERDOMAIN_VIRT_SLOT(2)
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 6b27af9eb5..3748e417ce 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -84,6 +84,9 @@ struct mapcache_vcpu {
uint32_t refcnt;
} hash[MAPHASH_ENTRIES];
+ /* Which of the vCPU's map_domain_pages() slots are in use. */
+ unsigned long multipage_inuse;
+
/* Accounting of a per-vCPU mapcache (vCPU-PT domains). */
struct mapcache cache;
/* Backing for cache.{inuse,garbage}. */
diff --git a/xen/arch/x86/include/asm/spec_ctrl.h b/xen/arch/x86/include/asm/spec_ctrl.h
index b62229ab5d..d9ad94ddb0 100644
--- a/xen/arch/x86/include/asm/spec_ctrl.h
+++ b/xen/arch/x86/include/asm/spec_ctrl.h
@@ -88,6 +88,7 @@ extern uint8_t default_scf;
extern int8_t opt_xpti_hwdom, opt_xpti_domu;
extern bool opt_vcpu_pt_hwdom, opt_vcpu_pt_hvm;
+extern bool opt_vmap_strict;
extern bool cpu_has_bug_l1tf;
extern int8_t opt_pv_l1tf_hwdom, opt_pv_l1tf_domu;
diff --git a/xen/arch/x86/spec_ctrl.c b/xen/arch/x86/spec_ctrl.c
index c86f5ff886..3c52916eda 100644
--- a/xen/arch/x86/spec_ctrl.c
+++ b/xen/arch/x86/spec_ctrl.c
@@ -86,6 +86,9 @@ bool __ro_after_init opt_bp_spec_reduce = true;
static bool __initdata opt_ibpb_alt;
+/* BUG() where map_domain_pages() would fall back to vmap() (vmap-strict). */
+bool __ro_after_init opt_vmap_strict;
+
/* Give each vCPU of an HVM domain its own per-domain area (vCPU-PT). */
bool __ro_after_init opt_vcpu_pt_hvm;
bool __ro_after_init opt_vcpu_pt_hwdom;
@@ -591,6 +594,8 @@ static int __init cf_check parse_asi(const char *s)
rc = -EINVAL;
}
}
+ else if ( (val = parse_boolean("vmap-strict", s, ss)) >= 0 )
+ opt_vmap_strict = val;
else if ( (val = parse_boolean("vcpu-pt", s, ss)) != -1 )
{
switch ( val )
@@ -840,6 +845,8 @@ static void __init print_details(enum ind_thunk thunk)
printk(" ASI features for PV VMs:%s\n",
sparse_dmap_pv() && !opt_xpti_domu
? " sparse-directmap" : " None");
+ printk(" ASI vmap-strict: %s\n",
+ opt_vmap_strict ? "enabled" : "disabled");
}
}
diff --git a/xen/include/xen/domain_page.h b/xen/include/xen/domain_page.h
index c89b149e54..bf45a4cf55 100644
--- a/xen/include/xen/domain_page.h
+++ b/xen/include/xen/domain_page.h
@@ -31,6 +31,27 @@ void *map_domain_page(mfn_t mfn);
*/
void unmap_domain_page(const void *ptr);
+/*
+ * Map @nr page frames, not necessarily physically contiguous, at
+ * consecutive virtual addresses. Returns the virtual address of the
+ * first frame, or NULL when no space is available. As with
+ * map_domain_page(), the mapping is only valid within the current VCPU,
+ * and must be unmapped by it; unmap_domain_pages() takes the same @nr,
+ * and accepts a pointer carrying an in-page offset. Implemented on x86
+ * only at present.
+ *
+ * @nr has an implementation-defined upper bound (x86:
+ * MAPCACHE_MULTIPAGE_ENTRIES). Exceeding it is a caller bug rather than a
+ * transient failure, and fails the mapping; callers needing more pages
+ * than a per-VCPU window holds want vmap() directly.
+ *
+ * unmap_domain_pages() is fatal given a pointer or an @nr which indexes
+ * outside the current VCPU's multipage slots. A wrong @nr within them
+ * is not detected: callers must pass the @nr they mapped with.
+ */
+void *map_domain_pages(const mfn_t *mfn, unsigned int nr);
+void unmap_domain_pages(const void *ptr, unsigned int nr);
+
/*
* Given a VA from map_domain_page(), return its underlying MFN.
*/
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 36/44] x86/shadow: use map_domain_pages() for cross-page emulated writes
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (34 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 35/44] x86/domain_page: introduce map_domain_pages() George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 37/44] x86/hvm: use map_domain_pages() for multi-frame emulated accesses George Dunlap
` (7 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel
Cc: Jan Beulich, Andrew Cooper, Tim Deegan, Roger Pau Monné,
Teddy Astie
sh_emulate_map_dest() picks its mapping method purely on whether the
write straddles a page boundary: a single page goes through
map_domain_page(), two pages through vmap() -- mapping the destination
in every context for the duration, those of domains on the sparse
directmap view included, whatever the guest's own configuration.
map_domain_pages() maps it only in the page tables of the guest's own
domain; use it for the two-page case.
The NULL return handling is unchanged: the primitive can fail only for
want of a free slot, and the emulation then bails out as unhandleable
exactly as before.
This is one of the two in-tree consumers of the primitive.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New in this version.
---
xen/arch/x86/mm/shadow/hvm.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/xen/arch/x86/mm/shadow/hvm.c b/xen/arch/x86/mm/shadow/hvm.c
index 0b23326214..b1c3d5e637 100644
--- a/xen/arch/x86/mm/shadow/hvm.c
+++ b/xen/arch/x86/mm/shadow/hvm.c
@@ -506,7 +506,7 @@ static void *sh_emulate_map_dest(struct vcpu *v, unsigned long vaddr,
/* Cross-page writes mean probably not a pagetable. */
sh_remove_shadows(d, sh_ctxt->mfn[1], 0, 0 /* Slow, can fail. */ );
- map = vmap(sh_ctxt->mfn, 2);
+ map = map_domain_pages(sh_ctxt->mfn, 2);
if ( !map )
{
put_page(mfn_to_page(sh_ctxt->mfn[0]));
@@ -670,7 +670,7 @@ static void sh_emulate_unmap_dest(struct vcpu *v, void *addr,
if ( unlikely(mfn_valid(sh_ctxt->mfn[1])) )
{
put_page(mfn_to_page(sh_ctxt->mfn[1]));
- vunmap((void *)((unsigned long)addr & PAGE_MASK));
+ unmap_domain_pages(addr, 2);
}
else
unmap_domain_page(addr);
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 37/44] x86/hvm: use map_domain_pages() for multi-frame emulated accesses
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (35 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 36/44] x86/shadow: use map_domain_pages() for cross-page emulated writes George Dunlap
@ 2026-11-03 22:13 ` George Dunlap
2026-11-03 22:14 ` [PATCH v6 38/44] x86/emulator: Refactor FXSAVE_AREA to use wrappers George Dunlap
` (6 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:13 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
hvmemul_map_linear_addr() picks its mapping method purely on the frame
count: one frame goes through map_domain_page(), more through vmap() --
mapping the guest data in every context for the duration, those of
domains on the sparse directmap view included, whatever the guest's own
configuration. map_domain_pages() maps them only in the page tables of
the guest's own domain; use it for the multi-frame case.
The NULL return handling is unchanged: the primitive can fail only for
want of a free slot, and the access then bails out as unhandleable
exactly as before.
This is one of the two in-tree consumers of the primitive. Other
long-lived global mappings of guest pages (the ioreq server pages among
them) remain in the vmap area.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New in this version.
---
xen/arch/x86/hvm/emulate.c | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/xen/arch/x86/hvm/emulate.c b/xen/arch/x86/hvm/emulate.c
index da7c08b5d9..7e7beccb93 100644
--- a/xen/arch/x86/hvm/emulate.c
+++ b/xen/arch/x86/hvm/emulate.c
@@ -735,7 +735,8 @@ static void *hvmemul_map_linear_addr(
/*
* The maximum access size depends on the number of adjacent mfns[] which
- * can be vmap()'d, accouting for possible misalignment within the region.
+ * can be mapped together, accounting for possible misalignment within
+ * the region.
* The higher level emulation callers are responsible for ensuring that
* mfns[] is large enough for the requested access size.
*/
@@ -824,9 +825,9 @@ static void *hvmemul_map_linear_addr(
/* Entire access within a single frame? */
if ( nr_frames == 1 )
mapping = map_domain_page(hvmemul_ctxt->mfn[0]);
- /* Multiple frames? Need to vmap(). */
- else if ( (mapping = vmap(hvmemul_ctxt->mfn,
- nr_frames)) == NULL )
+ /* Multiple frames? Need a multi-page mapping. */
+ else if ( (mapping = map_domain_pages(hvmemul_ctxt->mfn,
+ nr_frames)) == NULL )
goto unhandleable;
#ifndef NDEBUG /* Poision unused mfn[]s with INVALID_MFN. */
@@ -867,7 +868,7 @@ static void hvmemul_unmap_linear_addr(
if ( nr_frames == 1 )
unmap_domain_page(mapping);
else
- vunmap(mapping);
+ unmap_domain_pages(mapping, nr_frames);
for ( i = 0; i < nr_frames; i++ )
{
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 38/44] x86/emulator: Refactor FXSAVE_AREA to use wrappers
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (36 preceding siblings ...)
2026-11-03 22:13 ` [PATCH v6 37/44] x86/hvm: use map_domain_pages() for multi-frame emulated accesses George Dunlap
@ 2026-11-03 22:14 ` George Dunlap
2026-11-03 22:14 ` [PATCH v6 39/44] x86/mpx: Map/unmap xsave area in read_bndcfgu() George Dunlap
` (5 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:14 UTC (permalink / raw)
To: xen-devel
Cc: Alejandro Vallejo, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Alejandro Vallejo <alejandro.vallejo@cloud.com>
The FXSAVE/FXRSTOR emulation scratch buffer is the calling vCPU's own
XSAVE area, reached today through a raw pointer dereference. Route it
through the VCPU_MAP_XSAVE_AREA()/VCPU_UNMAP_XSAVE_AREA() seam
instead, so the buffer keeps working when guest XSAVE areas move
behind per-vCPU mappings. This path always operates on `current`, for
which the map has a fastpath; the unmap is a no-op outside Xen builds.
The macro pair is named MAP_FXSAVE_AREA()/UNMAP_FXSAVE_AREA() rather
than keeping the bare FXSAVE_AREA name, so that the mapping operation
is visible at the use sites.
blk_fxsave() saves directly into the caller's buffer when it is large
enough, so it needs a second variable: the save destination and the
mapped area are not the same pointer, and only the latter may be
handed to the unmap.
Signed-off-by: Alejandro Vallejo <alejandro.vallejo@cloud.com>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Revived from "x86: Address Space Isolation FPU preparations" v3 patch
08/12 (<20250110132823.24348-9-alejandro.vallejo@cloud.com>, Acked-by
Jan Beulich modulo the rename below; no v4 was ever posted).
Changes in v5:
- New in this version.
Changes since the previously posted version:
- Renamed FXSAVE_AREA to MAP_FXSAVE_AREA(), as requested by Jan
Beulich on the original thread ("the existing macro better would be
adjusted to become MAP_FXSAVE_AREA()"), making the map operation
visible at the use sites.
- Introduced the separate save-destination variable in blk_fxsave().
The original unmapped `fxsr` after that variable had been
repointed at the caller's buffer, so the unmap received a pointer
that was never mapped. Harmless while the unmap was a stub, but a
latent bug, and exactly the class of mistake Jan predicted in the
message quoted above when he objected to the mapping being
invisible at the use sites.
- Reworded the commit message.
- Unmap before raising #GP for a reserved MXCSR bit in blk_fxrstor:
generate_exception_if() leaves through the done: label, past the
unmap.
---
xen/arch/x86/x86_emulate/blk.c | 36 +++++++++++++++++++++++-----------
1 file changed, 25 insertions(+), 11 deletions(-)
diff --git a/xen/arch/x86/x86_emulate/blk.c b/xen/arch/x86/x86_emulate/blk.c
index 9fddf649ee..b3055c8084 100644
--- a/xen/arch/x86/x86_emulate/blk.c
+++ b/xen/arch/x86/x86_emulate/blk.c
@@ -11,9 +11,12 @@
!defined(X86EMUL_NO_SIMD)
# ifdef __XEN__
# include <asm/xstate.h>
-# define FXSAVE_AREA ((void *)¤t->arch.xsave_area->fpu_sse)
+/* Has a fastpath for `current`, so there's no actual map */
+# define MAP_FXSAVE_AREA() ((void *)VCPU_MAP_XSAVE_AREA(current))
+# define UNMAP_FXSAVE_AREA(x) VCPU_UNMAP_XSAVE_AREA(current, x)
# else
-# define FXSAVE_AREA get_fpu_save_area()
+# define MAP_FXSAVE_AREA() get_fpu_save_area()
+# define UNMAP_FXSAVE_AREA(x) ((void)(x))
# endif
#endif
@@ -254,7 +257,7 @@ int x86_emul_blk(
case blk_fxrstor:
{
- struct x86_fxsr *fxsr = FXSAVE_AREA;
+ struct x86_fxsr *fxsr = MAP_FXSAVE_AREA();
ASSERT(!data);
ASSERT(bytes == sizeof(*fxsr));
@@ -277,18 +280,26 @@ int x86_emul_blk(
(unsigned int)offsetof(struct x86_fxsr, rsvd)));
memset(fxsr->rsvd, 0, sizeof(*fxsr) - offsetof(struct x86_fxsr, rsvd));
- generate_exception_if(fxsr->mxcsr & ~mxcsr_mask, X86_EXC_GP, 0);
+ if ( fxsr->mxcsr & ~mxcsr_mask )
+ {
+ UNMAP_FXSAVE_AREA(fxsr);
+ generate_exception(X86_EXC_GP, 0);
+ }
if ( s->rex_prefix & REX_W )
asm volatile ( "fxrstorq %0" :: "m" (*fxsr) );
else
asm volatile ( "fxrstor %0" :: "m" (*fxsr) );
+
+ UNMAP_FXSAVE_AREA(fxsr);
+
break;
}
case blk_fxsave:
{
- struct x86_fxsr *fxsr = FXSAVE_AREA;
+ struct x86_fxsr *fxsr = MAP_FXSAVE_AREA();
+ struct x86_fxsr *dst = fxsr;
ASSERT(!data);
ASSERT(bytes == sizeof(*fxsr));
@@ -296,17 +307,20 @@ int x86_emul_blk(
if ( s->op_bytes < sizeof(*fxsr) )
/* Don't chance consuming uninitialized data. */
- memset(fxsr, 0, s->op_bytes);
+ memset(dst, 0, s->op_bytes);
else
- fxsr = ptr;
+ dst = ptr;
if ( s->rex_prefix & REX_W )
- asm volatile ( "fxsaveq %0" : "=m" (*fxsr) );
+ asm volatile ( "fxsaveq %0" : "=m" (*dst) );
else
- asm volatile ( "fxsave %0" : "=m" (*fxsr) );
+ asm volatile ( "fxsave %0" : "=m" (*dst) );
+
+ if ( dst != ptr ) /* i.e. s->op_bytes < sizeof(*fxsr) */
+ memcpy(ptr, dst, s->op_bytes);
+
+ UNMAP_FXSAVE_AREA(fxsr);
- if ( fxsr != ptr ) /* i.e. s->op_bytes < sizeof(*fxsr) */
- memcpy(ptr, fxsr, s->op_bytes);
break;
}
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 39/44] x86/mpx: Map/unmap xsave area in read_bndcfgu()
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (37 preceding siblings ...)
2026-11-03 22:14 ` [PATCH v6 38/44] x86/emulator: Refactor FXSAVE_AREA to use wrappers George Dunlap
@ 2026-11-03 22:14 ` George Dunlap
2026-11-03 22:14 ` [PATCH v6 40/44] x86/fpu: Pass explicit xsave areas to fpu_(f)xsave() George Dunlap
` (4 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:14 UTC (permalink / raw)
To: xen-devel
Cc: Alejandro Vallejo, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Alejandro Vallejo <alejandro.vallejo@cloud.com>
The BNDCSR read executes XSAVE into the local idle vCPU's save area as
scratch space and pulls BNDCFGU out of the result. Fetch that buffer
through the VCPU_MAP_XSAVE_AREA() seam rather than a raw dereference,
so the function is indifferent to where vCPU save areas live. This
was the last raw access to another vCPU's save-area pointer outside
the seam.
No functional change.
Signed-off-by: Alejandro Vallejo <alejandro.vallejo@cloud.com>
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Revived from "x86: Address Space Isolation FPU preparations" v3 patch
09/12 (<20250110132823.24348-10-alejandro.vallejo@cloud.com>; no v4
was ever posted).
Changes in v5:
- New in this version.
Changes since the previously posted version:
- Constified the vcpu pointer, as requested by Jan Beulich on the
original thread (the xstate pointer itself stays mutable: the
asm()s write through it, as Jan noted in withdrawing that half of
the request).
- Expanded the commit message.
---
xen/arch/x86/xstate.c | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/xen/arch/x86/xstate.c b/xen/arch/x86/xstate.c
index 8336414c52..2eae3f098d 100644
--- a/xen/arch/x86/xstate.c
+++ b/xen/arch/x86/xstate.c
@@ -1023,9 +1023,10 @@ int handle_xsetbv(u32 index, u64 new_bv)
uint64_t read_bndcfgu(void)
{
+ uint64_t bndcfgu = 0;
unsigned long cr0 = read_cr0();
- struct xsave_struct *xstate
- = idle_vcpu[smp_processor_id()]->arch.xsave_area;
+ const struct vcpu *v = idle_vcpu[smp_processor_id()];
+ struct xsave_struct *xstate = VCPU_MAP_XSAVE_AREA(v);
const struct xstate_bndcsr *bndcsr;
ASSERT(cpu_has_mpx);
@@ -1051,7 +1052,12 @@ uint64_t read_bndcfgu(void)
if ( cr0 & X86_CR0_TS )
write_cr0(cr0);
- return xstate->xsave_hdr.xstate_bv & X86_XCR0_BNDCSR ? bndcsr->bndcfgu : 0;
+ if ( xstate->xsave_hdr.xstate_bv & X86_XCR0_BNDCSR )
+ bndcfgu = bndcsr->bndcfgu;
+
+ VCPU_UNMAP_XSAVE_AREA(v, xstate);
+
+ return bndcfgu;
}
void xstate_set_init(uint64_t mask)
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 40/44] x86/fpu: Pass explicit xsave areas to fpu_(f)xsave()
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (38 preceding siblings ...)
2026-11-03 22:14 ` [PATCH v6 39/44] x86/mpx: Map/unmap xsave area in read_bndcfgu() George Dunlap
@ 2026-11-03 22:14 ` George Dunlap
2026-11-03 22:14 ` [PATCH v6 41/44] x86/fpu: Pass explicit xsave areas to fpu_(f)xrstor() George Dunlap
` (3 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:14 UTC (permalink / raw)
To: xen-devel
Cc: Alejandro Vallejo, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Alejandro Vallejo <alejandro.vallejo@cloud.com>
Have the save entry point fetch the vCPU's XSAVE area once, through
the VCPU_MAP_XSAVE_AREA() seam, and pass the resulting pointer down
explicitly: xsave() gains an area parameter in place of its internal
dereference, and fpu_fxsave() takes the legacy region directly. The
lower-level helpers no longer know where a vCPU's save area lives,
which is the property the coming relocation of guest save areas
relies on; the vcpu pointers become const on the way.
No functional change.
Signed-off-by: Alejandro Vallejo <alejandro.vallejo@cloud.com>
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Revived from "x86: Address Space Isolation FPU preparations" v3 patch
10/12 (<20250110132823.24348-11-alejandro.vallejo@cloud.com>, Acked-by
Jan Beulich; no v4 was ever posted).
Changes in v5:
- New in this version.
Changes since the previously posted version:
- Regenerated against the eager-FPU-only tree: the entry point is now
vcpu_save_fpu() (vcpu_xsave_mask()/_vcpu_save_fpu() no longer
exist), and the mask is always XSTATE_ALL.
- Expanded the commit message.
---
xen/arch/x86/i387.c | 18 ++++++++++++------
xen/arch/x86/include/asm/xstate.h | 2 +-
xen/arch/x86/xstate.c | 3 +--
3 files changed, 14 insertions(+), 9 deletions(-)
diff --git a/xen/arch/x86/i387.c b/xen/arch/x86/i387.c
index c6796bc97b..65491b5697 100644
--- a/xen/arch/x86/i387.c
+++ b/xen/arch/x86/i387.c
@@ -109,7 +109,8 @@ static inline void fpu_fxrstor(struct vcpu *v)
/*******************************/
/* Save x87 extended state */
-static inline void fpu_xsave(struct vcpu *v)
+static inline void fpu_xsave(const struct vcpu *v,
+ struct xsave_struct *xsave_area)
{
bool ok;
@@ -119,15 +120,14 @@ static inline void fpu_xsave(struct vcpu *v)
*/
ok = set_xcr0(v->arch.xcr0_accum | XSTATE_FP_SSE);
ASSERT(ok);
- xsave(v, XSTATE_ALL);
+ xsave(v, xsave_area, XSTATE_ALL);
ok = set_xcr0(v->arch.xcr0 ?: XSTATE_FP_SSE);
ASSERT(ok);
}
/* Save x87 FPU, MMX, SSE and SSE2 state */
-static inline void fpu_fxsave(struct vcpu *v)
+static inline void fpu_fxsave(const struct vcpu *v, fpusse_t *fpu_ctxt)
{
- fpusse_t *fpu_ctxt = &v->arch.xsave_area->fpu_sse;
unsigned int fip_width = v->domain->arch.x87_fip_width;
if ( fip_width != 4 )
@@ -192,15 +192,21 @@ void vcpu_restore_fpu(struct vcpu *v)
*/
void vcpu_save_fpu(struct vcpu *v)
{
+ struct xsave_struct *xsave_area;
+
ASSERT(!is_idle_vcpu(v));
/* This can happen, if a paravirtualised guest OS has set its CR0.TS. */
clts();
+ xsave_area = VCPU_MAP_XSAVE_AREA(v);
+
if ( cpu_has_xsave )
- fpu_xsave(v);
+ fpu_xsave(v, xsave_area);
else
- fpu_fxsave(v);
+ fpu_fxsave(v, &xsave_area->fpu_sse);
+
+ VCPU_UNMAP_XSAVE_AREA(v, xsave_area);
}
/* Initialize FPU's context save area */
diff --git a/xen/arch/x86/include/asm/xstate.h b/xen/arch/x86/include/asm/xstate.h
index 0519379edb..40bb6f152e 100644
--- a/xen/arch/x86/include/asm/xstate.h
+++ b/xen/arch/x86/include/asm/xstate.h
@@ -95,7 +95,7 @@ uint64_t get_xcr0(void);
void set_msr_xss(u64 xss);
uint64_t get_msr_xss(void);
uint64_t read_bndcfgu(void);
-void xsave(struct vcpu *v, uint64_t mask);
+void xsave(const struct vcpu *v, struct xsave_struct *ptr, uint64_t mask);
void xrstor(struct vcpu *v, uint64_t mask);
void xstate_set_init(uint64_t mask);
bool xsave_enabled(const struct vcpu *v);
diff --git a/xen/arch/x86/xstate.c b/xen/arch/x86/xstate.c
index 2eae3f098d..058bc4c0bc 100644
--- a/xen/arch/x86/xstate.c
+++ b/xen/arch/x86/xstate.c
@@ -304,9 +304,8 @@ void compress_xsave_states(struct vcpu *v, const void *src, unsigned int size)
VCPU_UNMAP_XSAVE_AREA(v, xstate);
}
-void xsave(struct vcpu *v, uint64_t mask)
+void xsave(const struct vcpu *v, struct xsave_struct *ptr, uint64_t mask)
{
- struct xsave_struct *ptr = v->arch.xsave_area;
uint32_t hmask = mask >> 32;
uint32_t lmask = mask;
unsigned int fip_width = v->domain->arch.x87_fip_width;
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 41/44] x86/fpu: Pass explicit xsave areas to fpu_(f)xrstor()
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (39 preceding siblings ...)
2026-11-03 22:14 ` [PATCH v6 40/44] x86/fpu: Pass explicit xsave areas to fpu_(f)xsave() George Dunlap
@ 2026-11-03 22:14 ` George Dunlap
2026-11-03 22:14 ` [PATCH v6 42/44] x86/xstate: classify xsave-area seam accesses by failure mode George Dunlap
` (2 subsequent siblings)
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:14 UTC (permalink / raw)
To: xen-devel
Cc: Alejandro Vallejo, Jan Beulich, Andrew Cooper,
Roger Pau Monné, Teddy Astie
From: Alejandro Vallejo <alejandro.vallejo@cloud.com>
Mirror of the save-side change: the restore entry point fetches the
vCPU's XSAVE area once, through the VCPU_MAP_XSAVE_AREA() seam, and
passes the pointer down explicitly. xrstor() gains an area parameter
in place of its internal dereference (its fault-recovery writes go
through the same mapped pointer), fpu_fxrstor() reduces to taking just
the legacy region, and the two direct xrstor() callers -- XSETBV
emulation and xstate_set_init() -- pass the pointers they already hold
mapped. In handle_xsetbv() this moves the xrstor inside the mapped
section, where it belongs.
No functional change.
Signed-off-by: Alejandro Vallejo <alejandro.vallejo@cloud.com>
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Revived from "x86: Address Space Isolation FPU preparations" v3 patch
11/12 (<20250110132823.24348-12-alejandro.vallejo@cloud.com>, Acked-by
Jan Beulich; no v4 was ever posted).
Changes in v5:
- New in this version.
Changes since the previously posted version:
- Regenerated against the eager-FPU-only tree: the entry point is now
vcpu_restore_fpu() (the lazy/nonlazy split no longer exists), and
the mask is always XSTATE_ALL.
- Moved handle_xsetbv()'s xrstor() call inside its map/unmap
bracket, which the explicit-area form makes possible.
- Expanded the commit message.
---
xen/arch/x86/i387.c | 19 ++++++++++++-------
xen/arch/x86/include/asm/xstate.h | 2 +-
xen/arch/x86/xstate.c | 7 +++----
3 files changed, 16 insertions(+), 12 deletions(-)
diff --git a/xen/arch/x86/i387.c b/xen/arch/x86/i387.c
index 65491b5697..b796687eeb 100644
--- a/xen/arch/x86/i387.c
+++ b/xen/arch/x86/i387.c
@@ -20,7 +20,8 @@
/* FPU Restore Functions */
/*******************************/
/* Restore x87 extended state */
-static inline void fpu_xrstor(struct vcpu *v, uint64_t mask)
+static inline void fpu_xrstor(const struct vcpu *v,
+ struct xsave_struct *xsave_area, uint64_t mask)
{
bool ok;
@@ -30,16 +31,14 @@ static inline void fpu_xrstor(struct vcpu *v, uint64_t mask)
*/
ok = set_xcr0(v->arch.xcr0_accum | XSTATE_FP_SSE);
ASSERT(ok);
- xrstor(v, mask);
+ xrstor(v, xsave_area, mask);
ok = set_xcr0(v->arch.xcr0 ?: XSTATE_FP_SSE);
ASSERT(ok);
}
/* Restore x87 FPU, MMX, SSE and SSE2 state */
-static inline void fpu_fxrstor(struct vcpu *v)
+static inline void fpu_fxrstor(const fpusse_t *fpu_ctxt)
{
- const fpusse_t *fpu_ctxt = &v->arch.xsave_area->fpu_sse;
-
/*
* Some CPUs don't save/restore FDP/FIP/FOP unless an exception
* is pending. Clear the x87 state here by setting it to fixed
@@ -174,12 +173,18 @@ static inline void fpu_fxsave(const struct vcpu *v, fpusse_t *fpu_ctxt)
/* Restore FPU state whenever VCPU is schduled in. */
void vcpu_restore_fpu(struct vcpu *v)
{
+ struct xsave_struct *xsave_area;
+
ASSERT(!is_idle_vcpu(v));
+ xsave_area = VCPU_MAP_XSAVE_AREA(v);
+
if ( cpu_has_xsave )
- fpu_xrstor(v, XSTATE_ALL);
+ fpu_xrstor(v, xsave_area, XSTATE_ALL);
else
- fpu_fxrstor(v);
+ fpu_fxrstor(&xsave_area->fpu_sse);
+
+ VCPU_UNMAP_XSAVE_AREA(v, xsave_area);
/* Xen doesn't need TS set, but the guest might. */
if ( is_pv_vcpu(v) && (v->arch.pv.ctrlreg[0] & X86_CR0_TS) )
diff --git a/xen/arch/x86/include/asm/xstate.h b/xen/arch/x86/include/asm/xstate.h
index 40bb6f152e..3d57aaadd8 100644
--- a/xen/arch/x86/include/asm/xstate.h
+++ b/xen/arch/x86/include/asm/xstate.h
@@ -96,7 +96,7 @@ void set_msr_xss(u64 xss);
uint64_t get_msr_xss(void);
uint64_t read_bndcfgu(void);
void xsave(const struct vcpu *v, struct xsave_struct *ptr, uint64_t mask);
-void xrstor(struct vcpu *v, uint64_t mask);
+void xrstor(const struct vcpu *v, struct xsave_struct *ptr, uint64_t mask);
void xstate_set_init(uint64_t mask);
bool xsave_enabled(const struct vcpu *v);
int __must_check validate_xstate(const struct domain *d,
diff --git a/xen/arch/x86/xstate.c b/xen/arch/x86/xstate.c
index 058bc4c0bc..3c9be271a3 100644
--- a/xen/arch/x86/xstate.c
+++ b/xen/arch/x86/xstate.c
@@ -378,11 +378,10 @@ void xsave(const struct vcpu *v, struct xsave_struct *ptr, uint64_t mask)
ptr->fpu_sse.x[FPU_WORD_SIZE_OFFSET] = fip_width;
}
-void xrstor(struct vcpu *v, uint64_t mask)
+void xrstor(const struct vcpu *v, struct xsave_struct *ptr, uint64_t mask)
{
uint32_t hmask = mask >> 32;
uint32_t lmask = mask;
- struct xsave_struct *ptr = v->arch.xsave_area;
unsigned int faults, prev_faults;
/*
@@ -1011,8 +1010,8 @@ int handle_xsetbv(u32 index, u64 new_bv)
clts();
asm ( "stmxcsr %0" : "=m" (xsave_area->fpu_sse.mxcsr) );
+ xrstor(curr, xsave_area, mask);
VCPU_UNMAP_XSAVE_AREA(curr, xsave_area);
- xrstor(curr, mask);
if ( cr0 & X86_CR0_TS )
write_cr0(cr0);
}
@@ -1079,7 +1078,7 @@ void xstate_set_init(uint64_t mask)
xstate = VCPU_MAP_XSAVE_AREA(v);
memset(&xstate->xsave_hdr, 0, sizeof(xstate->xsave_hdr));
- xrstor(v, mask);
+ xrstor(v, xstate, mask);
VCPU_UNMAP_XSAVE_AREA(v, xstate);
if ( cr0 & X86_CR0_TS )
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 42/44] x86/xstate: classify xsave-area seam accesses by failure mode
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (40 preceding siblings ...)
2026-11-03 22:14 ` [PATCH v6 41/44] x86/fpu: Pass explicit xsave areas to fpu_(f)xrstor() George Dunlap
@ 2026-11-03 22:14 ` George Dunlap
2026-11-03 22:14 ` [PATCH v6 43/44] x86/domain: restore FPU state after switching to the new page tables George Dunlap
2026-11-03 22:14 ` [PATCH v6 44/44] x86/xstate: move guest XSAVE areas behind the vCPU state window George Dunlap
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:14 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
The VCPU_MAP_XSAVE_AREA() seam currently cannot fail: the map is a
pointer fetch. Once guest XSAVE areas move behind per-vCPU mappings,
mapping another vCPU's multi-page area becomes an operation that can
run out of resources. Split the seam's users along that line ahead of
the change:
- expand_xsave_states() and compress_xsave_states() address the full
area; they gain an error return, which their four callers (the
vcpuextstate domctls and the HVM context save/load handlers)
propagate. The two deserialising callers (the setvcpuextstate
domctl and the HVM xsave-state load handler) restore the
previously-set XCR0 fields if the deserialisation fails, so a
failed restore leaves the vCPU's state as it was.
- Accessors touching only the legacy region and the XSAVE header --
vcpu_reset_fpu(), vcpu_setup_fpu(), hvm_save_cpu_ctxt() and
arch_get_info_guest() -- switch to a new
VCPU_MAP_XSAVE_AREA_PREFIX() variant. The prefix always fits the
area's first page, so that variant can remain infallible, and no
error plumbing is needed on the paths (INIT/SIPI, S3 resume) that
have no way to fail.
No functional change: both variants are still pointer fetches.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New in this version.
---
xen/arch/x86/domctl.c | 26 ++++++++++++++++++--------
xen/arch/x86/hvm/hvm.c | 27 ++++++++++++++++++---------
xen/arch/x86/i387.c | 8 ++++----
xen/arch/x86/include/asm/xstate.h | 13 +++++++++++--
xen/arch/x86/xstate.c | 18 +++++++++++++++---
5 files changed, 66 insertions(+), 26 deletions(-)
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index 04bcde5580..f0dad47a87 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1010,10 +1010,10 @@ long arch_do_domctl(
goto vcpuextstate_out;
}
- expand_xsave_states(v, xsave_area, xsave_size);
+ ret = expand_xsave_states(v, xsave_area, xsave_size);
- if ( copy_to_guest_offset(evc->buffer, offset, xsave_area,
- xsave_size) )
+ if ( !ret && copy_to_guest_offset(evc->buffer, offset,
+ xsave_area, xsave_size) )
ret = -EFAULT;
xfree(xsave_area);
}
@@ -1077,14 +1077,24 @@ long arch_do_domctl(
ret = -EINVAL; /* Not legitimate data. */
else
{
+ uint64_t old_xcr0, old_xcr0_accum;
+
vcpu_pause(v);
+ old_xcr0 = v->arch.xcr0;
+ old_xcr0_accum = v->arch.xcr0_accum;
v->arch.xcr0 = _xcr0;
v->arch.xcr0_accum = _xcr0_accum;
- compress_xsave_states(v, _xsave_area,
- evc->size - PV_XSAVE_HDR_SIZE);
+ ret = compress_xsave_states(v, _xsave_area,
+ evc->size - PV_XSAVE_HDR_SIZE);
- if ( is_hvm_domain(d) )
+ if ( ret )
+ {
+ /* Leave the vCPU's state as it was. */
+ v->arch.xcr0 = old_xcr0;
+ v->arch.xcr0_accum = old_xcr0_accum;
+ }
+ else if ( is_hvm_domain(d) )
hvmemul_cancel(v);
vcpu_unpause(v);
@@ -1468,10 +1478,10 @@ void arch_get_info_guest(struct vcpu *v, vcpu_guest_context_u c)
#define c(fld) (c.nat->fld)
#endif
- xsave_area = VCPU_MAP_XSAVE_AREA(v);
+ xsave_area = VCPU_MAP_XSAVE_AREA_PREFIX(v);
BUILD_BUG_ON(sizeof(c.nat->fpu_ctxt) != sizeof(xsave_area->fpu_sse));
memcpy(&c.nat->fpu_ctxt, &xsave_area->fpu_sse, sizeof(c.nat->fpu_ctxt));
- VCPU_UNMAP_XSAVE_AREA(v, xsave_area);
+ VCPU_UNMAP_XSAVE_AREA_PREFIX(v, xsave_area);
if ( is_pv_domain(d) )
c(flags = v->arch.pv.vgc_flags & ~VGCF_in_kernel);
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index eadccf3d79..42a7d03b3a 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -932,10 +932,10 @@ static int cf_check hvm_save_cpu_ctxt(struct vcpu *v, hvm_domain_context_t *h)
ctxt.ldtr_base = seg.base;
ctxt.ldtr_arbytes = seg.attr;
- xsave_area = VCPU_MAP_XSAVE_AREA(v);
+ xsave_area = VCPU_MAP_XSAVE_AREA_PREFIX(v);
BUILD_BUG_ON(sizeof(ctxt.fpu_regs) != sizeof(xsave_area->fpu_sse));
memcpy(ctxt.fpu_regs, &xsave_area->fpu_sse, sizeof(ctxt.fpu_regs));
- VCPU_UNMAP_XSAVE_AREA(v, xsave_area);
+ VCPU_UNMAP_XSAVE_AREA_PREFIX(v, xsave_area);
ctxt.flags = XEN_X86_FPU_INITIALISED;
return hvm_save_entry(CPU, v->vcpu_id, h, &ctxt);
@@ -1251,10 +1251,8 @@ static int cf_check hvm_save_cpu_xsave_states(
ctxt->xcr0 = v->arch.xcr0;
ctxt->xcr0_accum = v->arch.xcr0_accum;
- expand_xsave_states(v, &ctxt->save_area,
- size - offsetof(typeof(*ctxt), save_area));
-
- return 0;
+ return expand_xsave_states(v, &ctxt->save_area,
+ size - offsetof(typeof(*ctxt), save_area));
}
/*
@@ -1279,6 +1277,7 @@ static int cf_check hvm_load_cpu_xsave_states(
struct hvm_hw_cpu_xsave *ctxt;
const struct hvm_save_descriptor *desc;
unsigned int i, desc_start, desc_length;
+ uint64_t old_xcr0, old_xcr0_accum;
/* Which vcpu is this? */
vcpuid = hvm_load_instance(h);
@@ -1383,12 +1382,22 @@ static int cf_check hvm_load_cpu_xsave_states(
}
/* Checking finished */
+ old_xcr0 = v->arch.xcr0;
+ old_xcr0_accum = v->arch.xcr0_accum;
v->arch.xcr0 = ctxt->xcr0;
v->arch.xcr0_accum = ctxt->xcr0_accum;
- compress_xsave_states(v, &ctxt->save_area,
- size - offsetof(struct hvm_hw_cpu_xsave, save_area));
- return 0;
+ err = compress_xsave_states(v, &ctxt->save_area,
+ size - offsetof(struct hvm_hw_cpu_xsave,
+ save_area));
+ if ( err )
+ {
+ /* Leave the vCPU's state as it was. */
+ v->arch.xcr0 = old_xcr0;
+ v->arch.xcr0_accum = old_xcr0_accum;
+ }
+
+ return err;
}
#define HVM_CPU_MSR_SIZE(cnt) offsetof(struct hvm_msr, msr[cnt])
diff --git a/xen/arch/x86/i387.c b/xen/arch/x86/i387.c
index b796687eeb..d49e391e99 100644
--- a/xen/arch/x86/i387.c
+++ b/xen/arch/x86/i387.c
@@ -222,7 +222,7 @@ int vcpu_init_fpu(struct vcpu *v)
void vcpu_reset_fpu(struct vcpu *v)
{
- struct xsave_struct *xsave_area = VCPU_MAP_XSAVE_AREA(v);
+ struct xsave_struct *xsave_area = VCPU_MAP_XSAVE_AREA_PREFIX(v);
*xsave_area = (struct xsave_struct) {
.xsave_hdr.xstate_bv = X86_XCR0_X87,
@@ -233,19 +233,19 @@ void vcpu_reset_fpu(struct vcpu *v)
xsave_area->fpu_sse.fcw = FCW_RESET;
xsave_area->fpu_sse.ftw = FXSAVE_FTW_RESET;
- VCPU_UNMAP_XSAVE_AREA(v, xsave_area);
+ VCPU_UNMAP_XSAVE_AREA_PREFIX(v, xsave_area);
}
void vcpu_setup_fpu(struct vcpu *v, const void *data)
{
- struct xsave_struct *xsave_area = VCPU_MAP_XSAVE_AREA(v);
+ struct xsave_struct *xsave_area = VCPU_MAP_XSAVE_AREA_PREFIX(v);
*xsave_area = (struct xsave_struct) {
.fpu_sse = *(const fpusse_t*)data,
.xsave_hdr.xstate_bv = XSTATE_FP_SSE,
};
- VCPU_UNMAP_XSAVE_AREA(v, xsave_area);
+ VCPU_UNMAP_XSAVE_AREA_PREFIX(v, xsave_area);
}
/* Free FPU's context save area */
diff --git a/xen/arch/x86/include/asm/xstate.h b/xen/arch/x86/include/asm/xstate.h
index 3d57aaadd8..c23308cbe2 100644
--- a/xen/arch/x86/include/asm/xstate.h
+++ b/xen/arch/x86/include/asm/xstate.h
@@ -103,8 +103,8 @@ int __must_check validate_xstate(const struct domain *d,
uint64_t xcr0, uint64_t xcr0_accum,
const struct xsave_hdr *hdr);
int __must_check handle_xsetbv(u32 index, u64 new_bv);
-void expand_xsave_states(const struct vcpu *v, void *dest, unsigned int size);
-void compress_xsave_states(struct vcpu *v, const void *src, unsigned int size);
+int expand_xsave_states(const struct vcpu *v, void *dest, unsigned int size);
+int compress_xsave_states(struct vcpu *v, const void *src, unsigned int size);
/* extended state init and cleanup functions */
void xstate_free_save_area(struct vcpu *v);
@@ -171,4 +171,13 @@ xsave_area_compressed(const struct xsave_struct *xsave_area)
*/
#define VCPU_UNMAP_XSAVE_AREA(v, x) do { (void)(v); (x) = NULL; } while (0)
+/*
+ * Variant of VCPU_MAP_XSAVE_AREA() for accessors which touch nothing
+ * beyond the legacy region and the XSAVE header (the first
+ * XSTATE_AREA_MIN_SIZE bytes). That prefix always fits within the
+ * area's first page, so this variant cannot fail.
+ */
+#define VCPU_MAP_XSAVE_AREA_PREFIX(v) VCPU_MAP_XSAVE_AREA(v)
+#define VCPU_UNMAP_XSAVE_AREA_PREFIX(v, x) VCPU_UNMAP_XSAVE_AREA(v, x)
+
#endif /* __ASM_XSTATE_H */
diff --git a/xen/arch/x86/xstate.c b/xen/arch/x86/xstate.c
index 3c9be271a3..2dd1699aba 100644
--- a/xen/arch/x86/xstate.c
+++ b/xen/arch/x86/xstate.c
@@ -177,14 +177,19 @@ static void setup_xstate_comp(uint16_t *comp_offsets,
* It is the callers responsibility to ensure that there is xsave state to
* serialise, and that the provided buffer is exactly the right size.
*/
-void expand_xsave_states(const struct vcpu *v, void *dest, unsigned int size)
+int expand_xsave_states(const struct vcpu *v, void *dest, unsigned int size)
{
const struct xsave_struct *xstate = VCPU_MAP_XSAVE_AREA(v);
const void *src;
uint16_t comp_offsets[sizeof(xfeature_mask)*8];
- u64 xstate_bv = xstate->xsave_hdr.xstate_bv;
+ u64 xstate_bv;
u64 valid;
+ if ( !xstate )
+ return -ENOMEM;
+
+ xstate_bv = xstate->xsave_hdr.xstate_bv;
+
/* Check there is state to serialise (i.e. at least an XSAVE_HDR) */
BUG_ON(!v->arch.xcr0_accum);
/* Check there is the correct room to decompress into. */
@@ -233,6 +238,8 @@ void expand_xsave_states(const struct vcpu *v, void *dest, unsigned int size)
out:
VCPU_UNMAP_XSAVE_AREA(v, xstate);
+
+ return 0;
}
/*
@@ -245,13 +252,16 @@ void expand_xsave_states(const struct vcpu *v, void *dest, unsigned int size)
* It is the callers responsibility to ensure that the source buffer contains
* xsave state, is uncompressed, and is exactly the right size.
*/
-void compress_xsave_states(struct vcpu *v, const void *src, unsigned int size)
+int compress_xsave_states(struct vcpu *v, const void *src, unsigned int size)
{
struct xsave_struct *xstate = VCPU_MAP_XSAVE_AREA(v);
void *dest;
uint16_t comp_offsets[sizeof(xfeature_mask)*8];
u64 xstate_bv, valid;
+ if ( !xstate )
+ return -ENOMEM;
+
BUG_ON(!v->arch.xcr0_accum);
BUG_ON(size != xstate_uncompressed_size(v->arch.xcr0_accum));
ASSERT(!xsave_area_compressed(src));
@@ -302,6 +312,8 @@ void compress_xsave_states(struct vcpu *v, const void *src, unsigned int size)
out:
VCPU_UNMAP_XSAVE_AREA(v, xstate);
+
+ return 0;
}
void xsave(const struct vcpu *v, struct xsave_struct *ptr, uint64_t mask)
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 43/44] x86/domain: restore FPU state after switching to the new page tables
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (41 preceding siblings ...)
2026-11-03 22:14 ` [PATCH v6 42/44] x86/xstate: classify xsave-area seam accesses by failure mode George Dunlap
@ 2026-11-03 22:14 ` George Dunlap
2026-11-03 22:14 ` [PATCH v6 44/44] x86/xstate: move guest XSAVE areas behind the vCPU state window George Dunlap
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:14 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
__context_switch() restores the incoming vCPU's FPU state
(vcpu_restore_fpu(n)) while still running on the outgoing vCPU's page
tables, before write_ptbase(n). That dereferences n's XSAVE area
through whatever address it happens to have in p's address space --
fine today, when the area is globally mapped, but not once guest XSAVE
state moves to a per-vCPU mapping that need not exist in another vCPU's
address space (upcoming Address Space Isolation work).
Move the eager restore to just after write_ptbase(n), so it runs with
n's page tables loaded. The XCR0/XSS setup stays where it is (it
programs control registers / an MSR, independent of the page tables,
and XSS must be set before the supervisor-state xrstor). Nothing
between the save and the new restore point touches guest FPU state:
the ctxt_switch->to() hooks, psr and GDT handling are all FPU-free,
and Xen is built without SSE.
No functional change with the current global XSAVE-area mapping.
Assisted-by: Claude Code:claude-opus-4-8
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New in this version.
---
xen/arch/x86/domain.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 66d71ec41b..8222ad9e2a 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -2267,7 +2267,6 @@ static void __context_switch(void)
if ( cpu_has_xsaves && is_hvm_vcpu(n) )
set_msr_xss(n->arch.msrs->xss.raw);
}
- vcpu_restore_fpu(n);
nd->arch.ctxt_switch->to(n);
}
@@ -2283,7 +2282,10 @@ static void __context_switch(void)
write_ptbase(n);
if ( !is_idle_domain(nd) )
+ {
memcpy(stack_regs, vcpu_user_regs(n), CTXT_SWITCH_STACK_BYTES);
+ vcpu_restore_fpu(n);
+ }
#ifdef CONFIG_PV
/* Prefetch the VMCB if we expect to use it later in the context switch */
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH v6 44/44] x86/xstate: move guest XSAVE areas behind the vCPU state window
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
` (42 preceding siblings ...)
2026-11-03 22:14 ` [PATCH v6 43/44] x86/domain: restore FPU state after switching to the new page tables George Dunlap
@ 2026-11-03 22:14 ` George Dunlap
43 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-03 22:14 UTC (permalink / raw)
To: xen-devel; +Cc: Jan Beulich, Andrew Cooper, Roger Pau Monné, Teddy Astie
Guest vCPUs' XSAVE areas hold guest-owned register state, yet live in
always-mapped memory: xenheap objects when they fit a page, vmap-backed
allocations on hardware (AMX-class) where they do not. Under Address
Space Isolation both placements leave the state reachable from every
context. Move them to individual domain-heap pages, installed once at
vCPU creation at a fixed reservation (indexes 3-5) in the per-vCPU
state window, following the pattern set by the SVM VMCB and VMX MSR
list relocations. With this, all eight slots of the state window are
in use.
The seam introduced by the VCPU_MAP_XSAVE_AREA() wrappers now gains
its real implementation. The window address serves when v runs here
on its own page tables: v == current, and this_cpu(pgtable_vcpu) == v.
Neither test suffices alone. Inside the EFI runtime bracket, and after
a shadow-mode update_cr3() failure, `current`'s own page tables are not
loaded. While a CPU holds v's state lazily, v's page tables are
loaded, but an IPI syncing that state for another CPU can switch them
away at any time. The page-table test also structurally rules out one
vCPU reaching through a same-vcpu_id window address into another
domain's state, which a dispatch on `current` alone would have made
possible to get wrong silently. With the owning vCPU running on its
tables, or for an idle vCPU, the map is a pointer fetch, so the hot
paths map nothing. Otherwise (a paused, never-started or
lazily held vCPU) the full-area map uses map_domain_pages() and can
fail; those callers gained error handling in "x86/xstate: classify
xsave-area seam accesses by failure mode". The prefix variant maps
only the area's first page, which the legacy region and header always
fit, and hence cannot fail. A mapping of a single page is dropped with
unmap_vcpu_state_page(), which also flushes this CPU's translation of
it; unmap_domain_pages() flushes a multipage mapping itself.
__context_switch() saves the outgoing vCPU's FPU state after `current`
has moved to the incoming vCPU, with interrupts off, while the outgoing
vCPU's page tables -- and therefore its window -- are normally still
loaded. It takes the window address through a path of its own,
vcpu_save_fpu_ctxt_switch(): the transient-mapping fallback must not
run from there, as it would sync the local execution state and
re-enter __context_switch(). The shadow-mode paths which abandon a
vCPU's page tables early (update_cr3() failure in new_guest_cr3() and
_toggle_guest_pt()) leave the idle tables loaded with the outgoing
vCPU's state still in registers. Those paths have already crashed the
domain, so there is nothing worth saving, and the save is skipped there,
as the register frame's already is: both sit under one test of the
loaded page tables. Anything else failing it is a page-table tracking
bug rather than a recoverable state, so it is fatal in every build.
The window reservation is fixed at three pages, sized for the largest
save area current hardware produces (AMX TILEDATA); xstate_init()
refuses, with an explanatory panic, hardware whose enabled feature
set exceeds it. It does so on every boot, with or without asi=: every
guest vCPU's area moves.
Idle vCPUs keep a globally-reachable area (xenheap, or vmap space where
it exceeds a page, as with AMX) deliberately: it is scratch for
accesses which execute with a guest's page tables loaded
(read_bndcfgu(), xstate_set_init()) and holds no guest register state.
On such paths the area a guest's MPX state is transiently saved into
remains reachable from other contexts, as today.
The backing pages are caller-owned (no _PAGE_AVAIL0): the per-domain
teardown reclaims only the page tables, so xstate_free_save_area()
unmaps the pages from the window and frees them itself. Without
per-vCPU page tables (every PV domain) the window is in the per-domain
area that all its domain's vCPUs share, which outlives a vCPU whose
creation fails. A failed partial allocation unwinds in place, and
xstate_free_save_area() tolerates the resulting half-built state, which
arch_vcpu_create()'s failure path relies on.
Memory cost: sub-page areas (all non-AMX hardware) round up from a
2.7KiB xenheap object to a full page per vCPU.
The move does not depend on asi=, and applies to PV vCPUs as to HVM
ones. A context switch now calls vcpu_xsave_area_ctxt_switch() for the
save, and vcpu_map_xsave_area() and vcpu_unmap_xsave_area() around the
restore, out of line, instead of reading v->arch.xsave_area. The XRSTOR
following write_ptbase() reaches the area through a 4KiB, non-global PTE
in the per-domain area. As switch_cr3_cr4() has just flushed every
non-global translation, that takes a page walk on every switch to a
guest vCPU, where a sub-page area, a xenheap object, was reached through
a superpage of the full directmap; the walk goes through the L1 table
which the register frame's restore has just walked. On an Intel NUC, a
64-bit PV guest's vCPU context switch takes 15-30ns longer, with XPTI
off or on and against either the tree before the XSAVE work or the one
just before this change; code placement alone moves that measurement by
up to about 20ns between builds. That was measured while the PV
guest's register frames were still in struct vcpu.
On AMX-class hardware, where the area takes three pages, a full-area
map for a vCPU other than the one running here goes through
map_domain_pages(): XEN_DOMCTL_getvcpuextstate and
XEN_DOMCTL_setvcpuextstate make one for PV save, restore and migration,
as the HVM XSAVE save record does for HVM guests. The area is then
mapped in the calling vCPU's multipage slots, which only the calling
domain's page tables carry, and dropping the mapping flushes the
calling CPU's translations alone. Each of those operations can now
fail with -ENOMEM.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xenproject.org>
---
Changes in v5:
- New in this version.
xstate_init() now panics, with or without asi=, if the enabled XSAVE
features need more than the three pages the window reserves. No
current hardware does. Would you rather this did more, for instance
size the window's XSAVE range from the size found at boot, or fall back
to xenheap areas on such hardware?
---
xen/arch/x86/domain.c | 19 ++--
xen/arch/x86/i387.c | 23 +++-
xen/arch/x86/include/asm/config.h | 3 +
xen/arch/x86/include/asm/domain.h | 15 ++-
xen/arch/x86/include/asm/i387.h | 1 +
xen/arch/x86/include/asm/xstate.h | 60 +++++------
xen/arch/x86/xstate.c | 172 +++++++++++++++++++++++++++++-
7 files changed, 243 insertions(+), 50 deletions(-)
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 8222ad9e2a..f750ff4e6a 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -594,7 +594,11 @@ int arch_vcpu_create(struct vcpu *v)
BUILD_BUG_ON(VCPU_STATE_IDX_VMCB >= VCPU_STATE_NR_PAGES);
BUILD_BUG_ON(VCPU_STATE_IDX_N2VMCB >= VCPU_STATE_NR_PAGES);
BUILD_BUG_ON(VCPU_STATE_IDX_VMX_MSR >= VCPU_STATE_NR_PAGES);
+ BUILD_BUG_ON(VCPU_STATE_IDX_XSAVE + VCPU_STATE_XSAVE_PAGES >
+ VCPU_STATE_NR_PAGES);
BUILD_BUG_ON(VCPU_STATE_IDX_REGS >= VCPU_STATE_NR_PAGES);
+ BUILD_BUG_ON(VCPU_STATE_IDX_REGS < VCPU_STATE_IDX_XSAVE +
+ VCPU_STATE_XSAVE_PAGES);
BUILD_BUG_ON(VCPU_STATE_IDX_EMUL >= VCPU_STATE_NR_PAGES);
BUILD_BUG_ON(VCPU_STATE_IDX_EMUL == VCPU_STATE_IDX_REGS);
/* Page tables only; torn down with the rest of the per-domain area. */
@@ -2234,17 +2238,20 @@ static void __context_switch(void)
{
ASSERT(read_atomic(&p->dirty_cpu) == cpu);
/*
- * p's frame is reached through its state window, mapped while p's
- * own page tables are loaded, as they normally still are here. Not
- * so after the shadow-mode paths which abandon a vCPU's tables early
- * (update_cr3() failure), which have crashed the domain: the frame
- * is left unsaved. A transient mapping would re-enter this function.
+ * p's frame and XSAVE area are reached through its state window,
+ * mapped while p's own page tables are loaded, as they normally
+ * still are here. Not so after the shadow-mode paths which abandon
+ * a vCPU's tables early (update_cr3() failure), which have crashed
+ * the domain: both are left unsaved. A transient mapping would
+ * re-enter this function.
*/
if ( likely(this_cpu(pgtable_vcpu) == p) )
+ {
memcpy(vcpu_user_regs(p), stack_regs, CTXT_SWITCH_STACK_BYTES);
+ vcpu_save_fpu_ctxt_switch(p);
+ }
else
BUG_ON(!domain_shutting_down(pd));
- vcpu_save_fpu(p);
pd->arch.ctxt_switch->from(p);
}
diff --git a/xen/arch/x86/i387.c b/xen/arch/x86/i387.c
index d49e391e99..239404b37b 100644
--- a/xen/arch/x86/i387.c
+++ b/xen/arch/x86/i387.c
@@ -177,7 +177,9 @@ void vcpu_restore_fpu(struct vcpu *v)
ASSERT(!is_idle_vcpu(v));
+ /* v runs here on its own page tables: the map cannot fail. */
xsave_area = VCPU_MAP_XSAVE_AREA(v);
+ BUG_ON(!xsave_area);
if ( cpu_has_xsave )
fpu_xrstor(v, xsave_area, XSTATE_ALL);
@@ -195,25 +197,36 @@ void vcpu_restore_fpu(struct vcpu *v)
* On each context switch, save the necessary FPU info of VCPU being switch
* out. It dispatches saving operation based on CPU's capability.
*/
-void vcpu_save_fpu(struct vcpu *v)
+static void save_fpu(struct vcpu *v, struct xsave_struct *xsave_area)
{
- struct xsave_struct *xsave_area;
-
ASSERT(!is_idle_vcpu(v));
/* This can happen, if a paravirtualised guest OS has set its CR0.TS. */
clts();
- xsave_area = VCPU_MAP_XSAVE_AREA(v);
-
if ( cpu_has_xsave )
fpu_xsave(v, xsave_area);
else
fpu_fxsave(v, &xsave_area->fpu_sse);
+}
+/* Save the FPU state of v, the vCPU running here. */
+void vcpu_save_fpu(struct vcpu *v)
+{
+ /* As in vcpu_restore_fpu(): the fastpath, which cannot fail. */
+ struct xsave_struct *xsave_area = VCPU_MAP_XSAVE_AREA(v);
+
+ BUG_ON(!xsave_area);
+ save_fpu(v, xsave_area);
VCPU_UNMAP_XSAVE_AREA(v, xsave_area);
}
+/* As vcpu_save_fpu(), for __context_switch() switching v out. */
+void vcpu_save_fpu_ctxt_switch(struct vcpu *v)
+{
+ save_fpu(v, vcpu_xsave_area_ctxt_switch(v));
+}
+
/* Initialize FPU's context save area */
int vcpu_init_fpu(struct vcpu *v)
{
diff --git a/xen/arch/x86/include/asm/config.h b/xen/arch/x86/include/asm/config.h
index 52d1ebc0e7..498e3c91fa 100644
--- a/xen/arch/x86/include/asm/config.h
+++ b/xen/arch/x86/include/asm/config.h
@@ -325,6 +325,9 @@ extern unsigned long xen_phys_start;
#define VCPU_STATE_IDX_VMCB 0
#define VCPU_STATE_IDX_N2VMCB 1
#define VCPU_STATE_IDX_VMX_MSR 2
+#define VCPU_STATE_IDX_XSAVE 3
+/* Pages reserved for the XSAVE area; bounded at boot in xstate_init(). */
+#define VCPU_STATE_XSAVE_PAGES 3
#define VCPU_STATE_IDX_REGS 6
#define VCPU_STATE_IDX_EMUL 7
diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 3748e417ce..69ab854177 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -685,12 +685,19 @@ struct arch_vcpu
unsigned long cr3; /* (MA) value to install in HW CR3 */
/*
- * The save area for Processor Extended States and the bitmask of the
- * XSAVE/XRSTOR features. They are used by: 1) when a vcpu (which has
- * dirtied FPU/SSE) is scheduled out we XSAVE the states here; 2) in
- * #NM handler, we XRSTOR the states we XSAVE-ed;
+ * The save area for Processor Extended States, XSAVEd on context
+ * switch out and XRSTORed on context switch in. For guest vCPUs
+ * the backing pages (xsave_pg[], xsave_pgcnt of them) come from the
+ * domain heap and are mapped once, for the life of the vCPU, at a
+ * fixed spot in the per-vCPU state window; xsave_area holds that
+ * window address, valid whenever the owning vCPU's page tables are
+ * loaded. Idle vCPUs' areas are ordinary Xen allocations (xenheap,
+ * or vmap space when larger than a page), reachable from every
+ * context. Any other access goes through VCPU_MAP_XSAVE_AREA().
*/
struct xsave_struct *xsave_area;
+ struct page_info *xsave_pg[VCPU_STATE_XSAVE_PAGES];
+ unsigned int xsave_pgcnt;
uint64_t xcr0;
/* Accumulated eXtended features mask for using XSAVE/XRESTORE by Xen
* itself, as we can never know whether guest OS depends on content
diff --git a/xen/arch/x86/include/asm/i387.h b/xen/arch/x86/include/asm/i387.h
index 0717005d31..8d28de3ba8 100644
--- a/xen/arch/x86/include/asm/i387.h
+++ b/xen/arch/x86/include/asm/i387.h
@@ -29,6 +29,7 @@ struct ix87_env {
void vcpu_restore_fpu(struct vcpu *v);
void vcpu_save_fpu(struct vcpu *v);
+void vcpu_save_fpu_ctxt_switch(struct vcpu *v);
int vcpu_init_fpu(struct vcpu *v);
void vcpu_destroy_fpu(struct vcpu *v);
diff --git a/xen/arch/x86/include/asm/xstate.h b/xen/arch/x86/include/asm/xstate.h
index c23308cbe2..a906df11ad 100644
--- a/xen/arch/x86/include/asm/xstate.h
+++ b/xen/arch/x86/include/asm/xstate.h
@@ -130,46 +130,43 @@ xsave_area_compressed(const struct xsave_struct *xsave_area)
}
/*
- * Fetch a pointer to a vCPU's XSAVE area
+ * Fetch a pointer to a vCPU's XSAVE area.
*
- * TL;DR: If v == current, the mapping is guaranteed to already exist.
+ * Guest vCPUs' areas live on the domain heap, mapped once for the life
+ * of the vCPU at a fixed spot in the owning vCPU's state window. For a
+ * vCPU running here on its own page tables, and for an idle vCPU, the
+ * "map" is a pointer fetch.
*
- * Despite the name, this macro might not actually map anything. The only case
- * in which a mutation of page tables is strictly required is when ASI==on &&
- * v!=current. For everything else the mapping already exists and needs not
- * be created nor destroyed.
+ * Otherwise -- a paused, never-started or lazily held vCPU -- the area
+ * is genuinely mapped, and mapping the full, possibly multi-page area
+ * can fail: VCPU_MAP_XSAVE_AREA() may return NULL, and callers must
+ * handle that. The _PREFIX variant below cannot fail.
*
- * +-----------------+--------------+
- * | v == current | v != current |
- * +--------------+-----------------+--------------+
- * | ASI enabled | per-vCPU fixmap | actual map |
- * +--------------+-----------------+--------------+
- * | ASI disabled | directmap |
- * +--------------+--------------------------------+
- *
- * There MUST NOT be outstanding maps of XSAVE areas of the non-current vCPU
- * at the point of context switch. Otherwise, the unmap operation will
- * misbehave.
- *
- * TODO: Expand the macro to the ASI cases after infra to do so is in place.
+ * There MUST NOT be outstanding maps of another vCPU's XSAVE area at
+ * the point of context switch: such a mapping is only valid on the
+ * pCPU that created it, and only until the next reschedule.
*
* @param v Owner of the XSAVE area
*/
-#define VCPU_MAP_XSAVE_AREA(v) ((v)->arch.xsave_area)
+struct xsave_struct *vcpu_map_xsave_area(const struct vcpu *v);
+void vcpu_unmap_xsave_area(const struct vcpu *v, const void *area);
+struct xsave_struct *vcpu_map_xsave_area_prefix(const struct vcpu *v);
+void vcpu_unmap_xsave_area_prefix(const struct vcpu *v, const void *area);
+struct xsave_struct *vcpu_xsave_area_ctxt_switch(const struct vcpu *v);
+
+#define VCPU_MAP_XSAVE_AREA(v) vcpu_map_xsave_area(v)
/*
- * Drops the mapping of a vCPU's XSAVE area and nullifies its pointer on exit
- *
- * See VCPU_MAP_XSAVE_AREA() for additional information on the persistence of
- * these mappings. This macro only tears down the mappings in the ASI=on &&
- * v!=current case.
- *
- * TODO: Expand the macro to the ASI cases after infra to do so is in place.
+ * Drops the mapping of a vCPU's XSAVE area and nullifies the local
+ * pointer on exit.
*
* @param v Owner of the XSAVE area
* @param x XSAVE blob of v
*/
-#define VCPU_UNMAP_XSAVE_AREA(v, x) do { (void)(v); (x) = NULL; } while (0)
+#define VCPU_UNMAP_XSAVE_AREA(v, x) do { \
+ vcpu_unmap_xsave_area(v, x); \
+ (x) = NULL; \
+} while ( false )
/*
* Variant of VCPU_MAP_XSAVE_AREA() for accessors which touch nothing
@@ -177,7 +174,10 @@ xsave_area_compressed(const struct xsave_struct *xsave_area)
* XSTATE_AREA_MIN_SIZE bytes). That prefix always fits within the
* area's first page, so this variant cannot fail.
*/
-#define VCPU_MAP_XSAVE_AREA_PREFIX(v) VCPU_MAP_XSAVE_AREA(v)
-#define VCPU_UNMAP_XSAVE_AREA_PREFIX(v, x) VCPU_UNMAP_XSAVE_AREA(v, x)
+#define VCPU_MAP_XSAVE_AREA_PREFIX(v) vcpu_map_xsave_area_prefix(v)
+#define VCPU_UNMAP_XSAVE_AREA_PREFIX(v, x) do { \
+ vcpu_unmap_xsave_area_prefix(v, x); \
+ (x) = NULL; \
+} while ( false )
#endif /* __ASM_XSTATE_H */
diff --git a/xen/arch/x86/xstate.c b/xen/arch/x86/xstate.c
index 2dd1699aba..7543edc160 100644
--- a/xen/arch/x86/xstate.c
+++ b/xen/arch/x86/xstate.c
@@ -5,6 +5,7 @@
*
*/
+#include <xen/domain_page.h>
#include <xen/param.h>
#include <xen/percpu.h>
#include <xen/sched.h>
@@ -14,6 +15,7 @@
#include <asm/cpu-policy.h>
#include <asm/current.h>
#include <asm/i387.h>
+#include <asm/mm.h>
#include <asm/msr.h>
#include <asm/processor.h>
#include <asm/xstate.h>
@@ -555,9 +557,61 @@ int xstate_alloc_save_area(struct vcpu *v)
/* XSAVE/XRSTOR requires the save area be 64-byte-boundary aligned. */
BUILD_BUG_ON(__alignof(*save_area) < 64);
- save_area = _xvzalloc(size, __alignof(*save_area));
- if ( save_area == NULL )
- return -ENOMEM;
+
+ if ( !is_idle_vcpu(v) )
+ {
+ /*
+ * The pages are the vCPU's own (no _PAGE_AVAIL0): the area teardown
+ * leaves them alone, and xstate_free_save_area() frees them.
+ */
+ unsigned int nr = PFN_UP(size), i;
+ mfn_t mfns[VCPU_STATE_XSAVE_PAGES];
+
+ /* xstate_init() bounds the host size against the reservation. */
+ BUG_ON(nr > VCPU_STATE_XSAVE_PAGES);
+
+ for ( i = 0; i < nr; i++ )
+ {
+ struct page_info *pg = alloc_domheap_page(v->domain,
+ MEMF_no_owner);
+
+ if ( !pg )
+ {
+ while ( i-- )
+ {
+ free_domheap_page(v->arch.xsave_pg[i]);
+ v->arch.xsave_pg[i] = NULL;
+ }
+ return -ENOMEM;
+ }
+
+ v->arch.xsave_pg[i] = pg;
+ mfns[i] = page_to_mfn(pg);
+ clear_domain_page(mfns[i]);
+ }
+ v->arch.xsave_pgcnt = nr;
+
+ populate_perdomain_mapping(v, VCPU_STATE_ADDR(v, VCPU_STATE_IDX_XSAVE),
+ mfns, nr, __PAGE_HYPERVISOR_RW);
+
+ /*
+ * The window VA is not reachable from the creating context (v is
+ * still being built); seed the initial register values through a
+ * transient mapping of the first page instead.
+ */
+ save_area = map_domain_page(mfns[0]);
+ }
+ else
+ {
+ /*
+ * Idle vCPUs' areas must be reachable from every context: they are
+ * scratch for accesses made with a guest's page tables loaded
+ * (read_bndcfgu(), xstate_set_init()).
+ */
+ save_area = _xvzalloc(size, __alignof(*save_area));
+ if ( save_area == NULL )
+ return -ENOMEM;
+ }
/*
* We're creating a vCPU, so conceptually we should be choosing the
@@ -578,7 +632,15 @@ int xstate_alloc_save_area(struct vcpu *v)
save_area->fpu_sse.ftw = FXSAVE_FTW_RESET;
save_area->fpu_sse.mxcsr = MXCSR_DEFAULT;
- v->arch.xsave_area = save_area;
+ if ( !is_idle_vcpu(v) )
+ {
+ unmap_domain_page(save_area);
+ v->arch.xsave_area =
+ (void *)VCPU_STATE_ADDR(v, VCPU_STATE_IDX_XSAVE);
+ }
+ else
+ v->arch.xsave_area = save_area;
+
v->arch.xcr0 = 0;
v->arch.xcr0_accum = 0;
@@ -587,7 +649,103 @@ int xstate_alloc_save_area(struct vcpu *v)
void xstate_free_save_area(struct vcpu *v)
{
- XVFREE(v->arch.xsave_area);
+ if ( !is_idle_vcpu(v) )
+ {
+ unsigned int i, nr = v->arch.xsave_pgcnt;
+
+ if ( nr )
+ {
+ unmap_vcpu_state(v, VCPU_STATE_IDX_XSAVE, nr);
+ for ( i = 0; i < nr; i++ )
+ {
+ free_domheap_page(v->arch.xsave_pg[i]);
+ v->arch.xsave_pg[i] = NULL;
+ }
+ v->arch.xsave_pgcnt = 0;
+ }
+
+ v->arch.xsave_area = NULL;
+ }
+ else
+ XVFREE(v->arch.xsave_area);
+}
+
+struct xsave_struct *vcpu_map_xsave_area(const struct vcpu *v)
+{
+ mfn_t mfns[VCPU_STATE_XSAVE_PAGES];
+ unsigned int i;
+
+ if ( is_idle_vcpu(v) )
+ return v->arch.xsave_area;
+
+ /*
+ * The window VA is valid while v's own page tables are loaded, but that
+ * alone is not enough: while this CPU holds v's state lazily, an IPI
+ * syncing that state for another CPU can switch v's page tables away
+ * at any time. Hence the window only for v running here on its own
+ * page tables, which current's are not inside the EFI runtime bracket
+ * or after a shadow-mode update_cr3() failure.
+ */
+ if ( v == current && this_cpu(pgtable_vcpu) == v )
+ return v->arch.xsave_area;
+
+ for ( i = 0; i < v->arch.xsave_pgcnt; i++ )
+ mfns[i] = page_to_mfn(v->arch.xsave_pg[i]);
+
+ return map_domain_pages(mfns, v->arch.xsave_pgcnt);
+}
+
+void vcpu_unmap_xsave_area(const struct vcpu *v, const void *area)
+{
+ if ( !area || is_idle_vcpu(v) ||
+ area == (const void *)v->arch.xsave_area )
+ return;
+
+ /*
+ * A single page comes from the single-page mapcache, whose
+ * domain-wide form defers its flushes; unmap_domain_pages() flushes a
+ * multipage mapping itself.
+ */
+ if ( v->arch.xsave_pgcnt == 1 )
+ unmap_vcpu_state_page(area);
+ else
+ unmap_domain_pages(area, v->arch.xsave_pgcnt);
+}
+
+struct xsave_struct *vcpu_map_xsave_area_prefix(const struct vcpu *v)
+{
+ /* The legacy region and header always fit the area's first page. */
+ BUILD_BUG_ON(sizeof(struct xsave_struct) > PAGE_SIZE);
+
+ /* The window VA under vcpu_map_xsave_area()'s conditions. */
+ if ( is_idle_vcpu(v) ||
+ (v == current && this_cpu(pgtable_vcpu) == v) )
+ return v->arch.xsave_area;
+
+ return map_domain_page(page_to_mfn(v->arch.xsave_pg[0]));
+}
+
+void vcpu_unmap_xsave_area_prefix(const struct vcpu *v, const void *area)
+{
+ if ( !area || is_idle_vcpu(v) ||
+ area == (const void *)v->arch.xsave_area )
+ return;
+
+ unmap_vcpu_state_page(area);
+}
+
+/*
+ * The XSAVE area of @v for __context_switch() saving it: `current` has
+ * moved on to the incoming vCPU, but @v's page tables are still loaded, and
+ * interrupts are off, so the window VA serves. A transient mapping is not
+ * an option from there: it would sync the lazy state and re-enter
+ * __context_switch().
+ */
+struct xsave_struct *vcpu_xsave_area_ctxt_switch(const struct vcpu *v)
+{
+ BUG_ON(is_idle_vcpu(v) || this_cpu(pgtable_vcpu) != v);
+
+ return v->arch.xsave_area;
}
static bool valid_xcr0(uint64_t xcr0)
@@ -934,6 +1092,10 @@ void xstate_init(struct cpuinfo_x86 *c)
xsave_cntxt_size = cpuid_count_ebx(0xd, 0);
printk("xstate: size: %#x and states: %#"PRIx64"\n",
xsave_cntxt_size, xfeature_mask);
+
+ if ( PFN_UP(xsave_cntxt_size) > VCPU_STATE_XSAVE_PAGES )
+ panic("XSAVE area needs %lu pages, vCPU state window reserves %u\n",
+ PFN_UP(xsave_cntxt_size), VCPU_STATE_XSAVE_PAGES);
}
else
{
--
2.56.0
^ permalink raw reply related [flat|nested] 52+ messages in thread
* Re: [PATCH v6 01/44] x86/mm: allocate the per-domain L3 in arch_domain_create()
2026-11-03 22:13 ` [PATCH v6 01/44] x86/mm: allocate the per-domain L3 in arch_domain_create() George Dunlap
@ 2026-11-04 10:46 ` Jan Beulich
0 siblings, 0 replies; 52+ messages in thread
From: Jan Beulich @ 2026-11-04 10:46 UTC (permalink / raw)
To: George Dunlap; +Cc: Andrew Cooper, Roger Pau Monné, Teddy Astie, xen-devel
On 03.11.2026 23:13, George Dunlap wrote:
> init_xen_l4_slots() relies on d->arch.perdomain_l3_pg being non-NULL,
> or it will fill in a garbage value for slot 260. Currently, this
> is allocated only implicitly, as a side effect of calling
> create_perdomain_mapping(). But depending on the specific
> circumstances, this may not happen until several hypercalls after the
> domain is created. This is not robust: callers need to keep track of
> when exactly certain structures happen to be created to ensure that
> init_xen_l4_slots() operates properly.
>
> Since fcf91ce9de66 ("x86/pv: don't populate the GDT/LDT L3 slot at
> domain creation"), a PV domain gets its L3 with its first vCPU
> (pv_create_gdt_ldt_l1tab()). Only mapcache_domain_init() makes it
> earlier, and release builds skip that unless memory hotplug is enabled
> or the host's memory reaches past the part of the directmap PV guests
> can see. So in release builds on most hosts, an L4 that is promoted
> before XEN_DOMCTL_max_vcpus has created the domain's vCPUs gets a
> slot 260 entry made from a NULL page (0x803e900000000063). libxl is
> unaffected: it creates the vCPUs before it builds or restores a domain.
>
> Allocate it explicitly in arch_domain_create() instead. Introduce
> allocate_perdomain_l3(), which will allocate a page with the
> appropriate parameters and zero it. create_perdomain_mapping() now
> requires the L3 to exist, and BUG()s if it does not.
>
> We return a pg rather than setting d->arch.perdomain_l3_pg directly,
> because later we'll use the same function to initialize the per-vCPU
> perdomain L3.
>
> This removes the need to allocate it semi-implicitly (calling
> create_perdomain_mapping() with nr == 0) in hvm_domain_initialise().
>
> Fixes: fcf91ce9de66 ("x86/pv: don't populate the GDT/LDT L3 slot at domain creation")
> Assisted-by: Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH v6 25/44] x86/mm: introduce a per-vCPU state window in the perdomain area
2026-11-03 22:13 ` [PATCH v6 25/44] x86/mm: introduce a per-vCPU state window in the perdomain area George Dunlap
@ 2026-11-18 11:51 ` Jan Beulich
2026-11-19 12:15 ` George Dunlap
0 siblings, 1 reply; 52+ messages in thread
From: Jan Beulich @ 2026-11-18 11:51 UTC (permalink / raw)
To: George Dunlap; +Cc: Andrew Cooper, Roger Pau Monné, Teddy Astie, xen-devel
On 03.11.2026 23:13, George Dunlap wrote:
> --- a/xen/arch/x86/include/asm/config.h
> +++ b/xen/arch/x86/include/asm/config.h
> @@ -158,7 +158,7 @@
> /* Slot 260: per-domain mappings (including map cache). */
> #define PERDOMAIN_VIRT_START (PML4_ADDR(260))
> #define PERDOMAIN_SLOT_MBYTES (PML4_ENTRY_BYTES >> (20 + PAGETABLE_ORDER))
> -#define PERDOMAIN_SLOTS 3
> +#define PERDOMAIN_SLOTS 4
> #define PERDOMAIN_VIRT_SLOT(s) (PERDOMAIN_VIRT_START + (s) * \
> (PERDOMAIN_SLOT_MBYTES << 20))
> /* Slot 4: mirror of per-domain mappings (for compat xlat area accesses). */
> @@ -293,6 +293,25 @@ extern unsigned long xen_phys_start;
> #define ARG_XLAT_START(v) \
> (ARG_XLAT_VIRT_START + ((v)->vcpu_id << ARG_XLAT_VA_SHIFT))
>
> +/*
> + * Per-vCPU state window. The fourth per-domain-mapping sub-area: a small
> + * per-vCPU stride of page slots holding long-lived per-vCPU structures kept
> + * out of always-mapped memory, each mapped once for the life of the vCPU at
> + * a fixed index within the stride.
> + */
> +#define VCPU_STATE_SLOT 3
> +#define VCPU_STATE_VIRT_START PERDOMAIN_VIRT_SLOT(VCPU_STATE_SLOT)
> +#define VCPU_STATE_ORDER 3
As long as this remains <= GDT_LDT_VCPU_SHIFT, it won't further constrain
MAX_VIRT_CPUS (primarily relevant to PV, when right here you cover HVM
stuff only, but anyway). Nevertheless I wonder whether ...
> +#define VCPU_STATE_NR_PAGES (1U << VCPU_STATE_ORDER)
> +#define VCPU_STATE_VA_SHIFT (VCPU_STATE_ORDER + PAGE_SHIFT)
> +#define VCPU_STATE_START(v) \
> + (VCPU_STATE_VIRT_START + ((v)->vcpu_id << VCPU_STATE_VA_SHIFT))
> +#define VCPU_STATE_ADDR(v, idx) \
> + (VCPU_STATE_START(v) + (idx) * PAGE_SIZE)
> +#define VCPU_STATE_IDX_VMCB 0
> +#define VCPU_STATE_IDX_N2VMCB 1
> +#define VCPU_STATE_IDX_VMX_MSR 2
... we wouldn't better conserve on space use from the very start: VMX_MSR
could re-use either of the other two slots.
Jan
^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH v6 28/44] x86/svm: drop svm_vmcb_dump()'s paddr
2026-11-03 22:13 ` [PATCH v6 28/44] x86/svm: drop svm_vmcb_dump()'s paddr George Dunlap
@ 2026-11-18 11:54 ` Jan Beulich
0 siblings, 0 replies; 52+ messages in thread
From: Jan Beulich @ 2026-11-18 11:54 UTC (permalink / raw)
To: George Dunlap
Cc: Ross Lagerwall, Andrew Cooper, Roger Pau Monné,
Jason Andryuk, Teddy Astie, xen-devel
On 03.11.2026 23:13, George Dunlap wrote:
> svm_vmcb_dump() prints virt_to_maddr() of the VMCB, which is right only
> while VMCBs are reached through the directmap. Print the virtual
> address alone.
But what use is the virtual address if it's not a permanent one? The PA
is the only thing uniquely identifying a VMCB then.
Jan
^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH v6 15/44] xen/gunzip: process pending softirqs once per deflate block
2026-11-03 22:13 ` [PATCH v6 15/44] xen/gunzip: process pending softirqs once per deflate block George Dunlap
@ 2026-11-18 11:56 ` Jan Beulich
0 siblings, 0 replies; 52+ messages in thread
From: Jan Beulich @ 2026-11-18 11:56 UTC (permalink / raw)
To: George Dunlap
Cc: Andrew Cooper, Anthony PERARD, Michal Orzel, Julien Grall,
Roger Pau Monné, Stefano Stabellini, xen-devel
On 03.11.2026 23:13, George Dunlap wrote:
> Nothing in the gzip decompressor lets the CPU run softirqs. The NMI
> watchdog treats a CPU as stuck once its timer softirq has not run for
> the watchdog's timeout, 5s by default, so a decompression that runs
> that long panics the host.
>
> Decompression is normally quick, but it need not be. Every compressed
> deflate block builds its Huffman tables with xmalloc() and frees them
> with xfree(), and those calls reach the xenheap whenever a request is a
> page or more, or a free empties an xmalloc pool page. A subsequent
> change can make every xenheap free end in a TLB flush on all online
> CPUs. With it in effect, decompressing the PV dom0 kernel and reading
> its ELF headers on a 480-CPU host took 5.2s instead of 0.5s; with
> cpufreq=xen:performance max_cstate=1,0 dom0_nodes=0,relaxed added to
> the command line as well, the watchdog fired before decompression had
> finished, on each of eight boots.
>
> Process pending softirqs after each deflate block, as libelf does for
> each ELF note since c5089fae0f6f ("x86: avoid tripping watchdog when
> constructing dom0"), and as the PV and PVH dom0 builders do in their
> long loops. Every caller of perform_gunzip() runs during domain
> construction at boot, on the boot CPU with interrupts enabled and no
> lock held: x86's PV and PVH dom0 builders, which call
> process_pending_softirqs() themselves, and kernel_decompress(), which
> Arm uses for dom0 and for dom0less domains.
>
> Assisted-by: Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
Acked-by: Jan Beulich <jbeulich@suse.com>
^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH v6 22/44] xen/xmalloc: return an emptied region after dropping the pool lock
2026-11-03 22:13 ` [PATCH v6 22/44] xen/xmalloc: return an emptied region after dropping the pool lock George Dunlap
@ 2026-11-18 12:05 ` Jan Beulich
2026-11-19 14:11 ` George Dunlap
0 siblings, 1 reply; 52+ messages in thread
From: Jan Beulich @ 2026-11-18 12:05 UTC (permalink / raw)
To: George Dunlap
Cc: Andrew Cooper, Anthony PERARD, Michal Orzel, Julien Grall,
Roger Pau Monné, Stefano Stabellini, xen-devel
On 03.11.2026 23:13, George Dunlap wrote:
> When a free leaves a whole region of an xmalloc pool empty,
> xmem_pool_free() hands the region back through pool->put_mem() with
> pool->lock still held. For the xenheap pool that is
> free_xenheap_pages(), which takes the heap lock inside the pool lock
> and runs arch_xenheap_pages_free(). On x86 with a sparse directmap
> view configured, the latter takes the page out of the sparse view,
> at the cost of one to three rounds of TLB-flush IPIs to every CPU,
> while every other CPU's sub-page xmalloc() and xfree() waits on the
> pool lock.
>
> By then the region is on no free list and no longer counted in the
> pool,
This, I think, isn't entirely accurate, as pool->num_regions still
accounts for the region (and pool->used_size still accounts for the
bhdr). Yet at the same time the change you make also better mirrors
the ->get_mem() side, where accounting is done only after the call.
> so nothing else can reach it. Drop the lock first and return the
> region afterwards, as xmem_pool_alloc() already does when it calls
> pool->get_mem(), and say at both callbacks that they are called without
> the pool lock, so that they keep not relying on it.
>
> Assisted-by: Claude Code:claude-opus-5-5
> Signed-off-by: George Dunlap <gwd@xenproject.org>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Jan
^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH v6 25/44] x86/mm: introduce a per-vCPU state window in the perdomain area
2026-11-18 11:51 ` Jan Beulich
@ 2026-11-19 12:15 ` George Dunlap
0 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-19 12:15 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, Roger Pau Monné, Teddy Astie, xen-devel
On Wed, Nov 18, 2026 at 11:51 AM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 03.11.2026 23:13, George Dunlap wrote:
> > --- a/xen/arch/x86/include/asm/config.h
> > +++ b/xen/arch/x86/include/asm/config.h
> > @@ -158,7 +158,7 @@
> > /* Slot 260: per-domain mappings (including map cache). */
> > #define PERDOMAIN_VIRT_START (PML4_ADDR(260))
> > #define PERDOMAIN_SLOT_MBYTES (PML4_ENTRY_BYTES >> (20 + PAGETABLE_ORDER))
> > -#define PERDOMAIN_SLOTS 3
> > +#define PERDOMAIN_SLOTS 4
> > #define PERDOMAIN_VIRT_SLOT(s) (PERDOMAIN_VIRT_START + (s) * \
> > (PERDOMAIN_SLOT_MBYTES << 20))
> > /* Slot 4: mirror of per-domain mappings (for compat xlat area accesses). */
> > @@ -293,6 +293,25 @@ extern unsigned long xen_phys_start;
> > #define ARG_XLAT_START(v) \
> > (ARG_XLAT_VIRT_START + ((v)->vcpu_id << ARG_XLAT_VA_SHIFT))
> >
> > +/*
> > + * Per-vCPU state window. The fourth per-domain-mapping sub-area: a small
> > + * per-vCPU stride of page slots holding long-lived per-vCPU structures kept
> > + * out of always-mapped memory, each mapped once for the life of the vCPU at
> > + * a fixed index within the stride.
> > + */
> > +#define VCPU_STATE_SLOT 3
> > +#define VCPU_STATE_VIRT_START PERDOMAIN_VIRT_SLOT(VCPU_STATE_SLOT)
> > +#define VCPU_STATE_ORDER 3
>
> As long as this remains <= GDT_LDT_VCPU_SHIFT, it won't further constrain
> MAX_VIRT_CPUS (primarily relevant to PV, when right here you cover HVM
> stuff only, but anyway). Nevertheless I wonder whether ...
>
> > +#define VCPU_STATE_NR_PAGES (1U << VCPU_STATE_ORDER)
> > +#define VCPU_STATE_VA_SHIFT (VCPU_STATE_ORDER + PAGE_SHIFT)
> > +#define VCPU_STATE_START(v) \
> > + (VCPU_STATE_VIRT_START + ((v)->vcpu_id << VCPU_STATE_VA_SHIFT))
> > +#define VCPU_STATE_ADDR(v, idx) \
> > + (VCPU_STATE_START(v) + (idx) * PAGE_SIZE)
> > +#define VCPU_STATE_IDX_VMCB 0
> > +#define VCPU_STATE_IDX_N2VMCB 1
> > +#define VCPU_STATE_IDX_VMX_MSR 2
>
> ... we wouldn't better conserve on space use from the very start: VMX_MSR
> could re-use either of the other two slots.
Sure, we can have "AMD-specific", "Intel-specific", and "Common" slot
sections, where the first two are effectively a union.
-George
^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH v6 22/44] xen/xmalloc: return an emptied region after dropping the pool lock
2026-11-18 12:05 ` Jan Beulich
@ 2026-11-19 14:11 ` George Dunlap
0 siblings, 0 replies; 52+ messages in thread
From: George Dunlap @ 2026-11-19 14:11 UTC (permalink / raw)
To: Jan Beulich
Cc: Andrew Cooper, Anthony PERARD, Michal Orzel, Julien Grall,
Roger Pau Monné, Stefano Stabellini, xen-devel
On Wed, Nov 18, 2026 at 12:05 PM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 03.11.2026 23:13, George Dunlap wrote:
> > When a free leaves a whole region of an xmalloc pool empty,
> > xmem_pool_free() hands the region back through pool->put_mem() with
> > pool->lock still held. For the xenheap pool that is
> > free_xenheap_pages(), which takes the heap lock inside the pool lock
> > and runs arch_xenheap_pages_free(). On x86 with a sparse directmap
> > view configured, the latter takes the page out of the sparse view,
> > at the cost of one to three rounds of TLB-flush IPIs to every CPU,
> > while every other CPU's sub-page xmalloc() and xfree() waits on the
> > pool lock.
> >
> > By then the region is on no free list and no longer counted in the
> > pool,
>
> This, I think, isn't entirely accurate, as pool->num_regions still
> accounts for the region (and pool->used_size still accounts for the
> bhdr). Yet at the same time the change you make also better mirrors
> the ->get_mem() side, where accounting is done only after the call.
Ack
> > Assisted-by: Claude Code:claude-opus-5-5
> > Signed-off-by: George Dunlap <gwd@xenproject.org>
>
> Reviewed-by: Jan Beulich <jbeulich@suse.com>
Thanks,
-George
^ permalink raw reply [flat|nested] 52+ messages in thread
end of thread, other threads:[~2026-11-19 14:11 UTC | newest]
Thread overview: 52+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-03 22:13 [PATCH v6 00/44] ASI: HVM through X4 George Dunlap
2026-11-03 22:13 ` [PATCH v6 01/44] x86/mm: allocate the per-domain L3 in arch_domain_create() George Dunlap
2026-11-04 10:46 ` Jan Beulich
2026-11-03 22:13 ` [PATCH v6 02/44] x86/mm: purge unneeded destroy_perdomain_mapping() George Dunlap
2026-11-03 22:13 ` [PATCH v6 03/44] x86/mm: prepare destroy_perdomain_mapping() for per-vCPU perdomain areas George Dunlap
2026-11-03 22:13 ` [PATCH v6 04/44] x86/domain_page: drop redundant create_perdomain_mapping() call George Dunlap
2026-11-03 22:13 ` [PATCH v6 05/44] x86/mm: prepare create_perdomain_mapping() for per-vCPU perdomain areas George Dunlap
2026-11-03 22:13 ` [PATCH v6 06/44] x86/spec-ctrl: introduce Address Space Isolation command line option George Dunlap
2026-11-03 22:13 ` [PATCH v6 07/44] x86/mm: have init_xen_l4_slots() take a vCPU in place of the domain George Dunlap
2026-11-03 22:13 ` [PATCH v6 08/44] x86/hvm: introduce per-vCPU L3 page-table George Dunlap
2026-11-03 22:13 ` [PATCH v6 09/44] x86/domain_page: drop redundant is_pv_vcpu() checks George Dunlap
2026-11-03 22:13 ` [PATCH v6 10/44] x86/domain_page: prepare the mapcache for per-vCPU accounting George Dunlap
2026-11-03 22:13 ` [PATCH v6 11/44] x86/domain_page: move struct mapcache_vcpu to struct arch_vcpu George Dunlap
2026-11-03 22:13 ` [PATCH v6 12/44] x86/domain_page: don't unmap a mapcache entry that is not present George Dunlap
2026-11-03 22:13 ` [PATCH v6 13/44] x86/hvm: introduce per-vCPU mapcache for vCPU-PT domains George Dunlap
2026-11-03 22:13 ` [PATCH v6 14/44] x86/mm: let Xen's page-table walker operate on a given root George Dunlap
2026-11-03 22:13 ` [PATCH v6 15/44] xen/gunzip: process pending softirqs once per deflate block George Dunlap
2026-11-18 11:56 ` Jan Beulich
2026-11-03 22:13 ` [PATCH v6 16/44] x86/mm: add hooks for a sparse view of the directmap George Dunlap
2026-11-03 22:13 ` [PATCH v6 17/44] x86/mm: build and maintain " George Dunlap
2026-11-03 22:13 ` [PATCH v6 18/44] x86/domain_page: key the directmap fast paths on the loaded view George Dunlap
2026-11-03 22:13 ` [PATCH v6 19/44] x86/domain_page: clear a mapcache slot at its last unmap George Dunlap
2026-11-03 22:13 ` [PATCH v6 20/44] x86/mm: run guest contexts on the sparse directmap view George Dunlap
2026-11-03 22:13 ` [PATCH v6 21/44] x86/spec-ctrl: wire the sparse directmap view into the asi= option George Dunlap
2026-11-03 22:13 ` [PATCH v6 22/44] xen/xmalloc: return an emptied region after dropping the pool lock George Dunlap
2026-11-18 12:05 ` Jan Beulich
2026-11-19 14:11 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 23/44] x86/mm: split a sparse directmap superpage with a page being freed George Dunlap
2026-11-03 22:13 ` [PATCH v6 24/44] x86/mm: introduce populate_perdomain_mapping() George Dunlap
2026-11-03 22:13 ` [PATCH v6 25/44] x86/mm: introduce a per-vCPU state window in the perdomain area George Dunlap
2026-11-18 11:51 ` Jan Beulich
2026-11-19 12:15 ` George Dunlap
2026-11-03 22:13 ` [PATCH v6 26/44] x86/svm: introduce SVM_MAP_VMCB() and SVM_UNMAP_VMCB() George Dunlap
2026-11-03 22:13 ` [PATCH v6 27/44] x86/svm: restore guest debug registers in svm_do_resume() George Dunlap
2026-11-03 22:13 ` [PATCH v6 28/44] x86/svm: drop svm_vmcb_dump()'s paddr George Dunlap
2026-11-18 11:54 ` Jan Beulich
2026-11-03 22:13 ` [PATCH v6 29/44] x86/svm: allocate VMCBs from the domain heap behind the vCPU state window George Dunlap
2026-11-03 22:13 ` [PATCH v6 30/44] x86/vmx: move the guest MSR load/save list off always-mapped memory George Dunlap
2026-11-03 22:13 ` [PATCH v6 31/44] x86: access a vCPU's saved register frame through map/unmap helpers George Dunlap
2026-11-03 22:13 ` [PATCH v6 32/44] x86: move the saved register frame behind the vCPU state window George Dunlap
2026-11-03 22:13 ` [PATCH v6 33/44] x86/hvm: move the emulator's operand caches " George Dunlap
2026-11-03 22:13 ` [PATCH v6 34/44] x86/hvm: clear the vCPU's copy of an emulated I/O request once consumed George Dunlap
2026-11-03 22:13 ` [PATCH v6 35/44] x86/domain_page: introduce map_domain_pages() George Dunlap
2026-11-03 22:13 ` [PATCH v6 36/44] x86/shadow: use map_domain_pages() for cross-page emulated writes George Dunlap
2026-11-03 22:13 ` [PATCH v6 37/44] x86/hvm: use map_domain_pages() for multi-frame emulated accesses George Dunlap
2026-11-03 22:14 ` [PATCH v6 38/44] x86/emulator: Refactor FXSAVE_AREA to use wrappers George Dunlap
2026-11-03 22:14 ` [PATCH v6 39/44] x86/mpx: Map/unmap xsave area in read_bndcfgu() George Dunlap
2026-11-03 22:14 ` [PATCH v6 40/44] x86/fpu: Pass explicit xsave areas to fpu_(f)xsave() George Dunlap
2026-11-03 22:14 ` [PATCH v6 41/44] x86/fpu: Pass explicit xsave areas to fpu_(f)xrstor() George Dunlap
2026-11-03 22:14 ` [PATCH v6 42/44] x86/xstate: classify xsave-area seam accesses by failure mode George Dunlap
2026-11-03 22:14 ` [PATCH v6 43/44] x86/domain: restore FPU state after switching to the new page tables George Dunlap
2026-11-03 22:14 ` [PATCH v6 44/44] x86/xstate: move guest XSAVE areas behind the vCPU state window George Dunlap
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.