* [PATCH 0/4] target/i386: emulate LAM in TCG
@ 2026-11-10 13:01 Tim
2026-11-10 13:01 ` [PATCH 1/4] target/i386: mask the PML5 table address taken from CR3 Tim
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Tim @ 2026-11-10 13:01 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, richard.henderson, zhao1.liu, Tim
This series emulates LAM in TCG for x86_64 system mode and reports it
in "-cpu max". linux-user support will follow in a separate series.
Patch 1 keeps the CR3 LAM bits out of the PML5 table address. Patch 2
makes the large-page check in tlb_flush_range_locked() compare under
the flush mask, as agreed in [2]. Patch 3 is the LAM emulation, and
patch 4 adds a system-mode test.
The design follows the RFC thread [1]:
- On a TLB miss, get_physical_address() masks the address and runs
the canonical check and the page walk on the result. The entry is
installed under the tagged address, so each tag of a page has its
own entry. INVLPG and INVLPGA flush every tagged alias with
tlb_flush_page_bits_by_mmuidx(), and a change of CR4.LAM_SUP
flushes the TLB.
- Instruction fetches are not masked. If masking changes the address,
an entry filled by a data access is not executable, and an entry filled
by a fetch is not readable or writable.
- Exceptions do not report the tagged address, which answers
Richard's question [3]. On #PF, CR2 receives the masked address
(SDM Vol. 3A, section 4.4.3), and #GP reports no address.
- gdbstub, monitor, and qemu_plugin_translate_vaddr() translate a
tagged address as a data access does.
Known gaps:
- Data breakpoints on an untagged address do not fire for accesses
through a tagged alias.
- With LAM_U48 and 5-level paging, gdbstub, monitor, and
qemu_plugin_translate_vaddr() translate an instruction address at
or above 2^47 as a data address, not as a fetch does.
[1] https://lists.nongnu.org/archive/html/qemu-devel/2026-10/msg01840.html
[2] https://lists.nongnu.org/archive/html/qemu-devel/2026-11/msg01599.html
[3] https://lists.nongnu.org/archive/html/qemu-devel/2026-10/msg05620.html
Tim (4):
target/i386: mask the PML5 table address taken from CR3
accel/tcg: compare large pages under the flush mask
target/i386: emulate LAM in TCG
tests/tcg/x86_64: add a LAM system test
accel/tcg/cputlb.c | 4 +-
target/i386/cpu.c | 2 +-
target/i386/cpu.h | 46 ++
target/i386/helper.c | 8 +-
target/i386/tcg/system/excp_helper.c | 33 +-
target/i386/tcg/system/misc_helper.c | 24 +-
target/i386/tcg/system/svm_helper.c | 3 +-
tests/tcg/x86_64/system/boot.S | 14 +
tests/tcg/x86_64/system/lam.c | 797 +++++++++++++++++++++++++++
tests/tcg/x86_64/system/meson.build | 15 +
10 files changed, 934 insertions(+), 12 deletions(-)
create mode 100644 tests/tcg/x86_64/system/lam.c
--
2.47.3
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/4] target/i386: mask the PML5 table address taken from CR3
2026-11-10 13:01 [PATCH 0/4] target/i386: emulate LAM in TCG Tim
@ 2026-11-10 13:01 ` Tim
2026-11-10 13:01 ` [PATCH 2/4] accel/tcg: compare large pages under the flush mask Tim
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Tim @ 2026-11-10 13:01 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, richard.henderson, zhao1.liu, Tim
The 5-level page walks in mmu_translate() and
x86_cpu_translate_for_debug() take the PML5 table address from CR3
with ~0xfff, which keeps CR3 bits 63:52. The 4-level walks already
use PG_ADDRESS_MASK. Use PG_ADDRESS_MASK for the 5-level walks as
well, so that CR3 control bits above the physical address width, such
as the LAM bits 61 and 62, do not become part of the table address.
Signed-off-by: Tim <mdspan.github@gmail.com>
---
target/i386/helper.c | 2 +-
target/i386/tcg/system/excp_helper.c | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/target/i386/helper.c b/target/i386/helper.c
index cb25018..723e87f 100644
--- a/target/i386/helper.c
+++ b/target/i386/helper.c
@@ -297,7 +297,7 @@ bool x86_cpu_translate_for_debug(CPUState *cs, vaddr addr,
}
if (la57) {
- pml5e_addr = ((env->cr[3] & ~0xfff) +
+ pml5e_addr = ((env->cr[3] & PG_ADDRESS_MASK) +
(((addr >> 48) & 0x1ff) << 3)) & a20_mask;
pml5e = x86_ldq_phys(cs, pml5e_addr);
if (!(pml5e & PG_PRESENT_MASK)) {
diff --git a/target/i386/tcg/system/excp_helper.c b/target/i386/tcg/system/excp_helper.c
index 44be98a..6c48ccb 100644
--- a/target/i386/tcg/system/excp_helper.c
+++ b/target/i386/tcg/system/excp_helper.c
@@ -173,7 +173,8 @@ static bool mmu_translate(CPUX86State *env, const TranslateParams *in,
/*
* Page table level 5
*/
- pte_addr = (in->cr3 & ~0xfff) + (((addr >> 48) & 0x1ff) << 3);
+ pte_addr = (in->cr3 & PG_ADDRESS_MASK) +
+ (((addr >> 48) & 0x1ff) << 3);
if (!ptw_translate(&pte_trans, pte_addr)) {
return false;
}
--
2.47.3
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/4] accel/tcg: compare large pages under the flush mask
2026-11-10 13:01 [PATCH 0/4] target/i386: emulate LAM in TCG Tim
2026-11-10 13:01 ` [PATCH 1/4] target/i386: mask the PML5 table address taken from CR3 Tim
@ 2026-11-10 13:01 ` Tim
2026-11-10 13:01 ` [PATCH 3/4] target/i386: emulate LAM in TCG Tim
2026-11-10 13:01 ` [PATCH 4/4] tests/tcg/x86_64: add a LAM system test Tim
3 siblings, 0 replies; 5+ messages in thread
From: Tim @ 2026-11-10 13:01 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, richard.henderson, zhao1.liu, Tim
tlb_flush_range_locked() flushes entries whose address matches the
range under @mask, the low @bits of the address. Its large page
check, however, compares the range end with the recorded large page
in all bits. If the large page was entered under an address that
differs from the flushed range above @bits, the check fails, and the
large page is not flushed.
Compare under @mask as well. This matters for the x86 LAM emulation
in the next patch, which installs TLB entries under tagged addresses
and flushes every tagged alias on INVLPG of the untagged address with
tlb_flush_page_bits_by_mmuidx().
Signed-off-by: Tim <mdspan.github@gmail.com>
---
accel/tcg/cputlb.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c
index dc83f3b..ad2a5aa 100644
--- a/accel/tcg/cputlb.c
+++ b/accel/tcg/cputlb.c
@@ -691,8 +691,10 @@ static void tlb_flush_range_locked(CPUState *cpu, int midx,
* Check if we need to flush due to large pages.
* Because large_page_mask contains all 1's from the msb,
* we only need to test the end of the range.
+ * Compare under @mask, ignoring bits above @bits.
*/
- if (((addr + len - 1) & d->large_page_mask) == d->large_page_addr) {
+ if (((addr + len - 1) & d->large_page_mask & mask) ==
+ (d->large_page_addr & mask)) {
tlb_debug("forcing full flush midx %d ("
"%016" VADDR_PRIx "/%016" VADDR_PRIx ")\n",
midx, d->large_page_addr, d->large_page_mask);
--
2.47.3
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 3/4] target/i386: emulate LAM in TCG
2026-11-10 13:01 [PATCH 0/4] target/i386: emulate LAM in TCG Tim
2026-11-10 13:01 ` [PATCH 1/4] target/i386: mask the PML5 table address taken from CR3 Tim
2026-11-10 13:01 ` [PATCH 2/4] accel/tcg: compare large pages under the flush mask Tim
@ 2026-11-10 13:01 ` Tim
2026-11-10 13:01 ` [PATCH 4/4] tests/tcg/x86_64: add a LAM system test Tim
3 siblings, 0 replies; 5+ messages in thread
From: Tim @ 2026-11-10 13:01 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, richard.henderson, zhao1.liu, Tim
Linear Address Masking (LAM) lets software keep metadata in bits
62:57 or 62:48 of 64-bit data addresses. CR3.LAM_U57 and CR3.LAM_U48
enable it for user addresses, and CR4.LAM_SUP for supervisor
addresses. Implement it in TCG and add it to TCG_7_1_EAX_FEATURES.
get_physical_address() masks the address and runs the canonical check
and the page walk on the result. Since masking keeps bit 63, this is
the canonical check that the SDM specifies for LAM. CR2 receives the
masked address on #PF. The TLB entry is installed under the tagged
address, which is the same way target/arm does it for top-byte-ignore.
Instruction fetches are not masked, so if masking changes the address,
an entry filled by a data access is not executable and one filled by
a fetch is not readable or writable.
Since each tag of a page has its own entry, INVLPG and INVLPGA flush
every tagged alias with tlb_flush_page_bits_by_mmuidx(), and a change
of CR4.LAM_SUP flushes the TLB.
MOV to CR3 and VMRUN accept the CR3 LAM bits if CPUID reports LAM.
Masking checks CPUID too, since RSM and gdbstub load CR3 without
checking reserved bits.
x86_cpu_translate_for_debug() masks addresses as for a data access,
for gdbstub, monitor, and qemu_plugin_translate_vaddr(). With LAM_U48
and 5-level paging, it therefore translates an instruction address at
or above 2^47 differently from a fetch.
Signed-off-by: Tim <mdspan.github@gmail.com>
---
target/i386/cpu.c | 2 +-
target/i386/cpu.h | 46 ++++++++++++++++++++++++++++
target/i386/helper.c | 6 +++-
target/i386/tcg/system/excp_helper.c | 30 ++++++++++++++++--
target/i386/tcg/system/misc_helper.c | 24 +++++++++++++--
target/i386/tcg/system/svm_helper.c | 3 +-
6 files changed, 102 insertions(+), 9 deletions(-)
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index eab0d5a..ad5d90d 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1006,7 +1006,7 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t vendor1,
#define TCG_7_0_EDX_FEATURES (CPUID_7_0_EDX_FSRM | CPUID_7_0_EDX_KERNEL_FEATURES)
#define TCG_7_1_EAX_FEATURES (CPUID_7_1_EAX_FZRM | CPUID_7_1_EAX_FSRS | \
- CPUID_7_1_EAX_FSRC | CPUID_7_1_EAX_CMPCCXADD)
+ CPUID_7_1_EAX_FSRC | CPUID_7_1_EAX_CMPCCXADD | CPUID_7_1_EAX_LAM)
#define TCG_7_1_ECX_FEATURES 0
#define TCG_7_1_EDX_FEATURES CPUID_7_1_EDX_APXF
#define TCG_7_2_EDX_FEATURES 0
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 3945e90..f196d93 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -260,6 +260,9 @@ typedef enum X86Seg {
#define CR0_CD_MASK (1U << 30)
#define CR0_PG_MASK (1U << 31)
+#define CR3_LAM_U57_MASK (1ULL << 61)
+#define CR3_LAM_U48_MASK (1ULL << 62)
+
#define CR4_VME_MASK (1U << 0)
#define CR4_PVI_MASK (1U << 1)
#define CR4_TSD_MASK (1U << 2)
@@ -3111,6 +3114,49 @@ static inline uint64_t cr4_reserved_bits(CPUX86State *env)
return reserved_bits;
}
+/* Reserved CR3 bits in IA-32e mode */
+static inline uint64_t cr3_reserved_bits(CPUX86State *env)
+{
+ uint64_t reserved_bits = ~0ULL << env_archcpu(env)->phys_bits;
+
+ if (env->features[FEAT_7_1_EAX] & CPUID_7_1_EAX_LAM) {
+ reserved_bits &= ~(CR3_LAM_U57_MASK | CR3_LAM_U48_MASK);
+ }
+ return reserved_bits;
+}
+
+/* LAM width of @addr: 57 or 48, or 64 without LAM. IA-32e mode only. */
+static inline unsigned x86_lam_addr_bits(CPUX86State *env, uint64_t addr)
+{
+ if (!(env->features[FEAT_7_1_EAX] & CPUID_7_1_EAX_LAM)) {
+ return 64;
+ }
+ if (addr >> 63) {
+ if (!(env->cr[4] & CR4_LAM_SUP_MASK)) {
+ return 64;
+ }
+ return env->cr[4] & CR4_LA57_MASK ? 57 : 48;
+ }
+ if (env->cr[3] & CR3_LAM_U57_MASK) {
+ return 57;
+ }
+ if (env->cr[3] & CR3_LAM_U48_MASK) {
+ return 48;
+ }
+ return 64;
+}
+
+/* @addr with its LAM metadata masked, keeping bit 63. IA-32e mode only. */
+static inline uint64_t x86_lam_mask_addr(CPUX86State *env, uint64_t addr)
+{
+ unsigned lam_bits = x86_lam_addr_bits(env, addr);
+
+ if (lam_bits == 64) {
+ return addr;
+ }
+ return deposit64(sextract64(addr, 0, lam_bits), 63, 1, addr >> 63);
+}
+
static inline bool ctl_has_irq(CPUX86State *env)
{
uint32_t int_prio;
diff --git a/target/i386/helper.c b/target/i386/helper.c
index 723e87f..c680e01 100644
--- a/target/i386/helper.c
+++ b/target/i386/helper.c
@@ -204,7 +204,8 @@ void cpu_x86_update_cr4(CPUX86State *env, uint32_t new_cr4)
#endif
if ((new_cr4 ^ env->cr[4]) &
(CR4_PGE_MASK | CR4_PAE_MASK | CR4_PSE_MASK |
- CR4_SMEP_MASK | CR4_SMAP_MASK | CR4_LA57_MASK)) {
+ CR4_SMEP_MASK | CR4_SMAP_MASK | CR4_LA57_MASK |
+ CR4_LAM_SUP_MASK)) {
tlb_flush(env_cpu(env));
}
@@ -290,6 +291,9 @@ bool x86_cpu_translate_for_debug(CPUState *cs, vaddr addr,
uint64_t pml4e_addr, pml4e;
int32_t sext;
+ /* LAM: translate as for a data access */
+ addr = x86_lam_mask_addr(env, addr);
+
/* test virtual address sign extension */
sext = la57 ? (int64_t)addr >> 56 : (int64_t)addr >> 47;
if (sext != 0 && sext != -1) {
diff --git a/target/i386/tcg/system/excp_helper.c b/target/i386/tcg/system/excp_helper.c
index 6c48ccb..1c18eb7 100644
--- a/target/i386/tcg/system/excp_helper.c
+++ b/target/i386/tcg/system/excp_helper.c
@@ -581,15 +581,35 @@ static bool get_physical_address(CPUX86State *env, vaddr addr,
}
if (likely(env->cr[0] & CR0_PG_MASK || use_stage2)) {
+ int lam_prot_mask = 0;
+
in.cr3 = env->cr[3];
in.mmu_idx = mmu_idx;
in.ptw_idx = use_stage2 ? MMU_NESTED_IDX : MMU_PHYS_IDX;
in.pg_mode = get_pg_mode(env);
if (in.pg_mode & PG_MODE_LMA) {
- /* test virtual address sign extension */
+ vaddr masked = x86_lam_mask_addr(env, addr);
int shift = in.pg_mode & PG_MODE_LA57 ? 56 : 47;
- int64_t sext = (int64_t)addr >> shift;
+ int64_t sext;
+
+ /*
+ * LAM masks data addresses but not fetches. Both install
+ * the entry under the unmasked address, so if masking
+ * changes it, keep only the permissions of this access.
+ */
+ if (masked != addr) {
+ if (access_type == MMU_INST_FETCH) {
+ lam_prot_mask = PAGE_READ | PAGE_WRITE;
+ } else {
+ lam_prot_mask = PAGE_EXEC;
+ addr = masked;
+ in.addr = addr;
+ }
+ }
+
+ /* test virtual address sign extension */
+ sext = (int64_t)addr >> shift;
if (sext != 0 && sext != -1) {
*err = (TranslateFault){
.exception_index = EXCP0D_GPF,
@@ -599,7 +619,11 @@ static bool get_physical_address(CPUX86State *env, vaddr addr,
return false;
}
}
- return mmu_translate(env, &in, out, err, ra);
+ if (!mmu_translate(env, &in, out, err, ra)) {
+ return false;
+ }
+ out->prot &= ~lam_prot_mask;
+ return true;
}
break;
}
diff --git a/target/i386/tcg/system/misc_helper.c b/target/i386/tcg/system/misc_helper.c
index a77da59..be4f553 100644
--- a/target/i386/tcg/system/misc_helper.c
+++ b/target/i386/tcg/system/misc_helper.c
@@ -90,8 +90,7 @@ void HELPER(write_crN)(CPUX86State *env, int reg, target_ulong t0)
cpu_x86_update_cr0(env, t0);
break;
case 3:
- if ((env->efer & MSR_EFER_LMA) &&
- (t0 & ((~0ULL) << env_archcpu(env)->phys_bits))) {
+ if ((env->efer & MSR_EFER_LMA) && (t0 & cr3_reserved_bits(env))) {
cpu_vmexit(env, SVM_EXIT_ERR, 0, GETPC());
}
if (!(env->efer & MSR_EFER_LMA)) {
@@ -504,6 +503,27 @@ void HELPER(rdmsr)(CPUX86State *env)
void HELPER(flush_page)(CPUX86State *env, target_ulong addr)
{
+#ifdef TARGET_X86_64
+ if (env->hflags & HF_LMA_MASK) {
+ int shift = env->cr[4] & CR4_LA57_MASK ? 56 : 47;
+ int64_t sext = (int64_t)addr >> shift;
+ unsigned lam_bits;
+
+ /* no LAM for the operand: non-canonical is a no-op */
+ if (sext != 0 && sext != -1) {
+ return;
+ }
+
+ /* TLB entries are per tag: flush every tagged alias */
+ lam_bits = x86_lam_addr_bits(env, addr);
+ if (lam_bits < 64) {
+ tlb_flush_page_bits_by_mmuidx(env_cpu(env), addr,
+ MAKE_64BIT_MASK(0, NB_MMU_MODES),
+ lam_bits);
+ return;
+ }
+ }
+#endif
tlb_flush_page(env_cpu(env), addr);
}
diff --git a/target/i386/tcg/system/svm_helper.c b/target/i386/tcg/system/svm_helper.c
index 9c04042..0e3cf6f 100644
--- a/target/i386/tcg/system/svm_helper.c
+++ b/target/i386/tcg/system/svm_helper.c
@@ -315,8 +315,7 @@ void HELPER(vmrun)(CPUX86State *env, int aflag, int next_eip_addend)
cpu_vmexit(env, SVM_EXIT_ERR, 0, GETPC());
}
new_cr3 = x86_ldq_phys(cs, env->vm_vmcb + offsetof(struct vmcb, save.cr3));
- if ((env->efer & MSR_EFER_LMA) &&
- (new_cr3 & ((~0ULL) << cpu->phys_bits))) {
+ if ((env->efer & MSR_EFER_LMA) && (new_cr3 & cr3_reserved_bits(env))) {
cpu_vmexit(env, SVM_EXIT_ERR, 0, GETPC());
}
new_cr4 = x86_ldq_phys(cs, env->vm_vmcb + offsetof(struct vmcb, save.cr4));
--
2.47.3
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 4/4] tests/tcg/x86_64: add a LAM system test
2026-11-10 13:01 [PATCH 0/4] target/i386: emulate LAM in TCG Tim
` (2 preceding siblings ...)
2026-11-10 13:01 ` [PATCH 3/4] target/i386: emulate LAM in TCG Tim
@ 2026-11-10 13:01 ` Tim
3 siblings, 0 replies; 5+ messages in thread
From: Tim @ 2026-11-10 13:01 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, richard.henderson, zhao1.liu, Tim
Test LAM_U57, LAM_U48, and LAM_SUP at CPL 0: tagged aliases, the
canonical checks on the masked address, CR2 on #PF, TLB flushes on
CR3 and CR4.LAM_SUP writes and on INVLPG of an untagged 4 KiB or
2 MiB page, helpers and atomics, and instruction fetches through
tagged addresses.
The test runs with 4-level paging and with 5-level paging, for which
boot.S gains an LA57 option, each with and without LAM in CPUID.
Each run has its own build, which fails unless CPUID reports LAM
exactly when expected. The identity map uses 4 KiB pages, which
leaves the 2 MiB test page as the only large page in the TLB.
Signed-off-by: Tim <mdspan.github@gmail.com>
---
tests/tcg/x86_64/system/boot.S | 14 +
tests/tcg/x86_64/system/lam.c | 797 ++++++++++++++++++++++++++++
tests/tcg/x86_64/system/meson.build | 15 +
3 files changed, 826 insertions(+)
create mode 100644 tests/tcg/x86_64/system/lam.c
diff --git a/tests/tcg/x86_64/system/boot.S b/tests/tcg/x86_64/system/boot.S
index 7213aec..4ab554d 100644
--- a/tests/tcg/x86_64/system/boot.S
+++ b/tests/tcg/x86_64/system/boot.S
@@ -91,6 +91,10 @@ _start:
/* Enable PAE mode (bit 5). */
mov %cr4, %eax
btsl $5, %eax
+#ifdef LA57
+ /* Enable 5-level paging (bit 12). */
+ btsl $12, %eax
+#endif
mov %eax, %cr4
#define MSR_EFER 0xc0000080 /* extended feature register */
@@ -102,7 +106,11 @@ _start:
wrmsr
/* Enable paging */
+#ifdef LA57
+ mov $.Lpml5, %ecx
+#else
mov $.Lpml4, %ecx
+#endif
mov %ecx, %cr3
mov %cr0, %eax
@@ -272,3 +280,9 @@ i = 0
.align 4096
.Lpml4:
.quad .Lpdp + 7 /* 0-512 GB */
+
+#ifdef LA57
+.align 4096
+.Lpml5:
+ .quad .Lpml4 + 7
+#endif
diff --git a/tests/tcg/x86_64/system/lam.c b/tests/tcg/x86_64/system/lam.c
new file mode 100644
index 0000000..299119a
--- /dev/null
+++ b/tests/tcg/x86_64/system/lam.c
@@ -0,0 +1,797 @@
+/*
+ * Linear Address Masking (LAM) test, run at CPL 0. LA57 selects
+ * 5-level paging, and EXPECT_LAM whether CPUID must report LAM.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include <stdint.h>
+#include <minilib.h>
+
+#define CR3_LAM_U57 (1ULL << 61)
+#define CR3_LAM_U48 (1ULL << 62)
+#define CR4_LA57 (1ULL << 12)
+#define CR4_LAM_SUP (1ULL << 28)
+
+#define PTE_P 0x001ULL
+#define PTE_W 0x002ULL
+#define PTE_U 0x004ULL
+#define PTE_PS 0x080ULL
+#define PTE_FLAGS (PTE_P | PTE_W | PTE_U)
+
+#define VEC_NONE (-1)
+#define VEC_GP 13
+#define VEC_PF 14
+
+/* Identity map of the first 128 MiB with 4 KiB pages. */
+#define ID_PAGES (128 * 256)
+
+/* User-half test region: PML4 entry 1. */
+#define VA_BASE 0x0000008000000000ULL
+/* Supervisor-half test region: PML4 entry 256, same tables as VA_BASE. */
+#define VA_SUP 0xffff800000000000ULL
+/* Pages 0-2 are present, page 3 is not. */
+#define VA_PAGE(n) (VA_BASE + (n) * 4096ULL)
+/* A 2 MiB page after the 4 KiB pages. */
+#define VA_LP (VA_BASE + 0x200000ULL)
+
+#define LP0_PHYS 0x04000000ULL
+#define LP1_PHYS 0x06000000ULL
+
+/* Tags in the metadata bits of each LAM width. */
+#define TAG57(t) ((uint64_t)(t) << 57)
+#define TAG48(t) ((uint64_t)(t) << 48)
+
+/* VA_SUP with tag @t. SUP_BAD masks to a non-canonical address. */
+#ifdef LA57
+#define STAG(t) ((VA_SUP & ~TAG57(0x3f)) | TAG57(t))
+#define SUP_BAD ((1ULL << 63) | TAG57(0x2a) | 0x1000)
+#else
+#define STAG(t) ((1ULL << 63) | TAG48(t) | \
+ (VA_SUP & ((1ULL << 48) - 1)))
+#define SUP_BAD ((1ULL << 63) | TAG48(0x5a5a) | 0x1000)
+#endif
+
+/* Vector for an address that is non-canonical only with 4-level paging */
+#ifdef LA57
+#define VEC_NC4 VEC_PF
+#else
+#define VEC_NC4 VEC_GP
+#endif
+
+#define MARK(n) (0x1111111111111111ULL * (n))
+
+static uint64_t pml5[512] __attribute__((aligned(4096)));
+static uint64_t pml4[512] __attribute__((aligned(4096)));
+static uint64_t pdpt_id[512] __attribute__((aligned(4096)));
+static uint64_t pd_id[512] __attribute__((aligned(4096)));
+static uint64_t pt_id[ID_PAGES] __attribute__((aligned(4096)));
+static uint64_t pdpt_t[512] __attribute__((aligned(4096)));
+static uint64_t pd_t[512] __attribute__((aligned(4096)));
+static uint64_t pt_t[512] __attribute__((aligned(4096)));
+static uint8_t frames[4][4096] __attribute__((aligned(4096)));
+
+#ifdef LA57
+/* VA_BASE with bit 48 set: PML5 entry 1, mapped to frames[2]. */
+#define VA_HIGH (VA_BASE | (1ULL << 48))
+static uint64_t pml4_h[512] __attribute__((aligned(4096)));
+static uint64_t pdpt_h[512] __attribute__((aligned(4096)));
+static uint64_t pd_h[512] __attribute__((aligned(4096)));
+static uint64_t pt_h[512] __attribute__((aligned(4096)));
+#endif
+
+struct idt_gate {
+ uint16_t off0;
+ uint16_t sel;
+ uint8_t ist;
+ uint8_t type;
+ uint16_t off1;
+ uint32_t off2;
+ uint32_t zero;
+} __attribute__((packed));
+
+static struct idt_gate idt[32] __attribute__((aligned(16)));
+
+/* Written by the exception stubs. */
+volatile uint64_t fault_vec, fault_err, fault_rip, fault_cr2, fixup_rip;
+
+/* Exception stubs: record the fault, then resume at fixup_rip, or stop if 0. */
+asm(".text\n"
+ ".irp v, 0,1,2,3,4,5,6,7,9,15,16,18,19,20,22,23,24,25,26,27,28,31\n"
+ "isr_\\v:\n"
+ " pushq $0\n"
+ " pushq $\\v\n"
+ " jmp isr_common\n"
+ ".endr\n"
+ ".irp v, 8,10,11,12,13,14,17,21,29,30\n"
+ "isr_\\v:\n"
+ " pushq $\\v\n"
+ " jmp isr_common\n"
+ ".endr\n"
+ "isr_common:\n"
+ /* stack: rax, vector, error code, rip, cs, rflags, rsp, ss */
+ " pushq %rax\n"
+ " movq 8(%rsp), %rax\n"
+ " movq %rax, fault_vec(%rip)\n"
+ " movq 16(%rsp), %rax\n"
+ " movq %rax, fault_err(%rip)\n"
+ " movq 24(%rsp), %rax\n"
+ " movq %rax, fault_rip(%rip)\n"
+ " movq %cr2, %rax\n"
+ " movq %rax, fault_cr2(%rip)\n"
+ " movq fixup_rip(%rip), %rax\n"
+ " testq %rax, %rax\n"
+ " jz 1f\n"
+ " movq %rax, 24(%rsp)\n"
+ " movq $0, fixup_rip(%rip)\n"
+ " popq %rax\n"
+ " addq $16, %rsp\n"
+ " iretq\n"
+ "1: andq $-16, %rsp\n"
+ " call unexpected_exception\n"
+ "2: hlt\n"
+ " jmp 2b\n"
+ ".data\n"
+ ".align 8\n"
+ "isr_table:\n"
+ ".irp v, 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,"
+ "16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31\n"
+ " .quad isr_\\v\n"
+ ".endr\n"
+ ".text\n");
+
+static int passes, failures;
+static uint64_t base_cr3;
+
+void unexpected_exception(void)
+{
+ ml_printf("FAIL: unexpected exception %llu, error %llx, rip %llx, "
+ "cr2 %llx\n", fault_vec, fault_err, fault_rip, fault_cr2);
+ asm volatile("outw %w0, $0xf4" : : "a"(0x7f));
+ for (;;) {
+ asm volatile("hlt");
+ }
+}
+
+static const char *vec_name(long long vec)
+{
+ switch (vec) {
+ case VEC_NONE:
+ return "none";
+ case VEC_GP:
+ return "#GP";
+ case VEC_PF:
+ return "#PF";
+ default:
+ return "other";
+ }
+}
+
+static void check(int ok, const char *what)
+{
+ if (ok) {
+ passes++;
+ } else {
+ failures++;
+ ml_printf("FAIL: %s\n", what);
+ }
+}
+
+static uint64_t read_cr3(void)
+{
+ uint64_t v;
+
+ asm volatile("mov %%cr3, %0" : "=r"(v));
+ return v;
+}
+
+static uint64_t read_cr4(void)
+{
+ uint64_t v;
+
+ asm volatile("mov %%cr4, %0" : "=r"(v));
+ return v;
+}
+
+static void write_cr2(uint64_t v)
+{
+ asm volatile("mov %0, %%cr2" : : "r"(v) : "memory");
+}
+
+/* try_*: one operation, returning the vector it raised or VEC_NONE. */
+#define TRY_BEGIN \
+ "leaq 1f(%%rip), %%rax\n\t" \
+ "movq %%rax, fixup_rip(%%rip)\n\t"
+#define TRY_END \
+ "1:\n\t" \
+ "movq $0, fixup_rip(%%rip)\n\t"
+
+static long long try_load(uint64_t addr, uint64_t *val)
+{
+ uint64_t v = 0;
+
+ fault_vec = VEC_NONE;
+ asm volatile(TRY_BEGIN
+ "movq (%[a]), %[v]\n\t"
+ TRY_END
+ : [v] "+r"(v) : [a] "r"(addr) : "rax", "memory");
+ *val = v;
+ return (long long)fault_vec;
+}
+
+static long long try_store(uint64_t addr, uint64_t val)
+{
+ fault_vec = VEC_NONE;
+ asm volatile(TRY_BEGIN
+ "movq %[v], (%[a])\n\t"
+ TRY_END
+ : : [v] "r"(val), [a] "r"(addr) : "rax", "memory");
+ return (long long)fault_vec;
+}
+
+static long long try_write_cr3(uint64_t v)
+{
+ fault_vec = VEC_NONE;
+ asm volatile(TRY_BEGIN
+ "mov %[v], %%cr3\n\t"
+ TRY_END
+ : : [v] "r"(v) : "rax", "memory");
+ return (long long)fault_vec;
+}
+
+static long long try_write_cr4(uint64_t v)
+{
+ fault_vec = VEC_NONE;
+ asm volatile(TRY_BEGIN
+ "mov %[v], %%cr4\n\t"
+ TRY_END
+ : : [v] "r"(v) : "rax", "memory");
+ return (long long)fault_vec;
+}
+
+static long long try_invlpg(uint64_t addr)
+{
+ fault_vec = VEC_NONE;
+ asm volatile(TRY_BEGIN
+ "invlpg (%[a])\n\t"
+ TRY_END
+ : : [a] "r"(addr) : "rax", "memory");
+ return (long long)fault_vec;
+}
+
+/* Jump to an exec stub at @addr. *ran is its EAX, or 0 if none ran. */
+static long long try_exec(uint64_t addr, int *ran)
+{
+ uint64_t r;
+
+ fault_vec = VEC_NONE;
+ asm volatile("leaq 1f(%%rip), %%rdx\n\t"
+ "movq %%rdx, fixup_rip(%%rip)\n\t"
+ "xorl %%eax, %%eax\n\t"
+ "jmp *%[a]\n\t"
+ "1:\n\t"
+ "movq $0, fixup_rip(%%rip)\n\t"
+ : "=&a"(r) : [a] "r"(addr) : "rdx", "memory");
+ *ran = r;
+ return (long long)fault_vec;
+}
+
+/* movl $0x1234, %eax; jmp *%rdx, and the same with $0x2222 */
+static const uint8_t exec_stub[] = {
+ 0xb8, 0x34, 0x12, 0x00, 0x00, 0xff, 0xe2
+};
+#ifdef LA57
+static const uint8_t exec_stub2[] = {
+ 0xb8, 0x22, 0x22, 0x00, 0x00, 0xff, 0xe2
+};
+#endif
+#define EXEC_OFF 0x800
+
+static void expect_load(const char *what, uint64_t addr, long long vec,
+ uint64_t val)
+{
+ uint64_t v;
+ long long got = try_load(addr, &v);
+
+ if (got == vec && (vec != VEC_NONE || v == val)) {
+ passes++;
+ return;
+ }
+ failures++;
+ ml_printf("FAIL: %s: load %llx: exception %s, value %llx; "
+ "expected exception %s, value %llx\n",
+ what, addr, vec_name(got), v, vec_name(vec), val);
+}
+
+static void expect_store(const char *what, uint64_t addr, long long vec,
+ uint64_t val)
+{
+ long long got = try_store(addr, val);
+
+ if (got == vec) {
+ passes++;
+ return;
+ }
+ failures++;
+ ml_printf("FAIL: %s: store %llx: exception %s, expected %s\n",
+ what, addr, vec_name(got), vec_name(vec));
+}
+
+static void expect_fault(const char *what, long long got, long long vec)
+{
+ if (got == vec) {
+ passes++;
+ return;
+ }
+ failures++;
+ ml_printf("FAIL: %s: exception %s, expected %s\n",
+ what, vec_name(got), vec_name(vec));
+}
+
+static void set_lam_cr3(uint64_t bits)
+{
+ long long v = try_write_cr3(base_cr3 | bits);
+
+ if (v != VEC_NONE) {
+ ml_printf("FAIL: MOV to CR3 with %llx: exception %s\n",
+ bits, vec_name(v));
+ failures++;
+ }
+}
+
+static void set_lam_sup(int on)
+{
+ uint64_t cr4 = read_cr4();
+ long long v;
+
+ cr4 = on ? cr4 | CR4_LAM_SUP : cr4 & ~CR4_LAM_SUP;
+ v = try_write_cr4(cr4);
+ if (v != VEC_NONE) {
+ ml_printf("FAIL: MOV to CR4 with LAM_SUP=%d: exception %s\n",
+ on, vec_name(v));
+ failures++;
+ }
+}
+
+static uint64_t *phys(uint64_t pa)
+{
+ return (uint64_t *)pa;
+}
+
+static void setup_idt(void)
+{
+ struct {
+ uint16_t limit;
+ uint64_t base;
+ } __attribute__((packed)) idtr = { sizeof(idt) - 1, (uint64_t)idt };
+ const uint64_t *isr_table;
+
+ asm("leaq isr_table(%%rip), %0" : "=r"(isr_table));
+ for (int i = 0; i < 32; i++) {
+ uint64_t h = isr_table[i];
+
+ idt[i].off0 = h;
+ idt[i].sel = 0x8;
+ idt[i].ist = 0;
+ idt[i].type = 0x8e;
+ idt[i].off1 = h >> 16;
+ idt[i].off2 = h >> 32;
+ idt[i].zero = 0;
+ }
+ asm volatile("lidt %0" : : "m"(idtr));
+}
+
+static void setup_paging(void)
+{
+ for (int i = 0; i < 512; i++) {
+ pml5[i] = pml4[i] = pdpt_id[i] = pd_id[i] = 0;
+ pdpt_t[i] = pd_t[i] = pt_t[i] = 0;
+ }
+ for (int i = 0; i < ID_PAGES; i++) {
+ pt_id[i] = (uint64_t)i * 4096 | PTE_FLAGS;
+ }
+ for (int i = 0; i < ID_PAGES / 512; i++) {
+ pd_id[i] = (uint64_t)&pt_id[i * 512] | PTE_FLAGS;
+ }
+ pdpt_id[0] = (uint64_t)pd_id | PTE_FLAGS;
+ pml4[0] = (uint64_t)pdpt_id | PTE_FLAGS;
+
+ for (int i = 0; i < 3; i++) {
+ pt_t[i] = (uint64_t)frames[i] | PTE_FLAGS;
+ }
+ pd_t[0] = (uint64_t)pt_t | PTE_FLAGS;
+ pd_t[1] = LP0_PHYS | PTE_PS | PTE_FLAGS;
+ pdpt_t[0] = (uint64_t)pd_t | PTE_FLAGS;
+ pml4[1] = (uint64_t)pdpt_t | PTE_FLAGS;
+ pml4[256] = (uint64_t)pdpt_t | PTE_FLAGS;
+
+ for (int i = 0; i < 4; i++) {
+ for (int j = 0; j < 4096 / 8; j++) {
+ ((uint64_t *)frames[i])[j] = MARK(i + 1) + j;
+ }
+ }
+ *phys(LP0_PHYS) = 0xaaaaaaaaaaaaaaaaULL;
+ *phys(LP0_PHYS + 0x5000) = 0xaaaaaaaaaaaa5000ULL;
+ *phys(LP1_PHYS) = 0xbbbbbbbbbbbbbbbbULL;
+ *phys(LP1_PHYS + 0x5000) = 0xbbbbbbbbbbbb5000ULL;
+ for (int i = 0; i < (int)sizeof(exec_stub); i++) {
+ frames[0][EXEC_OFF + i] = exec_stub[i];
+ }
+
+#ifdef LA57
+ for (int i = 0; i < 512; i++) {
+ pml4_h[i] = pdpt_h[i] = pd_h[i] = pt_h[i] = 0;
+ }
+ pt_h[0] = (uint64_t)frames[2] | PTE_FLAGS;
+ pd_h[0] = (uint64_t)pt_h | PTE_FLAGS;
+ pdpt_h[0] = (uint64_t)pd_h | PTE_FLAGS;
+ pml4_h[1] = (uint64_t)pdpt_h | PTE_FLAGS;
+ for (int i = 0; i < (int)sizeof(exec_stub2); i++) {
+ frames[2][EXEC_OFF + i] = exec_stub2[i];
+ }
+ pml5[0] = (uint64_t)pml4 | PTE_FLAGS;
+ pml5[1] = (uint64_t)pml4_h | PTE_FLAGS;
+ pml5[511] = (uint64_t)pml4 | PTE_FLAGS;
+ base_cr3 = (uint64_t)pml5;
+#else
+ base_cr3 = (uint64_t)pml4;
+#endif
+ asm volatile("mov %0, %%cr3" : : "r"(base_cr3) : "memory");
+}
+
+static void restore_frames(void)
+{
+ pt_t[0] = (uint64_t)frames[0] | PTE_FLAGS;
+ pd_t[1] = LP0_PHYS | PTE_PS | PTE_FLAGS;
+ for (int j = 0; j < 4096 / 8; j++) {
+ ((uint64_t *)frames[0])[j] = MARK(1) + j;
+ }
+ for (int i = 0; i < (int)sizeof(exec_stub); i++) {
+ frames[0][EXEC_OFF + i] = exec_stub[i];
+ }
+}
+
+static int cpu_has_lam(void)
+{
+ uint32_t a, b, c, d;
+
+ asm volatile("cpuid" : "=a"(a), "=b"(b), "=c"(c), "=d"(d)
+ : "a"(7), "c"(1));
+ return (a >> 26) & 1;
+}
+
+static void test_no_lam_cpu(void)
+{
+ expect_load("no LAM: tagged load", VA_PAGE(0) | TAG57(1), VEC_GP, 0);
+}
+
+static void test_lam_off(void)
+{
+ set_lam_cr3(0);
+ set_lam_sup(0);
+ write_cr2(0xdead000);
+ expect_load("LAM off: untagged", VA_PAGE(0), VEC_NONE, MARK(1));
+ expect_load("LAM off: tag 62:57", VA_PAGE(0) | TAG57(0x2a), VEC_GP, 0);
+ check(fault_cr2 == 0xdead000, "LAM off: #GP left CR2 unchanged");
+ expect_load("LAM off: tag 56:48", VA_PAGE(0) | TAG48(0x10), VEC_NC4, 0);
+ expect_load("LAM off: supervisor tag", VA_SUP & ~(1ULL << 60), VEC_GP,
+ 0);
+}
+
+static void test_cr3_readback(void)
+{
+ set_lam_cr3(CR3_LAM_U57);
+ check(read_cr3() == (base_cr3 | CR3_LAM_U57), "CR3 reads back LAM_U57");
+ set_lam_cr3(CR3_LAM_U48);
+ check(read_cr3() == (base_cr3 | CR3_LAM_U48), "CR3 reads back LAM_U48");
+ set_lam_cr3(0);
+}
+
+static void test_u57(void)
+{
+ uint64_t a = VA_PAGE(0) | TAG57(0x2a);
+ uint64_t b = VA_PAGE(0) | TAG57(0x15);
+ uint64_t v;
+
+ set_lam_cr3(CR3_LAM_U57);
+
+ expect_load("U57: untagged", VA_PAGE(0), VEC_NONE, MARK(1));
+ expect_load("U57: tag 0x2a", a, VEC_NONE, MARK(1));
+ expect_load("U57: tag 0x3f", VA_PAGE(0) | TAG57(0x3f), VEC_NONE,
+ MARK(1));
+ expect_load("U57: tag 0x2a, page 1", VA_PAGE(1) | TAG57(0x2a) | 0x18,
+ VEC_NONE, MARK(2) + 3);
+
+ /* Aliases share memory, also when they alternate in one TLB slot. */
+ for (int i = 0; i < 8; i++) {
+ uint64_t w = i & 1 ? a : b;
+ uint64_t r = i & 1 ? b : a;
+
+ expect_store("U57: alias store", w + 8, VEC_NONE, 0x5150 + i);
+ expect_load("U57: alias load", r + 8, VEC_NONE, 0x5150 + i);
+ expect_load("U57: untagged load", VA_PAGE(0) + 8, VEC_NONE,
+ 0x5150 + i);
+ }
+
+ /* Bit 56, and with 4-level paging bits 55:47, must equal bit 63. */
+ expect_load("U57: bit 56 set", VA_PAGE(0) | (1ULL << 56), VEC_GP, 0);
+ expect_load("U57: tag and bit 56 set", a | (1ULL << 56), VEC_GP, 0);
+ expect_load("U57: bit 52 set", VA_PAGE(0) | (1ULL << 52), VEC_NC4, 0);
+ expect_load("U57: tag and bit 49 set", a | (1ULL << 49), VEC_NC4, 0);
+ expect_load("U57: bit 47 set", VA_PAGE(0) | (1ULL << 47), VEC_NC4, 0);
+
+ /* The supervisor half is not masked without LAM_SUP. */
+ expect_load("U57: supervisor untagged", VA_SUP, VEC_NONE, MARK(1));
+ expect_load("U57: supervisor tagged", VA_SUP & ~TAG57(0x3f), VEC_GP, 0);
+
+ /* #PF through a tagged address reports the masked address in CR2. */
+ expect_load("U57: #PF load", VA_PAGE(3) | TAG57(0x2a) | 0x40, VEC_PF,
+ 0);
+ check(fault_cr2 == (VA_PAGE(3) | 0x40), "U57: #PF load CR2 is masked");
+ check(fault_err == 0, "U57: #PF load error code");
+ expect_store("U57: #PF store", VA_PAGE(3) | TAG57(0x2a) | 0x48, VEC_PF,
+ 0);
+ check(fault_cr2 == (VA_PAGE(3) | 0x48), "U57: #PF store CR2 is masked");
+ check(fault_err == 2, "U57: #PF store error code");
+
+ /* Accesses that cross a page boundary. */
+ {
+ uint64_t want, cr2;
+
+ try_load(VA_PAGE(1) - 4, &want);
+ expect_load("U57: cross-page load", (VA_PAGE(1) - 4) | TAG57(9),
+ VEC_NONE, want);
+ expect_load("U57: cross-page #PF untagged", VA_PAGE(3) - 4, VEC_PF,
+ 0);
+ cr2 = fault_cr2;
+ expect_load("U57: cross-page #PF", (VA_PAGE(3) - 4) | TAG57(9),
+ VEC_PF, 0);
+ check(fault_cr2 == cr2, "U57: cross-page #PF CR2");
+ }
+
+ /* Helpers and atomics. */
+ {
+ uint64_t p = VA_PAGE(2) | TAG57(0x11);
+ uint64_t src = VA_PAGE(0) | TAG57(0x22);
+ uint64_t dst = (VA_PAGE(2) + 0x300) | TAG57(0x33);
+ uint64_t n = 64;
+ uint64_t old = 5, lo = 1, hi = 2;
+ uint8_t ok;
+
+ try_store(p + 0x100, 40);
+ asm volatile("lock xaddq %0, (%1)" : "+r"(old) : "r"(p + 0x100)
+ : "memory");
+ check(old == 40, "U57: lock xadd old value");
+ expect_load("U57: lock xadd result", VA_PAGE(2) + 0x100, VEC_NONE,
+ 45);
+
+ try_store(VA_PAGE(2) + 0x200, 1);
+ try_store(VA_PAGE(2) + 0x208, 2);
+ asm volatile("lock cmpxchg16b (%[p])\n\tsetz %[ok]"
+ : "+a"(lo), "+d"(hi), [ok] "=q"(ok)
+ : [p] "r"(p + 0x200), "b"(0x33ULL), "c"(0x44ULL)
+ : "memory");
+ check(ok, "U57: cmpxchg16b succeeded");
+ expect_load("U57: cmpxchg16b low", VA_PAGE(2) + 0x200, VEC_NONE,
+ 0x33);
+ expect_load("U57: cmpxchg16b high", VA_PAGE(2) + 0x208, VEC_NONE,
+ 0x44);
+
+ asm volatile("rep movsb" : "+S"(src), "+D"(dst), "+c"(n)
+ : : "memory");
+ expect_load("U57: rep movsb", VA_PAGE(2) + 0x300 + 16, VEC_NONE,
+ MARK(1) + 2);
+
+ try_store(VA_PAGE(2) + 0x400, 0);
+ asm volatile("fninit\n\tfxsave64 (%0)" : : "r"(p + 0x400)
+ : "memory");
+ try_load(VA_PAGE(2) + 0x400, &v);
+ check((v & 0xffff) == 0x37f, "U57: fxsave64 FCW");
+ }
+
+ /* Instruction fetches are not masked. */
+ {
+ uint64_t code = VA_PAGE(0) + EXEC_OFF;
+ int ran;
+
+ expect_fault("U57: exec untagged", try_exec(code, &ran), VEC_NONE);
+ check(ran == 0x1234, "U57: exec untagged ran");
+
+ set_lam_cr3(CR3_LAM_U57);
+ expect_fault("U57: exec tagged, empty TLB",
+ try_exec(code | TAG57(0x2a), &ran), VEC_GP);
+ check(!ran, "U57: exec tagged, empty TLB, did not run");
+
+ set_lam_cr3(CR3_LAM_U57);
+ expect_load("U57: data load before exec", code | TAG57(0x2a),
+ VEC_NONE, *(uint64_t *)&frames[0][EXEC_OFF]);
+ expect_fault("U57: exec tagged after data load",
+ try_exec(code | TAG57(0x2a), &ran), VEC_GP);
+ check(!ran, "U57: exec tagged after data load did not run");
+ }
+
+ /* Turning LAM off flushes entries for tagged addresses. */
+ expect_load("U57: before CR3 write", a, VEC_NONE, MARK(1));
+ set_lam_cr3(0);
+ expect_load("U57: after LAM_U57 cleared", a, VEC_GP, 0);
+}
+
+static void test_u48(void)
+{
+ uint64_t a = VA_PAGE(0) | TAG48(0x1234);
+ uint64_t b = VA_PAGE(0) | TAG48(0x7fff);
+
+ set_lam_cr3(CR3_LAM_U48);
+ expect_load("U48: tag 0x1234", a, VEC_NONE, MARK(1));
+ expect_load("U48: tag 0x7fff", b, VEC_NONE, MARK(1));
+ expect_store("U48: alias store", a + 16, VEC_NONE, 0x4848);
+ expect_load("U48: alias load", b + 16, VEC_NONE, 0x4848);
+ expect_load("U48: bit 47 set", VA_PAGE(0) | (1ULL << 47), VEC_GP, 0);
+ expect_load("U48: tag and bit 47 set", a | (1ULL << 47), VEC_GP, 0);
+ expect_load("U48: #PF", VA_PAGE(3) | TAG48(0x77) | 8, VEC_PF, 0);
+ check(fault_cr2 == (VA_PAGE(3) | 8), "U48: #PF CR2 is masked");
+
+ /* LAM_U57 takes precedence over LAM_U48. */
+ set_lam_cr3(CR3_LAM_U57 | CR3_LAM_U48);
+ expect_load("U57+U48: tag 62:57", VA_PAGE(0) | TAG57(5), VEC_NONE,
+ MARK(1));
+ expect_load("U57+U48: bit 50", VA_PAGE(0) | (1ULL << 50), VEC_NC4, 0);
+ set_lam_cr3(0);
+}
+
+static void test_sup(void)
+{
+ uint64_t a = STAG(0x1a);
+ uint64_t b = STAG(0x01);
+ uint64_t user = VA_PAGE(0) | TAG57(1);
+ uint64_t x = STAG(0x2e) & ~(1ULL << 50);
+
+ set_lam_cr3(0);
+ set_lam_sup(1);
+ check(read_cr4() & CR4_LAM_SUP, "CR4.LAM_SUP reads back");
+ expect_load("SUP: untagged", VA_SUP, VEC_NONE, MARK(1));
+ expect_load("SUP: tagged", a, VEC_NONE, MARK(1));
+ expect_store("SUP: alias store", b + 24, VEC_NONE, 0x5555);
+ expect_load("SUP: alias load", a + 24, VEC_NONE, 0x5555);
+ expect_load("SUP: highest address bit clear", SUP_BAD, VEC_GP, 0);
+ expect_load("SUP: user half not masked", user, VEC_GP, 0);
+ expect_load("SUP: #PF", STAG(0x22) + 3 * 4096 + 0x10, VEC_PF, 0);
+ check(fault_cr2 == (VA_SUP + 3 * 4096 + 0x10), "SUP: #PF CR2 is masked");
+
+ /*
+ * Bit 50 is metadata with 4-level paging. With 5-level paging it
+ * selects PML5 entry 0x1fb, which is not present.
+ */
+#ifdef LA57
+ expect_load("SUP: bit 50 clear", x, VEC_PF, 0);
+ check(fault_cr2 == (VA_SUP & ~(1ULL << 50)), "SUP: bit 50 clear CR2");
+#else
+ expect_load("SUP: bit 50 clear", x, VEC_NONE, MARK(1));
+#endif
+
+ /* Turning LAM_SUP off flushes entries for tagged addresses. */
+ expect_load("SUP: before CR4 write", a, VEC_NONE, MARK(1));
+ set_lam_sup(0);
+ expect_load("SUP: after LAM_SUP cleared", a, VEC_GP, 0);
+}
+
+#ifdef LA57
+/*
+ * With LAM_U48, fetches from VA_HIGH reach frames[2] and data accesses
+ * reach frames[0], in either order.
+ */
+static void test_u48_la57_split(void)
+{
+ uint64_t data = VA_HIGH, code = VA_HIGH + EXEC_OFF;
+ int ran;
+
+ set_lam_cr3(0);
+ expect_load("U48 LA57: LAM off, data", data, VEC_NONE, MARK(3));
+
+ set_lam_cr3(CR3_LAM_U48);
+ expect_fault("U48 LA57: fetch first", try_exec(code, &ran), VEC_NONE);
+ check(ran == 0x2222, "U48 LA57: fetch first ran frames[2]");
+ expect_load("U48 LA57: data after fetch", data, VEC_NONE, MARK(1));
+ expect_fault("U48 LA57: fetch after data", try_exec(code, &ran),
+ VEC_NONE);
+ check(ran == 0x2222, "U48 LA57: fetch after data ran frames[2]");
+ expect_load("U48 LA57: data after second fetch", data, VEC_NONE,
+ MARK(1));
+
+ set_lam_cr3(CR3_LAM_U48);
+ expect_load("U48 LA57: data first", data, VEC_NONE, MARK(1));
+ expect_fault("U48 LA57: fetch after data first", try_exec(code, &ran),
+ VEC_NONE);
+ check(ran == 0x2222, "U48 LA57: fetch after data first ran frames[2]");
+
+ set_lam_cr3(0);
+}
+#endif
+
+/* INVLPG of the untagged address must flush every tagged alias. */
+static void test_invlpg_4k(const char *what, uint64_t lam, int sup,
+ uint64_t base, uint64_t a, uint64_t b)
+{
+ restore_frames();
+ set_lam_cr3(lam);
+ set_lam_sup(sup);
+
+ expect_load(what, a, VEC_NONE, MARK(1));
+ expect_load(what, b, VEC_NONE, MARK(1));
+ expect_load(what, base, VEC_NONE, MARK(1));
+
+ pt_t[0] = (uint64_t)frames[3] | PTE_FLAGS;
+ expect_fault(what, try_invlpg(base), VEC_NONE);
+
+ expect_load(what, a, VEC_NONE, MARK(4));
+ expect_load(what, b, VEC_NONE, MARK(4));
+ expect_load(what, base, VEC_NONE, MARK(4));
+
+ /* INVLPG of a tagged operand is a no-op and does not fault. */
+ expect_fault(what, try_invlpg(a), VEC_NONE);
+
+ restore_frames();
+ set_lam_cr3(0);
+ set_lam_sup(0);
+}
+
+/* The same for a 2 MiB page, with INVLPG of a different 4 KiB part. */
+static void test_invlpg_2m(void)
+{
+ uint64_t t = TAG57(0x2a);
+
+ restore_frames();
+ set_lam_cr3(CR3_LAM_U57);
+
+ expect_load("INVLPG 2M: before", (VA_LP + 0x5000) | t, VEC_NONE,
+ 0xaaaaaaaaaaaa5000ULL);
+ pd_t[1] = LP1_PHYS | PTE_PS | PTE_FLAGS;
+ expect_fault("INVLPG 2M: invlpg", try_invlpg(VA_LP + 0x1000),
+ VEC_NONE);
+ expect_load("INVLPG 2M: after, other 4K part", (VA_LP + 0x5000) | t,
+ VEC_NONE, 0xbbbbbbbbbbbb5000ULL);
+ expect_load("INVLPG 2M: after, first 4K part", VA_LP | t, VEC_NONE,
+ 0xbbbbbbbbbbbbbbbbULL);
+
+ restore_frames();
+ set_lam_cr3(0);
+}
+
+int main(void)
+{
+ int lam;
+
+ setup_idt();
+ setup_paging();
+
+#ifdef LA57
+ check(read_cr4() & CR4_LA57, "CR4.LA57 is set");
+#endif
+ lam = cpu_has_lam();
+ check(lam == EXPECT_LAM,
+ EXPECT_LAM ? "CPUID reports LAM" : "CPUID reports no LAM");
+ if (!lam) {
+ ml_printf("CPUID reports no LAM\n");
+ test_no_lam_cpu();
+ } else {
+ ml_printf("CPUID reports LAM\n");
+ test_lam_off();
+ test_cr3_readback();
+ test_u57();
+ test_u48();
+ test_sup();
+ test_invlpg_4k("INVLPG U57", CR3_LAM_U57, 0, VA_PAGE(0),
+ VA_PAGE(0) | TAG57(0x2a), VA_PAGE(0) | TAG57(0x15));
+ test_invlpg_4k("INVLPG U48", CR3_LAM_U48, 0, VA_PAGE(0),
+ VA_PAGE(0) | TAG48(0x1234),
+ VA_PAGE(0) | TAG48(0x42));
+ test_invlpg_4k("INVLPG SUP", 0, 1, VA_SUP, STAG(0x1a), STAG(0x05));
+ test_invlpg_2m();
+#ifdef LA57
+ test_u48_la57_split();
+#endif
+ }
+
+ ml_printf("%d passed, %d failed\n", passes, failures);
+ return failures ? 1 : 0;
+}
diff --git a/tests/tcg/x86_64/system/meson.build b/tests/tcg/x86_64/system/meson.build
index 46a35d2..b9d18bd 100644
--- a/tests/tcg/x86_64/system/meson.build
+++ b/tests/tcg/x86_64/system/meson.build
@@ -48,6 +48,21 @@ tests += {
}
}
+# The LAM test takes exceptions on the current stack, without CR4.OSFXSR.
+lam_cflags = cflags + ['-mno-red-zone', '-mgeneral-regs-only']
+foreach paging, extra: {'': [], '-la57': ['-DLA57']}
+ foreach suffix, cpu: {'': 'max', '-off': 'max,lam=off'}
+ tests += {
+ 'lam.c': {
+ 'exe_name': 'lam' + paging + suffix,
+ 'cflags': lam_cflags + extra +
+ ['-DEXPECT_LAM=' + (suffix == '' ? '1' : '0')],
+ 'qemu_args': ['-cpu', cpu] + qemu_def_args,
+ }
+ }
+ endforeach
+endforeach
+
if 'qemu-system-x86_64' in emulators
tcg_tests += {
'x86_64-softmmu': {
--
2.47.3
^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-10 13:02 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-10 13:01 [PATCH 0/4] target/i386: emulate LAM in TCG Tim
2026-11-10 13:01 ` [PATCH 1/4] target/i386: mask the PML5 table address taken from CR3 Tim
2026-11-10 13:01 ` [PATCH 2/4] accel/tcg: compare large pages under the flush mask Tim
2026-11-10 13:01 ` [PATCH 3/4] target/i386: emulate LAM in TCG Tim
2026-11-10 13:01 ` [PATCH 4/4] tests/tcg/x86_64: add a LAM system test Tim
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.