* [PATCH v2 01/20] KVM: x86/xen: Fill the hypercall page with a single guest write
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 02/20] KVM: x86/xen: Add KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE David Woodhouse
` (18 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
Writing the hypercall page as 128 separate 32-byte guest writes pushes
128 entries into the dirty ring for the same gfn. The ring's headroom
above the soft limit is KVM_DIRTY_RING_RSVD_ENTRIES (64) plus
kvm_cpu_dirty_log_size(), so when the CPU assists with dirty logging
(PML adds 512) a single WRMSR fits; without that assist (e.g. VMX with
PML disabled) one guest WRMSR can overflow it: a guest-reachable
WARN_ON_ONCE() in kvm_dirty_ring_push(), and an unharvested entry
overwritten, losing a page from dirty tracking during live migration.
Build the page in a buffer and write it with one call.
Fixes: fb04a1eddb1a ("KVM: X86: Implement ring-based dirty memory tracking")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kvm/xen.c | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 6aa00f746cba..5875d10ae3a9 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -1345,7 +1345,8 @@ int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data)
*/
if (kvm_xen_hypercall_enabled(kvm)) {
u8 instructions[32];
- int i;
+ u8 *page;
+ int ret, i;
if (page_num)
return 1;
@@ -1362,13 +1363,20 @@ int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data)
/* int3 to pad */
memset(instructions + 9, 0xcc, sizeof(instructions) - 9);
+ page = kmalloc(PAGE_SIZE, GFP_KERNEL);
+ if (!page)
+ return -ENOMEM;
+
for (i = 0; i < PAGE_SIZE / sizeof(instructions); i++) {
*(u32 *)&instructions[1] = i;
- if (kvm_vcpu_write_guest(vcpu,
- page_addr + (i * sizeof(instructions)),
- instructions, sizeof(instructions)))
- return 1;
+ memcpy(page + (i * sizeof(instructions)), instructions,
+ sizeof(instructions));
}
+
+ ret = kvm_vcpu_write_guest(vcpu, page_addr, page, PAGE_SIZE);
+ kfree(page);
+ if (ret)
+ return 1;
} else {
/*
* Note, truncation is a non-issue as 'lm' is guaranteed to be
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 02/20] KVM: x86/xen: Add KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
2026-11-05 2:00 ` [PATCH v2 01/20] KVM: x86/xen: Fill the hypercall page with a single guest write David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 03/20] KVM: x86/xen: Extract delivery of event to vCPU into a separate helper David Woodhouse
` (17 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
Commit 3617c0ee7dec ("KVM: x86/xen: Only write Xen hypercall page for
guest writes to MSR") blocked host-initiated writes from triggering the
Xen hypercall page setup, to fix an SRCU usage violation when the
hypercall MSR index collides with a real MSR written during vCPU reset.
However, some VMMs legitimately need to trigger hypercall page setup
from host context. For example, a VMM may intercept the guest's MSR
write to track an epoch (for kexec/crash recovery), and then replay the
write as a host-initiated KVM_SET_MSRS to populate the hypercall page.
The host_initiated check breaks this use case.
Add KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE as a vCPU attribute
which invokes the write explicitly. The work moves into a helper
called with xen_lock held — which the attribute handler already holds,
and which now covers the whole operation, serializing the writes to
guest memory — in the shape the original posting's review asked for:
handled in the attribute case-statement, with no locking games of its
own. The helper returns raw errnos for each caller to map onto its own
contract: the attribute returns them to userspace as they are, while
the guest WRMSR raises #GP for anything but -ENOMEM. That includes a
fault reading the VMM-supplied blob, which previously escaped to
userspace as -EFAULT: by that point it is indistinguishable from the
guest's own bad page_addr, and #GP serves them both. The helper also
reverts the latched long_mode when the shared_info re-init fails —
on both entry points, including the guest WRMSR path — where the old
code left it latched, letting a retry of the same write skip the
re-init and "succeed" with the wallclock still published at the
offset for the old bitness.
The Fixes: tag documents which commit regressed the use case; since
this patch adds new UAPI (an attribute and a capability bit), it is
not suitable for backporting to stable kernels.
Fixes: 3617c0ee7dec ("KVM: x86/xen: Only write Xen hypercall page for guest writes to MSR")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
Documentation/virt/kvm/api.rst | 22 +++-
arch/x86/include/uapi/asm/kvm.h | 3 +
arch/x86/kvm/x86.c | 3 +-
arch/x86/kvm/xen.c | 73 ++++++++---
.../selftests/kvm/x86/xen_vmcall_test.c | 118 ++++++++++++++++++
5 files changed, 203 insertions(+), 16 deletions(-)
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index e95c9cda085d..70818f1353c3 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -5848,6 +5848,23 @@ KVM_XEN_VCPU_ATTR_TYPE_UPCALL_VECTOR
vector configured with HVM_PARAM_CALLBACK_IRQ. It is disabled by
setting the vector to zero.
+KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE
+ This attribute is available when the KVM_CAP_XEN_HVM ioctl indicates
+ support for KVM_XEN_HVM_CONFIG_WRITE_HYPERCALL_PAGE. It populates the
+ Xen hypercall page at the guest physical address given in ``gpa``,
+ just as a guest write to the hypercall MSR would. This is for VMMs
+ which intercept the guest's MSR write (e.g. to track an epoch for
+ kexec/crash recovery) and then need to perform the write themselves:
+ host-initiated writes through KVM_SET_MSRS do not populate the page.
+ As for the guest's own write, the vCPU's current long mode is latched
+ as the VM-wide Xen long_mode, and the shared_info wallclock is
+ re-published if that changed. The value is interpreted exactly as the
+ guest's MSR write would be, not as a plain address: when Xen hypercall
+ interception is enabled, any non-zero bits below the page size make
+ the write fail with -EINVAL; without interception, those bits select
+ which page of the VMM-provided hypercall blob is written, and a page
+ number beyond the blob fails with -EINVAL.
+
4.129 KVM_XEN_VCPU_GET_ATTR
---------------------------
@@ -5861,7 +5878,8 @@ KVM_XEN_VCPU_ATTR_TYPE_UPCALL_VECTOR
Allows Xen vCPU attributes to be read. For the structure and types,
see KVM_XEN_VCPU_SET_ATTR above.
-The KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_ADJUST type may not be used
+The KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_ADJUST and
+KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE types may not be used
with the KVM_XEN_VCPU_GET_ATTR ioctl.
4.130 KVM_ARM_MTE_COPY_TAGS
@@ -9509,6 +9527,8 @@ PVHVM guests. Valid flags are::
#define KVM_XEN_HVM_CONFIG_EVTCHN_SEND (1 << 5)
#define KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG (1 << 6)
#define KVM_XEN_HVM_CONFIG_PVCLOCK_TSC_UNSTABLE (1 << 7)
+ #define KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA (1 << 8)
+ #define KVM_XEN_HVM_CONFIG_WRITE_HYPERCALL_PAGE (1 << 9)
The KVM_XEN_HVM_CONFIG_HYPERCALL_MSR flag indicates that the KVM_XEN_HVM_CONFIG
ioctl is available, for the guest to set its hypercall page.
diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kvm.h
index 00ce265391e1..e53a26370e50 100644
--- a/arch/x86/include/uapi/asm/kvm.h
+++ b/arch/x86/include/uapi/asm/kvm.h
@@ -596,6 +596,7 @@ struct kvm_x86_mce {
#define KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG (1 << 6)
#define KVM_XEN_HVM_CONFIG_PVCLOCK_TSC_UNSTABLE (1 << 7)
#define KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA (1 << 8)
+#define KVM_XEN_HVM_CONFIG_WRITE_HYPERCALL_PAGE (1 << 9)
#define KVM_XEN_MSR_MIN_INDEX 0x40000000u
#define KVM_XEN_MSR_MAX_INDEX 0x4fffffffu
@@ -704,6 +705,8 @@ struct kvm_xen_vcpu_attr {
#define KVM_XEN_VCPU_ATTR_TYPE_UPCALL_VECTOR 0x8
/* Available with KVM_CAP_XEN_HVM / KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA */
#define KVM_XEN_VCPU_ATTR_TYPE_VCPU_INFO_HVA 0x9
+/* Available with KVM_CAP_XEN_HVM / KVM_XEN_HVM_CONFIG_WRITE_HYPERCALL_PAGE */
+#define KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE 0xa
/* Secure Encrypted Virtualization command */
enum sev_cmd_id {
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 32563a91fba9..6a1f0404f029 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2315,7 +2315,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
KVM_XEN_HVM_CONFIG_EVTCHN_2LEVEL |
KVM_XEN_HVM_CONFIG_EVTCHN_SEND |
KVM_XEN_HVM_CONFIG_PVCLOCK_TSC_UNSTABLE |
- KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA;
+ KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA |
+ KVM_XEN_HVM_CONFIG_WRITE_HYPERCALL_PAGE;
if (sched_info_on())
r |= KVM_XEN_HVM_CONFIG_RUNSTATE |
KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG;
diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 5875d10ae3a9..2718da1320e0 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -33,6 +33,7 @@
static int kvm_xen_set_evtchn(struct kvm_xen_evtchn *xe, struct kvm *kvm);
static int kvm_xen_setattr_evtchn(struct kvm *kvm, struct kvm_xen_hvm_attr *data);
static bool kvm_xen_hcall_evtchn_send(struct kvm_vcpu *vcpu, u64 param, u64 *r);
+static int __kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data);
DEFINE_STATIC_KEY_DEFERRED_FALSE(kvm_xen_enabled, HZ);
@@ -1177,6 +1178,15 @@ int kvm_xen_vcpu_set_attr(struct kvm_vcpu *vcpu, struct kvm_xen_vcpu_attr *data)
}
break;
+ case KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE:
+ /* KVM cannot read a protected guest's bitness to latch it. */
+ if (vcpu->arch.guest_state_protected) {
+ r = -EINVAL;
+ break;
+ }
+ r = __kvm_xen_write_hypercall_page(vcpu, data->u.gpa);
+ break;
+
default:
break;
}
@@ -1312,30 +1322,42 @@ int kvm_xen_vcpu_get_attr(struct kvm_vcpu *vcpu, struct kvm_xen_vcpu_attr *data)
return r;
}
-int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data)
+/*
+ * Latch the guest's bitness, re-publish the wallclock in the right
+ * place for it if needed, and fill the hypercall page. Called with
+ * xen_lock held, which serializes the writes to guest memory, and
+ * within a kvm->srcu read section.
+ *
+ * Returns raw errnos for each caller to map onto its own contract:
+ * -EINVAL for an out-of-range page_num; -EFAULT for a fault writing
+ * the page, whether through the guest's page_addr or from the VMM's
+ * blob address; -ENOMEM from the page allocations.
+ */
+static int __kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data)
{
struct kvm *kvm = vcpu->kvm;
u32 page_num = data & ~PAGE_MASK;
u64 page_addr = data & PAGE_MASK;
bool lm = is_long_mode(vcpu);
- int r = 0;
- mutex_lock(&kvm->arch.xen.xen_lock);
+ lockdep_assert_held(&kvm->arch.xen.xen_lock);
+
if (kvm->arch.xen.long_mode != lm) {
kvm->arch.xen.long_mode = lm;
/*
* Re-initialize shared_info to put the wallclock in the
- * correct place.
+ * correct place. On failure, revert the latched long_mode:
+ * leaving it latched would make a retry of the same write
+ * skip this block and "succeed" with the wallclock still
+ * published at the offset for the old bitness.
*/
if (kvm->arch.xen.shinfo_cache.active &&
- kvm_xen_shared_info_init(kvm))
- r = 1;
+ kvm_xen_shared_info_init(kvm)) {
+ kvm->arch.xen.long_mode = !lm;
+ return -EFAULT;
+ }
}
- mutex_unlock(&kvm->arch.xen.xen_lock);
-
- if (r)
- return r;
/*
* If Xen hypercall intercept is enabled, fill the hypercall
@@ -1349,7 +1371,7 @@ int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data)
int ret, i;
if (page_num)
- return 1;
+ return -EINVAL;
/* mov imm32, %eax */
instructions[0] = 0xb8;
@@ -1376,7 +1398,7 @@ int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data)
ret = kvm_vcpu_write_guest(vcpu, page_addr, page, PAGE_SIZE);
kfree(page);
if (ret)
- return 1;
+ return ret;
} else {
/*
* Note, truncation is a non-issue as 'lm' is guaranteed to be
@@ -1390,7 +1412,7 @@ int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data)
int ret;
if (page_num >= blob_size)
- return 1;
+ return -EINVAL;
blob_addr += page_num * PAGE_SIZE;
@@ -1401,11 +1423,34 @@ int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data)
ret = kvm_vcpu_write_guest(vcpu, page_addr, page, PAGE_SIZE);
kfree(page);
if (ret)
- return 1;
+ return ret;
}
return 0;
}
+/* The guest WRMSR entry point. */
+int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data)
+{
+ struct kvm *kvm = vcpu->kvm;
+ int r;
+
+ mutex_lock(&kvm->arch.xen.xen_lock);
+ r = __kvm_xen_write_hypercall_page(vcpu, data);
+ mutex_unlock(&kvm->arch.xen.xen_lock);
+
+ /*
+ * Any failure to populate the page raises #GP; only -ENOMEM is
+ * signalled upwards. The one deliberate change: a fault reading
+ * the VMM-supplied blob previously escaped to userspace as
+ * -EFAULT, but by this point it is indistinguishable from the
+ * guest's own bad page_addr, and #GP serves them both.
+ */
+ if (r && r != -ENOMEM)
+ return 1;
+
+ return r;
+}
+
int kvm_xen_hvm_config(struct kvm *kvm, struct kvm_xen_hvm_config *xhc)
{
/* Only some feature flags need to be *enabled* by userspace */
diff --git a/tools/testing/selftests/kvm/x86/xen_vmcall_test.c b/tools/testing/selftests/kvm/x86/xen_vmcall_test.c
index 2585087cdf5c..85d836ddbdb8 100644
--- a/tools/testing/selftests/kvm/x86/xen_vmcall_test.c
+++ b/tools/testing/selftests/kvm/x86/xen_vmcall_test.c
@@ -12,6 +12,8 @@
#include "processor.h"
#include "hyperv.h"
+#include <string.h>
+
#define HCALL_REGION_GPA 0xc0000000ULL
#define HCALL_REGION_SLOT 10
@@ -26,6 +28,10 @@
#define HVCALL_SIGNAL_EVENT 0x005d
#define HV_STATUS_INVALID_ALIGNMENT 4
+enum {
+ TEST_WRITE_HYPERCALL_PAGE = 1,
+};
+
static void guest_code(void)
{
unsigned long rax = INPUTVALUE;
@@ -76,18 +82,74 @@ static void guest_code(void)
"r"(r8));
GUEST_ASSERT(rax == HV_STATUS_INVALID_ALIGNMENT);
+ /*
+ * Test KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE: ask userspace
+ * to set up MSR filtering, then write the MSR. The WRMSR will exit
+ * to userspace (not populate the page). Userspace verifies the page
+ * is empty, uses the attr to populate it, then resumes us.
+ */
+ GUEST_SYNC(TEST_WRITE_HYPERCALL_PAGE);
+
+ __asm__ __volatile__("wrmsr" : : "c" (XEN_HYPERCALL_MSR),
+ "a" (HCALL_REGION_GPA & 0xffffffff),
+ "d" (HCALL_REGION_GPA >> 32));
+
+ /* Userspace populated the page via the attr — verify it works */
+ rax = INPUTVALUE;
+ rdi = ARGVALUE(1);
+ rsi = ARGVALUE(2);
+ rdx = ARGVALUE(3);
+ r10 = ARGVALUE(4);
+ r8 = ARGVALUE(5);
+ r9 = ARGVALUE(6);
+ __asm__ __volatile__("call *%1" : "=a"(rax) :
+ "r"(HCALL_REGION_GPA + INPUTVALUE * 32),
+ "a"(rax), "D"(rdi), "S"(rsi), "d"(rdx),
+ "r"(r10), "r"(r8), "r"(r9));
+ GUEST_ASSERT(rax == RETVALUE);
+
GUEST_DONE();
}
+static void setup_msr_filter(struct kvm_vm *vm)
+{
+ uint64_t deny_bits = 0;
+ struct kvm_msr_filter filter = {
+ .flags = KVM_MSR_FILTER_DEFAULT_ALLOW,
+ .ranges = {
+ {
+ .flags = KVM_MSR_FILTER_WRITE,
+ .nmsrs = 1,
+ .base = XEN_HYPERCALL_MSR,
+ .bitmap = (uint8_t *)&deny_bits,
+ },
+ },
+ };
+
+ vm_ioctl(vm, KVM_X86_SET_MSR_FILTER, &filter);
+}
+
int main(int argc, char *argv[])
{
unsigned int xen_caps;
struct kvm_vcpu *vcpu;
struct kvm_vm *vm;
+ bool msr_filter_ready = false;
+ bool attr_tested = false;
xen_caps = kvm_check_cap(KVM_CAP_XEN_HVM);
TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL);
+ /*
+ * Without these, skip only the WRITE_HYPERCALL_PAGE leg: the MSR
+ * filter is never installed, so the guest's second WRMSR fills
+ * the page through KVM as the first did, and the rest of the
+ * test still runs on older kernels.
+ */
+ bool have_whp = (xen_caps & KVM_XEN_HVM_CONFIG_WRITE_HYPERCALL_PAGE) &&
+ kvm_check_cap(KVM_CAP_X86_USER_SPACE_MSR) &&
+ kvm_check_cap(KVM_CAP_X86_MSR_FILTER);
+
vm = vm_create_with_one_vcpu(&vcpu, guest_code);
vcpu_set_hv_cpuid(vcpu);
@@ -123,6 +185,42 @@ int main(int argc, char *argv[])
continue;
}
+ if (run->exit_reason == KVM_EXIT_X86_WRMSR) {
+ /* MSR filter caught the Xen hypercall MSR write */
+ TEST_ASSERT(msr_filter_ready,
+ "Unexpected WRMSR exit before filter setup");
+ TEST_ASSERT_EQ(run->msr.index, XEN_HYPERCALL_MSR);
+
+ /*
+ * The MSR filter should have prevented KVM from
+ * populating the page. Verify it's empty.
+ */
+ uint8_t *hcall_page = addr_gpa2hva(vm, HCALL_REGION_GPA);
+
+ TEST_ASSERT_EQ(hcall_page[0], 0);
+
+ /*
+ * Now use the attr to populate the page, as a
+ * VMM would after intercepting the MSR write.
+ */
+ struct kvm_xen_vcpu_attr attr = {
+ .type = KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE,
+ .u.gpa = HCALL_REGION_GPA,
+ };
+ vcpu_ioctl(vcpu, KVM_XEN_VCPU_SET_ATTR, &attr);
+
+ /* Verify the page is now populated */
+ TEST_ASSERT_EQ(hcall_page[0], 0xb8);
+ TEST_ASSERT_EQ(*(uint32_t *)&hcall_page[1], 0);
+ TEST_ASSERT_EQ(hcall_page[INPUTVALUE * 32], 0xb8);
+ TEST_ASSERT_EQ(*(uint32_t *)&hcall_page[INPUTVALUE * 32 + 1],
+ INPUTVALUE);
+ attr_tested = true;
+
+ run->msr.error = 0;
+ continue;
+ }
+
TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
switch (get_ucall(vcpu, &uc)) {
@@ -130,6 +228,21 @@ int main(int argc, char *argv[])
REPORT_GUEST_ASSERT(uc);
/* NOT REACHED */
case UCALL_SYNC:
+ TEST_ASSERT_EQ(uc.args[1], TEST_WRITE_HYPERCALL_PAGE);
+
+ if (!have_whp)
+ break;
+
+ /*
+ * Guest is about to write the Xen MSR. Clear the
+ * hypercall page, install an MSR filter to intercept
+ * the write, and enable userspace MSR exits.
+ */
+ memset(addr_gpa2hva(vm, HCALL_REGION_GPA), 0, PAGE_SIZE);
+ vm_enable_cap(vm, KVM_CAP_X86_USER_SPACE_MSR,
+ KVM_MSR_EXIT_REASON_FILTER);
+ setup_msr_filter(vm);
+ msr_filter_ready = true;
break;
case UCALL_DONE:
goto done;
@@ -138,6 +251,11 @@ int main(int argc, char *argv[])
}
}
done:
+ if (have_whp)
+ TEST_ASSERT(attr_tested,
+ "WRITE_HYPERCALL_PAGE attr was never exercised");
+ else
+ pr_info("Skipped KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE\n");
kvm_vm_free(vm);
return 0;
}
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 03/20] KVM: x86/xen: Extract delivery of event to vCPU into a separate helper
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
2026-11-05 2:00 ` [PATCH v2 01/20] KVM: x86/xen: Fill the hypercall page with a single guest write David Woodhouse
2026-11-05 2:00 ` [PATCH v2 02/20] KVM: x86/xen: Add KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 04/20] KVM: x86/xen: Explicitly tag "shared info" page as never being dirty tracked David Woodhouse
` (16 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: Sean Christopherson <seanjc@google.com>
Move the marking of an event as pending in the target vCPU's vcpu_info
into a separate helper, __kvm_xen_set_evtchn_fast(), invoked after the
shared_info processing is complete and the shared_info cache's read
lock has been dropped. This makes both halves easier to read.
Signed-off-by: Sean Christopherson <seanjc@google.com>
[dwmw2: rebased onto the -EAGAIN series, and reworded. Sean's series
first converted the GPC locks to read_trylock(), which this series
does not carry, so the helper takes the plain read lock and the
original's trylock failure path does not apply. The caller's
kvm->srcu section extends across the helper call, since
kvm_gpc_check() on the vcpu_info cache consults the memslot
generation and the irqfd path enters holding only irq_srcu; in
Sean's series that was covered by a guard(srcu) spanning the whole
function. The original said "no functional change intended", but
here the lock scopes do change: kvm_xen_check_poller() and the
upcall vector injection now run outside the GPC read locks (and
with interrupts restored), deliberately, so that the vCPU kick and
the APIC map walk are not nested inside them.]
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kvm/xen.c | 158 ++++++++++++++++++++++++++++-----------------
1 file changed, 97 insertions(+), 61 deletions(-)
diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 2718da1320e0..4517eeb24909 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -1884,6 +1884,89 @@ static void kvm_xen_check_poller(struct kvm_vcpu *vcpu, int port)
}
}
+static void __kvm_xen_set_evtchn_fast(struct kvm_vcpu *vcpu, int port_word_bit,
+ bool has_64bit_shinfo)
+{
+ struct gfn_to_pfn_cache *gpc = &vcpu->arch.xen.vcpu_info_cache;
+ unsigned long vi_pending_sel_ofs;
+ bool inject_upcall = false;
+ bool kick_vcpu = false;
+ unsigned long flags;
+ bool old;
+
+ vi_pending_sel_ofs = has_64bit_shinfo ?
+ offsetof(struct vcpu_info, evtchn_pending_sel) :
+ offsetof(struct compat_vcpu_info, evtchn_pending_sel);
+
+ /* Now switch to the vCPU's vcpu_info to set the index and pending_sel */
+ read_lock_irqsave(&gpc->lock, flags);
+ if (!kvm_gpc_check(gpc, sizeof(struct vcpu_info))) {
+ /*
+ * Could not access the vcpu_info. Set the bit in-kernel and
+ * prod the vCPU to deliver it for itself. Note that an MSI is
+ * *not* an acceptable substitute here even if the vCPU has an
+ * upcall vector: the pending bit only exists in
+ * evtchn_pending_sel so far, and the vCPU has to be forced
+ * out of the guest so that kvm_xen_inject_pending_events()
+ * can refresh the cache and propagate it to the guest.
+ */
+ if (!test_and_set_bit(port_word_bit, &vcpu->arch.xen.evtchn_pending_sel))
+ kick_vcpu = true;
+ goto out_unlock;
+ }
+ inject_upcall = true;
+
+ /*
+ * Explicitly use a 32-bit btsl instead of test_and_set_bit(), which
+ * would use btsq on x86-64. The vcpu_info is guest-controlled and only
+ * required to be 32-bit aligned, so a 64-bit access could generate a
+ * split-lock #AC.
+ *
+ * Note, this does not apply to the test_and_set_bit() on pending_bits
+ * in the caller: that is in the per-VM shared_info, which is page
+ * aligned, so the access is guaranteed to be 64-bit aligned.
+ */
+ old = GEN_BINARY_RMWcc(LOCK_PREFIX "btsl",
+ *(u32 *)(gpc->khva + vi_pending_sel_ofs),
+ c, "Ir", port_word_bit);
+ if (!old) {
+ struct vcpu_info *vi = gpc->khva;
+
+ /* No need for compat handling */
+ BUILD_BUG_ON(offsetof(struct vcpu_info, evtchn_upcall_pending) !=
+ offsetof(struct compat_vcpu_info, evtchn_upcall_pending));
+
+ WRITE_ONCE(vi->evtchn_upcall_pending, 1);
+ kick_vcpu = true;
+ }
+
+out_unlock:
+ read_unlock_irqrestore(&gpc->lock, flags);
+
+ /*
+ * Deliver the upcall or kick the vCPU only after dropping the GPC
+ * read lock. Both paths end up in kvm_vcpu_kick(), and the MSI
+ * delivery also walks the APIC map and takes APIC locks; none of
+ * that wants to be nested inside the GPC read lock, which a
+ * concurrent invalidation may be waiting to take for write.
+ */
+ if (!kick_vcpu)
+ return;
+
+ /*
+ * For the per-vCPU lapic vector, deliver it as MSI — but only if the
+ * vcpu_info was actually updated above. If it wasn't, the vCPU must
+ * be kicked instead (see above).
+ */
+ if (inject_upcall && vcpu->arch.xen.upcall_vector) {
+ kvm_xen_inject_vcpu_vector(vcpu);
+ return;
+ }
+
+ kvm_make_request(KVM_REQ_UNBLOCK, vcpu);
+ kvm_vcpu_kick(vcpu);
+}
+
/*
* The return value from this function is propagated to kvm_set_irq() API,
* so it returns:
@@ -1898,11 +1981,10 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, struct kvm *kvm)
{
struct gfn_to_pfn_cache *gpc = &kvm->arch.xen.shinfo_cache;
bool has_64bit_shinfo = kvm_xen_has_64bit_shinfo(kvm);
- unsigned long *pending_bits, *mask_bits, vi_pending_sel_ofs;
+ unsigned long *pending_bits, *mask_bits;
struct kvm_vcpu *vcpu;
unsigned long flags;
int port_word_bit;
- bool kick_vcpu = false;
int vcpu_idx, idx, rc;
vcpu_idx = READ_ONCE(xe->vcpu_idx);
@@ -1924,23 +2006,19 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, struct kvm *kvm)
read_lock_irqsave(&gpc->lock, flags);
if (!kvm_gpc_check(gpc, PAGE_SIZE))
- goto out_rcu;
+ goto out_unlock;
if (has_64bit_shinfo) {
struct shared_info *shinfo = gpc->khva;
pending_bits = (unsigned long *)&shinfo->evtchn_pending;
mask_bits = (unsigned long *)&shinfo->evtchn_mask;
port_word_bit = xe->port / 64;
-
- vi_pending_sel_ofs = offsetof(struct vcpu_info, evtchn_pending_sel);
} else {
struct compat_shared_info *shinfo = gpc->khva;
pending_bits = (unsigned long *)&shinfo->evtchn_pending;
mask_bits = (unsigned long *)&shinfo->evtchn_mask;
port_word_bit = xe->port / 32;
- vi_pending_sel_ofs = offsetof(struct compat_vcpu_info, evtchn_pending_sel);
-
/* test_and_set_bit() needs 64-bit alignment, but that's OK */
BUILD_BUG_ON(offsetof(struct compat_shared_info, evtchn_pending) & 7);
}
@@ -1956,66 +2034,24 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, struct kvm *kvm)
rc = 0; /* It was already raised */
} else if (test_bit(xe->port, mask_bits)) {
rc = -ENOTCONN; /* Masked */
- kvm_xen_check_poller(vcpu, xe->port);
} else {
- bool old;
-
rc = 1; /* Delivered to the bitmap in shared_info. */
- /* Now switch to the vCPU's vcpu_info to set the index and pending_sel */
- read_unlock_irqrestore(&gpc->lock, flags);
- gpc = &vcpu->arch.xen.vcpu_info_cache;
-
- read_lock_irqsave(&gpc->lock, flags);
- if (!kvm_gpc_check(gpc, sizeof(struct vcpu_info))) {
- /*
- * Could not access the vcpu_info. Set the bit in-kernel
- * and prod the vCPU to deliver it for itself.
- */
- if (!test_and_set_bit(port_word_bit, &vcpu->arch.xen.evtchn_pending_sel))
- kick_vcpu = true;
- goto out_rcu;
- }
-
- /*
- * Explicitly use a 32-bit btsl instead of test_and_set_bit(),
- * which would use btsq on x86-64. The vcpu_info is guest-
- * controlled and only required to be 32-bit aligned, so a
- * 64-bit access could generate a split-lock #AC.
- *
- * Note, this does not apply to the test_and_set_bit() on
- * pending_bits above: that is in the per-VM shared_info, which
- * is page aligned, so the access is guaranteed to be 64-bit
- * aligned.
- */
- old = GEN_BINARY_RMWcc(LOCK_PREFIX "btsl",
- *(u32 *)(gpc->khva + vi_pending_sel_ofs),
- c, "Ir", port_word_bit);
- if (!old) {
- struct vcpu_info *vi = gpc->khva;
-
- /* No need for compat handling */
- BUILD_BUG_ON(offsetof(struct vcpu_info, evtchn_upcall_pending) !=
- offsetof(struct compat_vcpu_info, evtchn_upcall_pending));
-
- WRITE_ONCE(vi->evtchn_upcall_pending, 1);
- kick_vcpu = true;
- }
-
- /* For the per-vCPU lapic vector, deliver it as MSI. */
- if (kick_vcpu && vcpu->arch.xen.upcall_vector) {
- kvm_xen_inject_vcpu_vector(vcpu);
- kick_vcpu = false;
- }
}
- out_rcu:
+ out_unlock:
read_unlock_irqrestore(&gpc->lock, flags);
- srcu_read_unlock(&kvm->srcu, idx);
- if (kick_vcpu) {
- kvm_make_request(KVM_REQ_UNBLOCK, vcpu);
- kvm_vcpu_kick(vcpu);
- }
+ /*
+ * Both of these can kick the vCPU, so keep them outside the GPC
+ * read lock; an invalidation may be waiting to take it for write.
+ */
+ if (rc == -ENOTCONN)
+ kvm_xen_check_poller(vcpu, xe->port);
+ else if (rc == 1)
+ __kvm_xen_set_evtchn_fast(vcpu, port_word_bit,
+ has_64bit_shinfo);
+
+ srcu_read_unlock(&kvm->srcu, idx);
return rc;
}
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 04/20] KVM: x86/xen: Explicitly tag "shared info" page as never being dirty tracked
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (2 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 03/20] KVM: x86/xen: Extract delivery of event to vCPU into a separate helper David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 05/20] KVM: x86/xen: Don't dirty track "vCPU info" page David Woodhouse
` (15 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: Sean Christopherson <seanjc@google.com>
Explicitly mark the Xen shared info page as never being dirty tracked so
that higher-level gpc APIs can be added to automatically take care of
things like dirty tracking, without reintroducing the bug fixed by
commit 55749769fe60 ("KVM: x86: Fix wall clock writes in Xen shared_info
not to mark page dirty"). And because the code _looks_ buggy.
No functional change intended.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kvm/xen.c | 2 +-
include/linux/kvm_host.h | 15 +++++++++++----
include/linux/kvm_types.h | 1 +
virt/kvm/pfncache.c | 4 +++-
4 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 4517eeb24909..d3773416cd44 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -2485,7 +2485,7 @@ void kvm_xen_init_vm(struct kvm *kvm)
{
mutex_init(&kvm->arch.xen.xen_lock);
xa_init(&kvm->arch.xen.evtchn_ports);
- kvm_gpc_init(&kvm->arch.xen.shinfo_cache, kvm);
+ __kvm_gpc_init(&kvm->arch.xen.shinfo_cache, kvm, true);
}
void kvm_xen_destroy_vm(struct kvm *kvm)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index e7fc94e2910a..cadd0318400e 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1459,16 +1459,23 @@ int kvm_vcpu_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa, const void *data,
unsigned long len);
/**
- * kvm_gpc_init - initialize gfn_to_pfn_cache.
+ * __kvm_gpc_init - initialize gfn_to_pfn_cache.
*
* @gpc: struct gfn_to_pfn_cache object.
* @kvm: pointer to kvm instance.
+ * @never_dirty: the cache never needs to mark its page dirty in its memslot.
*
* This sets up a gfn_to_pfn_cache by initializing locks and assigning the
* immutable attributes. Note, the cache must be zero-allocated (or zeroed by
- * the caller before init).
+ * the caller before init). Most callers want the kvm_gpc_init() wrapper.
*/
-void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm);
+void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm,
+ bool never_dirty);
+
+static inline void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm)
+{
+ __kvm_gpc_init(gpc, kvm, false);
+}
/**
* kvm_gpc_activate - prepare a cached kernel mapping and HPA for a given guest
@@ -1979,7 +1986,7 @@ static inline void kvm_gpc_mark_dirty_in_slot(struct gfn_to_pfn_cache *gpc)
{
lockdep_assert_held(&gpc->lock);
- if (!gpc->memslot)
+ if (!gpc->memslot || gpc->never_dirty)
return;
mark_page_dirty_in_slot(gpc->kvm, gpc->memslot, gpa_to_gfn(gpc->gpa));
diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h
index a568d8e6f4e8..e850adc3f47e 100644
--- a/include/linux/kvm_types.h
+++ b/include/linux/kvm_types.h
@@ -94,6 +94,7 @@ struct gfn_to_pfn_cache {
kvm_pfn_t pfn;
bool active;
bool valid;
+ bool never_dirty;
};
#ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE
diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c
index 3659686b97c2..17a30da00d40 100644
--- a/virt/kvm/pfncache.c
+++ b/virt/kvm/pfncache.c
@@ -382,7 +382,8 @@ int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsigned long len)
return __kvm_gpc_refresh(gpc, gpc->gpa, uhva);
}
-void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm)
+void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm,
+ bool never_dirty)
{
rwlock_init(&gpc->lock);
mutex_init(&gpc->refresh_lock);
@@ -392,6 +393,7 @@ void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm)
gpc->gpa = INVALID_GPA;
gpc->uhva = KVM_HVA_ERR_BAD;
gpc->active = gpc->valid = false;
+ gpc->never_dirty = never_dirty;
}
static int __kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned long uhva,
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 05/20] KVM: x86/xen: Don't dirty track "vCPU info" page
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (3 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 04/20] KVM: x86/xen: Explicitly tag "shared info" page as never being dirty tracked David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 06/20] KVM: x86: Request the guest TLB flush from record_steal_time() David Woodhouse
` (14 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: Sean Christopherson <seanjc@google.com>
Give the Xen per-vCPU info page the same treatment as the per-VM shared
info page, and never mark it dirty, as KVM clearly relies on userspace to
assume the page is always dirty. While the page is marked dirty on writes
via kvm_xen_inject_pending_events(), it's not marked dirty when written by
__kvm_xen_set_evtchn_fast().
Furthermore, as was the case with the shared info page, writes in the event
channel fastpath may be done without an active vCPU, e.g. when called via
timer callback or irqfd injection. I.e. attempting to fix the fastpath
would run afoul of the same issue that was fixed by
commit 55749769fe60 ("KVM: x86: Fix wall clock writes in Xen shared_info
not to mark page dirty").
This also covers the pvclock writes kvm_guest_time_update() makes
through the same cache; userspace assuming the page is always dirty
covers those equally.
Signed-off-by: Sean Christopherson <seanjc@google.com>
[dwmw2: also update the KVM_GET_DIRTY_LOG note in api.rst, which
named only the shared_info page]
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
Documentation/virt/kvm/api.rst | 5 +++--
arch/x86/kvm/xen.c | 3 +--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 70818f1353c3..6f0500f7ffe2 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -385,8 +385,9 @@ The bits in the dirty bitmap are cleared before the ioctl returns, unless
KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2 is enabled. For more information,
see the description of the capability.
-Note that the Xen shared_info page, if configured, shall always be assumed
-to be dirty. KVM will not explicitly mark it such.
+Note that the Xen shared_info page and vcpu_info area, if configured,
+shall always be assumed to be dirty. KVM will not explicitly mark them
+such.
4.10 KVM_RUN
diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index d3773416cd44..e54988522b8b 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -706,7 +706,6 @@ void kvm_xen_inject_pending_events(struct kvm_vcpu *v)
WRITE_ONCE(vi->evtchn_upcall_pending, 1);
}
- kvm_gpc_mark_dirty_in_slot(gpc);
read_unlock_irqrestore(&gpc->lock, flags);
/* For the per-vCPU lapic vector, deliver it as MSI. */
@@ -2464,7 +2463,7 @@ void kvm_xen_init_vcpu(struct kvm_vcpu *vcpu)
kvm_gpc_init(&vcpu->arch.xen.runstate_cache, vcpu->kvm);
kvm_gpc_init(&vcpu->arch.xen.runstate2_cache, vcpu->kvm);
- kvm_gpc_init(&vcpu->arch.xen.vcpu_info_cache, vcpu->kvm);
+ __kvm_gpc_init(&vcpu->arch.xen.vcpu_info_cache, vcpu->kvm, true);
kvm_gpc_init(&vcpu->arch.xen.vcpu_time_info_cache, vcpu->kvm);
}
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 06/20] KVM: x86: Request the guest TLB flush from record_steal_time()
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (4 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 05/20] KVM: x86/xen: Don't dirty track "vCPU info" page David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 07/20] KVM: selftests: Actually deactivate the HVA-based shared_info cache in xen_shinfo_test David Woodhouse
` (13 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
record_steal_time() performs a TLB flush on the guest's behalf when the
guest sets KVM_VCPU_FLUSH_TLB. With shadow paging,
kvm_vcpu_flush_tlb_guest() ends up in mmu_sync_children(), which can
reschedule via cond_resched_rwlock_write(). That is why the flush
currently sits between a user_access_end() and a fresh
user_access_begin(): it cannot be done while the guest page is mapped
for access.
Raise KVM_REQ_TLB_FLUSH_GUEST instead of flushing inline, and move the
processing of KVM_REQ_STEAL_UPDATE ahead of where the TLB flush requests
are serviced in vcpu_enter_guest() so that the flush still happens before
the vCPU enters the guest. There is no requirement for it to complete any
earlier than that.
kvm_make_request() on the vCPU which is running is nothing more than a
barrier and a set_bit(), so unlike the flush itself it can be done from
any context — which lets the surrounding access be simplified later.
Note that the xchg() must stay where it is: it atomically consumes any
KVM_VCPU_FLUSH_TLB which the guest has set, as required by commit
b043138246a4 ("x86/KVM: Make sure KVM_VCPU_FLUSH_TLB flag is not missed").
Reported-by: Sashiko AI review <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260530061932.7849D1F00893@smtp.kernel.org
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kvm/x86.c | 26 +++++++++++++++++++++++---
1 file changed, 23 insertions(+), 3 deletions(-)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 6a1f0404f029..3c26c0eb9865 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2117,8 +2117,22 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
trace_kvm_pv_tlb_flush(vcpu->vcpu_id,
st_preempted & KVM_VCPU_FLUSH_TLB);
+ /*
+ * Request the flush instead of performing it inline. With
+ * shadow paging kvm_vcpu_flush_tlb_guest() can reschedule
+ * (via mmu_sync_children()), so it could not be called while
+ * the user access above was open. Requesting is just a bit
+ * set on the running vCPU, and the request is serviced
+ * before the vCPU enters the guest; nothing requires the
+ * flush to have completed any earlier than that.
+ *
+ * Note the xchg above must remain as-is: it atomically
+ * consumes any KVM_VCPU_FLUSH_TLB the guest set, as required
+ * by commit b043138246a4 ("x86/KVM: Make sure
+ * KVM_VCPU_FLUSH_TLB flag is not missed").
+ */
if (st_preempted & KVM_VCPU_FLUSH_TLB)
- kvm_vcpu_flush_tlb_guest(vcpu);
+ kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
if (!user_access_begin(st, sizeof(*st)))
goto dirty;
@@ -8168,6 +8182,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
if (unlikely(r))
goto out;
}
+ /*
+ * Process the steal time update before the TLB flush requests
+ * are serviced below; record_steal_time() requests
+ * KVM_REQ_TLB_FLUSH_GUEST on the guest's behalf rather than
+ * performing the flush itself.
+ */
+ if (kvm_check_request(KVM_REQ_STEAL_UPDATE, vcpu))
+ record_steal_time(vcpu);
if (kvm_check_request(KVM_REQ_MMU_SYNC, vcpu))
kvm_mmu_sync_roots(vcpu);
if (kvm_check_request(KVM_REQ_LOAD_MMU_PGD, vcpu))
@@ -8217,8 +8239,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
r = 1;
goto out;
}
- if (kvm_check_request(KVM_REQ_STEAL_UPDATE, vcpu))
- record_steal_time(vcpu);
if (kvm_check_request(KVM_REQ_PMU, vcpu))
kvm_pmu_handle_event(vcpu);
if (kvm_check_request(KVM_REQ_PMI, vcpu))
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 07/20] KVM: selftests: Actually deactivate the HVA-based shared_info cache in xen_shinfo_test
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (5 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 06/20] KVM: x86: Request the guest TLB flush from record_steal_time() David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 08/20] KVM: pfncache: Use atomic SRCU for readers instead of a rwlock David Woodhouse
` (12 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
The "deactivate by HVA" leg of juggle_shinfo_state() uses attribute
type KVM_XEN_ATTR_TYPE_SHARED_INFO with .u.shared_info.hva = 0. The
hva field shares a union with gfn, so this activates the shared_info
cache at guest GFN 0 — zero is not KVM_XEN_INVALID_GFN — instead of
deactivating it.
The juggle thread therefore cycles through an unintended fourth
state, with three consequences:
- Every "deactivation" splats wallclock data over guest frame 0.
- The HVA-based deactivation path is never tested at all.
- In the TEST_LOCKING_POLL_RACE phase, a SCHEDOP_poll which samples
the shared_info in this state finds evtchn_pending[0] bit 0 clear
(the real shared_info page has it set; frame 0 does not) and
blocks in kvm_vcpu_halt() — the only state in which that poll can
block, as a deactivated cache fails kvm_gpc_check() and reports
the port pending. If the timer event is then dropped, nothing
ever wakes the guest and the test hangs.
Use KVM_XEN_ATTR_TYPE_SHARED_INFO_HVA, where hva = 0 deactivates.
Fixes: 9397b5334af1 ("KVM: selftests: map Xen's shared_info page using HVA rather than GFN")
Reported-by: sashiko-bot@kernel.org
Closes: https://lore.kernel.org/all/sashiko-outbox-177931@kernel.org
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
tools/testing/selftests/kvm/x86/xen_shinfo_test.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/testing/selftests/kvm/x86/xen_shinfo_test.c b/tools/testing/selftests/kvm/x86/xen_shinfo_test.c
index aaa414c677ad..fb3c834c7cb7 100644
--- a/tools/testing/selftests/kvm/x86/xen_shinfo_test.c
+++ b/tools/testing/selftests/kvm/x86/xen_shinfo_test.c
@@ -410,7 +410,7 @@ static void *juggle_shinfo_state(void *arg)
};
struct kvm_xen_hvm_attr cache_deactivate_hva = {
- .type = KVM_XEN_ATTR_TYPE_SHARED_INFO,
+ .type = KVM_XEN_ATTR_TYPE_SHARED_INFO_HVA,
.u.shared_info.hva = 0
};
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 08/20] KVM: pfncache: Use atomic SRCU for readers instead of a rwlock
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (6 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 07/20] KVM: selftests: Actually deactivate the HVA-based shared_info cache in xen_shinfo_test David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 09/20] KVM: x86: Use gfn_to_pfn_cache for steal time / preempted status David Woodhouse
` (11 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
Replace the per-cache rwlock with a dedicated per-VM atomic SRCU
domain (kvm->gpc_srcu) for the read side. Readers now run under
srcu_read_lock_atomic() alone: preemption disabled and non-blocking
enforced, which every GPC read-side section already satisfied as a
former rwlock critical section. It works in any context (including
hardirq and sched-out paths) and never fails or spins, eliminating
the read_trylock() contortions in the atomic paths, the double-lock
dance with the lockdep subclass hack in the runstate update, and the
PREEMPT_RT problems inherent to taking an rwlock in those contexts.
In particular, kvm_xen_set_evtchn_fast() is called from hardirq
context (timer callback, kvm_arch_set_irq_inatomic()), where taking
gpc->lock is a sleeping-lock-in-atomic-context bug on PREEMPT_RT.
The mutators' reader drains use synchronize_srcu_atomic(), which
spins for the bounded readers instead of sleeping: no path waiting
out GPC readers ever schedules. That satisfies the OOM-reaper
non-blocking requirement in the invalidation walk, whose wait must
happen even on unblockable ranges — GPC readers never allocate, take
mmap_lock, or sleep, so waiting them out cannot deadlock against
reclaim. SRCU rather than plain RCU so that wait covers only this
VM's GPC readers, not a grace period across every pCPU in the
system; the read-side cost is a this_cpu_inc() and a barrier.
Without gpc->lock, the refresh can no longer rely on holding it to
make its final invalidation check and its publish atomic with
respect to the mmu_notifier walk, and can no longer treat "GPC_VALID
already clear" as "no readers remain". Three state bits in a single
atomic word replace it: GPC_BECOMING_VALID lets the walk veto an
in-flight refresh's pending publish (the publish is a cmpxchg which
fails if the bit was cleared, closing the window between the
refresh's last check and its publish); GPC_INVALIDATING records that
a grace period for the previously published mapping is still owed:
whoever clears GPC_VALID leaves the marker standing until their
grace-period wait completes, so that a later mutator — or a later
mmu_notifier walk, whose return is what lets the primary MMU free
the page — finding GPC_VALID already clear still performs a wait of
its own rather than trusting one it cannot see finish. The comments
in pfncache.c and kvm_types.h carry the full protocol.
As before, a refresh which resolves to the same uHVA in the same
memslot leaves the cache valid and undisturbed, skipping the grace
period: routine host memory management (page migration, NUMA
balancing, KSM) usually re-backs the same uHVA, and a grace period
in the reader's refresh path for each such event would add seconds
to a guest boot.
Not suitable for stable backport despite the Fixes tag (it depends
on the atomic SRCU reader flavour, new in this cycle); the tag
documents where the PREEMPT_RT sleeping-lock problem was introduced.
Fixes: 14243b387137 ("KVM: x86/xen: Add KVM_IRQ_ROUTING_XEN_EVTCHN and event channel delivery")
Reported-by: syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=208f7f3e5f59c11aeb90
Reported-by: syzbot+919877893c9d28162dc2@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=919877893c9d28162dc2
Tested-by: syzbot+919877893c9d28162dc2@syzkaller.appspotmail.com
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kvm/x86.c | 10 +-
arch/x86/kvm/xen.c | 146 ++++++------
include/linux/kvm_host.h | 26 ++-
include/linux/kvm_types.h | 37 ++-
include/linux/srcu.h | 7 +
virt/kvm/kvm_main.c | 9 +
virt/kvm/pfncache.c | 481 ++++++++++++++++++++++++++++++--------
7 files changed, 528 insertions(+), 188 deletions(-)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 3c26c0eb9865..f5f3459ed4a5 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1747,18 +1747,18 @@ static void kvm_setup_guest_pvclock(struct pvclock_vcpu_time_info *ref_hv_clock,
{
struct pvclock_vcpu_time_info *guest_hv_clock;
struct pvclock_vcpu_time_info hv_clock;
- unsigned long flags;
+ int idx;
memcpy(&hv_clock, ref_hv_clock, sizeof(hv_clock));
- read_lock_irqsave(&gpc->lock, flags);
+ idx = srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu);
while (!kvm_gpc_check(gpc, offset + sizeof(*guest_hv_clock))) {
- read_unlock_irqrestore(&gpc->lock, flags);
+ srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx);
if (kvm_gpc_refresh(gpc, offset + sizeof(*guest_hv_clock)))
return;
- read_lock_irqsave(&gpc->lock, flags);
+ idx = srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu);
}
guest_hv_clock = (void *)(gpc->khva + offset);
@@ -1783,7 +1783,7 @@ static void kvm_setup_guest_pvclock(struct pvclock_vcpu_time_info *ref_hv_clock,
guest_hv_clock->version = ++hv_clock.version;
kvm_gpc_mark_dirty_in_slot(gpc);
- read_unlock_irqrestore(&gpc->lock, flags);
+ srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx);
trace_kvm_pvclock_update(vcpu->vcpu_id, &hv_clock);
}
diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index e54988522b8b..13e287bc17bb 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -46,16 +46,17 @@ static int kvm_xen_shared_info_init(struct kvm *kvm)
u64 wall_nsec;
int ret = 0;
int idx = srcu_read_lock(&kvm->srcu);
+ int gpc_idx;
- read_lock_irq(&gpc->lock);
+ gpc_idx = srcu_read_lock_atomic(&kvm->gpc_srcu);
while (!kvm_gpc_check(gpc, PAGE_SIZE)) {
- read_unlock_irq(&gpc->lock);
+ srcu_read_unlock_atomic(&kvm->gpc_srcu, gpc_idx);
ret = kvm_gpc_refresh(gpc, PAGE_SIZE);
if (ret)
goto out;
- read_lock_irq(&gpc->lock);
+ gpc_idx = srcu_read_lock_atomic(&kvm->gpc_srcu);
}
/*
@@ -98,7 +99,7 @@ static int kvm_xen_shared_info_init(struct kvm *kvm)
smp_wmb();
wc->version = wc_version + 1;
- read_unlock_irq(&gpc->lock);
+ srcu_read_unlock_atomic(&kvm->gpc_srcu, gpc_idx);
out:
srcu_read_unlock(&kvm->srcu, idx);
@@ -155,22 +156,21 @@ static int xen_get_guest_pvclock(struct kvm_vcpu *vcpu,
struct gfn_to_pfn_cache *gpc,
unsigned int offset)
{
- unsigned long flags;
- int r;
+ int idx, r;
- read_lock_irqsave(&gpc->lock, flags);
+ idx = srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu);
while (!kvm_gpc_check(gpc, offset + sizeof(*hv_clock))) {
- read_unlock_irqrestore(&gpc->lock, flags);
+ srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx);
r = kvm_gpc_refresh(gpc, offset + sizeof(*hv_clock));
if (r)
return r;
- read_lock_irqsave(&gpc->lock, flags);
+ idx = srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu);
}
memcpy(hv_clock, gpc->khva + offset, sizeof(*hv_clock));
- read_unlock_irqrestore(&gpc->lock, flags);
+ srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx);
/*
* Sanity check TSC shift+multiplier to verify the guest's view of time
@@ -325,12 +325,12 @@ static void kvm_xen_update_runstate_guest(struct kvm_vcpu *v, bool atomic)
struct gfn_to_pfn_cache *gpc2 = &vx->runstate2_cache;
size_t user_len, user_len1, user_len2;
struct vcpu_runstate_info rs;
- unsigned long flags;
size_t times_ofs;
uint8_t *update_bit = NULL;
uint64_t entry_time;
uint64_t *rs_times;
int *rs_state;
+ int idx;
/*
* The only difference between 32-bit and 64-bit versions of the
@@ -417,20 +417,12 @@ static void kvm_xen_update_runstate_guest(struct kvm_vcpu *v, bool atomic)
retry:
/*
- * Attempt to obtain the GPC lock on *both* (if there are two)
- * gfn_to_pfn caches that cover the region.
+ * Check *both* (if there are two) gfn_to_pfn caches that cover
+ * the region, under a single SRCU read-side critical section.
*/
- if (atomic) {
- local_irq_save(flags);
- if (!read_trylock(&gpc1->lock)) {
- local_irq_restore(flags);
- return;
- }
- } else {
- read_lock_irqsave(&gpc1->lock, flags);
- }
+ idx = srcu_read_lock_atomic(&v->kvm->gpc_srcu);
while (!kvm_gpc_check(gpc1, user_len1)) {
- read_unlock_irqrestore(&gpc1->lock, flags);
+ srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx);
/* When invoked from kvm_sched_out() we cannot sleep */
if (atomic)
@@ -439,7 +431,7 @@ static void kvm_xen_update_runstate_guest(struct kvm_vcpu *v, bool atomic)
if (kvm_gpc_refresh(gpc1, user_len1))
return;
- read_lock_irqsave(&gpc1->lock, flags);
+ idx = srcu_read_lock_atomic(&v->kvm->gpc_srcu);
}
if (likely(!user_len2)) {
@@ -459,24 +451,11 @@ static void kvm_xen_update_runstate_guest(struct kvm_vcpu *v, bool atomic)
} else {
/*
* The guest's runstate_info is split across two pages and we
- * need to hold and validate both GPCs simultaneously. We can
- * declare a lock ordering GPC1 > GPC2 because nothing else
- * takes them more than one at a time. Set a subclass on the
- * gpc1 lock to make lockdep shut up about it.
+ * need to validate both GPCs simultaneously. They are both
+ * covered by the single SRCU read-side critical section above.
*/
- lock_set_subclass(&gpc1->lock.dep_map, 1, _THIS_IP_);
- if (atomic) {
- if (!read_trylock(&gpc2->lock)) {
- read_unlock_irqrestore(&gpc1->lock, flags);
- return;
- }
- } else {
- read_lock(&gpc2->lock);
- }
-
if (!kvm_gpc_check(gpc2, user_len2)) {
- read_unlock(&gpc2->lock);
- read_unlock_irqrestore(&gpc1->lock, flags);
+ srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx);
/* When invoked from kvm_sched_out() we cannot sleep */
if (atomic)
@@ -575,13 +554,11 @@ static void kvm_xen_update_runstate_guest(struct kvm_vcpu *v, bool atomic)
smp_wmb();
}
- if (user_len2) {
+ if (user_len2)
kvm_gpc_mark_dirty_in_slot(gpc2);
- read_unlock(&gpc2->lock);
- }
kvm_gpc_mark_dirty_in_slot(gpc1);
- read_unlock_irqrestore(&gpc1->lock, flags);
+ srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx);
}
void kvm_xen_update_runstate(struct kvm_vcpu *v, int state)
@@ -656,7 +633,7 @@ void kvm_xen_inject_pending_events(struct kvm_vcpu *v)
{
unsigned long evtchn_pending_sel = READ_ONCE(v->arch.xen.evtchn_pending_sel);
struct gfn_to_pfn_cache *gpc = &v->arch.xen.vcpu_info_cache;
- unsigned long flags;
+ int idx;
if (!evtchn_pending_sel)
return;
@@ -666,14 +643,14 @@ void kvm_xen_inject_pending_events(struct kvm_vcpu *v)
* does anyway. Page it in and retry the instruction. We're just a
* little more honest about it.
*/
- read_lock_irqsave(&gpc->lock, flags);
+ idx = srcu_read_lock_atomic(&v->kvm->gpc_srcu);
while (!kvm_gpc_check(gpc, sizeof(struct vcpu_info))) {
- read_unlock_irqrestore(&gpc->lock, flags);
+ srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx);
if (kvm_gpc_refresh(gpc, sizeof(struct vcpu_info)))
return;
- read_lock_irqsave(&gpc->lock, flags);
+ idx = srcu_read_lock_atomic(&v->kvm->gpc_srcu);
}
/* Now gpc->khva is a valid kernel address for the vcpu_info */
@@ -706,7 +683,7 @@ void kvm_xen_inject_pending_events(struct kvm_vcpu *v)
WRITE_ONCE(vi->evtchn_upcall_pending, 1);
}
- read_unlock_irqrestore(&gpc->lock, flags);
+ srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx);
/* For the per-vCPU lapic vector, deliver it as MSI. */
if (v->arch.xen.upcall_vector)
@@ -716,8 +693,8 @@ void kvm_xen_inject_pending_events(struct kvm_vcpu *v)
int __kvm_xen_has_interrupt(struct kvm_vcpu *v)
{
struct gfn_to_pfn_cache *gpc = &v->arch.xen.vcpu_info_cache;
- unsigned long flags;
u8 rc = 0;
+ int idx;
/*
* If the global upcall vector (HVMIRQ_callback_vector) is set and
@@ -742,9 +719,9 @@ int __kvm_xen_has_interrupt(struct kvm_vcpu *v)
*/
guard(srcu)(&v->kvm->srcu);
- read_lock_irqsave(&gpc->lock, flags);
+ idx = srcu_read_lock_atomic(&v->kvm->gpc_srcu);
while (!kvm_gpc_check(gpc, sizeof(struct vcpu_info))) {
- read_unlock_irqrestore(&gpc->lock, flags);
+ srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx);
/*
* This function gets called from kvm_vcpu_block() after setting the
@@ -764,11 +741,11 @@ int __kvm_xen_has_interrupt(struct kvm_vcpu *v)
*/
return 0;
}
- read_lock_irqsave(&gpc->lock, flags);
+ idx = srcu_read_lock_atomic(&v->kvm->gpc_srcu);
}
rc = ((struct vcpu_info *)gpc->khva)->evtchn_upcall_pending;
- read_unlock_irqrestore(&gpc->lock, flags);
+ srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx);
return rc;
}
@@ -803,6 +780,13 @@ int kvm_xen_hvm_set_attr(struct kvm *kvm, struct kvm_xen_hvm_attr *data)
mutex_lock(&kvm->arch.xen.xen_lock);
+ /*
+ * The cache's own readers hold kvm->gpc_srcu, but activating
+ * it resolves the gfn through kvm_memslots(), which is
+ * protected by kvm->srcu. (Holding kvm->srcu across the
+ * deactivation is also fine: its grace-period wait is on the
+ * gpc_srcu domain, not this one.)
+ */
idx = srcu_read_lock(&kvm->srcu);
if (data->type == KVM_XEN_ATTR_TYPE_SHARED_INFO) {
@@ -937,6 +921,14 @@ int kvm_xen_vcpu_set_attr(struct kvm_vcpu *vcpu, struct kvm_xen_vcpu_attr *data)
int idx, r = -ENOENT;
mutex_lock(&vcpu->kvm->arch.xen.xen_lock);
+
+ /*
+ * Activating the caches below resolves gfns through kvm_memslots(),
+ * and the runstate cases read it via kvm_gpc_check(); both are
+ * protected by kvm->srcu (the caches' readers hold kvm->gpc_srcu,
+ * which does not cover the memslots, and whose grace-period waits
+ * are safe to perform with kvm->srcu held).
+ */
idx = srcu_read_lock(&vcpu->kvm->srcu);
switch (data->type) {
@@ -1532,14 +1524,13 @@ static bool wait_pending_event(struct kvm_vcpu *vcpu, bool has_64bit_shinfo,
struct kvm *kvm = vcpu->kvm;
struct gfn_to_pfn_cache *gpc = &kvm->arch.xen.shinfo_cache;
unsigned long *pending_bits;
- unsigned long flags;
bool ret = true;
- int idx, i;
+ int idx, gpc_idx, i;
idx = srcu_read_lock(&kvm->srcu);
- read_lock_irqsave(&gpc->lock, flags);
+ gpc_idx = srcu_read_lock_atomic(&kvm->gpc_srcu);
if (!kvm_gpc_check(gpc, PAGE_SIZE))
- goto out_rcu;
+ goto out_gpc_srcu;
ret = false;
if (has_64bit_shinfo) {
@@ -1557,8 +1548,8 @@ static bool wait_pending_event(struct kvm_vcpu *vcpu, bool has_64bit_shinfo,
}
}
- out_rcu:
- read_unlock_irqrestore(&gpc->lock, flags);
+ out_gpc_srcu:
+ srcu_read_unlock_atomic(&kvm->gpc_srcu, gpc_idx);
srcu_read_unlock(&kvm->srcu, idx);
return ret;
@@ -1883,6 +1874,15 @@ static void kvm_xen_check_poller(struct kvm_vcpu *vcpu, int port)
}
}
+/*
+ * Called with kvm->srcu held, as kvm_gpc_check() consults the memslots.
+ *
+ * The caller passes its own has_64bit_shinfo snapshot, from which it also
+ * derived port_word_bit; re-reading the mode here could race with a
+ * concurrent mode flip and pair a 64-bit bit index with the compat
+ * evtchn_pending_sel offset (or vice versa), writing outside the intended
+ * field within the guest's vcpu_info.
+ */
static void __kvm_xen_set_evtchn_fast(struct kvm_vcpu *vcpu, int port_word_bit,
bool has_64bit_shinfo)
{
@@ -1890,15 +1890,15 @@ static void __kvm_xen_set_evtchn_fast(struct kvm_vcpu *vcpu, int port_word_bit,
unsigned long vi_pending_sel_ofs;
bool inject_upcall = false;
bool kick_vcpu = false;
- unsigned long flags;
bool old;
+ int idx;
vi_pending_sel_ofs = has_64bit_shinfo ?
offsetof(struct vcpu_info, evtchn_pending_sel) :
offsetof(struct compat_vcpu_info, evtchn_pending_sel);
/* Now switch to the vCPU's vcpu_info to set the index and pending_sel */
- read_lock_irqsave(&gpc->lock, flags);
+ idx = srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu);
if (!kvm_gpc_check(gpc, sizeof(struct vcpu_info))) {
/*
* Could not access the vcpu_info. Set the bit in-kernel and
@@ -1940,14 +1940,17 @@ static void __kvm_xen_set_evtchn_fast(struct kvm_vcpu *vcpu, int port_word_bit,
}
out_unlock:
- read_unlock_irqrestore(&gpc->lock, flags);
+ srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx);
/*
* Deliver the upcall or kick the vCPU only after dropping the GPC
* read lock. Both paths end up in kvm_vcpu_kick(), and the MSI
* delivery also walks the APIC map and takes APIC locks; none of
- * that wants to be nested inside the GPC read lock, which a
- * concurrent invalidation may be waiting to take for write.
+ * that wants to be nested inside the GPC read-side critical
+ * section, which must be no longer than the accesses to gpc->khva
+ * above. Invalidation waits for a grace period, so holding the
+ * read lock across the kick would extend how long a memory
+ * invalidation is blocked.
*/
if (!kick_vcpu)
return;
@@ -1982,9 +1985,8 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, struct kvm *kvm)
bool has_64bit_shinfo = kvm_xen_has_64bit_shinfo(kvm);
unsigned long *pending_bits, *mask_bits;
struct kvm_vcpu *vcpu;
- unsigned long flags;
int port_word_bit;
- int vcpu_idx, idx, rc;
+ int vcpu_idx, idx, gpc_idx, rc;
vcpu_idx = READ_ONCE(xe->vcpu_idx);
if (vcpu_idx >= 0)
@@ -2003,9 +2005,9 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, struct kvm *kvm)
idx = srcu_read_lock(&kvm->srcu);
- read_lock_irqsave(&gpc->lock, flags);
+ gpc_idx = srcu_read_lock_atomic(&kvm->gpc_srcu);
if (!kvm_gpc_check(gpc, PAGE_SIZE))
- goto out_unlock;
+ goto out_gpc_srcu;
if (has_64bit_shinfo) {
struct shared_info *shinfo = gpc->khva;
@@ -2037,12 +2039,12 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe, struct kvm *kvm)
rc = 1; /* Delivered to the bitmap in shared_info. */
}
- out_unlock:
- read_unlock_irqrestore(&gpc->lock, flags);
+ out_gpc_srcu:
+ srcu_read_unlock_atomic(&kvm->gpc_srcu, gpc_idx);
/*
* Both of these can kick the vCPU, so keep them outside the GPC
- * read lock; an invalidation may be waiting to take it for write.
+ * read-side critical section; invalidation waits for a grace period.
*/
if (rc == -ENOTCONN)
kvm_xen_check_poller(vcpu, xe->port);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index cadd0318400e..3ed0b8c9c102 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -798,6 +798,16 @@ struct kvm {
/* For management / invalidation of gfn_to_pfn_caches */
spinlock_t gpc_lock;
struct list_head gpc_list;
+ /*
+ * Protects the contents of active gfn_to_pfn_caches. Readers hold it
+ * across kvm_gpc_check() and any subsequent access to gpc->khva; the
+ * invalidation and refresh paths clear GPC_VALID in gpc->state and
+ * then wait for a grace period before mutating anything a reader
+ * might be looking at.
+ * A dedicated domain (rather than kvm->srcu) keeps those waits from
+ * being lengthened by unrelated memslot readers.
+ */
+ struct srcu_struct gpc_srcu;
/*
* created_vcpus is protected by kvm->lock, and is incremented
@@ -1520,12 +1530,9 @@ int kvm_gpc_activate_hva(struct gfn_to_pfn_cache *gpc, unsigned long hva, unsign
* @return: %true if the cache is still valid and the address matches.
* %false if the cache is not valid.
*
- * Callers outside IN_GUEST_MODE context should hold a read lock on @gpc->lock
- * while calling this function, and then continue to hold the lock until the
- * access is complete.
- *
- * Callers in IN_GUEST_MODE may do so without locking, although they should
- * still hold a read lock on kvm->scru for the memslot checks.
+ * Callers must hold a read lock on @gpc->kvm->gpc_srcu across this function
+ * and any subsequent access to the target page, and must hold a read lock on
+ * kvm->srcu for the memslot checks.
*/
bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len);
@@ -1542,8 +1549,8 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len);
* This will attempt to refresh a gfn_to_pfn_cache. Note that a successful
* return from this function does not mean the page can be immediately
* accessed because it may have raced with an invalidation. Callers must
- * still lock and check the cache status, as this function does not return
- * with the lock still held to permit access.
+ * still check the cache status under a kvm->gpc_srcu read lock, via
+ * kvm_gpc_check(), before accessing the target page.
*/
int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsigned long len);
@@ -1984,7 +1991,8 @@ static inline bool kvm_is_gpa_in_memslot(struct kvm *kvm, gpa_t gpa)
static inline void kvm_gpc_mark_dirty_in_slot(struct gfn_to_pfn_cache *gpc)
{
- lockdep_assert_held(&gpc->lock);
+ RCU_LOCKDEP_WARN(!srcu_read_lock_held(&gpc->kvm->gpc_srcu),
+ "kvm_gpc_mark_dirty_in_slot() without kvm->gpc_srcu read lock");
if (!gpc->memslot || gpc->never_dirty)
return;
diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h
index e850adc3f47e..22393f97266a 100644
--- a/include/linux/kvm_types.h
+++ b/include/linux/kvm_types.h
@@ -88,15 +88,48 @@ struct gfn_to_pfn_cache {
struct kvm_memory_slot *memslot;
struct kvm *kvm;
struct list_head list;
- rwlock_t lock;
struct mutex refresh_lock;
void *khva;
kvm_pfn_t pfn;
bool active;
- bool valid;
+ /*
+ * GPC_VALID, GPC_BECOMING_VALID and GPC_INVALIDATING live in a
+ * single atomic word so that a refresh can publish
+ * (BECOMING_VALID => VALID) with a single conditional RMW, an
+ * invalidation can veto that publish by clearing both of those bits
+ * (setting INVALIDATING in the same operation if VALID was set, to
+ * record that its grace period is still owed), and a mutator can
+ * consume the whole state with one xchg to learn whether it must
+ * wait for readers before touching anything they see. See the
+ * comments in pfncache.c.
+ *
+ * 'active' deliberately stays separate: the invalidation and
+ * mutation paths operate on the word without having to care about
+ * racing with deactivation, which is ordered by refresh_lock
+ * instead.
+ */
+ atomic_t state;
bool never_dirty;
};
+/* The cache is valid: pfn and khva may be used by readers. */
+#define GPC_VALID 0x1
+/*
+ * A refresh is in flight and intends to publish. Set before the uhva is
+ * resolved, so an invalidation of that uhva can veto the publish even
+ * though the cache is not yet valid.
+ */
+#define GPC_BECOMING_VALID 0x2
+/*
+ * GPC_VALID has been cleared but the grace period for readers of the
+ * previously published mapping has not yet completed. Nothing may mutate
+ * reader-visible fields (pfn, khva, uhva, gpa, memslot) until whoever
+ * next needs to do so has waited it out. Set by whichever party clears
+ * GPC_VALID; cleared after a synchronize_srcu() which began after the
+ * VALID clear.
+ */
+#define GPC_INVALIDATING 0x4
+
#ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE
/*
* Memory caches are used to preallocate memory ahead of various MMU flows,
diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index 1a8a465a5650..f7a6bbcabc4e 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -706,6 +706,13 @@ DEFINE_LOCK_GUARD_1(srcu, struct srcu_struct,
DECLARE_LOCK_GUARD_1_ATTRS(srcu, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
#define class_srcu_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu, _T)
+DEFINE_LOCK_GUARD_1(srcu_atomic, struct srcu_struct,
+ _T->idx = srcu_read_lock_atomic(_T->lock),
+ srcu_read_unlock_atomic(_T->lock, _T->idx),
+ int idx)
+DECLARE_LOCK_GUARD_1_ATTRS(srcu_atomic, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
+#define class_srcu_atomic_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_atomic, _T)
+
DEFINE_LOCK_GUARD_1(srcu_fast, struct srcu_struct,
_T->scp = srcu_read_lock_fast(_T->lock),
srcu_read_unlock_fast(_T->lock, _T->scp),
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index cf4cb586f140..a3b4ca1ae88d 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1160,6 +1160,8 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
goto out_err_no_srcu;
if (init_srcu_struct(&kvm->irq_srcu))
goto out_err_no_irq_srcu;
+ if (init_srcu_struct_atomic(&kvm->gpc_srcu))
+ goto out_err_no_gpc_srcu;
r = kvm_init_irq_routing(kvm);
if (r)
@@ -1240,6 +1242,8 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
kfree(kvm_get_bus_for_destruction(kvm, i));
kvm_free_irq_routing(kvm);
out_err_no_irq_routing:
+ cleanup_srcu_struct(&kvm->gpc_srcu);
+out_err_no_gpc_srcu:
cleanup_srcu_struct(&kvm->irq_srcu);
out_err_no_irq_srcu:
cleanup_srcu_struct(&kvm->srcu);
@@ -1316,6 +1320,11 @@ static void kvm_destroy_vm(struct kvm *kvm)
kvm_free_memslots(kvm, &kvm->__memslots[i][0]);
kvm_free_memslots(kvm, &kvm->__memslots[i][1]);
}
+ /*
+ * All gfn_to_pfn_caches have been deactivated by kvm_arch_destroy_vm()
+ * above, so there can be no readers or pending invalidations left.
+ */
+ cleanup_srcu_struct(&kvm->gpc_srcu);
cleanup_srcu_struct(&kvm->irq_srcu);
srcu_barrier(&kvm->srcu);
cleanup_srcu_struct(&kvm->srcu);
diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c
index 17a30da00d40..1c98b7bdbf09 100644
--- a/virt/kvm/pfncache.c
+++ b/virt/kvm/pfncache.c
@@ -26,35 +26,149 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start,
unsigned long end)
{
struct gfn_to_pfn_cache *gpc;
+ bool cleared = false;
spin_lock(&kvm->gpc_lock);
+
+ /*
+ * Order the caller's earlier mn_active_invalidate_count increment
+ * (and gpc_invalidate_seq bump, for range_end) before the state
+ * reads below. This is a store->load ordering, which neither the
+ * unlock/lock the increment travelled through (RCtso) nor the
+ * acquire on the state read provides generically. It pairs with
+ * smp_mb__after_atomic() in hva_to_pfn_retry(): whichever side's
+ * full barrier executes second guarantees that either this walk
+ * observes GPC_BECOMING_VALID and vetoes the pending publish, or
+ * the refresh observes the elevated count / bumped seq and
+ * retries — closing the store-buffering window in which each
+ * side misses the other and a stale pfn gets published. (On x86
+ * both sides' locked RMW operations already provide this; generic
+ * code must not rely on that.)
+ */
+ smp_mb__after_spinlock();
+
list_for_each_entry(gpc, &kvm->gpc_list, list) {
- read_lock_irq(&gpc->lock);
+ /*
+ * A cache covers only a single page, keyed by its uhva, so
+ * there is no length to consider.
+ *
+ * The acquire pairs with the release-publish in
+ * hva_to_pfn_retry() (for GPC_VALID) and with the
+ * smp_mb__before_atomic() before the GPC_BECOMING_VALID
+ * announce (for that bit): whichever is seen, the uhva read
+ * below is guaranteed to see the value stored before it. A
+ * stale uhva paired with a fresh state bit could otherwise
+ * cause a cache whose new uhva is in the invalidated range
+ * to be skipped.
+ *
+ * uhva is read once, with READ_ONCE(), and only after the
+ * state (the acquire provides that ordering): it is written
+ * (with WRITE_ONCE()) by __kvm_gpc_refresh() without holding
+ * gpc_lock. Note the refresh only mutates uhva while the
+ * state word is 0 — after its xchg, before its announce —
+ * so this walk cannot observe a half-updated cache: if
+ * either state bit is visible, the uhva is current.
+ *
+ * Keying on GPC_BECOMING_VALID as well as GPC_VALID means a
+ * cache whose refresh is in flight is *not* skipped: its
+ * uhva is already known at that point, and clearing the bit
+ * vetoes the pending publish. Testing only GPC_VALID would
+ * leave the window between the refresh's last invalidation
+ * check and its publish unprotected. (For an invalidation
+ * which begins before that check, this veto is belt and
+ * braces: the mn_active_invalidate_count / gpc_invalidate_seq
+ * check makes the refresh retry anyway. The window after
+ * that check has only the veto.)
+ *
+ * Keying on GPC_INVALIDATING too means a cache whose
+ * VALID-clearing party is still mid-grace-period is not
+ * skipped either: see the comment at the 'cleared' update
+ * below.
+ */
+ unsigned long gpc_uhva;
+ int old, new;
- /* Only a single page so no need to care about length */
- if (gpc->valid && !is_error_noslot_pfn(gpc->pfn) &&
- gpc->uhva >= start && gpc->uhva < end) {
- read_unlock_irq(&gpc->lock);
+ old = atomic_read_acquire(&gpc->state);
+ if (!(old & (GPC_VALID | GPC_BECOMING_VALID | GPC_INVALIDATING)))
+ continue;
+ gpc_uhva = READ_ONCE(gpc->uhva);
+ if (gpc_uhva >= start && gpc_uhva < end) {
/*
- * There is a small window here where the cache could
- * be modified, and invalidation would no longer be
- * necessary. Hence check again whether invalidation
- * is still necessary once the write lock has been
- * acquired.
+ * Clear VALID and BECOMING_VALID; if VALID was set,
+ * also set INVALIDATING so that anyone else who wants
+ * to mutate reader-visible fields before our grace
+ * period below completes knows to wait for one of
+ * their own. (A cache which was merely becoming
+ * valid has no readers: the veto alone suffices.)
+ *
+ * 'old' is seeded from the read above; if the state
+ * changed since (e.g. the refresh published), the
+ * cmpxchg fails, reloads it, and goes around.
*/
+ do {
+ new = old & ~(GPC_VALID | GPC_BECOMING_VALID);
+ if (old & GPC_VALID)
+ new |= GPC_INVALIDATING;
+ } while (!atomic_try_cmpxchg(&gpc->state, &old, new));
- write_lock_irq(&gpc->lock);
- if (gpc->valid && !is_error_noslot_pfn(gpc->pfn) &&
- gpc->uhva >= start && gpc->uhva < end)
- gpc->valid = false;
- write_unlock_irq(&gpc->lock);
- continue;
+ /*
+ * Wait below not only when this walk itself cleared
+ * GPC_VALID, but also when GPC_INVALIDATING alone
+ * was found: the party which cleared GPC_VALID (a
+ * concurrent walk, or a refresh/deactivation mid
+ * grace-period — they keep the marker up until their
+ * wait completes) may still be inside its own
+ * synchronize, with readers outstanding on the old
+ * mapping. Those readers are covered by *its* grace
+ * period, which this walk's caller does not wait
+ * for; it must begin (and finish) one of its own
+ * before the caller frees the page.
+ */
+ if (old & (GPC_VALID | GPC_INVALIDATING))
+ cleared = true;
}
-
- read_unlock_irq(&gpc->lock);
}
spin_unlock(&kvm->gpc_lock);
+
+ /*
+ * Readers may still be using the old mapping, having sampled
+ * GPC_VALID before it was cleared. Wait for them all to drain
+ * before the caller proceeds to zap the page tables; like a TLB
+ * shootdown, this guarantees no access via the stale mapping
+ * once the invalidation completes.
+ *
+ * This wait is safe even on unblockable ranges (the OOM reaper):
+ * the constraint there is not "no sleeping" but "no blocking on
+ * anything which may itself depend on memory allocation to make
+ * progress" (see the reasoning in commit 312364f3534c and its
+ * discussion). A GPC read-side critical section has no such
+ * dependency: readers never allocate, never take mmap_lock, and
+ * never sleep.
+ *
+ * The atomic variant: this stalls progress of the primary MMU
+ * operation which triggered the invalidation, and the readers being
+ * drained are bounded, short, non-sleeping sections, so the wait
+ * spins (preemption disabled) rather than sleeping — and never
+ * queues a callback, so it depends on no allocation at all. That
+ * satisfies even the strictest reading of the OOM-reaper
+ * requirement; it is also the only wait primitive permitted on a
+ * domain whose readers are the atomic flavor.
+ */
+ if (cleared)
+ synchronize_srcu_atomic(&kvm->gpc_srcu);
+
+ /*
+ * Note the GPC_INVALIDATING markers set above are deliberately NOT
+ * cleared here: this walk dropped gpc_lock before waiting, so it
+ * cannot tell whether the marker it would clear is still the one
+ * it set (consider a republish-and-reinvalidate cycle completing
+ * entirely within this synchronize_srcu()). Leaving it standing is
+ * merely conservative: a later walk over the same uhva pays a
+ * redundant grace period, and the next mutation of the cache (a
+ * refresh or deactivation, which performs its own wait) retires
+ * it. Readers ignore the bit.
+ */
}
static bool kvm_gpc_is_valid_len(gpa_t gpa, unsigned long uhva,
@@ -74,14 +188,31 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
{
struct kvm_memslots *slots = kvm_memslots(gpc->kvm);
- if (!gpc->active)
+ RCU_LOCKDEP_WARN(!srcu_read_lock_held(&gpc->kvm->gpc_srcu),
+ "kvm_gpc_check() without kvm->gpc_srcu read lock");
+
+ /*
+ * Check valid *first*. The acquire pairs with the release-publish
+ * in hva_to_pfn_retry(), so every field read below — and any use
+ * of gpc->khva by the caller — is guaranteed to be from the
+ * published generation, not a stale value reordered from before
+ * the publish. The fields are then stable for the remainder of
+ * the RCU read-side critical section, because every mutator
+ * clears valid and waits a full grace period before changing
+ * anything.
+ */
+ if (!(atomic_read_acquire(&gpc->state) & GPC_VALID))
+ return false;
+
+ if (!READ_ONCE(gpc->active))
return false;
/*
* If the page was cached from a memslot, make sure the memslots have
* not been re-configured.
*/
- if (!kvm_is_error_gpa(gpc->gpa) && gpc->generation != slots->generation)
+ if (!kvm_is_error_gpa(gpc->gpa) &&
+ READ_ONCE(gpc->generation) != slots->generation)
return false;
if (kvm_is_error_hva(gpc->uhva))
@@ -90,9 +221,6 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
if (!kvm_gpc_is_valid_len(gpc->gpa, gpc->uhva, len))
return false;
- if (!gpc->valid)
- return false;
-
return true;
}
@@ -134,8 +262,8 @@ static inline bool mmu_notifier_retry_cache(struct kvm *kvm, unsigned long gpc_s
* is elevated.
*
* Note, it does not matter that mn_active_invalidate_count
- * is not protected by gpc->lock. It is guaranteed to
- * be elevated before the mmu_notifier acquires gpc->lock, and
+ * is not protected by any lock the refresher holds. It is
+ * guaranteed to be elevated before the mmu_notifier walk, and
* isn't dropped until after gpc_invalidate_seq is updated.
*/
if (kvm->mn_active_invalidate_count)
@@ -156,10 +284,13 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)
{
/* Note, the new page offset may be different than the old! */
void *old_khva = (void *)PAGE_ALIGN_DOWN((uintptr_t)gpc->khva);
+ void *old_khva_exact = gpc->khva;
+ kvm_pfn_t old_pfn = gpc->pfn;
kvm_pfn_t new_pfn = KVM_PFN_ERR_FAULT;
void *new_khva = NULL;
unsigned long gpc_seq;
struct page *page;
+ int state;
struct kvm_follow_pfn kfp = {
.slot = gpc->memslot,
@@ -171,21 +302,42 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)
lockdep_assert_held(&gpc->refresh_lock);
- lockdep_assert_held_write(&gpc->lock);
-
- /*
- * Invalidate the cache prior to dropping gpc->lock, the gpa=>uhva
- * assets have already been updated and so a concurrent check() from a
- * different task may not fail the gpa/uhva/generation checks.
- */
- gpc->valid = false;
+ WARN_ON_ONCE(atomic_read(&gpc->state) & GPC_VALID);
do {
+ /*
+ * Announce the intent to publish before the uhva is resolved
+ * and before any lock is dropped. From here on an
+ * invalidation of this uhva will clear the bit, and the
+ * cmpxchg below will then fail and send us around again.
+ * This is what makes the check-and-publish atomic with
+ * respect to the mmu_notifier walk without holding a lock
+ * across it.
+ *
+ * The barrier orders the caller's gpc->uhva store before the
+ * bit becoming visible (atomic_or() alone is unordered, and
+ * x86's lock-or being a full barrier must not be relied on
+ * in generic code). It pairs with the atomic_read_acquire()
+ * in gfn_to_pfn_cache_invalidate_start(): a walker which
+ * observes the bit observes the uhva this refresh is
+ * resolving, and so aims its veto correctly.
+ */
+ smp_mb__before_atomic();
+ atomic_or(GPC_BECOMING_VALID, &gpc->state);
+
+ /*
+ * Order the announce store before the gpc_invalidate_seq
+ * and mn_active_invalidate_count reads which follow
+ * (store->load; atomic_or() alone is unordered). Pairs
+ * with smp_mb__after_spinlock() in
+ * gfn_to_pfn_cache_invalidate_start(); see the comment
+ * there for the store-buffering window this closes.
+ */
+ smp_mb__after_atomic();
+
gpc_seq = gpc->kvm->gpc_invalidate_seq;
smp_rmb();
- write_unlock_irq(&gpc->lock);
-
/*
* If the previous iteration "failed" due to an mmu_notifier
* event, release the pfn and unmap the kernel virtual address
@@ -213,9 +365,15 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)
/*
* Obtain a new kernel mapping if KVM itself will access the
* pfn. Note, kmap() and memremap() can both sleep, so this
- * too must be done outside of gpc->lock!
+ * can sleep, which is fine: this path holds no spinning locks.
*/
- if (new_pfn == gpc->pfn)
+ /*
+ * Compare against the *published* pfn captured at entry, via
+ * a local: gpc->pfn must not be consulted mid-loop. (A
+ * previous iteration's vetoed publish attempt must not be
+ * able to leak its doomed pfn into this comparison.)
+ */
+ if (new_pfn == old_pfn)
new_khva = old_khva;
else
new_khva = gpc_map(new_pfn);
@@ -225,18 +383,39 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)
goto out_error;
}
- write_lock_irq(&gpc->lock);
+ if (mmu_notifier_retry_cache(gpc->kvm, gpc_seq))
+ continue;
+
+ gpc->pfn = new_pfn;
+ gpc->khva = new_khva + offset_in_page(gpc->uhva);
/*
- * Other tasks must wait for _this_ refresh to complete before
- * attempting to refresh.
+ * Publish: BECOMING_VALID => VALID, but only if the bit is
+ * still set. If an invalidation of this uhva landed at any
+ * point since it was set — including in the window after the
+ * retry check just above — the bit is gone, the cmpxchg fails
+ * and we go around again. The release ordering pairs with the
+ * atomic_read_acquire() in kvm_gpc_check() so the pfn/khva
+ * stores are visible before GPC_VALID is. (The stores must
+ * precede the VALID publish; a reader which observes VALID
+ * must observe the fields it covers.)
*/
- WARN_ON_ONCE(gpc->valid);
- } while (mmu_notifier_retry_cache(gpc->kvm, gpc_seq));
+ state = GPC_BECOMING_VALID;
+ if (atomic_try_cmpxchg_release(&gpc->state, &state, GPC_VALID))
+ break;
- gpc->valid = true;
- gpc->pfn = new_pfn;
- gpc->khva = new_khva + offset_in_page(gpc->uhva);
+ /*
+ * Vetoed: an invalidation cleared GPC_BECOMING_VALID after
+ * the retry check above. No reader ever saw the values just
+ * stored (VALID was never set for them), but the loop's own
+ * cleanup and aliasing logic must not see them either —
+ * restore the published values. This runs under refresh_lock
+ * with the cache unpublished, so it is single-writer plain
+ * stores.
+ */
+ gpc->pfn = old_pfn;
+ gpc->khva = old_khva_exact;
+ } while (1);
/*
* Put the reference to the _new_ page. The page is now tracked by the
@@ -248,8 +427,19 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)
return 0;
out_error:
- write_lock_irq(&gpc->lock);
-
+ /*
+ * Withdraw the announcement: no publish is coming. Leaving the bit
+ * set would be harmless — an invalidation walk would merely veto a
+ * publish which was never going to happen, and the next refresh or
+ * deactivation starts by xchg'ing the whole state word to zero —
+ * but clearing it here keeps the invariant that GPC_BECOMING_VALID
+ * means a refresh is between announce and publish. Only VALID can
+ * have been set here by anyone else, and only by our own publish
+ * (which didn't happen), and INVALIDATING only by whoever clears
+ * VALID; the sole concurrent writer is an invalidation walk also
+ * clearing this bit, so a plain andnot cannot lose an update.
+ */
+ atomic_andnot(GPC_BECOMING_VALID, &gpc->state);
return -EFAULT;
}
@@ -259,7 +449,7 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
bool unmap_old = false;
unsigned long old_uhva;
kvm_pfn_t old_pfn;
- bool hva_change = false;
+ bool must_drain;
void *old_khva;
int ret;
@@ -269,13 +459,103 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
lockdep_assert_held(&gpc->refresh_lock);
- write_lock_irq(&gpc->lock);
+ if (!gpc->active)
+ return -EINVAL;
- if (!gpc->active) {
- ret = -EINVAL;
- goto out_unlock;
+ /*
+ * Resolve the target uHVA (and memslot, for a GPA-based cache) before
+ * deciding whether anything needs to be invalidated at all.
+ *
+ * If the cache is still valid and this refresh resolves to exactly the
+ * same uHVA, in the same memslot, for the same GPA, then nothing which
+ * a reader can observe is changing: the gPA => uHVA translation is
+ * unchanged, and GPC_VALID being set is precisely the assertion that
+ * the second stage (uHVA => PFN, and the kernel mapping of it) is
+ * still good. No mapping is retired, so there is nothing for a grace
+ * period to wait for. At most the memslot generation needs updating,
+ * which no reader consults except via kvm_gpc_check() itself.
+ *
+ * All of the following must hold to take this shortcut:
+ *
+ * - GPC_VALID: the PFN and its kernel mapping are still good.
+ * - the resolved uHVA is unchanged. kvm_gpc_check() validates
+ * 'offset + len <= PAGE_SIZE' from gpc->uhva/gpa on the reader's
+ * behalf, and the reader then accesses gpc->khva for 'len' bytes
+ * with no recheck; moving the offset under a live reader would let
+ * it run off the end of the page.
+ * - the memslot is unchanged. kvm_gpc_mark_dirty_in_slot() uses
+ * gpc->memslot, so replacing it under a reader could mark the wrong
+ * slot dirty and thus lose a dirty page for live migration.
+ * - the gPA is unchanged, as it provides the gfn for dirty tracking.
+ */
+ if (atomic_read(&gpc->state) & GPC_VALID) {
+ struct kvm_memory_slot *new_slot = gpc->memslot;
+ unsigned long new_uhva = KVM_HVA_ERR_BAD;
+ u64 new_generation = gpc->generation;
+
+ if (kvm_is_error_gpa(gpa)) {
+ new_uhva = uhva;
+ } else {
+ struct kvm_memslots *slots = kvm_memslots(gpc->kvm);
+ gfn_t gfn = gpa_to_gfn(gpa);
+
+ new_generation = slots->generation;
+ new_slot = __gfn_to_memslot(slots, gfn);
+ if (new_slot)
+ new_uhva = gfn_to_hva_memslot(new_slot, gfn) +
+ offset_in_page(gpa);
+ }
+
+ if (!kvm_is_error_hva(new_uhva) && new_uhva == gpc->uhva &&
+ new_slot == gpc->memslot && gpa == gpc->gpa) {
+ /*
+ * Nothing to invalidate. A concurrent reader may be
+ * using the cache right now and can safely continue
+ * to do so; only the memslot generation, which is
+ * read solely by kvm_gpc_check(), may need updating.
+ */
+ if (new_generation != gpc->generation)
+ WRITE_ONCE(gpc->generation, new_generation);
+
+ return 0;
+ }
}
+ /*
+ * Take the cache invalid and wait for all current readers to
+ * drain before mutating anything they might be looking at. Once
+ * the grace period has elapsed, this task (serialized by
+ * refresh_lock) owns all the cache fields exclusively: readers
+ * check valid (with an acquire load) inside their SRCU read-side
+ * critical sections and back off. This mirrors what a TLB
+ * shootdown does for the hardware page tables.
+ *
+ * The clear and the test are one operation on purpose, and the wait
+ * triggers on GPC_INVALIDATING as well as GPC_VALID: an mmu_notifier
+ * invalidation which cleared GPC_VALID a moment ago is not
+ * necessarily finished — its grace period may still be running, with
+ * readers which sampled GPC_VALID before the clear still
+ * dereferencing gpc->khva. GPC_INVALIDATING is set by whoever clears
+ * GPC_VALID precisely so that this path knows those readers may
+ * exist. Our own grace-period wait below necessarily begins after
+ * that clear, so waiting here covers them; only then may
+ * reader-visible fields (uhva, gpa, memslot, and later pfn/khva) be
+ * mutated or the old mapping torn down.
+ *
+ * The xchg leaves GPC_INVALIDATING *standing* (rather than clearing
+ * the word outright) for the duration of the wait, for the same
+ * reason it is tested: an mmu_notifier walk which arrives while
+ * this grace period is still running must not read a zero state,
+ * skip the cache, and let its caller free the page under readers
+ * only this wait covers. The marker comes down only once the drain
+ * is complete and the readers it advertised cannot exist.
+ */
+ must_drain = atomic_xchg(&gpc->state, GPC_INVALIDATING) &
+ (GPC_VALID | GPC_INVALIDATING);
+ if (must_drain)
+ synchronize_srcu_atomic(&gpc->kvm->gpc_srcu);
+ atomic_set(&gpc->state, 0);
+
old_pfn = gpc->pfn;
old_khva = (void *)PAGE_ALIGN_DOWN((uintptr_t)gpc->khva);
old_uhva = PAGE_ALIGN_DOWN(gpc->uhva);
@@ -285,10 +565,14 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
gpc->gpa = INVALID_GPA;
gpc->memslot = NULL;
- gpc->uhva = PAGE_ALIGN_DOWN(uhva);
-
- if (gpc->uhva != old_uhva)
- hva_change = true;
+ /*
+ * uhva stores are WRITE_ONCE(): the invalidation walks read
+ * it without holding refresh_lock. This mutation window is
+ * safe because the state word is 0 here (cleared by the
+ * xchg above, announced again only in hva_to_pfn_retry()
+ * after these stores), so no walk will match this cache.
+ */
+ WRITE_ONCE(gpc->uhva, PAGE_ALIGN_DOWN(uhva));
} else {
struct kvm_memslots *slots = kvm_memslots(gpc->kvm);
@@ -301,43 +585,31 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
gpc->gpa = gpa;
gpc->generation = slots->generation;
gpc->memslot = __gfn_to_memslot(slots, gfn);
- gpc->uhva = gfn_to_hva_memslot(gpc->memslot, gfn);
+ WRITE_ONCE(gpc->uhva, gfn_to_hva_memslot(gpc->memslot, gfn));
if (kvm_is_error_hva(gpc->uhva)) {
ret = -EFAULT;
goto out;
}
-
- /*
- * Even if the GPA and/or the memslot generation changed, the
- * HVA may still be the same.
- */
- if (gpc->uhva != old_uhva)
- hva_change = true;
} else {
- gpc->uhva = old_uhva;
+ WRITE_ONCE(gpc->uhva, old_uhva);
}
}
/* Note: the offset must be correct before calling hva_to_pfn_retry() */
- gpc->uhva += page_offset;
+ WRITE_ONCE(gpc->uhva, gpc->uhva + page_offset);
/*
- * If the userspace HVA changed or the PFN was already invalid,
- * drop the lock and do the HVA to PFN lookup again.
+ * Always redo the HVA to PFN lookup: an invalidation of our uhva
+ * may have raced with (or followed) the valid-clearing above, in
+ * which case the notifier walk skipped this cache (valid was
+ * already false) and the old pfn may already be stale. The
+ * gpc_invalidate_seq check in hva_to_pfn_retry() is what detects
+ * that race, so the lookup path is the only safe way to publish.
+ * If the mapping is in fact unchanged, hva_to_pfn_retry() reuses
+ * the existing kernel mapping for the same pfn.
*/
- if (!gpc->valid || hva_change) {
- ret = hva_to_pfn_retry(gpc);
- } else {
- /*
- * If the HVA→PFN mapping was already valid, don't unmap it.
- * But do update gpc->khva because the offset within the page
- * may have changed.
- */
- gpc->khva = old_khva + page_offset;
- ret = 0;
- goto out_unlock;
- }
+ ret = hva_to_pfn_retry(gpc);
out:
/*
@@ -346,17 +618,12 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
* valid, leave it as is.
*/
if (ret) {
- gpc->valid = false;
+ WARN_ON_ONCE(atomic_read(&gpc->state) & GPC_VALID);
gpc->pfn = KVM_PFN_ERR_FAULT;
gpc->khva = NULL;
}
- /* Detect a pfn change before dropping the lock! */
unmap_old = (old_pfn != gpc->pfn);
-
-out_unlock:
- write_unlock_irq(&gpc->lock);
-
if (unmap_old)
gpc_unmap(old_pfn, old_khva);
@@ -385,14 +652,14 @@ int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsigned long len)
void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm,
bool never_dirty)
{
- rwlock_init(&gpc->lock);
mutex_init(&gpc->refresh_lock);
gpc->kvm = kvm;
gpc->pfn = KVM_PFN_ERR_FAULT;
gpc->gpa = INVALID_GPA;
gpc->uhva = KVM_HVA_ERR_BAD;
- gpc->active = gpc->valid = false;
+ gpc->active = false;
+ atomic_set(&gpc->state, 0);
gpc->never_dirty = never_dirty;
}
@@ -407,7 +674,7 @@ static int __kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned
guard(mutex)(&gpc->refresh_lock);
if (!gpc->active) {
- if (KVM_BUG_ON(gpc->valid, kvm))
+ if (KVM_BUG_ON(atomic_read(&gpc->state) & GPC_VALID, kvm))
return -EIO;
spin_lock(&kvm->gpc_lock);
@@ -417,11 +684,11 @@ static int __kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned
/*
* Activate the cache after adding it to the list, a concurrent
* refresh must not establish a mapping until the cache is
- * reachable by mmu_notifier events.
+ * reachable by mmu_notifier events. (Refreshes are serialized
+ * by refresh_lock, which we hold; the store ordering matters
+ * only against the notifier walk, which holds gpc_lock.)
*/
- write_lock_irq(&gpc->lock);
- gpc->active = true;
- write_unlock_irq(&gpc->lock);
+ WRITE_ONCE(gpc->active, true);
}
return __kvm_gpc_refresh(gpc, gpa, uhva);
}
@@ -456,13 +723,28 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc)
if (gpc->active) {
/*
- * Deactivate the cache before removing it from the list, KVM
- * must stall mmu_notifier events until all users go away, i.e.
- * until gpc->lock is dropped and refresh is guaranteed to fail.
+ * Mark the cache inactive and invalid, and wait for all
+ * current readers to drain, before tearing down the mapping
+ * they may have been using. Refreshes are excluded by
+ * refresh_lock, which we hold.
+ */
+ WRITE_ONCE(gpc->active, false);
+
+ /*
+ * As in __kvm_gpc_refresh(): clear and test in one
+ * operation, waiting on GPC_INVALIDATING as well as
+ * GPC_VALID. An invalidation which cleared GPC_VALID first
+ * may still be inside its own synchronize_srcu() with
+ * readers outstanding — it left GPC_INVALIDATING set to say
+ * so — and the unmap below must not run under those readers.
+ * And, also as there, leave GPC_INVALIDATING standing until
+ * the drain completes, so that an mmu_notifier walk arriving
+ * mid-wait does not read a zero state and skip the cache.
*/
- write_lock_irq(&gpc->lock);
- gpc->active = false;
- gpc->valid = false;
+ if (atomic_xchg(&gpc->state, GPC_INVALIDATING) &
+ (GPC_VALID | GPC_INVALIDATING))
+ synchronize_srcu_atomic(&kvm->gpc_srcu);
+ atomic_set(&gpc->state, 0);
/*
* Leave the GPA => uHVA cache intact, it's protected by the
@@ -475,7 +757,6 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc)
old_pfn = gpc->pfn;
gpc->pfn = KVM_PFN_ERR_FAULT;
- write_unlock_irq(&gpc->lock);
spin_lock(&kvm->gpc_lock);
list_del(&gpc->list);
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 09/20] KVM: x86: Use gfn_to_pfn_cache for steal time / preempted status
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (7 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 08/20] KVM: pfncache: Use atomic SRCU for readers instead of a rwlock David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 10/20] KVM: pfncache: Add guest-mode pinning (GUEST_USES_PFN successor) David Woodhouse
` (10 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: Carsten Stollmaier <stollmc@amazon.com>
This largely reverts commit 7e2175ebd695 ("KVM: x86: Fix recording of
guest steal time / preempted status"), which dropped the use of the
gfn_to_pfn_cache because it was not integrated with the MMU notifiers
at the time. That shortcoming has long since been addressed, making
the GPC work correctly for this use case.
Aside from cleaning up the last open-coded assembler access to user
addresses and associated explicit asm exception fixups, moving back
to the now-functional GPC also resolves an issue with contention on
the mmap_lock with userfaultfd. The contention issue is as follows:
On vcpu_run, before entering the guest, the update of the steal time
information causes a page-fault if the page is not present. In our
scenario, this gets handled by do_user_addr_fault() and successively
handle_userfault() because the region is registered to that.
Since handle_userfault() uses TASK_INTERRUPTIBLE, it is interruptible
by signals. But do_user_addr_fault() then busy-retries if the pending
signal is non-fatal, which leads to heavy contention of the mmap_lock.
By restoring the use of GPC for accessing the guest steal time, the
contention is avoided and refreshing the GPC happens when the vCPU is
next scheduled.
Since the gfn_to_pfn_cache gives a kernel mapping rather than a
userspace HVA, accesses are now plain C instead of unsafe_put_user()
et al. READ_ONCE()/WRITE_ONCE() and an smp_wmb() before the final
version increment replace the ordering the old inline assembly
provided implicitly.
kvm_steal_time_set_preempted(), called from the scheduler path, just
takes the atomic SRCU read lock (unconditionally safe there) and
bails if the cache is not currently valid: setting the preempted
flag is best-effort anyway — the old code used
copy_to_user_nofault(), which could equally silently fail — and
kvm_arch_vcpu_load() raises KVM_REQ_STEAL_UPDATE, so the cache is
refreshed from task context before the vCPU next enters the guest.
MSR disable (kvm_steal_time_reset()) leaves last_steal alone: it is
the baseline against the vCPU task's monotonic run_delay, and zeroing
it would dump the task's lifetime run_delay into the guest's steal
counter when the MSR is next enabled (e.g. after INIT-SIPI or kexec).
[dwmw2: Port to the SRCU-protected GPC. The original was written
against a gpc->lock rwlock and needed a read_trylock() dance
to be safe from the scheduler path on PREEMPT_RT; the atomic
SRCU read lock is unconditionally safe there, so all of that
goes away.]
Signed-off-by: Carsten Stollmaier <stollmc@amazon.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Co-developed-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/include/asm/kvm_host.h | 2 +-
arch/x86/kvm/msrs.c | 7 +-
arch/x86/kvm/x86.c | 140 ++++++++++++++++----------------
3 files changed, 76 insertions(+), 73 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 1ee4b127004d..4b0c18c77771 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -872,7 +872,7 @@ struct kvm_vcpu_arch {
u8 preempted;
u64 msr_val;
u64 last_steal;
- struct gfn_to_hva_cache cache;
+ struct gfn_to_pfn_cache cache;
} st;
u64 l1_tsc_offset;
diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c
index e91650af44c2..23407f8c441d 100644
--- a/arch/x86/kvm/msrs.c
+++ b/arch/x86/kvm/msrs.c
@@ -1728,8 +1728,13 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
vcpu->arch.st.msr_val = data;
- if (!(data & KVM_MSR_ENABLED))
+ if (!(data & KVM_MSR_ENABLED)) {
+ kvm_gpc_deactivate(&vcpu->arch.st.cache);
break;
+ }
+
+ kvm_gpc_activate(&vcpu->arch.st.cache, data & ~KVM_MSR_ENABLED,
+ sizeof(struct kvm_steal_time));
kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index f5f3459ed4a5..321f29789f9f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2058,12 +2058,11 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_service_local_tlb_flush_requests);
static void record_steal_time(struct kvm_vcpu *vcpu)
{
- struct gfn_to_hva_cache *ghc = &vcpu->arch.st.cache;
- struct kvm_steal_time __user *st;
- struct kvm_memslots *slots;
- gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
+ struct gfn_to_pfn_cache *gpc = &vcpu->arch.st.cache;
+ struct kvm_steal_time *st;
u64 steal;
u32 version;
+ int idx;
if (kvm_xen_msr_enabled(vcpu->kvm)) {
kvm_xen_runstate_set_running(vcpu);
@@ -2076,55 +2075,41 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
if (WARN_ON_ONCE(current->mm != vcpu->kvm->mm))
return;
- slots = kvm_memslots(vcpu->kvm);
+ /* We rely on the fact that it fits in a single page. */
+ BUILD_BUG_ON((sizeof(*st) - 1) & KVM_STEAL_VALID_BITS);
- if (unlikely(slots->generation != ghc->generation ||
- gpa != ghc->gpa ||
- kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
- /* We rely on the fact that it fits in a single page. */
- BUILD_BUG_ON((sizeof(*st) - 1) & KVM_STEAL_VALID_BITS);
+ idx = srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu);
+ while (!kvm_gpc_check(gpc, sizeof(*st))) {
+ srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx);
- if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st)) ||
- kvm_is_error_hva(ghc->hva) || !ghc->memslot)
+ if (kvm_gpc_refresh(gpc, sizeof(*st)))
return;
+
+ idx = srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu);
}
- st = (struct kvm_steal_time __user *)ghc->hva;
+ st = gpc->khva;
+
/*
* Doing a TLB flush here, on the guest's behalf, can avoid
* expensive IPIs.
*/
if (guest_pv_has(vcpu, KVM_FEATURE_PV_TLB_FLUSH)) {
- u8 st_preempted = 0;
- int err = -EFAULT;
-
- if (!user_access_begin(st, sizeof(*st)))
- return;
-
- asm volatile("1: xchgb %0, %2\n"
- "xor %1, %1\n"
- "2:\n"
- _ASM_EXTABLE_UA(1b, 2b)
- : "+q" (st_preempted),
- "+&r" (err),
- "+m" (st->preempted));
- if (err)
- goto out;
-
- user_access_end();
+ u8 st_preempted = xchg(&st->preempted, 0);
vcpu->arch.st.preempted = 0;
trace_kvm_pv_tlb_flush(vcpu->vcpu_id,
st_preempted & KVM_VCPU_FLUSH_TLB);
/*
- * Request the flush instead of performing it inline. With
- * shadow paging kvm_vcpu_flush_tlb_guest() can reschedule
- * (via mmu_sync_children()), so it could not be called while
- * the user access above was open. Requesting is just a bit
- * set on the running vCPU, and the request is serviced
- * before the vCPU enters the guest; nothing requires the
- * flush to have completed any earlier than that.
+ * Request the flush instead of performing it inline. This
+ * runs inside an atomic gpc_srcu read-side section, with
+ * preemption disabled, and with shadow paging
+ * kvm_vcpu_flush_tlb_guest() can reschedule (via
+ * mmu_sync_children()). Requesting is just a bit set on the
+ * running vCPU, and the request is serviced before the vCPU
+ * enters the guest; nothing requires the flush to have
+ * completed any earlier than that.
*
* Note the xchg above must remain as-is: it atomically
* consumes any KVM_VCPU_FLUSH_TLB the guest set, as required
@@ -2133,39 +2118,33 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
*/
if (st_preempted & KVM_VCPU_FLUSH_TLB)
kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
-
- if (!user_access_begin(st, sizeof(*st)))
- goto dirty;
} else {
- if (!user_access_begin(st, sizeof(*st)))
- return;
-
- unsafe_put_user(0, &st->preempted, out);
+ WRITE_ONCE(st->preempted, 0);
vcpu->arch.st.preempted = 0;
}
- unsafe_get_user(version, &st->version, out);
+ version = READ_ONCE(st->version);
if (version & 1)
version += 1; /* first time write, random junk */
version += 1;
- unsafe_put_user(version, &st->version, out);
+ WRITE_ONCE(st->version, version);
smp_wmb();
- unsafe_get_user(steal, &st->steal, out);
+ steal = READ_ONCE(st->steal);
steal += current->sched_info.run_delay -
vcpu->arch.st.last_steal;
vcpu->arch.st.last_steal = current->sched_info.run_delay;
- unsafe_put_user(steal, &st->steal, out);
+ WRITE_ONCE(st->steal, steal);
+
+ smp_wmb();
version += 1;
- unsafe_put_user(version, &st->version, out);
+ WRITE_ONCE(st->version, version);
- out:
- user_access_end();
- dirty:
- mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa));
+ kvm_gpc_mark_dirty_in_slot(gpc);
+ srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx);
}
static inline bool kvm_can_mwait_in_guest(void)
@@ -2640,11 +2619,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
{
- struct gfn_to_hva_cache *ghc = &vcpu->arch.st.cache;
- struct kvm_steal_time __user *st;
- struct kvm_memslots *slots;
- static const u8 preempted = KVM_VCPU_PREEMPTED;
- gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
+ struct gfn_to_pfn_cache *gpc = &vcpu->arch.st.cache;
+ struct kvm_steal_time *st;
/*
* The vCPU can be marked preempted if and only if the VM-Exit was on
@@ -2669,20 +2645,37 @@ static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
if (unlikely(current->mm != vcpu->kvm->mm))
return;
- slots = kvm_memslots(vcpu->kvm);
-
- if (unlikely(slots->generation != ghc->generation ||
- gpa != ghc->gpa ||
- kvm_is_error_hva(ghc->hva) || !ghc->memslot))
- return;
-
- st = (struct kvm_steal_time __user *)ghc->hva;
- BUILD_BUG_ON(sizeof(st->preempted) != sizeof(preempted));
-
- if (!copy_to_user_nofault(&st->preempted, &preempted, sizeof(preempted)))
+ /*
+ * This is called from the scheduler path (via kvm_sched_out), so
+ * the cache cannot be refreshed here. Setting 'preempted' is
+ * best-effort anyway; the old HVA-based code used
+ * copy_to_user_nofault(), which could also silently fail. If the
+ * cache is invalid, it will be refreshed from task context by
+ * record_steal_time() before the vCPU next enters the guest, since
+ * kvm_arch_vcpu_load() unconditionally raises KVM_REQ_STEAL_UPDATE.
+ */
+ guard(srcu_atomic)(&vcpu->kvm->gpc_srcu);
+ if (kvm_gpc_check(gpc, sizeof(*st))) {
+ st = gpc->khva;
+ WRITE_ONCE(st->preempted, KVM_VCPU_PREEMPTED);
vcpu->arch.st.preempted = KVM_VCPU_PREEMPTED;
- mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa));
+ kvm_gpc_mark_dirty_in_slot(gpc);
+ }
+}
+
+static void kvm_steal_time_reset(struct kvm_vcpu *vcpu)
+{
+ kvm_gpc_deactivate(&vcpu->arch.st.cache);
+ vcpu->arch.st.preempted = 0;
+ vcpu->arch.st.msr_val = 0;
+ /*
+ * Leave last_steal alone: it is the baseline against the vCPU
+ * task's cumulative sched_info.run_delay, which is never reset.
+ * Zeroing it here would cause the first record_steal_time() after
+ * a vCPU reset (INIT-SIPI, kexec) to account the task's entire
+ * lifetime run_delay as steal time in one jump.
+ */
}
void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
@@ -9460,6 +9453,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
kvm_gpc_init(&vcpu->arch.pv_time, vcpu->kvm);
+ kvm_gpc_init(&vcpu->arch.st.cache, vcpu->kvm);
+
if (!irqchip_in_kernel(vcpu->kvm) || kvm_vcpu_is_reset_bsp(vcpu))
kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE);
else
@@ -9569,6 +9564,8 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
kvm_clear_async_pf_completion_queue(vcpu);
kvm_mmu_unload(vcpu);
+ kvm_steal_time_reset(vcpu);
+
kvmclock_reset(vcpu);
for_each_possible_cpu(cpu)
@@ -9689,7 +9686,8 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
kvm_make_request(KVM_REQ_EVENT, vcpu);
vcpu->arch.apf.msr_en_val = 0;
vcpu->arch.apf.msr_int_val = 0;
- vcpu->arch.st.msr_val = 0;
+
+ kvm_steal_time_reset(vcpu);
kvmclock_reset(vcpu);
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 10/20] KVM: pfncache: Add guest-mode pinning (GUEST_USES_PFN successor)
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (8 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 09/20] KVM: x86: Use gfn_to_pfn_cache for steal time / preempted status David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:29 ` sashiko-bot
2026-11-05 2:00 ` [PATCH v2 11/20] KVM: nVMX: Implement cache for L1 MSR bitmap David Woodhouse
` (9 subsequent siblings)
19 siblings, 1 reply; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
Reinstate support for caches whose pfn a vCPU uses directly from guest
mode (stuffed into hardware control structures such as vmcs02 fields),
addressing the reasons the original KVM_GUEST_USES_PFN was removed as
incomplete/broken¹²:
- The kick now comes with a guarantee. The original
KVM_REQ_OUTSIDE_GUEST_MODE kick did "absolutely nothing to
guarantee KVM refreshes the cache before re-entering the guest"¹.
Now, a vCPU pins with a cmpxchg which succeeds only while
GPC_VALID holds, the invalidation walk reads the pin in the same
atomic word it uses to veto in-flight publishes, and re-pinning
requires a refresh which cannot complete until
invalidate_range_end(): a forced-out vCPU cannot re-enter with
the stale mapping.
- The service request is per-cache, given at init³
(kvm_gpc_init_for_vcpu()), so e.g. nVMX can request
KVM_REQ_GET_NESTED_STATE_PAGES rather than KVM guessing what a
kicked vCPU must do before re-entering.
- No special-casing of non-blockable invalidations. The original
stripped KVM_REQUEST_WAIT for the OOM reaper, assuming a reaped
task's vCPUs are stopped; that is false for process_mrelease()
and opened the very use-after-free the kick exists to prevent⁴.
The wait is unconditionally safe: kvm_make_vcpus_request_mask()
spin-waits for IPI acks with preemption disabled, never sleeping.
- Pin lifecycle has single-writer discipline: only the pinning vCPU
sets or clears its pin, except that a mutator's xchg of the state
word may consume it, inheriting the kick-and-wait duty. A stale
pin costs at most a spurious request bit.
No users yet; the nVMX conversion⁵ builds on this.
¹ https://lore.kernel.org/all/ZBEEQtmtNPaEqU1i@google.com
² https://lore.kernel.org/all/ZQiR8IpqOZrOpzHC@google.com
³ https://lore.kernel.org/all/agJnqh7ACEkB-ftx@google.com
⁴ per Jann Horn's analysis in ²
⁵ https://lore.kernel.org/all/20260102142429.896101-1-griffoul@gmail.com
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kvm/xen.c | 4 +-
include/linux/kvm_host.h | 64 +++++++++++++-
include/linux/kvm_types.h | 28 ++++--
virt/kvm/pfncache.c | 181 ++++++++++++++++++++++++++++++++++++--
4 files changed, 259 insertions(+), 18 deletions(-)
diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 13e287bc17bb..132f0c379537 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -2465,7 +2465,7 @@ void kvm_xen_init_vcpu(struct kvm_vcpu *vcpu)
kvm_gpc_init(&vcpu->arch.xen.runstate_cache, vcpu->kvm);
kvm_gpc_init(&vcpu->arch.xen.runstate2_cache, vcpu->kvm);
- __kvm_gpc_init(&vcpu->arch.xen.vcpu_info_cache, vcpu->kvm, true);
+ __kvm_gpc_init(&vcpu->arch.xen.vcpu_info_cache, vcpu->kvm, true, NULL, 0);
kvm_gpc_init(&vcpu->arch.xen.vcpu_time_info_cache, vcpu->kvm);
}
@@ -2486,7 +2486,7 @@ void kvm_xen_init_vm(struct kvm *kvm)
{
mutex_init(&kvm->arch.xen.xen_lock);
xa_init(&kvm->arch.xen.evtchn_ports);
- __kvm_gpc_init(&kvm->arch.xen.shinfo_cache, kvm, true);
+ __kvm_gpc_init(&kvm->arch.xen.shinfo_cache, kvm, true, NULL, 0);
}
void kvm_xen_destroy_vm(struct kvm *kvm)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 3ed0b8c9c102..008012d74d58 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1474,19 +1474,79 @@ int kvm_vcpu_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa, const void *data,
* @gpc: struct gfn_to_pfn_cache object.
* @kvm: pointer to kvm instance.
* @never_dirty: the cache never needs to mark its page dirty in its memslot.
+ * @vcpu: the vCPU which may pin this cache's pfn for direct use from
+ * guest mode, or NULL for a cache which is never pinned.
+ * @vcpu_req: service request posted to @vcpu when a pinned cache is
+ * invalidated; must be non-zero when @vcpu is given (request
+ * 0 is a real request, so it cannot mean "none"). Most
+ * callers want the kvm_gpc_init_for_vcpu() wrapper.
*
* This sets up a gfn_to_pfn_cache by initializing locks and assigning the
* immutable attributes. Note, the cache must be zero-allocated (or zeroed by
* the caller before init). Most callers want the kvm_gpc_init() wrapper.
*/
void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm,
- bool never_dirty);
+ bool never_dirty, struct kvm_vcpu *vcpu, u32 vcpu_req);
static inline void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm)
{
- __kvm_gpc_init(gpc, kvm, false);
+ __kvm_gpc_init(gpc, kvm, false, NULL, 0);
}
+/**
+ * kvm_gpc_init_for_vcpu - initialize a cache whose pfn a vCPU may pin for
+ * direct use from guest mode.
+ *
+ * @gpc: struct gfn_to_pfn_cache object.
+ * @vcpu: the vCPU which will pin and directly access this cache.
+ * @req: service request to post to @vcpu when the cache is invalidated
+ * while pinned. The invalidation separately forces @vcpu out of
+ * guest mode (KVM_REQ_OUTSIDE_GUEST_MODE, with WAIT) and does not
+ * complete until it has left; @req says what the vCPU must then do
+ * before re-entering. Its handler must re-establish the mapping
+ * (via a refresh, which cannot complete until the invalidation
+ * ends) and re-pin before the pfn may be used from guest mode
+ * again, or exit to userspace.
+ */
+static inline void kvm_gpc_init_for_vcpu(struct gfn_to_pfn_cache *gpc,
+ struct kvm_vcpu *vcpu, u32 req)
+{
+ __kvm_gpc_init(gpc, vcpu->kvm, false, vcpu, req);
+}
+
+/**
+ * kvm_gpc_pin_for_guest - pin a valid cache's pfn for use from guest mode.
+ *
+ * @gpc: struct gfn_to_pfn_cache object (initialized with
+ * kvm_gpc_init_for_vcpu()).
+ *
+ * Marks the cache as being used directly by the vCPU in guest mode, so
+ * that an invalidation will kick the vCPU (with the request given at init
+ * time) and wait for it to be out of guest mode before completing.
+ * Succeeds only while the cache is valid: on success the pfn is
+ * guaranteed to remain mapped and unreclaimed until the pin is released
+ * or the vCPU is kicked. Must be called by @gpc->vcpu itself.
+ *
+ * Returns false if the cache is not currently valid; the caller must
+ * refresh and retry, or bail. Also returns false, permanently, for a
+ * cache initialized without a vCPU: a retry loop must not treat pin
+ * failure as transient unless the cache was initialized with
+ * kvm_gpc_init_for_vcpu().
+ */
+bool kvm_gpc_pin_for_guest(struct gfn_to_pfn_cache *gpc);
+
+/**
+ * kvm_gpc_unpin_for_guest - release a guest-mode pin.
+ *
+ * @gpc: struct gfn_to_pfn_cache object.
+ *
+ * Must be called by @gpc->vcpu itself, after it has left guest mode (or
+ * before entering it), when the pfn is no longer stuffed into any
+ * hardware control structure. Idempotent: a pin already consumed by a
+ * mutator's teardown is harmless to release again.
+ */
+void kvm_gpc_unpin_for_guest(struct gfn_to_pfn_cache *gpc);
+
/**
* kvm_gpc_activate - prepare a cached kernel mapping and HPA for a given guest
* physical address.
diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h
index 22393f97266a..bdee01f22a2d 100644
--- a/include/linux/kvm_types.h
+++ b/include/linux/kvm_types.h
@@ -87,21 +87,34 @@ struct gfn_to_pfn_cache {
unsigned long uhva;
struct kvm_memory_slot *memslot;
struct kvm *kvm;
+ /*
+ * A cache initialized with kvm_gpc_init_for_vcpu() may be pinned for
+ * direct use from guest mode (its pfn stuffed into hardware control
+ * structures such as vmcs02 fields) by setting GPC_GUEST_USING while
+ * GPC_VALID holds. When such a cache is invalidated, @vcpu is kicked
+ * out of guest mode with @vcpu_req, whose handler must re-pin (via a
+ * refresh which cannot complete until the invalidation ends) or exit.
+ */
+ struct kvm_vcpu *vcpu;
+ u32 vcpu_req;
struct list_head list;
struct mutex refresh_lock;
void *khva;
kvm_pfn_t pfn;
bool active;
/*
- * GPC_VALID, GPC_BECOMING_VALID and GPC_INVALIDATING live in a
- * single atomic word so that a refresh can publish
- * (BECOMING_VALID => VALID) with a single conditional RMW, an
+ * GPC_VALID, GPC_BECOMING_VALID, GPC_INVALIDATING and
+ * GPC_GUEST_USING live in a single atomic word so that a refresh can
+ * publish (BECOMING_VALID => VALID) with a single conditional RMW, an
* invalidation can veto that publish by clearing both of those bits
* (setting INVALIDATING in the same operation if VALID was set, to
- * record that its grace period is still owed), and a mutator can
- * consume the whole state with one xchg to learn whether it must
- * wait for readers before touching anything they see. See the
- * comments in pfncache.c.
+ * record that its grace period is still owed) and observe in the same
+ * read whether a vCPU has the pfn pinned for guest-mode use
+ * (GUEST_USING, set by the vCPU with a cmpxchg only while VALID
+ * holds, cleared only by that vCPU or by a mutator's xchg), and a
+ * mutator can consume the whole state with one xchg to learn whether
+ * it must wait for readers and/or kick the pinning vCPU before
+ * touching anything they see. See the comments in pfncache.c.
*
* 'active' deliberately stays separate: the invalidation and
* mutation paths operate on the word without having to care about
@@ -129,6 +142,7 @@ struct gfn_to_pfn_cache {
* VALID clear.
*/
#define GPC_INVALIDATING 0x4
+#define GPC_GUEST_USING 0x8
#ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE
/*
diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c
index 1c98b7bdbf09..5cb30525e577 100644
--- a/virt/kvm/pfncache.c
+++ b/virt/kvm/pfncache.c
@@ -25,7 +25,9 @@
void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start,
unsigned long end)
{
+ DECLARE_BITMAP(vcpu_bitmap, KVM_MAX_VCPUS);
struct gfn_to_pfn_cache *gpc;
+ bool evict_vcpus = false;
bool cleared = false;
spin_lock(&kvm->gpc_lock);
@@ -127,10 +129,60 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start,
*/
if (old & (GPC_VALID | GPC_INVALIDATING))
cleared = true;
+
+ /*
+ * If a vCPU has the pfn pinned for direct use from
+ * guest mode, it must be forced out of guest mode
+ * before the invalidation may complete. The pin bit
+ * was set with a cmpxchg only while GPC_VALID held,
+ * so this same read of 'old' cannot miss it. It is
+ * NOT cleared here: only the vCPU itself (on request
+ * service, before re-pinning) or a mutator's xchg
+ * (which inherits the kick duty) may clear it. A
+ * stale pin costs at most a spurious request bit:
+ * kvm_make_vcpus_request_mask() only IPIs and waits
+ * for targets actually in guest mode.
+ */
+ if (old & GPC_GUEST_USING) {
+ if (!evict_vcpus) {
+ evict_vcpus = true;
+ bitmap_zero(vcpu_bitmap, KVM_MAX_VCPUS);
+ }
+ __set_bit(gpc->vcpu->vcpu_idx, vcpu_bitmap);
+ /*
+ * Set this cache's own service request (what
+ * the vCPU must do before re-entering); the
+ * force-out below is common to all of them.
+ */
+ __kvm_make_request(gpc->vcpu_req, gpc->vcpu);
+ }
}
}
spin_unlock(&kvm->gpc_lock);
+ /*
+ * Force pinned vCPUs out of guest mode before waiting for the
+ * kernel readers below and before the caller zaps the page tables.
+ * KVM_REQ_OUTSIDE_GUEST_MODE carries KVM_REQUEST_WAIT, so this
+ * does not return until the vCPUs have left guest mode; they cannot
+ * re-enter with the stale mapping, because re-pinning requires
+ * GPC_VALID, which requires a refresh, which cannot complete until
+ * invalidate_range_end().
+ *
+ * This is safe on unblockable (OOM reaper) ranges too, with no
+ * special-casing: kvm_make_vcpus_request_mask() runs with
+ * preemption disabled and spin-waits for IPI acks; it never sleeps.
+ * Do not be tempted to skip the wait for !blockable ranges on the
+ * assumption that a reaped task's vCPUs are already stopped: that
+ * is false for process_mrelease(), which only requires
+ * SIGNAL_GROUP_EXIT, and a vCPU still in guest mode after this
+ * walk returns is exactly the use-after-free this kick exists to
+ * prevent.
+ */
+ if (evict_vcpus)
+ kvm_make_vcpus_request_mask(kvm, KVM_REQ_OUTSIDE_GUEST_MODE,
+ vcpu_bitmap);
+
/*
* Readers may still be using the old mapping, having sampled
* GPC_VALID before it was cleared. Wait for them all to drain
@@ -224,6 +276,107 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
return true;
}
+/*
+ * A mutator's xchg of the state word to zero may have consumed a live
+ * GPC_GUEST_USING pin. Clearing that bit obliges the clearer to do what
+ * the invalidation walk would have done: post the cache's service
+ * request and force the vCPU out of guest mode, waiting until it has
+ * left, before mutating or tearing down what the guest may be using.
+ *
+ * The service request is posted unconditionally: consuming a pin means
+ * whatever physical address the vCPU had latched (e.g. into a hardware
+ * control structure) is stale, and only the request handler makes that
+ * good. That holds even when the pinning vCPU is the caller itself —
+ * a vCPU refreshing its own pinned cache may re-map at a different pfn,
+ * and without the request nothing would rewrite the stale address
+ * before the next guest entry. Only the waited force-out is skipped in
+ * that case: a vCPU running a refresh is by definition not in guest
+ * mode. (The vCPU's own unpin is the one silent path, in
+ * kvm_gpc_unpin_for_guest(), because unpinning asserts the address is
+ * no longer latched anywhere.)
+ */
+static void gpc_kick_pinned_vcpu(struct gfn_to_pfn_cache *gpc, int old_state)
+{
+ DECLARE_BITMAP(vcpu_bitmap, KVM_MAX_VCPUS);
+
+ if (!(old_state & GPC_GUEST_USING))
+ return;
+ if (WARN_ON_ONCE(!gpc->vcpu))
+ return;
+
+ __kvm_make_request(gpc->vcpu_req, gpc->vcpu);
+
+ if (kvm_get_running_vcpu() == gpc->vcpu)
+ return;
+
+ /*
+ * As in the invalidation walk: force the vCPU out of guest mode and
+ * wait until it has left (KVM_REQ_OUTSIDE_GUEST_MODE carries
+ * KVM_REQUEST_WAIT). A plain kvm_vcpu_kick() would not wait.
+ */
+ bitmap_zero(vcpu_bitmap, KVM_MAX_VCPUS);
+ __set_bit(gpc->vcpu->vcpu_idx, vcpu_bitmap);
+ kvm_make_vcpus_request_mask(gpc->kvm, KVM_REQ_OUTSIDE_GUEST_MODE,
+ vcpu_bitmap);
+}
+
+bool kvm_gpc_pin_for_guest(struct gfn_to_pfn_cache *gpc)
+{
+ int old, new;
+
+ /*
+ * Fail closed on a cache initialized without a vCPU: proceeding
+ * would arm the gpc->vcpu->vcpu_idx dereference in the
+ * invalidation walk, crashing inside the mmu_notifier rather
+ * than in the misusing caller.
+ */
+ if (WARN_ON_ONCE(!gpc->vcpu))
+ return false;
+ WARN_ON_ONCE(kvm_get_running_vcpu() != gpc->vcpu);
+
+ /*
+ * Set GPC_GUEST_USING only while GPC_VALID holds: the pin then
+ * cannot race with an invalidation, because the invalidation walk
+ * reads the whole state word in one atomic operation — either it
+ * sees VALID (and this pin, if set) and kicks; or the publish which
+ * set VALID has not happened and the walk's veto of
+ * GPC_BECOMING_VALID prevents it. The acquire pairs with the
+ * release-publish so the pinned pfn/khva are the published ones.
+ */
+ old = atomic_read_acquire(&gpc->state);
+ do {
+ if (!(old & GPC_VALID))
+ return false;
+ new = old | GPC_GUEST_USING;
+ } while (!atomic_try_cmpxchg(&gpc->state, &old, new));
+
+ return true;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gpc_pin_for_guest);
+
+void kvm_gpc_unpin_for_guest(struct gfn_to_pfn_cache *gpc)
+{
+ /*
+ * Only the pinning vCPU itself may unpin — except on teardown
+ * paths (vCPU destruction), which run with no running-vCPU
+ * context but with the vCPU necessarily out of guest mode.
+ */
+ WARN_ON_ONCE(gpc->vcpu && kvm_get_running_vcpu() &&
+ kvm_get_running_vcpu() != gpc->vcpu);
+
+ /*
+ * The vCPU's last access to the pfn must be visible before the pin
+ * is seen clear by an invalidator deciding it need not kick
+ * (atomic_andnot() alone is unordered). In practice the VM-exit
+ * which preceded this call already serialized the guest's accesses,
+ * but the API should not depend on its callers' exit paths. A pin
+ * already consumed by a mutator's xchg makes this a harmless no-op.
+ */
+ smp_mb__before_atomic();
+ atomic_andnot(GPC_GUEST_USING, &gpc->state);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gpc_unpin_for_guest);
+
static void *gpc_map(kvm_pfn_t pfn)
{
if (pfn_valid(pfn))
@@ -449,7 +602,7 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
bool unmap_old = false;
unsigned long old_uhva;
kvm_pfn_t old_pfn;
- bool must_drain;
+ int old_state;
void *old_khva;
int ret;
@@ -550,9 +703,9 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
* only this wait covers. The marker comes down only once the drain
* is complete and the readers it advertised cannot exist.
*/
- must_drain = atomic_xchg(&gpc->state, GPC_INVALIDATING) &
- (GPC_VALID | GPC_INVALIDATING);
- if (must_drain)
+ old_state = atomic_xchg(&gpc->state, GPC_INVALIDATING);
+ gpc_kick_pinned_vcpu(gpc, old_state);
+ if (old_state & (GPC_VALID | GPC_INVALIDATING))
synchronize_srcu_atomic(&gpc->kvm->gpc_srcu);
atomic_set(&gpc->state, 0);
@@ -650,7 +803,7 @@ int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsigned long len)
}
void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm,
- bool never_dirty)
+ bool never_dirty, struct kvm_vcpu *vcpu, u32 vcpu_req)
{
mutex_init(&gpc->refresh_lock);
@@ -661,6 +814,18 @@ void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm,
gpc->active = false;
atomic_set(&gpc->state, 0);
gpc->never_dirty = never_dirty;
+
+ /*
+ * A vCPU-owned cache must come with a service request for the
+ * invalidation walk to post; request 0 is a real request
+ * (KVM_REQ_TLB_FLUSH on x86), so a missing one cannot be allowed
+ * to alias it. Refuse the vcpu rather than arm that footgun —
+ * the pin API then fails closed on !gpc->vcpu.
+ */
+ if (WARN_ON_ONCE(vcpu && !vcpu_req))
+ vcpu = NULL;
+ gpc->vcpu = vcpu;
+ gpc->vcpu_req = vcpu_req;
}
static int __kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned long uhva,
@@ -718,6 +883,7 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc)
struct kvm *kvm = gpc->kvm;
kvm_pfn_t old_pfn;
void *old_khva;
+ int old_state;
guard(mutex)(&gpc->refresh_lock);
@@ -741,8 +907,9 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc)
* the drain completes, so that an mmu_notifier walk arriving
* mid-wait does not read a zero state and skip the cache.
*/
- if (atomic_xchg(&gpc->state, GPC_INVALIDATING) &
- (GPC_VALID | GPC_INVALIDATING))
+ old_state = atomic_xchg(&gpc->state, GPC_INVALIDATING);
+ gpc_kick_pinned_vcpu(gpc, old_state);
+ if (old_state & (GPC_VALID | GPC_INVALIDATING))
synchronize_srcu_atomic(&kvm->gpc_srcu);
atomic_set(&gpc->state, 0);
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v2 10/20] KVM: pfncache: Add guest-mode pinning (GUEST_USES_PFN successor)
2026-11-05 2:00 ` [PATCH v2 10/20] KVM: pfncache: Add guest-mode pinning (GUEST_USES_PFN successor) David Woodhouse
@ 2026-11-05 2:29 ` sashiko-bot
0 siblings, 0 replies; 25+ messages in thread
From: sashiko-bot @ 2026-11-05 2:29 UTC (permalink / raw)
To: David Woodhouse, kvm; +Cc: linux-rt-devel, linux-kernel
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] KVM: pfncache: gpc_kick_pinned_vcpu() re-posts vcpu_req on self-refresh
--
commit 9027e5fc5e4260b0089300dc0cf3173581bbcfed
Author: David Woodhouse <dwmw@amazon.co.uk>
KVM: pfncache: Add guest-mode pinning (GUEST_USES_PFN successor)
This commit adds guest-mode pinning support to gfn_to_pfn_cache so that
caches whose pfn is used directly from guest mode force the vCPU out of
guest mode and post a per-cache service request upon invalidation or
mutation.
> diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c
> index 1c98b7bdbf09c..5cb30525e577c 100644
> --- a/virt/kvm/pfncache.c
> +++ b/virt/kvm/pfncache.c
> @@ -25,7 +25,9 @@
> void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start,
> unsigned long end)
> {
> + DECLARE_BITMAP(vcpu_bitmap, KVM_MAX_VCPUS);
> struct gfn_to_pfn_cache *gpc;
> + bool evict_vcpus = false;
> bool cleared = false;
>
> spin_lock(&kvm->gpc_lock);
> @@ -127,10 +129,60 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start,
> */
> if (old & (GPC_VALID | GPC_INVALIDATING))
> cleared = true;
> +
> + /*
> + * If a vCPU has the pfn pinned for direct use from
> + * guest mode, it must be forced out of guest mode
> + * before the invalidation may complete. The pin bit
> + * was set with a cmpxchg only while GPC_VALID held,
> + * so this same read of 'old' cannot miss it. It is
> + * NOT cleared here: only the vCPU itself (on request
> + * service, before re-pinning) or a mutator's xchg
> + * (which inherits the kick duty) may clear it. A
> + * stale pin costs at most a spurious request bit:
> + * kvm_make_vcpus_request_mask() only IPIs and waits
> + * for targets actually in guest mode.
> + */
> + if (old & GPC_GUEST_USING) {
> + if (!evict_vcpus) {
> + evict_vcpus = true;
> + bitmap_zero(vcpu_bitmap, KVM_MAX_VCPUS);
> + }
> + __set_bit(gpc->vcpu->vcpu_idx, vcpu_bitmap);
> + /*
> + * Set this cache's own service request (what
> + * the vCPU must do before re-entering); the
> + * force-out below is common to all of them.
> + */
> + __kvm_make_request(gpc->vcpu_req, gpc->vcpu);
> + }
> }
> }
> spin_unlock(&kvm->gpc_lock);
[ ... ]
> @@ -224,6 +276,107 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
> return true;
> }
>
> +/*
> + * A mutator's xchg of the state word to zero may have consumed a live
> + * GPC_GUEST_USING pin. Clearing that bit obliges the clearer to do what
> + * the invalidation walk would have done: post the cache's service
> + * request and force the vCPU out of guest mode, waiting until it has
> + * left, before mutating or tearing down what the guest may be using.
> + *
> + * The service request is posted unconditionally: consuming a pin means
> + * whatever physical address the vCPU had latched (e.g. into a hardware
> + * control structure) is stale, and only the request handler makes that
> + * good. That holds even when the pinning vCPU is the caller itself —
> + * a vCPU refreshing its own pinned cache may re-map at a different pfn,
> + * and without the request nothing would rewrite the stale address
> + * before the next guest entry. Only the waited force-out is skipped in
> + * that case: a vCPU running a refresh is by definition not in guest
> + * mode. (The vCPU's own unpin is the one silent path, in
> + * kvm_gpc_unpin_for_guest(), because unpinning asserts the address is
> + * no longer latched anywhere.)
> + */
> +static void gpc_kick_pinned_vcpu(struct gfn_to_pfn_cache *gpc, int old_state)
> +{
> + DECLARE_BITMAP(vcpu_bitmap, KVM_MAX_VCPUS);
> +
> + if (!(old_state & GPC_GUEST_USING))
> + return;
> + if (WARN_ON_ONCE(!gpc->vcpu))
> + return;
> +
> + __kvm_make_request(gpc->vcpu_req, gpc->vcpu);
> +
> + if (kvm_get_running_vcpu() == gpc->vcpu)
> + return;
[Severity: Medium]
When gfn_to_pfn_cache_invalidate_start() invalidates a pinned cache, it
clears GPC_VALID, sets GPC_INVALIDATING, leaves GPC_GUEST_USING set in
gpc->state, and posts gpc->vcpu_req to gpc->vcpu:
do {
new = old & ~(GPC_VALID | GPC_BECOMING_VALID);
if (old & GPC_VALID)
new |= GPC_INVALIDATING;
} while (!atomic_try_cmpxchg(&gpc->state, &old, new));
When gpc->vcpu subsequently services gpc->vcpu_req on the next guest entry
attempt, kvm_check_request() clears gpc->vcpu_req and the request handler
calls kvm_gpc_refresh() (or kvm_gpc_activate()) before re-pinning with
kvm_gpc_pin_for_guest(), as described in kvm_gpc_init_for_vcpu().
Because GPC_VALID is clear, __kvm_gpc_refresh() does not take the fast path
and instead consumes the leftover GPC_GUEST_USING bit via atomic_xchg():
old_state = atomic_xchg(&gpc->state, GPC_INVALIDATING);
gpc_kick_pinned_vcpu(gpc, old_state);
Won't gpc_kick_pinned_vcpu() see GPC_GUEST_USING in old_state and
unconditionally re-post gpc->vcpu_req to gpc->vcpu right in the middle of
handling that very request, causing the subsequent guest entry to abort and
run the request handler a second time (or loop if the handler does not hit
the __kvm_gpc_refresh() fast path on the next pass)?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261105020104.2507083-1-dwmw2@infradead.org?part=10
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v2 11/20] KVM: nVMX: Implement cache for L1 MSR bitmap
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (9 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 10/20] KVM: pfncache: Add guest-mode pinning (GUEST_USES_PFN successor) David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 12/20] KVM: nVMX: Use pinned pfncache for L1 APIC virtualization pages David Woodhouse
` (8 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: Fred Griffoul <fgriffo@amazon.co.uk>
Introduce a gfn_to_pfn_cache to optimize L1 MSR bitmap access by
replacing map/unmap operations. This optimization reduces overhead
during L2 VM-entry where nested_vmx_prepare_msr_bitmap() merges L1's
MSR intercepts with L0's requirements.
The current implementation maps and unmaps the page on every bitmap
merge, which is a significant performance impact with unmanaged guest
memory (where the map is a memremap/memunmap cycle).
The cache is initialized once at vCPU creation and deactivated when
VMX operation ends; re-entering VMX operation reactivates it through
the normal check/activate path. Readers run under kvm->gpc_srcu,
following the check/activate/retry pattern; KVM only ever reads the
L1 bitmap, so the cache is mapped read-only (which L1 may
legitimately demand for its bitmap) and never marked dirty.
This exports the core pfncache API to kvm-intel.ko, which is its
first modular user.
[dwmw2: Port from the rwlock-protected pfncache to the SRCU reader
protocol, rebase over the kvm_vcpu_map_local_readonly CLASS()
conversion, introduce the lock/unlock helpers in x86.h as
kvm_gpc_lock_page()/kvm_gpc_unlock() for later nSVM use, map
the cache read-only, init at vCPU creation.]
Reported-by: sashiko-bot@kernel.org
Closes: https://lore.kernel.org/all/20260920215613.1CC0C1F000FF@smtp.kernel.org
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
Co-developed-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kvm/vmx/nested.c | 30 +++++++++++++++++++++++++----
arch/x86/kvm/vmx/nested.h | 1 +
arch/x86/kvm/vmx/vmx.c | 2 ++
arch/x86/kvm/vmx/vmx.h | 2 ++
arch/x86/kvm/x86.h | 33 ++++++++++++++++++++++++++++++++
arch/x86/kvm/xen.c | 4 ++--
include/linux/kvm_host.h | 40 +++++++++++++++++++++++++++------------
include/linux/kvm_types.h | 12 ++++++++++++
virt/kvm/pfncache.c | 37 +++++++++++++++++++++++++++++++-----
9 files changed, 138 insertions(+), 23 deletions(-)
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index 516ba03bf608..bb0baa79c28c 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -370,6 +370,8 @@ static void free_nested(struct kvm_vcpu *vcpu)
kfree(vmx->nested.cached_shadow_vmcs12);
vmx->nested.cached_shadow_vmcs12 = NULL;
+ kvm_gpc_deactivate(&vmx->nested.msr_bitmap_cache);
+
nested_put_vmcs12_pages(vcpu);
kvm_mmu_free_roots(vcpu->kvm, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL);
@@ -390,6 +392,24 @@ void nested_vmx_free_vcpu(struct kvm_vcpu *vcpu)
vcpu_put(vcpu);
}
+/*
+ * Initialize the nested gfn_to_pfn_caches, once per vCPU. Unlike the
+ * allocations in enter_vmx_operation(), the caches persist across
+ * VMXOFF/VMXON cycles: free_nested() only deactivates them, and reuse
+ * goes through kvm_gpc_activate() without re-init.
+ */
+void nested_vmx_init_caches(struct kvm_vcpu *vcpu)
+{
+ struct vcpu_vmx *vmx = to_vmx(vcpu);
+
+ /*
+ * KVM only ever reads the L1 MSR bitmap, which L1 may keep in
+ * memory that its VMM maps read-only: map the cache read-only
+ * (which also means never marking it dirty).
+ */
+ __kvm_gpc_init(&vmx->nested.msr_bitmap_cache, vcpu->kvm, false, true, NULL, 0);
+}
+
#define EPTP_PA_MASK GENMASK_ULL(51, 12)
static bool nested_ept_root_matches(hpa_t root_hpa, u64 root_eptp, u64 eptp)
@@ -734,7 +754,7 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,
struct vmcs12 *vmcs12)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
- int msr;
+ int msr, idx;
unsigned long *msr_bitmap_l1;
unsigned long *msr_bitmap_l0 = vmx->nested.vmcs02.msr_bitmap;
@@ -759,11 +779,11 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,
return true;
}
- CLASS(kvm_vcpu_map_local_readonly, m)(vcpu, gpa_to_gfn(vmcs12->msr_bitmap));
- if (m.ret)
+ idx = kvm_gpc_lock_page(&vmx->nested.msr_bitmap_cache, vmcs12->msr_bitmap);
+ if (idx < 0)
return false;
- msr_bitmap_l1 = (unsigned long *)m.map.hva;
+ msr_bitmap_l1 = (unsigned long *)vmx->nested.msr_bitmap_cache.khva;
/*
* To keep the control flow simple, pay eight 8-byte writes (sixteen
@@ -843,6 +863,8 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,
nested_vmx_merge_pmu_msr_bitmaps(vcpu, msr_bitmap_l1, msr_bitmap_l0);
+ kvm_gpc_unlock(&vmx->nested.msr_bitmap_cache, idx);
+
vmx->nested.force_msr_bitmap_recalc = false;
return true;
diff --git a/arch/x86/kvm/vmx/nested.h b/arch/x86/kvm/vmx/nested.h
index c6de848bd9ce..e27299b9f30b 100644
--- a/arch/x86/kvm/vmx/nested.h
+++ b/arch/x86/kvm/vmx/nested.h
@@ -24,6 +24,7 @@ __init int nested_vmx_hardware_setup(int (*exit_handlers[])(struct kvm_vcpu *));
void nested_vmx_set_vmcs_shadowing_bitmap(void);
int nested_vmx_check_restored_vmcs12(struct kvm_vcpu *vcpu);
void nested_vmx_free_vcpu(struct kvm_vcpu *vcpu);
+void nested_vmx_init_caches(struct kvm_vcpu *vcpu);
enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
bool from_vmentry);
bool nested_vmx_reflect_vmexit(struct kvm_vcpu *vcpu);
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 82eb891f684c..590095bcb665 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -7613,6 +7613,8 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu)
WRITE_ONCE(to_kvm_vmx(vcpu->kvm)->pid_table[vcpu->vcpu_id],
__pa(&vmx->vt.pi_desc) | PID_TABLE_ENTRY_VALID);
+ nested_vmx_init_caches(vcpu);
+
return 0;
free_vmcs:
diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h
index 74cd50f29fa4..c6827368c84a 100644
--- a/arch/x86/kvm/vmx/vmx.h
+++ b/arch/x86/kvm/vmx/vmx.h
@@ -146,6 +146,8 @@ struct nested_vmx {
* Guest pages referred to in the vmcs02 with host-physical
* pointers, so we must keep them pinned while L2 runs.
*/
+ struct gfn_to_pfn_cache msr_bitmap_cache;
+
struct kvm_host_map apic_access_page_map;
struct kvm_host_map virtual_apic_map;
struct kvm_host_map pi_desc_map;
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index 436e2fb1396c..c48f3aeb4d64 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -14,6 +14,39 @@
#define KVM_MAX_MCE_BANKS 32
+/*
+ * Map a single page of L1 guest memory at @gpa into a gfn_to_pfn_cache
+ * and enter an SRCU read-side critical section protecting the mapping.
+ * Returns the SRCU index (>= 0) to pass to kvm_gpc_unlock(), or a
+ * negative error code. While locked, the cache's khva and pfn are
+ * guaranteed to remain valid: an invalidation cannot complete until the
+ * reader drains.
+ */
+static inline int kvm_gpc_lock_page(struct gfn_to_pfn_cache *gpc, gpa_t gpa)
+{
+ int idx, err;
+
+ if (!PAGE_ALIGNED(gpa))
+ return -EINVAL;
+retry:
+ idx = srcu_read_lock_atomic(&gpc->kvm->gpc_srcu);
+ if (!kvm_gpc_check(gpc, PAGE_SIZE) || gpc->gpa != gpa) {
+ srcu_read_unlock_atomic(&gpc->kvm->gpc_srcu, idx);
+ err = kvm_gpc_activate(gpc, gpa, PAGE_SIZE);
+ if (err)
+ return err;
+
+ goto retry;
+ }
+
+ return idx;
+}
+
+static inline void kvm_gpc_unlock(struct gfn_to_pfn_cache *gpc, int idx)
+{
+ srcu_read_unlock_atomic(&gpc->kvm->gpc_srcu, idx);
+}
+
int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops);
void kvm_x86_vendor_exit(void);
diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 132f0c379537..ecfeeea57c66 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -2465,7 +2465,7 @@ void kvm_xen_init_vcpu(struct kvm_vcpu *vcpu)
kvm_gpc_init(&vcpu->arch.xen.runstate_cache, vcpu->kvm);
kvm_gpc_init(&vcpu->arch.xen.runstate2_cache, vcpu->kvm);
- __kvm_gpc_init(&vcpu->arch.xen.vcpu_info_cache, vcpu->kvm, true, NULL, 0);
+ __kvm_gpc_init(&vcpu->arch.xen.vcpu_info_cache, vcpu->kvm, true, false, NULL, 0);
kvm_gpc_init(&vcpu->arch.xen.vcpu_time_info_cache, vcpu->kvm);
}
@@ -2486,7 +2486,7 @@ void kvm_xen_init_vm(struct kvm *kvm)
{
mutex_init(&kvm->arch.xen.xen_lock);
xa_init(&kvm->arch.xen.evtchn_ports);
- __kvm_gpc_init(&kvm->arch.xen.shinfo_cache, kvm, true, NULL, 0);
+ __kvm_gpc_init(&kvm->arch.xen.shinfo_cache, kvm, true, false, NULL, 0);
}
void kvm_xen_destroy_vm(struct kvm *kvm)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 008012d74d58..c604b7bb0b4c 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1471,26 +1471,35 @@ int kvm_vcpu_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa, const void *data,
/**
* __kvm_gpc_init - initialize gfn_to_pfn_cache.
*
- * @gpc: struct gfn_to_pfn_cache object.
- * @kvm: pointer to kvm instance.
- * @never_dirty: the cache never needs to mark its page dirty in its memslot.
- * @vcpu: the vCPU which may pin this cache's pfn for direct use from
- * guest mode, or NULL for a cache which is never pinned.
- * @vcpu_req: service request posted to @vcpu when a pinned cache is
- * invalidated; must be non-zero when @vcpu is given (request
- * 0 is a real request, so it cannot mean "none"). Most
- * callers want the kvm_gpc_init_for_vcpu() wrapper.
+ * @gpc: struct gfn_to_pfn_cache object.
+ * @kvm: pointer to kvm instance.
+ * @never_dirty: KVM never marks the cached page dirty in its memslot;
+ * userspace must assume the page is always dirty (as for
+ * the Xen shared_info page, written on every event channel
+ * delivery). kvm_gpc_mark_dirty_in_slot() becomes a no-op.
+ * @readonly: map without FOLL_WRITE; KVM must never write through
+ * @gpc->khva. For guest pages KVM only reads, which the
+ * guest may place in memory its VMM maps read-only.
+ * Implies @never_dirty.
+ * @vcpu: the vCPU which may pin this cache's pfn for direct use
+ * from guest mode, or NULL. Callers use the
+ * kvm_gpc_init() / kvm_gpc_init_for_vcpu() wrappers
+ * rather than passing this directly.
+ * @vcpu_req: service request posted to @vcpu when the cache is
+ * invalidated while pinned; required (non-zero) when
+ * @vcpu is given.
*
* This sets up a gfn_to_pfn_cache by initializing locks and assigning the
* immutable attributes. Note, the cache must be zero-allocated (or zeroed by
* the caller before init). Most callers want the kvm_gpc_init() wrapper.
*/
void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm,
- bool never_dirty, struct kvm_vcpu *vcpu, u32 vcpu_req);
+ bool never_dirty, bool readonly,
+ struct kvm_vcpu *vcpu, u32 vcpu_req);
static inline void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm)
{
- __kvm_gpc_init(gpc, kvm, false, NULL, 0);
+ __kvm_gpc_init(gpc, kvm, false, false, NULL, 0);
}
/**
@@ -1511,7 +1520,7 @@ static inline void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm)
static inline void kvm_gpc_init_for_vcpu(struct gfn_to_pfn_cache *gpc,
struct kvm_vcpu *vcpu, u32 req)
{
- __kvm_gpc_init(gpc, vcpu->kvm, false, vcpu, req);
+ __kvm_gpc_init(gpc, vcpu->kvm, false, false, vcpu, req);
}
/**
@@ -2054,6 +2063,13 @@ static inline void kvm_gpc_mark_dirty_in_slot(struct gfn_to_pfn_cache *gpc)
RCU_LOCKDEP_WARN(!srcu_read_lock_held(&gpc->kvm->gpc_srcu),
"kvm_gpc_mark_dirty_in_slot() without kvm->gpc_srcu read lock");
+ /*
+ * Nobody may write through (and hence dirty) a read-only cache:
+ * its mapping was created without FOLL_WRITE and may lack write
+ * permission entirely.
+ */
+ WARN_ON_ONCE(gpc->readonly);
+
if (!gpc->memslot || gpc->never_dirty)
return;
diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h
index bdee01f22a2d..193dd5b64c2a 100644
--- a/include/linux/kvm_types.h
+++ b/include/linux/kvm_types.h
@@ -122,7 +122,19 @@ struct gfn_to_pfn_cache {
* instead.
*/
atomic_t state;
+ /*
+ * KVM never marks the cached page dirty in its memslot; userspace
+ * must assume the page is always dirty (as for the Xen shared_info
+ * page, written on every event channel delivery).
+ */
bool never_dirty;
+ /*
+ * The cache is mapped without FOLL_WRITE and KVM must never write
+ * through khva: for guest pages KVM only reads (e.g. the L1 MSR
+ * bitmap), which the guest may legitimately place in memory its VMM
+ * maps read-only. Implies never_dirty (there is nothing to mark).
+ */
+ bool readonly;
};
/* The cache is valid: pfn and khva may be used by readers. */
diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c
index 5cb30525e577..42adfdccd6fe 100644
--- a/virt/kvm/pfncache.c
+++ b/virt/kvm/pfncache.c
@@ -223,6 +223,20 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start,
*/
}
+/*
+ * As gfn_to_hva_memslot(), but tolerate a KVM_MEM_READONLY slot for a
+ * cache which promises never to write through its mapping.
+ */
+static unsigned long gpc_gfn_to_hva_memslot(struct gfn_to_pfn_cache *gpc,
+ struct kvm_memory_slot *slot,
+ gfn_t gfn)
+{
+ if (gpc->readonly)
+ return gfn_to_hva_memslot_prot(slot, gfn, NULL);
+
+ return gfn_to_hva_memslot(slot, gfn);
+}
+
static bool kvm_gpc_is_valid_len(gpa_t gpa, unsigned long uhva,
unsigned long len)
{
@@ -275,6 +289,7 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
return true;
}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gpc_check);
/*
* A mutator's xchg of the state word to zero may have consumed a live
@@ -448,7 +463,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)
struct kvm_follow_pfn kfp = {
.slot = gpc->memslot,
.gfn = gpa_to_gfn(gpc->gpa),
- .flags = FOLL_WRITE,
+ .flags = gpc->readonly ? 0 : FOLL_WRITE,
.hva = gpc->uhva,
.refcounted_page = &page,
};
@@ -655,7 +670,7 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
new_generation = slots->generation;
new_slot = __gfn_to_memslot(slots, gfn);
if (new_slot)
- new_uhva = gfn_to_hva_memslot(new_slot, gfn) +
+ new_uhva = gpc_gfn_to_hva_memslot(gpc, new_slot, gfn) +
offset_in_page(gpa);
}
@@ -738,7 +753,8 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
gpc->gpa = gpa;
gpc->generation = slots->generation;
gpc->memslot = __gfn_to_memslot(slots, gfn);
- WRITE_ONCE(gpc->uhva, gfn_to_hva_memslot(gpc->memslot, gfn));
+ WRITE_ONCE(gpc->uhva,
+ gpc_gfn_to_hva_memslot(gpc, gpc->memslot, gfn));
if (kvm_is_error_hva(gpc->uhva)) {
ret = -EFAULT;
@@ -801,9 +817,11 @@ int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsigned long len)
return __kvm_gpc_refresh(gpc, gpc->gpa, uhva);
}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gpc_refresh);
void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm,
- bool never_dirty, struct kvm_vcpu *vcpu, u32 vcpu_req)
+ bool never_dirty, bool readonly,
+ struct kvm_vcpu *vcpu, u32 vcpu_req)
{
mutex_init(&gpc->refresh_lock);
@@ -813,7 +831,13 @@ void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm,
gpc->uhva = KVM_HVA_ERR_BAD;
gpc->active = false;
atomic_set(&gpc->state, 0);
- gpc->never_dirty = never_dirty;
+ /*
+ * A read-only cache maps without FOLL_WRITE, so writing through
+ * gpc->khva would hit a mapping which may lack write permission;
+ * there is never anything to mark dirty.
+ */
+ gpc->never_dirty = never_dirty || readonly;
+ gpc->readonly = readonly;
/*
* A vCPU-owned cache must come with a service request for the
@@ -827,6 +851,7 @@ void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm,
gpc->vcpu = vcpu;
gpc->vcpu_req = vcpu_req;
}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_gpc_init);
static int __kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned long uhva,
unsigned long len)
@@ -869,6 +894,7 @@ int kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned long len)
return __kvm_gpc_activate(gpc, gpa, KVM_HVA_ERR_BAD, len);
}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gpc_activate);
int kvm_gpc_activate_hva(struct gfn_to_pfn_cache *gpc, unsigned long uhva, unsigned long len)
{
@@ -932,3 +958,4 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc)
gpc_unmap(old_pfn, old_khva);
}
}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gpc_deactivate);
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 12/20] KVM: nVMX: Use pinned pfncache for L1 APIC virtualization pages
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (10 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 11/20] KVM: nVMX: Implement cache for L1 MSR bitmap David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:33 ` sashiko-bot
2026-11-05 2:00 ` [PATCH v2 13/20] KVM: x86: Post KVM_REQ_GET_NESTED_STATE_PAGES on memslot updates David Woodhouse
` (7 subsequent siblings)
19 siblings, 1 reply; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: Fred Griffoul <fgriffo@amazon.co.uk>
Replace kvm_host_map usage with gfn_to_pfn_cache for the L1 pages whose
physical addresses are handed to the CPU in vmcs02 for direct use while
running L2: the APIC-access page, the virtual-APIC page, and the posted
interrupt descriptor. This eliminates the map/unmap (with unmanaged
memory: memremap/memunmap) cycle on every L2 VM-entry/exit.
The caches are initialized once at vCPU creation with
kvm_gpc_init_for_vcpu(), naming KVM_REQ_GET_NESTED_STATE_PAGES as the
service request, and pinned when written into vmcs02 by
nested_get_vmcs12_pages(). An invalidation of a pinned cache posts
that request, forces the vCPU out of guest mode and waits; the
handler re-runs nested_get_vmcs12_pages(). Pins are dropped at
nested VM-exit (including the vmentry_fail_vmexit path, which does
not pass through __nested_vmx_vmexit()).
No validity check is added to the guest-entry path: a stale physical
address latched in vmcs02 always implies a pending request. Every
path which consumes a pin posts the request, and the handler cannot
lose it: any failure of vmx_get_nested_state_pages() re-posts it
before exiting to userspace, so a persistently failing configuration
bounces to userspace rather than ever entering L2 stale. This closes
the historical KVM_GUEST_USES_PFN gaps¹ without polling cache
validity on every entry.
Resolution failure distinguishes three cases. A slot mid-update (the
pfncache reports -EFAULT; finding the KVM_MEMSLOT_INVALID placeholder
still installed identifies the failure as transient) re-posts the
request and continues, retrying until the update commits or reverts.
At entry time (VMLAUNCH/VMRESUME, or resume from
KVM_SET_NESTED_STATE, tracked in nested.resume_pending — which dies
with the request at nested VM-exit, lest a later lazy request
inherit entry-time semantics) a genuinely unbackable page is a
misconfiguration: KVM_EXIT_INTERNAL_ERROR, as ever. On lazy
revalidation it means userspace yanked the backing from under a
running L2: for the APIC-access page and the posted interrupt
descriptor, clear the corresponding vmcs02 control instead of
killing the VM, so L2's accesses become ordinary EPT violations
handled as MMIO; the successful re-resolution path sets the control
back, so a later update re-backing the gfn heals the degradation
without waiting for a nested VM-entry. The virtual-APIC page allows
no such degradation (the TPR shadow is live, and the entry-time
INVALID_GPA fallback would deliver a VMfailValid to an L1 whose
VMLAUNCH retired long ago), and an unbacked gfn cannot be an error
either: a routine DELETE-then-recreate reshuffle has a committed gap
in which the gfn is genuinely absent, and the lazy revalidation runs
inside that gap precisely because the DELETE's commit posted the
request. So a vAPIC loss while L2 runs is ridden out like the
in-flight window: the pending request keeps L2 un-enterable until a
later update re-backs the gfn, and a userspace which never does
leaves its vCPU retrying in KVM_RUN, as for RET_PF_RETRY. The PAE
PDPTR reload runs only at
entry-time resolution: PDPTRs latch at MOV CR3 and VM-entry, so a
lazy revalidation must not re-derive them.
Host-side (khva) readers do not need the pin.
vmx_complete_nested_posted_interrupt() resolves both caches before
entering either one's atomic read section, and
vmx_has_nested_events(), which runs under set_current_state() in the
block loop where a refresh must not sleep, uses a non-sleeping
try-lock and reports "event pending" for an unusable cache, posting
KVM_REQ_GET_NESTED_STATE_PAGES itself (a bare memslot-generation
bump fails the check without anything else posting it): the bounce
through vcpu_run re-establishes the cache and re-evaluates for real,
and a failed activation deactivates the cache so an unbackable page
cannot livelock that conservative answer. The try-lock takes
kvm->srcu across its memslot-generation check; callers in the
vcpu_block() loop run with the vCPU's srcu lock dropped.
Dirty marking moves from the unmap path to gfn-based marking on
every L2 exit (__vmx_handle_exit() already did that, making
unmap-time marking redundant); the one software writer,
__kvm_apic_update_irr(), marks at the write site.
¹ https://lore.kernel.org/all/ZBEEQtmtNPaEqU1i@google.com
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
Co-developed-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kvm/vmx/nested.c | 483 ++++++++++++++++++++++++++++++++++----
arch/x86/kvm/vmx/vmx.c | 11 +-
arch/x86/kvm/vmx/vmx.h | 21 +-
3 files changed, 458 insertions(+), 57 deletions(-)
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index bb0baa79c28c..23f1fff97917 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -320,14 +320,140 @@ static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs)
kvm_reset_dirty_registers(vcpu);
}
+/*
+ * Map a page of L1 memory, pin it for direct use by the guest (i.e. by
+ * the CPU while running the L2 guest, via a physical address in vmcs02),
+ * and return the host physical address to write into vmcs02. The pin
+ * guarantees that if the mapping is invalidated, this vCPU is forced out
+ * of guest mode and KVM_REQ_GET_NESTED_STATE_PAGES is posted so that the
+ * address is re-established (or the vCPU exits to userspace) before the
+ * next entry to guest mode.
+ */
+static int nested_gpc_hpa(struct gfn_to_pfn_cache *gpc, gpa_t gpa, hpa_t *hpa)
+{
+ int idx;
+
+ do {
+ idx = kvm_gpc_lock_page(gpc, gpa);
+ if (idx == -EFAULT) {
+ struct kvm_memory_slot *slot;
+
+ /*
+ * An unbackable gfn and a memslot mid-update (a
+ * DELETE or MOVE keeps a KVM_MEMSLOT_INVALID
+ * placeholder installed until it commits) both fail
+ * uHVA resolution with -EFAULT. Distinguish them:
+ * finding the placeholder in the active memslots
+ * (kvm->srcu is held by every caller) means the
+ * failure is transient, and the callers' -EAGAIN
+ * handling — re-post the request, don't enter L2 —
+ * is what must ride it out; only a truly absent
+ * slot means the page is unbacked.
+ */
+ slot = gfn_to_memslot(gpc->kvm, gpa_to_gfn(gpa));
+ if (slot && (slot->flags & KVM_MEMSLOT_INVALID))
+ return -EAGAIN;
+ }
+ if (idx < 0)
+ return idx;
+
+ /*
+ * The pin can fail only if an invalidation cleared the
+ * valid bit after kvm_gpc_lock_page() checked it; go back
+ * around to refresh (which will not complete until the
+ * invalidation is over) and try again.
+ */
+ if (kvm_gpc_pin_for_guest(gpc))
+ break;
+
+ kvm_gpc_unlock(gpc, idx);
+ } while (1);
+
+ *hpa = pfn_to_hpa(gpc->pfn);
+ kvm_gpc_unlock(gpc, idx);
+ return 0;
+}
+
+/*
+ * Try to lock an already-configured cache for host-side (khva) access,
+ * without sleeping. Returns -ENOENT if the cache is not active (e.g.
+ * posted interrupts not configured for this L2), -EWOULDBLOCK if it is
+ * active but currently invalidated (only a sleeping refresh would make
+ * it usable). Does not pin; the mapping is stable only until
+ * kvm_gpc_unlock().
+ */
+static int nested_gpc_try_lock_if_active(struct gfn_to_pfn_cache *gpc)
+{
+ int idx, srcu_idx;
+ bool usable;
+
+ /*
+ * The caches are initialized at vCPU creation, so gpc->kvm is
+ * always valid; !active covers a cache never activated (or since
+ * deactivated) for this L2. Deactivation only happens on this
+ * vCPU (under vcpu->mutex), so the check cannot race with the
+ * lock which follows it.
+ */
+ if (!gpc->active)
+ return -ENOENT;
+
+ idx = srcu_read_lock_atomic(&gpc->kvm->gpc_srcu);
+ if (!gpc->active) {
+ srcu_read_unlock_atomic(&gpc->kvm->gpc_srcu, idx);
+ return -ENOENT;
+ }
+
+ /*
+ * kvm_gpc_check() consults kvm_memslots() (for the generation of
+ * a GPA-based cache), which is protected by kvm->srcu — and not
+ * every caller holds that: kvm_arch_vcpu_runnable() runs from the
+ * vcpu_block() loop with the vCPU's srcu lock deliberately
+ * dropped. Take it across the check; the khva the caller then
+ * reads is protected by gpc_srcu, not kvm->srcu.
+ */
+ srcu_idx = srcu_read_lock(&gpc->kvm->srcu);
+ usable = kvm_gpc_check(gpc, PAGE_SIZE);
+ srcu_read_unlock(&gpc->kvm->srcu, srcu_idx);
+
+ if (!usable) {
+ srcu_read_unlock_atomic(&gpc->kvm->gpc_srcu, idx);
+ return -EWOULDBLOCK;
+ }
+
+ return idx;
+}
+
+/*
+ * As above, but refresh the cache if it has been invalidated. May sleep;
+ * callers must be in a sleepable context (in particular, NOT under
+ * set_current_state() in a block loop, and not in a VM-exit fastpath).
+ */
+static int nested_gpc_lock_if_active(struct gfn_to_pfn_cache *gpc)
+{
+ int idx, err;
+
+ while ((idx = nested_gpc_try_lock_if_active(gpc)) == -EWOULDBLOCK) {
+ err = kvm_gpc_refresh(gpc, PAGE_SIZE);
+ if (err)
+ return err;
+ }
+
+ return idx;
+}
+
+static struct pi_desc *nested_pi_desc(struct vcpu_vmx *vmx)
+{
+ return (struct pi_desc *)((u8 *)vmx->nested.pi_desc_cache.khva +
+ vmx->nested.pi_desc_offset);
+}
+
static void nested_put_vmcs12_pages(struct kvm_vcpu *vcpu)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
- kvm_vcpu_unmap(vcpu, &vmx->nested.apic_access_page_map);
- kvm_vcpu_unmap(vcpu, &vmx->nested.virtual_apic_map);
- kvm_vcpu_unmap(vcpu, &vmx->nested.pi_desc_map);
- vmx->nested.pi_desc = NULL;
+ kvm_gpc_unpin_for_guest(&vmx->nested.apic_access_page_cache);
+ kvm_gpc_unpin_for_guest(&vmx->nested.virtual_apic_cache);
+ kvm_gpc_unpin_for_guest(&vmx->nested.pi_desc_cache);
}
/*
@@ -346,6 +472,7 @@ static void free_nested(struct kvm_vcpu *vcpu)
return;
kvm_clear_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
+ vmx->nested.resume_pending = false;
vmx->nested.vmxon = false;
vmx->nested.smm.vmxon = false;
@@ -370,10 +497,13 @@ static void free_nested(struct kvm_vcpu *vcpu)
kfree(vmx->nested.cached_shadow_vmcs12);
vmx->nested.cached_shadow_vmcs12 = NULL;
- kvm_gpc_deactivate(&vmx->nested.msr_bitmap_cache);
-
nested_put_vmcs12_pages(vcpu);
+ kvm_gpc_deactivate(&vmx->nested.pi_desc_cache);
+ kvm_gpc_deactivate(&vmx->nested.virtual_apic_cache);
+ kvm_gpc_deactivate(&vmx->nested.apic_access_page_cache);
+ kvm_gpc_deactivate(&vmx->nested.msr_bitmap_cache);
+
kvm_mmu_free_roots(vcpu->kvm, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL);
nested_release_evmcs(vcpu);
@@ -408,6 +538,20 @@ void nested_vmx_init_caches(struct kvm_vcpu *vcpu)
* (which also means never marking it dirty).
*/
__kvm_gpc_init(&vmx->nested.msr_bitmap_cache, vcpu->kvm, false, true, NULL, 0);
+
+ /*
+ * The pfns of these three caches are handed to the CPU in vmcs02
+ * fields, for direct use while running the L2 guest. Pinning them
+ * for guest use makes an invalidation force this vCPU out of
+ * guest mode and post KVM_REQ_GET_NESTED_STATE_PAGES, whose
+ * handler re-establishes them before the next entry.
+ */
+ kvm_gpc_init_for_vcpu(&vmx->nested.apic_access_page_cache, vcpu,
+ KVM_REQ_GET_NESTED_STATE_PAGES);
+ kvm_gpc_init_for_vcpu(&vmx->nested.virtual_apic_cache, vcpu,
+ KVM_REQ_GET_NESTED_STATE_PAGES);
+ kvm_gpc_init_for_vcpu(&vmx->nested.pi_desc_cache, vcpu,
+ KVM_REQ_GET_NESTED_STATE_PAGES);
}
#define EPTP_PA_MASK GENMASK_ULL(51, 12)
@@ -3467,41 +3611,117 @@ static bool nested_get_evmcs_page(struct kvm_vcpu *vcpu)
}
#endif
-static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
+static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu, bool lazy)
{
struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
struct vcpu_vmx *vmx = to_vmx(vcpu);
- struct kvm_host_map *map;
+ hpa_t hpa;
- if (!vcpu->arch.pdptrs_from_userspace &&
+ if (!lazy && !vcpu->arch.pdptrs_from_userspace &&
!nested_cpu_has_ept(vmcs12) && is_pae_paging(vcpu)) {
/*
* Reload the guest's PDPTRs since after a migration
* the guest CR3 might be restored prior to setting the nested
- * state which can lead to a load of wrong PDPTRs.
+ * state which can lead to a load of wrong PDPTRs. Only at
+ * entry-time resolution: PDPTRs latch at MOV CR3 and
+ * VM-entry, so a lazy revalidation (a memslot update while
+ * L2 runs) must not re-derive them at a moment the
+ * architecture does not allow.
*/
- if (CC(!load_pdptrs(vcpu, vcpu->arch.cr3)))
+ if (CC(!load_pdptrs(vcpu, vcpu->arch.cr3))) {
+ /*
+ * Report the unusable PDPT rather than return to
+ * userspace with a stale exit_reason.
+ */
+ vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
+ vcpu->run->internal.suberror =
+ KVM_INTERNAL_ERROR_EMULATION;
+ vcpu->run->internal.ndata = 0;
return false;
+ }
}
if (nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES)) {
- map = &vmx->nested.apic_access_page_map;
-
- if (!kvm_vcpu_map(vcpu, gpa_to_gfn(vmcs12->apic_access_addr), map)) {
- vmcs_write64(APIC_ACCESS_ADDR, pfn_to_hpa(map->pfn));
- } else {
+ int err = nested_gpc_hpa(&vmx->nested.apic_access_page_cache,
+ vmcs12->apic_access_addr, &hpa);
+ if (!err) {
+ vmcs_write64(APIC_ACCESS_ADDR, hpa);
+ /*
+ * Undo any earlier lazy degradation: once the page
+ * is backed again the vmcs02 control must match
+ * vmcs12, and prepare_vmcs02_early() only runs at
+ * nested VM-entry, which a vCPU parked in L2 may
+ * not reach for a long time.
+ */
+ secondary_exec_controls_setbit(vmx,
+ SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES);
+ } else if (err == -EAGAIN) {
+ kvm_gpc_unpin_for_guest(&vmx->nested.apic_access_page_cache);
+ kvm_gpc_deactivate(&vmx->nested.apic_access_page_cache);
+ goto retry;
+ } else if (!lazy) {
+ /*
+ * Entry-time resolution (VMLAUNCH/VMRESUME, or resume
+ * from KVM_SET_NESTED_STATE): an unbackable
+ * APIC-access page is a misconfiguration to report,
+ * not a transient to ride out.
+ */
pr_debug_ratelimited("%s: no backing for APIC-access address in vmcs12\n",
__func__);
+ vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
+ vcpu->run->internal.suberror =
+ KVM_INTERNAL_ERROR_EMULATION;
+ vcpu->run->internal.ndata = 0;
+ kvm_gpc_unpin_for_guest(&vmx->nested.apic_access_page_cache);
+ kvm_gpc_deactivate(&vmx->nested.apic_access_page_cache);
return false;
+ } else {
+ /*
+ * Lazy revalidation (a memslot update or invalidation
+ * while L2 runs) found the gfn unbacked: userspace
+ * yanked the backing from under a running L2. Turn
+ * off the APIC-access match for vmcs02 instead of
+ * killing the VM; L2's accesses to the page then take
+ * ordinary EPT violations on an unbacked gfn and are
+ * handled (or reported) as MMIO, exactly as a
+ * non-nested guest touching the same gpa would be.
+ * prepare_vmcs02_early() recomputes the control from
+ * vmcs12 on the next nested entry, and a later
+ * memslot update which re-backs the gfn re-posts the
+ * request and re-latches the page, so the
+ * degradation heals itself.
+ */
+ kvm_gpc_unpin_for_guest(&vmx->nested.apic_access_page_cache);
+ kvm_gpc_deactivate(&vmx->nested.apic_access_page_cache);
+ secondary_exec_controls_clearbit(vmx,
+ SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES);
}
}
if (nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW)) {
- map = &vmx->nested.virtual_apic_map;
+ int vapic_err = nested_gpc_hpa(&vmx->nested.virtual_apic_cache,
+ vmcs12->virtual_apic_page_addr, &hpa);
+
+ /*
+ * On failure, deactivate the cache: an active-but-invalid
+ * cache would read as "might have an event" to the
+ * non-sleeping check in vmx_has_nested_events(), causing
+ * spurious wakeups for as long as the page remains
+ * unbackable.
+ */
+ if (vapic_err) {
+ kvm_gpc_unpin_for_guest(&vmx->nested.virtual_apic_cache);
+ kvm_gpc_deactivate(&vmx->nested.virtual_apic_cache);
+ }
- if (!kvm_vcpu_map(vcpu, gpa_to_gfn(vmcs12->virtual_apic_page_addr), map)) {
- vmcs_write64(VIRTUAL_APIC_PAGE_ADDR, pfn_to_hpa(map->pfn));
+ if (vapic_err == -EAGAIN)
+ goto retry;
+
+ if (!vapic_err) {
+ vmcs_write64(VIRTUAL_APIC_PAGE_ADDR, hpa);
+ /* As for the APIC-access page: undo a lazy degrade. */
+ exec_controls_setbit(vmx, CPU_BASED_TPR_SHADOW);
} else if (nested_cpu_has(vmcs12, CPU_BASED_CR8_LOAD_EXITING) &&
nested_cpu_has(vmcs12, CPU_BASED_CR8_STORE_EXITING) &&
!nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES)) {
@@ -3514,6 +3734,27 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
* only possibility we have.
*/
exec_controls_clearbit(vmx, CPU_BASED_TPR_SHADOW);
+ } else if (lazy) {
+ /*
+ * Unlike the two sibling pages, a vAPIC loss cannot
+ * be degraded around: the TPR shadow is live. Nor is
+ * the entry-time INVALID_GPA fallback usable here —
+ * the resulting hardware VM-entry failure would be
+ * delivered as VMfailValid to an L1 whose VMLAUNCH
+ * retired long ago, rewinding L2. Nor can an unbacked
+ * gfn be reported as an error: a routine userspace
+ * DELETE-then-recreate reshuffle has a committed gap
+ * in which the gfn is genuinely absent, and this
+ * revalidation runs inside it precisely because the
+ * DELETE's commit posted the request. Ride it out
+ * exactly like the in-flight window: keep the
+ * request pending so L2 cannot be entered, and let
+ * the recreate (whose commit posts again) supply the
+ * page. A userspace which never re-backs the gfn
+ * leaves the vCPU retrying in KVM_RUN indefinitely —
+ * as for RET_PF_RETRY, that is its own doing.
+ */
+ goto retry;
} else {
/*
* Write an illegal value to VIRTUAL_APIC_PAGE_ADDR to
@@ -3524,14 +3765,15 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
}
if (nested_cpu_has_posted_intr(vmcs12)) {
- map = &vmx->nested.pi_desc_map;
-
- if (!kvm_vcpu_map(vcpu, gpa_to_gfn(vmcs12->posted_intr_desc_addr), map)) {
- vmx->nested.pi_desc =
- (struct pi_desc *)(((void *)map->hva) +
- offset_in_page(vmcs12->posted_intr_desc_addr));
+ int pi_err = nested_gpc_hpa(&vmx->nested.pi_desc_cache,
+ vmcs12->posted_intr_desc_addr & PAGE_MASK, &hpa);
+ if (!pi_err) {
+ vmx->nested.pi_desc_offset =
+ offset_in_page(vmcs12->posted_intr_desc_addr);
vmcs_write64(POSTED_INTR_DESC_ADDR,
- pfn_to_hpa(map->pfn) + offset_in_page(vmcs12->posted_intr_desc_addr));
+ hpa + offset_in_page(vmcs12->posted_intr_desc_addr));
+ /* As for the APIC-access page: undo a lazy degrade. */
+ pin_controls_setbit(vmx, PIN_BASED_POSTED_INTR);
} else {
/*
* Defer the KVM_INTERNAL_EXIT until KVM tries to
@@ -3539,7 +3781,10 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
* descriptor. (Note that KVM may do this when it
* should not, per the architectural specification.)
*/
- vmx->nested.pi_desc = NULL;
+ kvm_gpc_unpin_for_guest(&vmx->nested.pi_desc_cache);
+ kvm_gpc_deactivate(&vmx->nested.pi_desc_cache);
+ if (pi_err == -EAGAIN)
+ goto retry;
pin_controls_clearbit(vmx, PIN_BASED_POSTED_INTR);
}
}
@@ -3549,9 +3794,27 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
exec_controls_clearbit(vmx, CPU_BASED_USE_MSR_BITMAPS);
return true;
+
+retry:
+ /*
+ * The gfn's memslot is being moved or deleted: present but
+ * flagged invalid, so the lookup failure is transient and the
+ * gfn must not (yet) be treated as unbacked. Re-post the
+ * request and report success: the pending request prevents
+ * any VM entry (vcpu_enter_guest() bails and reprocesses
+ * requests), so L2 cannot run with the partially-updated
+ * vmcs02, and the vCPU retries here until the memslot update
+ * completes or reverts. This mirrors RET_PF_RETRY for
+ * ordinary guest faults on an invalid slot; as there, a
+ * userspace which never completes the update leaves the vCPU
+ * retrying indefinitely, with signals and request processing
+ * still serviced.
+ */
+ kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
+ return true;
}
-static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
+static bool __vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
{
#ifdef CONFIG_KVM_HYPERV
/*
@@ -3567,9 +3830,39 @@ static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
}
#endif
- if (is_guest_mode(vcpu) && !nested_get_vmcs12_pages(vcpu))
+ if (is_guest_mode(vcpu) &&
+ !nested_get_vmcs12_pages(vcpu, !to_vmx(vcpu)->nested.resume_pending))
return false;
+ /*
+ * Success consumes the resume classification (a -EAGAIN retry
+ * "succeeds" too: the resume did all it could against a memslot
+ * mid-update, and the re-posted request's consumption is a fresh
+ * revalidation like any other).
+ */
+ to_vmx(vcpu)->nested.resume_pending = false;
+ return true;
+}
+
+static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
+{
+ if (!__vmx_get_nested_state_pages(vcpu)) {
+ /*
+ * The caller (vcpu_enter_guest) consumed
+ * KVM_REQ_GET_NESTED_STATE_PAGES before calling in, and a
+ * failure here may leave a previously latched — and, after
+ * an invalidation, stale and unreferenced — physical
+ * address in vmcs02. Re-post the request so that no path
+ * can re-enter L2 without retrying this function: the
+ * invariant is that a stale latched address always implies
+ * a pending request. If the failure persists, the vCPU
+ * bounces to userspace on every KVM_RUN rather than ever
+ * entering L2, which is the invariant doing its job.
+ */
+ kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
+ return false;
+ }
+
return true;
}
@@ -3698,7 +3991,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
prepare_vmcs02_early(vmx, &vmx->vmcs01, vmcs12);
if (from_vmentry) {
- if (unlikely(!nested_get_vmcs12_pages(vcpu))) {
+ if (unlikely(!nested_get_vmcs12_pages(vcpu, false))) {
vmx_switch_vmcs(vcpu, &vmx->vmcs01);
return NVMX_VMENTRY_KVM_INTERNAL_ERROR;
}
@@ -3736,6 +4029,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
* to nested_get_vmcs12_pages before the next VM-entry. The MSRs
* have already been set at vmentry time and should not be reset.
*/
+ vmx->nested.resume_pending = true;
kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
}
@@ -3792,13 +4086,19 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
vmentry_fail_vmexit:
vmx_switch_vmcs(vcpu, &vmx->vmcs01);
+ /*
+ * Release any guest-mode pins taken by nested_get_vmcs12_pages()
+ * during this failed entry: this path returns to L1 without going
+ * through __nested_vmx_vmexit(), which is where they are normally
+ * dropped. (Idempotent if a later vmexit drops them again.)
+ */
+ nested_put_vmcs12_pages(vcpu);
+
if (!from_vmentry) {
nested_ept_uninit_mmu_context(vcpu);
return NVMX_VMENTRY_VMEXIT;
}
- nested_put_vmcs12_pages(vcpu);
-
load_vmcs12_host_state(vcpu, vmcs12);
vmcs12->vm_exit_reason = exit_reason.full;
if (enable_shadow_vmcs || nested_vmx_is_evmptr12_valid(vmx))
@@ -4054,6 +4354,8 @@ static void vmcs12_save_pending_event(struct kvm_vcpu *vcpu,
static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
+ int pi_idx, vapic_idx;
+ struct pi_desc *pi_desc;
int max_irr;
void *vapic_page;
u16 status;
@@ -4061,22 +4363,58 @@ static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu)
if (!vmx->nested.pi_pending)
return 0;
- if (!vmx->nested.pi_desc)
+ /*
+ * Resolve both caches before touching the descriptor: taking one
+ * cache's lock inside the other's would sleep in refresh (and
+ * wait a grace period on the gpc_srcu domain this task already
+ * reads) with preemption disabled. The vAPIC lock may go unused
+ * (no pending vector), and its absence only matters once a
+ * vector is found, so a failure is remembered, not acted on.
+ */
+retry:
+ vapic_idx = nested_gpc_lock_if_active(&vmx->nested.virtual_apic_cache);
+
+ pi_idx = nested_gpc_try_lock_if_active(&vmx->nested.pi_desc_cache);
+ if (pi_idx == -EWOULDBLOCK) {
+ if (vapic_idx >= 0)
+ kvm_gpc_unlock(&vmx->nested.virtual_apic_cache,
+ vapic_idx);
+ if (kvm_gpc_refresh(&vmx->nested.pi_desc_cache, PAGE_SIZE))
+ goto mmio_needed;
+ goto retry;
+ }
+ if (pi_idx < 0) {
+ if (vapic_idx >= 0)
+ kvm_gpc_unlock(&vmx->nested.virtual_apic_cache,
+ vapic_idx);
goto mmio_needed;
+ }
+ pi_desc = nested_pi_desc(vmx);
vmx->nested.pi_pending = false;
- if (!pi_test_and_clear_on(vmx->nested.pi_desc))
+ if (!pi_test_and_clear_on(pi_desc)) {
+ kvm_gpc_unlock(&vmx->nested.pi_desc_cache, pi_idx);
+ if (vapic_idx >= 0)
+ kvm_gpc_unlock(&vmx->nested.virtual_apic_cache,
+ vapic_idx);
return 0;
+ }
- max_irr = pi_find_highest_vector(vmx->nested.pi_desc);
+ max_irr = pi_find_highest_vector(pi_desc);
if (max_irr > 0) {
- vapic_page = vmx->nested.virtual_apic_map.hva;
- if (!vapic_page)
+ if (vapic_idx < 0) {
+ /* The ON bit is already cleared: mark the write. */
+ kvm_gpc_mark_dirty_in_slot(&vmx->nested.pi_desc_cache);
+ kvm_gpc_unlock(&vmx->nested.pi_desc_cache, pi_idx);
goto mmio_needed;
+ }
+ vapic_page = vmx->nested.virtual_apic_cache.khva;
+
+ __kvm_apic_update_irr(pi_desc->pir, vapic_page, &max_irr);
+
+ kvm_gpc_mark_dirty_in_slot(&vmx->nested.virtual_apic_cache);
- __kvm_apic_update_irr(vmx->nested.pi_desc->pir,
- vapic_page, &max_irr);
status = vmcs_read16(GUEST_INTR_STATUS);
if ((u8)max_irr > ((u8)status & 0xff)) {
status &= ~0xff;
@@ -4085,8 +4423,10 @@ static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu)
}
}
- kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.virtual_apic_map);
- kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.pi_desc_map);
+ if (vapic_idx >= 0)
+ kvm_gpc_unlock(&vmx->nested.virtual_apic_cache, vapic_idx);
+ kvm_gpc_mark_dirty_in_slot(&vmx->nested.pi_desc_cache);
+ kvm_gpc_unlock(&vmx->nested.pi_desc_cache, pi_idx);
return 0;
mmio_needed:
@@ -4205,8 +4545,9 @@ static bool nested_vmx_preemption_timer_pending(struct kvm_vcpu *vcpu)
static bool vmx_has_nested_events(struct kvm_vcpu *vcpu, bool for_injection)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
- void *vapic = vmx->nested.virtual_apic_map.hva;
- int max_irr, vppr;
+ int max_irr, vppr, idx;
+ struct pi_desc *pi_desc;
+ void *vapic;
if (nested_vmx_preemption_timer_pending(vcpu) ||
vmx->nested.mtf_pending)
@@ -4225,20 +4566,57 @@ static bool vmx_has_nested_events(struct kvm_vcpu *vcpu, bool for_injection)
__vmx_interrupt_blocked(vcpu))
return false;
- if (!vapic)
+ /*
+ * This is called from non-sleeping contexts (notably the
+ * kvm_vcpu_block() loop, under set_current_state()), so it must
+ * not refresh an unusable cache. Claim a pending event instead,
+ * and post KVM_REQ_GET_NESTED_STATE_PAGES: the bounce through
+ * vcpu_run services the request in a sleepable context,
+ * re-establishing the cache, and re-evaluates events for real.
+ * The request must be posted here rather than assumed pending —
+ * an invalidation posts one, but a cache can also fail the check
+ * on a bare memslot-generation bump (e.g. a dirty-logging toggle,
+ * KVM_MR_FLAGS_ONLY), where nothing else does, and without it
+ * the woken vCPU would re-enter, HLT, and spin back here. A
+ * spurious wakeup is safe; sleeping here is not, and returning
+ * false could miss a wakeup.
+ */
+ idx = nested_gpc_try_lock_if_active(&vmx->nested.virtual_apic_cache);
+ if (idx == -EWOULDBLOCK) {
+ kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
+ return true;
+ }
+ if (idx < 0)
return false;
+ vapic = vmx->nested.virtual_apic_cache.khva;
vppr = *((u32 *)(vapic + APIC_PROCPRI));
+ kvm_gpc_unlock(&vmx->nested.virtual_apic_cache, idx);
+
max_irr = vmx_get_rvi();
if ((max_irr & 0xf0) > (vppr & 0xf0))
return true;
- if (vmx->nested.pi_pending && vmx->nested.pi_desc &&
- pi_test_on(vmx->nested.pi_desc)) {
- max_irr = pi_find_highest_vector(vmx->nested.pi_desc);
- if (max_irr > 0 && (max_irr & 0xf0) > (vppr & 0xf0))
+ if (vmx->nested.pi_pending) {
+ idx = nested_gpc_try_lock_if_active(&vmx->nested.pi_desc_cache);
+ if (idx == -EWOULDBLOCK) {
+ kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
return true;
+ }
+ if (idx < 0)
+ return false;
+
+ pi_desc = nested_pi_desc(vmx);
+ if (pi_test_on(pi_desc)) {
+ max_irr = pi_find_highest_vector(pi_desc);
+ if (max_irr > 0 && (max_irr & 0xf0) > (vppr & 0xf0)) {
+ kvm_gpc_unlock(&vmx->nested.pi_desc_cache, idx);
+ return true;
+ }
+ }
+
+ kvm_gpc_unlock(&vmx->nested.pi_desc_cache, idx);
}
return false;
@@ -5121,6 +5499,15 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
#endif
}
+ /*
+ * The resume classification dies with the request: a stale
+ * resume_pending would give the next request (e.g. posted by an
+ * invalidation after L1 re-enters L2) entry-time semantics,
+ * re-deriving PDPTRs mid-L2 and treating a transiently
+ * unbackable page as a terminal error.
+ */
+ vmx->nested.resume_pending = false;
+
/* Service pending TLB flush requests for L2 before switching to L1. */
kvm_service_local_tlb_flush_requests(vcpu);
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 590095bcb665..bb766380276e 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -6428,11 +6428,14 @@ static void vmx_flush_pml_buffer(struct kvm_vcpu *vcpu)
static void nested_vmx_mark_all_vmcs12_pages_dirty(struct kvm_vcpu *vcpu)
{
- struct vcpu_vmx *vmx = to_vmx(vcpu);
+ struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
- kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.apic_access_page_map);
- kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.virtual_apic_map);
- kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.pi_desc_map);
+ if (nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES))
+ kvm_vcpu_mark_page_dirty(vcpu, gpa_to_gfn(vmcs12->apic_access_addr));
+ if (nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW))
+ kvm_vcpu_mark_page_dirty(vcpu, gpa_to_gfn(vmcs12->virtual_apic_page_addr));
+ if (nested_cpu_has_posted_intr(vmcs12))
+ kvm_vcpu_mark_page_dirty(vcpu, gpa_to_gfn(vmcs12->posted_intr_desc_addr));
}
static void vmx_dump_sel(char *name, uint32_t sel)
diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h
index c6827368c84a..7d8bced38ca0 100644
--- a/arch/x86/kvm/vmx/vmx.h
+++ b/arch/x86/kvm/vmx/vmx.h
@@ -73,6 +73,12 @@ struct pt_desc {
struct nested_vmx {
/* Has the level1 guest done vmxon? */
bool vmxon;
+ /*
+ * The pending KVM_REQ_GET_NESTED_STATE_PAGES was posted by
+ * KVM_SET_NESTED_STATE: an unbackable page is then a userspace
+ * error to report, not a transient state to ride out.
+ */
+ bool resume_pending;
gpa_t vmxon_ptr;
bool pml_full;
@@ -146,13 +152,18 @@ struct nested_vmx {
* Guest pages referred to in the vmcs02 with host-physical
* pointers, so we must keep them pinned while L2 runs.
*/
- struct gfn_to_pfn_cache msr_bitmap_cache;
+ struct gfn_to_pfn_cache apic_access_page_cache;
+ struct gfn_to_pfn_cache virtual_apic_cache;
+ struct gfn_to_pfn_cache pi_desc_cache;
- struct kvm_host_map apic_access_page_map;
- struct kvm_host_map virtual_apic_map;
- struct kvm_host_map pi_desc_map;
+ /*
+ * L1's MSR bitmap, read by KVM when merging into the bitmap the
+ * CPU actually uses (vmcs02 points at the merged copy). Never
+ * pinned, never written through.
+ */
+ struct gfn_to_pfn_cache msr_bitmap_cache;
- struct pi_desc *pi_desc;
+ u64 pi_desc_offset;
bool pi_pending;
u16 posted_intr_nv;
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v2 12/20] KVM: nVMX: Use pinned pfncache for L1 APIC virtualization pages
2026-11-05 2:00 ` [PATCH v2 12/20] KVM: nVMX: Use pinned pfncache for L1 APIC virtualization pages David Woodhouse
@ 2026-11-05 2:33 ` sashiko-bot
0 siblings, 0 replies; 25+ messages in thread
From: sashiko-bot @ 2026-11-05 2:33 UTC (permalink / raw)
To: David Woodhouse, kvm; +Cc: linux-rt-devel, linux-kernel
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] KVM: nVMX: spurious L2 unhalt in vmx_has_nested_events() on invalid cache
- [High] KVM: nVMX: fatal exit and PI loss in vmx_complete_nested_posted_interrupt()
- [Medium] KVM: nVMX: loss of entry-time semantics on retry in nested_get_vmcs12_pages()
- [Low] KVM: nVMX: redundant request repost in nested_gpc_hpa() when refreshing pin
--
commit 2e4c94957b2cebc2ce8ce9d33fd7500248768210
Author: Fred Griffoul <fgriffo@amazon.co.uk>
KVM: nVMX: Use pinned pfncache for L1 APIC virtualization pages
This commit replaces kvm_host_map with pinned gfn_to_pfn_cache instances for
the L1 APIC-access page, virtual-APIC page, and posted interrupt descriptor
in vmcs02. Caches are pinned for guest use on nested entry or state
revalidation, and invalidated caches trigger KVM_REQ_GET_NESTED_STATE_PAGES
to refresh or degrade the corresponding vmcs02 controls.
> diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
> index bb0baa79c28cb..23f1fff97917a 100644
> --- a/arch/x86/kvm/vmx/nested.c
> +++ b/arch/x86/kvm/vmx/nested.c
> @@ -320,14 +320,140 @@ static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs)
> kvm_reset_dirty_registers(vcpu);
> }
>
> +/*
> + * Map a page of L1 memory, pin it for direct use by the guest (i.e. by
> + * the CPU while running the L2 guest, via a physical address in vmcs02),
> + * and return the host physical address to write into vmcs02. The pin
> + * guarantees that if the mapping is invalidated, this vCPU is forced out
> + * of guest mode and KVM_REQ_GET_NESTED_STATE_PAGES is posted so that the
> + * address is re-established (or the vCPU exits to userspace) before the
> + * next entry to guest mode.
> + */
> +static int nested_gpc_hpa(struct gfn_to_pfn_cache *gpc, gpa_t gpa, hpa_t *hpa)
> +{
> + int idx;
> +
> + do {
> + idx = kvm_gpc_lock_page(gpc, gpa);
[Severity: Low]
When KVM_REQ_GET_NESTED_STATE_PAGES runs to refresh an already-pinned cache
after an mmu_notifier invalidation or a memslot generation change,
GPC_GUEST_USING is still set in gpc->state because
gfn_to_pfn_cache_invalidate_start() preserves GPC_GUEST_USING in the state
word.
When nested_gpc_hpa() calls kvm_gpc_lock_page() -> kvm_gpc_activate() ->
__kvm_gpc_refresh() without unpinning first, __kvm_gpc_refresh() executes:
virt/kvm/pfncache.c:__kvm_gpc_refresh() {
...
old_state = atomic_xchg(&gpc->state, GPC_INVALIDATING);
gpc_kick_pinned_vcpu(gpc, old_state);
...
}
Because GPC_GUEST_USING is still present in old_state, gpc_kick_pinned_vcpu()
unconditionally re-posts KVM_REQ_GET_NESTED_STATE_PAGES even though the
request handler is already in the middle of servicing it. Once
vmx_get_nested_state_pages() finishes, vcpu_enter_guest() sees the newly
re-posted request at kvm_vcpu_exit_request() and aborts guest entry to run a
redundant second pass.
Should nested_gpc_hpa() call kvm_gpc_unpin_for_guest(gpc) before
kvm_gpc_lock_page()?
[ ... ]
> @@ -3467,41 +3611,117 @@ static bool nested_get_evmcs_page(struct kvm_vcpu *vcpu)
> }
> #endif
>
> -static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
> +static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu, bool lazy)
> {
[ ... ]
> if (nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW)) {
> - map = &vmx->nested.virtual_apic_map;
> + int vapic_err = nested_gpc_hpa(&vmx->nested.virtual_apic_cache,
> + vmcs12->virtual_apic_page_addr, &hpa);
> +
> + /*
> + * On failure, deactivate the cache: an active-but-invalid
> + * cache would read as "might have an event" to the
> + * non-sleeping check in vmx_has_nested_events(), causing
> + * spurious wakeups for as long as the page remains
> + * unbackable.
> + */
> + if (vapic_err) {
> + kvm_gpc_unpin_for_guest(&vmx->nested.virtual_apic_cache);
> + kvm_gpc_deactivate(&vmx->nested.virtual_apic_cache);
> + }
>
> - if (!kvm_vcpu_map(vcpu, gpa_to_gfn(vmcs12->virtual_apic_page_addr), map)) {
> - vmcs_write64(VIRTUAL_APIC_PAGE_ADDR, pfn_to_hpa(map->pfn));
> + if (vapic_err == -EAGAIN)
> + goto retry;
> +
> + if (!vapic_err) {
> + vmcs_write64(VIRTUAL_APIC_PAGE_ADDR, hpa);
> + /* As for the APIC-access page: undo a lazy degrade. */
> + exec_controls_setbit(vmx, CPU_BASED_TPR_SHADOW);
> } else if (nested_cpu_has(vmcs12, CPU_BASED_CR8_LOAD_EXITING) &&
> nested_cpu_has(vmcs12, CPU_BASED_CR8_STORE_EXITING) &&
> !nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES)) {
> @@ -3514,6 +3734,27 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
> * only possibility we have.
> */
> exec_controls_clearbit(vmx, CPU_BASED_TPR_SHADOW);
> + } else if (lazy) {
[ ... ]
> + goto retry;
> + } else {
> /*
> * Write an illegal value to VIRTUAL_APIC_PAGE_ADDR to
> * force VM-Entry to fail.
> */
> vmcs_write64(VIRTUAL_APIC_PAGE_ADDR, INVALID_GPA);
> }
> }
[ ... ]
> +retry:
> + /*
> + * The gfn's memslot is being moved or deleted: present but
> + * flagged invalid, so the lookup failure is transient and the
> + * gfn must not (yet) be treated as unbacked. Re-post the
> + * request and report success: the pending request prevents
> + * any VM entry (vcpu_enter_guest() bails and reprocesses
> + * requests), so L2 cannot run with the partially-updated
> + * vmcs02, and the vCPU retries here until the memslot update
> + * completes or reverts. This mirrors RET_PF_RETRY for
> + * ordinary guest faults on an invalid slot; as there, a
> + * userspace which never completes the update leaves the vCPU
> + * retrying indefinitely, with signals and request processing
> + * still serviced.
> + */
> + kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
> + return true;
> }
>
> -static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
> +static bool __vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
> {
> #ifdef CONFIG_KVM_HYPERV
> /*
> @@ -3567,9 +3830,39 @@ static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
> }
> #endif
>
> - if (is_guest_mode(vcpu) && !nested_get_vmcs12_pages(vcpu))
> + if (is_guest_mode(vcpu) &&
> + !nested_get_vmcs12_pages(vcpu, !to_vmx(vcpu)->nested.resume_pending))
> return false;
>
> + /*
> + * Success consumes the resume classification (a -EAGAIN retry
> + * "succeeds" too: the resume did all it could against a memslot
> + * mid-update, and the re-posted request's consumption is a fresh
> + * revalidation like any other).
> + */
> + to_vmx(vcpu)->nested.resume_pending = false;
> + return true;
> +}
[Severity: Medium]
Does taking the retry path on -EAGAIN during entry-time resolution lose
entry-time (!lazy) semantics for the retry?
There appear to be two ways this happens:
1. During VMLAUNCH/VMRESUME, nested_vmx_enter_non_root_mode() calls
nested_get_vmcs12_pages(vcpu, false) while vmx->nested.resume_pending is
false. If nested_gpc_hpa() returns -EAGAIN because a memslot is mid-update,
nested_get_vmcs12_pages() posts KVM_REQ_GET_NESTED_STATE_PAGES and returns
true without setting resume_pending.
2. During KVM_SET_NESTED_STATE resume, __vmx_get_nested_state_pages() calls
nested_get_vmcs12_pages(vcpu, false) because resume_pending is true, but when
nested_get_vmcs12_pages() jumps to retry on -EAGAIN and returns true,
__vmx_get_nested_state_pages() clears resume_pending = false.
In both cases, the re-posted request runs nested_get_vmcs12_pages() with
lazy = true. If the in-flight memslot operation was a deletion (or if a
subsequent page in vmcs12 such as virtual_apic_page_addr had not been
checked yet on the first pass and is unbacked), the retry applies lazy
semantics: an unbacked apic_access_addr silently degrades
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES instead of exiting with
KVM_EXIT_INTERNAL_ERROR, and an unbacked virtual_apic_page_addr spins
forever in KVM_RUN via goto retry instead of writing INVALID_GPA to
VIRTUAL_APIC_PAGE_ADDR to deliver VMfailValid to L1.
Additionally, in nested_get_vmcs12_pages(), the CR8 load/store exiting check:
} else if (nested_cpu_has(vmcs12, CPU_BASED_CR8_LOAD_EXITING) &&
nested_cpu_has(vmcs12, CPU_BASED_CR8_STORE_EXITING) &&
!nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES)) {
exec_controls_clearbit(vmx, CPU_BASED_TPR_SHADOW);
} else if (lazy) {
is evaluated before checking lazy, and without checking whether other APICv
controls that require CPU_BASED_TPR_SHADOW (such as
SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE, SECONDARY_EXEC_APIC_REGISTER_VIRT,
SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY, or PIN_BASED_POSTED_INTR) are enabled.
If L2 is running with x2APIC virtualization and CR8 load/store exiting when a
lazy vAPIC revalidation fails during a memslot delete-then-recreate, clearing
CPU_BASED_TPR_SHADOW while SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE remains set
in vmcs02 will cause hardware VMRESUME to fail with
VMXERR_ENTRY_INVALID_CONTROL_FIELD instead of riding out the gap via
goto retry.
Should entry-time resolution preserve !lazy across -EAGAIN retries, and
should the lazy branch precede the CPU_BASED_TPR_SHADOW clearing fallback?
[ ... ]
> @@ -4054,6 +4354,8 @@ static void vmcs12_save_pending_event(struct kvm_vcpu *vcpu,
> static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu)
> {
> struct vcpu_vmx *vmx = to_vmx(vcpu);
> + int pi_idx, vapic_idx;
> + struct pi_desc *pi_desc;
> int max_irr;
> void *vapic_page;
> u16 status;
>
> if (!vmx->nested.pi_pending)
> return 0;
>
> - if (!vmx->nested.pi_desc)
> + /*
> + * Resolve both caches before touching the descriptor: taking one
> + * cache's lock inside the other's would sleep in refresh (and
> + * wait a grace period on the gpc_srcu domain this task already
> + * reads) with preemption disabled. The vAPIC lock may go unused
> + * (no pending vector), and its absence only matters once a
> + * vector is found, so a failure is remembered, not acted on.
> + */
> +retry:
> + vapic_idx = nested_gpc_lock_if_active(&vmx->nested.virtual_apic_cache);
> +
> + pi_idx = nested_gpc_try_lock_if_active(&vmx->nested.pi_desc_cache);
> + if (pi_idx == -EWOULDBLOCK) {
> + if (vapic_idx >= 0)
> + kvm_gpc_unlock(&vmx->nested.virtual_apic_cache,
> + vapic_idx);
> + if (kvm_gpc_refresh(&vmx->nested.pi_desc_cache, PAGE_SIZE))
> + goto mmio_needed;
> + goto retry;
> + }
> + if (pi_idx < 0) {
> + if (vapic_idx >= 0)
> + kvm_gpc_unlock(&vmx->nested.virtual_apic_cache,
> + vapic_idx);
> goto mmio_needed;
> + }
>
> + pi_desc = nested_pi_desc(vmx);
> vmx->nested.pi_pending = false;
>
> - if (!pi_test_and_clear_on(vmx->nested.pi_desc))
> + if (!pi_test_and_clear_on(pi_desc)) {
> + kvm_gpc_unlock(&vmx->nested.pi_desc_cache, pi_idx);
> + if (vapic_idx >= 0)
> + kvm_gpc_unlock(&vmx->nested.virtual_apic_cache,
> + vapic_idx);
> return 0;
> + }
>
> - max_irr = pi_find_highest_vector(vmx->nested.pi_desc);
> + max_irr = pi_find_highest_vector(pi_desc);
> if (max_irr > 0) {
> - vapic_page = vmx->nested.virtual_apic_map.hva;
> - if (!vapic_page)
> + if (vapic_idx < 0) {
> + /* The ON bit is already cleared: mark the write. */
> + kvm_gpc_mark_dirty_in_slot(&vmx->nested.pi_desc_cache);
> + kvm_gpc_unlock(&vmx->nested.pi_desc_cache, pi_idx);
> goto mmio_needed;
> + }
[Severity: High]
What happens in vmx_complete_nested_posted_interrupt() when
nested_get_vmcs12_pages() has deferred cache resolution via goto retry?
In nested_get_vmcs12_pages(), when resolving apic_access_page_cache,
virtual_apic_cache, or pi_desc_cache encounters an in-flight memslot update
(-EAGAIN) or a lazy unbacked virtual_apic_cache, it deactivates that cache,
re-posts KVM_REQ_GET_NESTED_STATE_PAGES, and returns true.
In vcpu_enter_guest(), because vmx_get_nested_state_pages() returns true,
execution continues to kvm_check_and_inject_events() ->
vmx_check_nested_events() -> vmx_complete_nested_posted_interrupt() before
kvm_vcpu_exit_request() is checked:
arch/x86/kvm/x86.c:vcpu_enter_guest() {
...
if (kvm_check_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu)) {
if (unlikely(!static_call(kvm_x86_get_nested_state_pages)(vcpu))) {
r = 0;
goto out;
}
}
...
if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
kvm_xen_has_interrupt(vcpu)) {
...
if (kvm_check_and_inject_events(vcpu, & Bool) < 0) {
r = 0;
goto out;
}
}
...
if (kvm_vcpu_exit_request(vcpu)) {
vcpu->mode = OUTSIDE_GUEST_MODE;
smp_wmb();
local_irq_enable();
preempt_enable();
kvm_vcpu_srcu_read_lock(vcpu);
r = 1;
goto cancel_injection;
}
}
(Similarly, if L2 is restored in KVM_MP_STATE_HALTED via KVM_SET_NESTED_STATE,
vcpu_block() calls kvm_check_nested_events() before vcpu_enter_guest() has
ever run KVM_REQ_GET_NESTED_STATE_PAGES.)
If vmx->nested.pi_pending is true when vmx_complete_nested_posted_interrupt()
is called while KVM_REQ_GET_NESTED_STATE_PAGES is still pending:
1. If pi_desc_cache was deactivated (pi_idx == -ENOENT) or its refresh hits
an in-flight KVM_MEMSLOT_INVALID slot, vmx_complete_nested_posted_interrupt()
jumps to mmio_needed and calls kvm_handle_memory_failure(), terminating the
VM with KVM_EXIT_INTERNAL_ERROR instead of waiting for the pending retry.
2. If pi_desc_cache is active and valid, but virtual_apic_cache was
deactivated by nested_get_vmcs12_pages() (or its refresh failed during a
memslot update, so vapic_idx < 0), vmx_complete_nested_posted_interrupt()
clears vmx->nested.pi_pending = false and clears POSTED_INTR_ON in pi_desc
before checking if (vapic_idx < 0) and jumping to mmio_needed, losing the
posted interrupt notification and killing the VM.
3. Because nested_put_vmcs12_pages() on nested VM-exit only unpins the caches
without deactivating them, if apic_access_page_cache hits -EAGAIN at
VMLAUNCH/VMRESUME of a new L2, nested_get_vmcs12_pages() jumps to retry
before updating virtual_apic_cache and pi_desc_cache, leaving them active
with the previous L2's GPAs and pi_desc_offset when
vmx_complete_nested_posted_interrupt() runs.
Should vmx_complete_nested_posted_interrupt() (or vmx_check_nested_events())
defer processing while KVM_REQ_GET_NESTED_STATE_PAGES is pending, and avoid
clearing pi_pending or POSTED_INTR_ON until both caches are locked?
[ ... ]
> @@ -4225,20 +4566,57 @@ static bool vmx_has_nested_events(struct kvm_vcpu *vcpu, bool for_injection)
> __vmx_interrupt_blocked(vcpu))
> return false;
>
> - if (!vapic)
> + /*
> + * This is called from non-sleeping contexts (notably the
> + * kvm_vcpu_block() loop, under set_current_state()), so it must
> + * not refresh an unusable cache. Claim a pending event instead,
> + * and post KVM_REQ_GET_NESTED_STATE_PAGES: the bounce through
> + * vcpu_run services the request in a sleepable context,
> + * re-establishing the cache, and re-evaluates events for real.
> + * The request must be posted here rather than assumed pending —
> + * an invalidation posts one, but a cache can also fail the check
> + * on a bare memslot-generation bump (e.g. a dirty-logging toggle,
> + * KVM_MR_FLAGS_ONLY), where nothing else does, and without it
> + * the woken vCPU would re-enter, HLT, and spin back here. A
> + * spurious wakeup is safe; sleeping here is not, and returning
> + * false could miss a wakeup.
> + */
> + idx = nested_gpc_try_lock_if_active(&vmx->nested.virtual_apic_cache);
> + if (idx == -EWOULDBLOCK) {
> + kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
> + return true;
> + }
> + if (idx < 0)
> return false;
>
> + vapic = vmx->nested.virtual_apic_cache.khva;
> vppr = *((u32 *)(vapic + APIC_PROCPRI));
>
> + kvm_gpc_unlock(&vmx->nested.virtual_apic_cache, idx);
> +
> max_irr = vmx_get_rvi();
> if ((max_irr & 0xf0) > (vppr & 0xf0))
> return true;
>
> - if (vmx->nested.pi_pending && vmx->nested.pi_desc &&
> - pi_test_on(vmx->nested.pi_desc)) {
> - max_irr = pi_find_highest_vector(vmx->nested.pi_desc);
> - if (max_irr > 0 && (max_irr & 0xf0) > (vppr & 0xf0))
> + if (vmx->nested.pi_pending) {
> + idx = nested_gpc_try_lock_if_active(&vmx->nested.pi_desc_cache);
> + if (idx == -EWOULDBLOCK) {
> + kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
> return true;
> + }
> + if (idx < 0)
> + return false;
[Severity: High]
Won't returning true on -EWOULDBLOCK here permanently unhalt a halted L2
vCPU instead of just bouncing through vcpu_run() to refresh the cache?
When a halted L2 vCPU is blocked in kvm_vcpu_block() and an mmu_notifier
invalidates virtual_apic_cache or a dirty-logging toggle bumps the memslot
generation, kvm_vcpu_check_block() -> kvm_arch_vcpu_runnable() ->
kvm_vcpu_has_events() -> vmx_has_nested_events() sees -EWOULDBLOCK, posts
KVM_REQ_GET_NESTED_STATE_PAGES, and returns true.
However, kvm_vcpu_block() returns to vcpu_block(), which immediately
re-checks kvm_arch_vcpu_runnable(vcpu) before returning to vcpu_run():
arch/x86/kvm/x86.c:vcpu_block() {
bool hv_timer;
if (!kvm_arch_vcpu_runnable(vcpu)) {
...
kvm_vcpu_halt(vcpu, &hv_timer);
...
if (!kvm_arch_vcpu_runnable(vcpu))
return 1;
}
...
switch(vcpu->arch.mp_state) {
case KVM_MP_STATE_HALTED:
case KVM_MP_STATE_AP_RESET_HOLD:
kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE);
...
}
Because KVM_REQ_GET_NESTED_STATE_PAGES is only serviced in
vcpu_enter_guest() (which only runs when mp_state == KVM_MP_STATE_RUNNABLE),
nothing refreshes virtual_apic_cache or pi_desc_cache between the two
kvm_arch_vcpu_runnable() calls in vcpu_block(). The second call in
vcpu_block() invokes vmx_has_nested_events() again, still gets -EWOULDBLOCK,
and returns true again. vcpu_block() then transitions the vCPU from
KVM_MP_STATE_HALTED to KVM_MP_STATE_RUNNABLE, resuming L2 execution past HLT
without any architectural wake event.
The same thing happens when L2 executes HLT while a cache is invalid:
kvm_emulate_halt() advances RIP past HLT and calls __kvm_emulate_halt(),
which checks kvm_vcpu_has_events(vcpu) and sets KVM_MP_STATE_RUNNABLE
immediately instead of halting.
Conversely, if a vCPU is restored in L2 with KVM_MP_STATE_HALTED via
KVM_SET_NESTED_STATE (or enters L2 halted via RSM),
nested_vmx_enter_non_root_mode(vcpu, false) defers activating the caches to
KVM_REQ_GET_NESTED_STATE_PAGES. Because vcpu_run() calls vcpu_block()
directly when mp_state != KVM_MP_STATE_RUNNABLE, the caches remain inactive
(!gpc->active), so nested_gpc_try_lock_if_active() returns -ENOENT and
vmx_has_nested_events() returns false, missing pending virtual or posted
interrupts until something else wakes the vCPU.
Should vmx_has_nested_events() refresh the cache when called from a
sleepable TASK_RUNNING context (similar to __kvm_xen_has_interrupt() checking
task_is_running(current) && !in_atomic()), or should vcpu_block() service
KVM_REQ_GET_NESTED_STATE_PAGES before re-evaluating kvm_arch_vcpu_runnable()?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261105020104.2507083-1-dwmw2@infradead.org?part=12
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v2 13/20] KVM: x86: Post KVM_REQ_GET_NESTED_STATE_PAGES on memslot updates
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (11 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 12/20] KVM: nVMX: Use pinned pfncache for L1 APIC virtualization pages David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 14/20] KVM: selftests: Add nested VMX APIC cache invalidation test David Woodhouse
` (6 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
The physical addresses which nested page setup latches into hardware
control structures (vmcs02's APIC-access, virtual-APIC and posted
interrupt descriptor addresses) are derived from gPA→uHVA
translations which a memslot update can change. Software readers of
the underlying gfn_to_pfn_caches catch that lazily, via the memslot
generation check at their next use — but the CPU's use of a latched
address from guest mode is continuous and checks nothing. A vCPU
running L2 across a memslot move keeps using the old translation
until a nested VM-exit happens to re-run nested page setup. (A
staleness problem, not a lifetime one: freeing the page is the
mmu_notifier's business. The replaced kvm_host_map code had the same
staleness with no remedy at all.)
Post the request from the memslot update itself — the single point
where the generation changes, and a slow path by definition — rather
than checking generations in the VM-entry path forever. The request
is what survives racing with a concurrent VM-entry (a bare kick
landing before vcpu->mode is set would be lost), and a vCPU which
resolved its pages against the old memslots but has not yet entered
guest mode is invisible to any is_guest_mode() filter, so it is
posted unconditionally to every vCPU. TD vCPUs share vmx_nested_ops
but are not struct vcpu_vmx; the VMX handler bails for them up
front.
The posting site is kvm_arch_commit_memory_region(), once per
user-visible update with the final memslots in place — not
kvm_arch_memslots_updated(), which also runs when a DELETE or MOVE
installs its transient KVM_MEMSLOT_INVALID placeholder, where the
handler could only retry, and skipping FLAGS_ONLY updates, which
cannot change any translation. kvm_arch_memslots_updated() keeps a
bare kick for the steal_time cache's check-at-use sites.
The handlers distinguish the request's origin, extending VMX's
nested.resume_pending to nSVM. A resume into L2 from saved state
(KVM_SET_NESTED_STATE after migration, or RSM) re-derives PAE PDPTRs
and re-runs the MSRPM merge; a failure there is a userspace error
(or, in a narrow window, a concurrent memslot update's transient
KVM_MEMSLOT_INVALID placeholder), reported as KVM_EXIT_INTERNAL_ERROR
with the request re-posted and resume_pending kept, so a further
KVM_RUN bounces back out rather than entering L2 with the restore
work undone — and so a userspace which retries recovers once a
transient failure clears. A memslot-posted request re-latches the vmcs02 addresses on
VMX and is a no-op on SVM, which latches no L1-owned physical
address in hardware (the merged MSRPM and IOPM are kernel pages);
neither vendor re-derives PDPTRs for it, as those latch only at MOV
CR3 and VM-entry. The classification dies with the request: nested
VM-exit clears resume_pending on both vendors, so a request posted
after L1 re-enters L2 cannot inherit entry-time semantics.
Accordingly, downgrade the WARN in svm_get_nested_state_pages(): a
request with nothing to do is now expected.
The memslot-move mode of the vmx_apic_update_test selftest, added
later in this series, exercises this path.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kvm/svm/nested.c | 45 ++++++++++++++++++++++++++++++++++-----
arch/x86/kvm/svm/svm.h | 12 +++++++++++
arch/x86/kvm/vmx/nested.c | 12 +++++++++--
arch/x86/kvm/x86.c | 44 +++++++++++++++++++++++++++++++++++++-
4 files changed, 105 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index af0ef678433d..ab187abc11b5 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1078,8 +1078,10 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)
if (ret)
return ret;
- if (!from_vmrun)
+ if (!from_vmrun) {
+ svm->nested.resume_pending = true;
kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
+ }
svm_set_gif(svm, true);
@@ -1340,6 +1342,7 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
kvm_warn_on_nested_run_pending(vcpu);
kvm_clear_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
+ svm->nested.resume_pending = false;
/* in case we halted in L2 */
kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE);
@@ -1568,6 +1571,7 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
}
kvm_clear_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
+ svm->nested.resume_pending = false;
}
static int nested_svm_exit_handled_msr(struct vcpu_svm *svm)
@@ -2113,6 +2117,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
if (kvm_vcpu_apicv_active(vcpu))
kvm_make_request(KVM_REQ_APICV_UPDATE, vcpu);
+ svm->nested.resume_pending = true;
kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
out_free:
kfree(save);
@@ -2123,7 +2128,21 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu)
{
- if (WARN_ON(!is_guest_mode(vcpu)))
+ struct vcpu_svm *svm = to_svm(vcpu);
+
+ /*
+ * Memslot updates post this request to every vCPU (to make any
+ * vCPU which has guest pages latched in hardware re-resolve them
+ * against the new memslots), so it can arrive with nothing to
+ * do: outside guest mode, or (unlike VMX) even with L2
+ * running, because SVM latches no L1-owned physical address in
+ * hardware (the merged MSRPM and the IOPM are kernel pages).
+ * The restore work below (PDPTR re-derivation, MSRPM merge)
+ * runs only when the request came from a migration restore;
+ * running it for a memslot update would re-latch a live L2's
+ * PDPTRs at a moment the architecture does not allow.
+ */
+ if (!is_guest_mode(vcpu) || !svm->nested.resume_pending)
return true;
if (is_pae_paging(vcpu)) {
@@ -2139,16 +2158,32 @@ static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu)
kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR);
else if (!vcpu->arch.pdptrs_from_userspace &&
CC(!load_pdptrs(vcpu, vcpu->arch.cr3)))
- return false;
+ goto internal_error;
}
if (!nested_svm_merge_msrpm(vcpu))
- return false;
+ goto internal_error;
if (kvm_hv_verify_vp_assist(vcpu))
- return false;
+ goto internal_error;
+ /* Success consumes the resume classification. */
+ svm->nested.resume_pending = false;
return true;
+
+internal_error:
+ /*
+ * Restored state names an unusable page. (In a narrow window, a
+ * concurrent memslot update's KVM_MEMSLOT_INVALID placeholder
+ * lands here too; a userspace which retries KVM_RUN recovers
+ * once the update commits.) The caller fills in the
+ * KVM_EXIT_INTERNAL_ERROR exit and re-posts the request; what is
+ * SVM's to keep is resume_pending, still set, so that no
+ * subsequent KVM_RUN can enter L2 with the restore work undone:
+ * the vCPU bounces back here (and to userspace) until the
+ * failure is resolved or the vCPU state is replaced.
+ */
+ return false;
}
static gpa_t svm_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 6d0242966dc8..1b8f3f2057d9 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -236,6 +236,18 @@ struct svm_nested_state {
bool initialized;
+ /*
+ * The pending KVM_REQ_GET_NESTED_STATE_PAGES was posted by a
+ * migration restore (KVM_SET_NESTED_STATE or RSM back into L2),
+ * where PDPTRs must be re-derived and the MSRPM merge re-run
+ * before re-entering L2, and a failure is a userspace error to
+ * report. Unlike VMX, SVM latches no L1-owned physical address
+ * in hardware (the merged MSRPM and the IOPM are kernel pages),
+ * so a request posted by a memslot update finds nothing to
+ * re-latch and the handler no-ops on !resume_pending.
+ */
+ bool resume_pending;
+
/*
* Indicates whether MSR bitmap for L2 needs to be rebuilt due to
* changes in MSR bitmap for L1 or switching to a different L2. Note,
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index 23f1fff97917..6c8004009bcd 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -3846,6 +3846,14 @@ static bool __vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
{
+ /*
+ * Memslot updates post this request to every vCPU, including TD
+ * vCPUs, which are not struct vcpu_vmx and have no nested state
+ * to re-resolve.
+ */
+ if (is_td_vcpu(vcpu))
+ return true;
+
if (!__vmx_get_nested_state_pages(vcpu)) {
/*
* The caller (vcpu_enter_guest) consumed
@@ -5501,8 +5509,8 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
/*
* The resume classification dies with the request: a stale
- * resume_pending would give the next request (e.g. posted by an
- * invalidation after L1 re-enters L2) entry-time semantics,
+ * resume_pending would give the next request (e.g. posted by a
+ * memslot update after L1 re-enters L2) entry-time semantics,
* re-deriving PDPTRs mid-L2 and treating a transiently
* unbackable page as a terminal error.
*/
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 321f29789f9f..892c6f81d704 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10282,7 +10282,16 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen)
*/
kvm_mmu_invalidate_mmio_sptes(kvm, gen);
- /* Force re-initialization of steal_time cache */
+ /*
+ * Force re-initialization of the steal_time cache: its
+ * check-at-use sites only see the new generation once the vCPU
+ * goes around its run loop. This runs on both memslot swaps of a
+ * DELETE/MOVE; the re-resolution of latched nested-state pages
+ * is posted from kvm_arch_commit_memory_region() instead, once
+ * per update and only when the final memslots are in place, so
+ * that its handler never chases an in-flight update's
+ * KVM_MEMSLOT_INVALID placeholder.
+ */
kvm_for_each_vcpu(i, vcpu, kvm)
kvm_vcpu_kick(vcpu);
}
@@ -10452,6 +10461,39 @@ void kvm_arch_commit_memory_region(struct kvm *kvm,
const struct kvm_memory_slot *new,
enum kvm_mr_change change)
{
+ /*
+ * Force re-resolution of any nested-state pages whose physical
+ * addresses a vCPU has latched in hardware control structures
+ * (e.g. vmcs02) from a gfn_to_pfn_cache. Software readers of
+ * such caches catch the generation bump lazily, via
+ * kvm_gpc_check() at their next use; the CPU's use from guest
+ * mode is continuous and checks nothing, so the vCPU must be
+ * told to re-resolve and re-latch before it next enters the
+ * guest. The request is the artifact that survives racing with
+ * a concurrent VM-entry (a bare kick landing before vcpu->mode
+ * is set would be lost); its handler re-runs nested page setup,
+ * whose gPA lookups then see the new generation and refresh.
+ *
+ * This is posted from commit, not from kvm_arch_memslots_updated(),
+ * so it runs once per user-visible update with the final memslots
+ * in place: a request serviced during a DELETE/MOVE's
+ * invalidation phase would find the KVM_MEMSLOT_INVALID
+ * placeholder and could only retry. A FLAGS_ONLY update (e.g. a
+ * dirty-log toggle) cannot change any translation — base_gfn and
+ * userspace_addr are immutable — so it posts nothing.
+ *
+ * The kick is deliberately fire-and-forget (no KVM_REQUEST_WAIT):
+ * a vCPU still in L2 may keep using the old translation until it
+ * services the request, which matches the staleness semantics the
+ * replaced kvm_host_map code had, unboundedly. Lifetime is not at
+ * stake — the underlying page is freed only via the mmu_notifier,
+ * whose invalidation forces pinned vCPUs out of guest mode and
+ * waits. Making the ioctl wait here would buy only a tighter
+ * staleness bound, at a vCPU-roundtrip cost on every update.
+ */
+ if (change != KVM_MR_FLAGS_ONLY)
+ kvm_make_all_cpus_request(kvm, KVM_REQ_GET_NESTED_STATE_PAGES);
+
if (change == KVM_MR_DELETE)
kvm_page_track_delete_slot(kvm, old);
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 14/20] KVM: selftests: Add nested VMX APIC cache invalidation test
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (12 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 13/20] KVM: x86: Post KVM_REQ_GET_NESTED_STATE_PAGES on memslot updates David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:25 ` sashiko-bot
2026-11-05 2:00 ` [PATCH v2 15/20] KVM: nSVM: Use a gfn_to_pfn_cache for the vmcb12 page David Woodhouse
` (5 subsequent siblings)
19 siblings, 1 reply; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: Fred Griffoul <fgriffo@amazon.co.uk>
Add a selftest for the nested VMX APIC page caches: an L2 guest
writes the APIC-access page while a second thread repeatedly
invalidates the underlying mapping, either with
madvise(MADV_DONTNEED) (MMU notifier path) or by moving the memslot
away and back (memslot update path), so the GPA under test is
repeatedly unbacked and re-latched.
The test asserts that L2 keeps running across the invalidations with
no guest-visible errors, that at least one invalidation actually ran
concurrently, and that the access only ever bounces to userspace as
MMIO in the memslot-move variant (while the slot is moved away the
GPA is unbacked; madvise never unbacks the GPA, so an MMIO exit
there means KVM lost the mapping).
Signed-off-by: Fred Griffoul <fgriffo@amazon.co.uk>
[dwmw2: Adapt to selftest API drift since the v4 posting: prepare_vmcs()
no longer takes an L2 stack (4c396f19de5d), vm_vaddr_t is now
gva_t, and the prepare_eptp()/nested_map*() EPT helpers were
replaced by vm_enable_tdp()/tdp_map()/
tdp_identity_map_default_memslots().]
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
tools/testing/selftests/kvm/Makefile.kvm | 1 +
.../selftests/kvm/x86/vmx_apic_update_test.c | 323 ++++++++++++++++++
2 files changed, 324 insertions(+)
create mode 100644 tools/testing/selftests/kvm/x86/vmx_apic_update_test.c
diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 17a4256e9081..dcfe815f4a02 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -161,6 +161,7 @@ TEST_GEN_PROGS_x86 += x86/triple_fault_event_test
TEST_GEN_PROGS_x86 += x86/recalc_apic_map_test
TEST_GEN_PROGS_x86 += x86/aperfmperf_test
TEST_GEN_PROGS_x86 += x86/guest_memfd_conversions_test
+TEST_GEN_PROGS_x86 += x86/vmx_apic_update_test
TEST_GEN_PROGS_x86 += access_tracking_perf_test
TEST_GEN_PROGS_x86 += coalesced_io_test
TEST_GEN_PROGS_x86 += dirty_log_perf_test
diff --git a/tools/testing/selftests/kvm/x86/vmx_apic_update_test.c b/tools/testing/selftests/kvm/x86/vmx_apic_update_test.c
new file mode 100644
index 000000000000..357e10c88cd0
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/vmx_apic_update_test.c
@@ -0,0 +1,323 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * vmx_apic_update_test
+ *
+ * Copyright (C) 2025, Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Test L2 guest APIC access page writes with concurrent MMU
+ * notification and memslot move updates.
+ */
+#include <limits.h>
+#include <pthread.h>
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "vmx.h"
+
+#define VAPIC_GPA 0xc0000000
+#define VAPIC_SLOT 1
+
+
+#define L2_DELAY (100)
+
+static void l2_guest_code(void)
+{
+ uint32_t *vapic_addr = (uint32_t *) (VAPIC_GPA + 0x80);
+
+ /* Unroll the loop to avoid any compiler side effect */
+
+ WRITE_ONCE(*vapic_addr, 1 << 0);
+ udelay(msecs_to_usecs(L2_DELAY));
+
+ WRITE_ONCE(*vapic_addr, 1 << 1);
+ udelay(msecs_to_usecs(L2_DELAY));
+
+ WRITE_ONCE(*vapic_addr, 1 << 2);
+ udelay(msecs_to_usecs(L2_DELAY));
+
+ WRITE_ONCE(*vapic_addr, 1 << 3);
+ udelay(msecs_to_usecs(L2_DELAY));
+
+ WRITE_ONCE(*vapic_addr, 1 << 4);
+ udelay(msecs_to_usecs(L2_DELAY));
+
+ WRITE_ONCE(*vapic_addr, 1 << 5);
+ udelay(msecs_to_usecs(L2_DELAY));
+
+ WRITE_ONCE(*vapic_addr, 1 << 6);
+ udelay(msecs_to_usecs(L2_DELAY));
+
+ WRITE_ONCE(*vapic_addr, 0);
+ udelay(msecs_to_usecs(L2_DELAY));
+
+ /* Exit to L1 */
+ vmcall();
+}
+
+static void l1_guest_code(struct vmx_pages *vmx_pages)
+{
+ uint32_t control, exit_reason;
+
+ prepare_for_vmx_operation(vmx_pages);
+ load_vmcs(vmx_pages);
+ prepare_vmcs(vmx_pages, l2_guest_code);
+
+ /* Enable APIC access */
+ control = vmread(CPU_BASED_VM_EXEC_CONTROL);
+ control |= CPU_BASED_ACTIVATE_SECONDARY_CONTROLS;
+ vmwrite(CPU_BASED_VM_EXEC_CONTROL, control);
+ control = vmread(SECONDARY_VM_EXEC_CONTROL);
+ control |= SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
+ vmwrite(SECONDARY_VM_EXEC_CONTROL, control);
+ vmwrite(APIC_ACCESS_ADDR, VAPIC_GPA);
+
+ GUEST_SYNC1(0);
+ vmlaunch();
+again:
+ exit_reason = vmread(VM_EXIT_REASON);
+ if (exit_reason == EXIT_REASON_APIC_ACCESS) {
+ uint64_t guest_rip = vmread(GUEST_RIP);
+ uint64_t instr_len = vmread(VM_EXIT_INSTRUCTION_LEN);
+
+ vmwrite(GUEST_RIP, guest_rip + instr_len);
+ vmresume();
+ goto again;
+ }
+
+ GUEST_SYNC1(exit_reason);
+ GUEST_ASSERT(exit_reason == EXIT_REASON_VMCALL);
+ GUEST_DONE();
+}
+
+static const char *progname;
+static int update_period_ms = L2_DELAY / 4;
+
+struct update_control {
+ pthread_mutex_t mutex;
+ pthread_cond_t start_cond;
+ struct kvm_vm *vm;
+ bool running;
+ bool started;
+ int updates;
+};
+
+static void wait_for_start_signal(struct update_control *ctrl)
+{
+ pthread_mutex_lock(&ctrl->mutex);
+ while (!ctrl->started)
+ pthread_cond_wait(&ctrl->start_cond, &ctrl->mutex);
+
+ pthread_mutex_unlock(&ctrl->mutex);
+ printf("%s: starting update\n", progname);
+}
+
+static bool is_running(struct update_control *ctrl)
+{
+ return READ_ONCE(ctrl->running);
+}
+
+static void set_running(struct update_control *ctrl, bool running)
+{
+ WRITE_ONCE(ctrl->running, running);
+}
+
+static void signal_thread_start(struct update_control *ctrl)
+{
+ pthread_mutex_lock(&ctrl->mutex);
+ if (!ctrl->started) {
+ ctrl->started = true;
+ pthread_cond_signal(&ctrl->start_cond);
+ }
+ pthread_mutex_unlock(&ctrl->mutex);
+}
+
+static void *update_madvise(void *arg)
+{
+ struct update_control *ctrl = arg;
+ void *hva;
+
+ wait_for_start_signal(ctrl);
+
+ hva = addr_gpa2hva(ctrl->vm, VAPIC_GPA);
+ memset(hva, 0x45, ctrl->vm->page_size);
+
+ while (is_running(ctrl)) {
+ usleep(update_period_ms * 1000);
+ madvise(hva, ctrl->vm->page_size, MADV_DONTNEED);
+ ctrl->updates++;
+ }
+
+ return NULL;
+}
+
+static void *update_move_memslot(void *arg)
+{
+ struct update_control *ctrl = arg;
+ uint64_t gpa = VAPIC_GPA;
+
+ wait_for_start_signal(ctrl);
+
+ /*
+ * Alternate the slot between VAPIC_GPA and an alternate address,
+ * so that the GPA under test is repeatedly unbacked (L2's APIC
+ * access bounces to userspace as MMIO) and re-backed (KVM must
+ * re-latch the page). Moving strictly away would change
+ * vCPU-visible state only once.
+ */
+ while (is_running(ctrl)) {
+ usleep(update_period_ms * 1000);
+ gpa = (gpa == VAPIC_GPA) ? VAPIC_GPA + 0x10000 : VAPIC_GPA;
+ vm_mem_region_move(ctrl->vm, VAPIC_SLOT, gpa);
+ ctrl->updates++;
+ }
+
+ return NULL;
+}
+
+static void run(void * (*update)(void *), const char *name, bool mmio_ok)
+{
+ struct kvm_vm *vm;
+ struct kvm_vcpu *vcpu;
+ struct update_control ctrl;
+ struct ucall uc;
+ gva_t vmx_pages_gva;
+ pthread_t update_thread;
+ bool done = false;
+
+ vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code);
+
+ /* Set up nested EPT */
+ vm_enable_tdp(vm);
+
+ /* Allocate VMX pages */
+ vcpu_alloc_vmx(vm, &vmx_pages_gva);
+
+ /* Allocate memory and create VAPIC memslot */
+ vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, VAPIC_GPA,
+ VAPIC_SLOT, 1, 0);
+
+ /* Allocate guest page table */
+ virt_map(vm, VAPIC_GPA, VAPIC_GPA, 1);
+
+ /* Identity-map everything, including the VAPIC page, in the EPT */
+ tdp_identity_map_default_memslots(vm);
+ tdp_map(vm, VAPIC_GPA, VAPIC_GPA, vm->page_size);
+
+ vcpu_args_set(vcpu, 1, vmx_pages_gva);
+
+ pthread_mutex_init(&ctrl.mutex, NULL);
+ pthread_cond_init(&ctrl.start_cond, NULL);
+ ctrl.vm = vm;
+ ctrl.running = true;
+ ctrl.started = false;
+ ctrl.updates = 0;
+
+ kvm_pthread_create(&update_thread, NULL, update, &ctrl);
+
+ printf("%s: running %s (tsc_khz %lu)\n", progname, name, guest_tsc_khz);
+
+ while (!done) {
+ vcpu_run(vcpu);
+
+ switch (vcpu->run->exit_reason) {
+ case KVM_EXIT_IO:
+ switch (get_ucall(vcpu, &uc)) {
+ case UCALL_SYNC:
+ printf("%s: sync(%ld)\n", progname, uc.args[0]);
+ if (uc.args[0] == 0)
+ signal_thread_start(&ctrl);
+ break;
+ case UCALL_ABORT:
+ REPORT_GUEST_ASSERT(uc);
+ /* NOT REACHED */
+ case UCALL_DONE:
+ done = true;
+ break;
+ default:
+ TEST_ASSERT(false, "Unknown ucall %lu", uc.cmd);
+ }
+ break;
+ case KVM_EXIT_MMIO:
+ /*
+ * While the slot is moved away, the GPA under test
+ * is unbacked and the access legitimately bounces
+ * to userspace as MMIO (until the updater moves it
+ * back). madvise() never unbacks the GPA, so there
+ * an MMIO exit means KVM lost the mapping: fail.
+ */
+ TEST_ASSERT(mmio_ok,
+ "%s: unexpected APIC MMIO access at 0x%llx",
+ progname, vcpu->run->mmio.phys_addr);
+ printf
+ ("%s: APIC MMIO access at 0x%llx (memslot move effect)\n",
+ progname, vcpu->run->mmio.phys_addr);
+ break;
+ default:
+ TEST_FAIL("%s: Unexpected exit reason: %d (flags 0x%x)",
+ progname,
+ vcpu->run->exit_reason, vcpu->run->flags);
+ }
+ }
+
+ set_running(&ctrl, false);
+ if (!ctrl.started)
+ signal_thread_start(&ctrl);
+ kvm_pthread_join(update_thread, NULL);
+ printf("%s: completed with %d updates\n", progname, ctrl.updates);
+ TEST_ASSERT(ctrl.updates > 0,
+ "%s: %s performed no updates; nothing was tested",
+ progname, name);
+
+ pthread_mutex_destroy(&ctrl.mutex);
+ pthread_cond_destroy(&ctrl.start_cond);
+ kvm_vm_free(vm);
+}
+
+int main(int argc, char *argv[])
+{
+ int opt_madvise = 0;
+ int opt_memslot_move = 0;
+
+ TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX));
+ TEST_REQUIRE(kvm_cpu_has_ept());
+
+ if (argc == 1) {
+ opt_madvise = 1;
+ opt_memslot_move = 1;
+ } else {
+ int opt;
+
+ while ((opt = getopt(argc, argv, "amp:")) != -1) {
+ switch (opt) {
+ case 'a':
+ opt_madvise = 1;
+ break;
+ case 'm':
+ opt_memslot_move = 1;
+ break;
+ case 'p':
+ update_period_ms = atoi(optarg);
+ TEST_ASSERT(update_period_ms > 0 &&
+ update_period_ms < INT_MAX / 1000,
+ "Update period must be a positive number of milliseconds, not '%s'",
+ optarg);
+ break;
+ default:
+ exit(1);
+ }
+ }
+ }
+
+ TEST_ASSERT(opt_madvise
+ || opt_memslot_move, "No update test configured");
+
+ progname = argv[0];
+
+ if (opt_madvise)
+ run(update_madvise, "madvise", false);
+
+ if (opt_memslot_move)
+ run(update_move_memslot, "move memslot", true);
+
+ return 0;
+}
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v2 14/20] KVM: selftests: Add nested VMX APIC cache invalidation test
2026-11-05 2:00 ` [PATCH v2 14/20] KVM: selftests: Add nested VMX APIC cache invalidation test David Woodhouse
@ 2026-11-05 2:25 ` sashiko-bot
0 siblings, 0 replies; 25+ messages in thread
From: sashiko-bot @ 2026-11-05 2:25 UTC (permalink / raw)
To: David Woodhouse, kvm; +Cc: linux-rt-devel, linux-kernel
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Medium] KVM: selftests: missing virtualize APIC accesses check in main()
- [Medium] KVM: selftests: memslot move races with vmlaunch() and vmresume() in run()
- [Medium] KVM: selftests: vmx_apic_update_test fails to verify APIC-access exits
--
commit c3febf61f5d64e784b8c55b816b4a00a658a0208
Author: Fred Griffoul <fgriffo@amazon.co.uk>
KVM: selftests: Add nested VMX APIC cache invalidation test
This commit adds a selftest for nested VMX APIC-access page cache
invalidation while an L2 guest writes to the APIC-access page concurrently
with either madvise(MADV_DONTNEED) or memslot moves.
> diff --git a/tools/testing/selftests/kvm/x86/vmx_apic_update_test.c b/tools/testing/selftests/kvm/x86/vmx_apic_update_test.c
> new file mode 100644
> index 0000000000000..357e10c88cd0a
> --- /dev/null
> +++ b/tools/testing/selftests/kvm/x86/vmx_apic_update_test.c
[ ... ]
> +static void l1_guest_code(struct vmx_pages *vmx_pages)
> +{
> + uint32_t control, exit_reason;
> +
> + prepare_for_vmx_operation(vmx_pages);
> + load_vmcs(vmx_pages);
> + prepare_vmcs(vmx_pages, l2_guest_code);
> +
> + /* Enable APIC access */
> + control = vmread(CPU_BASED_VM_EXEC_CONTROL);
> + control |= CPU_BASED_ACTIVATE_SECONDARY_CONTROLS;
> + vmwrite(CPU_BASED_VM_EXEC_CONTROL, control);
> + control = vmread(SECONDARY_VM_EXEC_CONTROL);
> + control |= SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES;
> + vmwrite(SECONDARY_VM_EXEC_CONTROL, control);
> + vmwrite(APIC_ACCESS_ADDR, VAPIC_GPA);
> +
> + GUEST_SYNC1(0);
> + vmlaunch();
> +again:
> + exit_reason = vmread(VM_EXIT_REASON);
> + if (exit_reason == EXIT_REASON_APIC_ACCESS) {
> + uint64_t guest_rip = vmread(GUEST_RIP);
> + uint64_t instr_len = vmread(VM_EXIT_INSTRUCTION_LEN);
> +
> + vmwrite(GUEST_RIP, guest_rip + instr_len);
> + vmresume();
> + goto again;
> + }
> +
> + GUEST_SYNC1(exit_reason);
> + GUEST_ASSERT(exit_reason == EXIT_REASON_VMCALL);
> + GUEST_DONE();
> +}
[ ... ]
> +static void *update_move_memslot(void *arg)
> +{
> + struct update_control *ctrl = arg;
> + uint64_t gpa = VAPIC_GPA;
> +
> + wait_for_start_signal(ctrl);
> +
> + /*
> + * Alternate the slot between VAPIC_GPA and an alternate address,
> + * so that the GPA under test is repeatedly unbacked (L2's APIC
> + * access bounces to userspace as MMIO) and re-backed (KVM must
> + * re-latch the page). Moving strictly away would change
> + * vCPU-visible state only once.
> + */
> + while (is_running(ctrl)) {
> + usleep(update_period_ms * 1000);
> + gpa = (gpa == VAPIC_GPA) ? VAPIC_GPA + 0x10000 : VAPIC_GPA;
> + vm_mem_region_move(ctrl->vm, VAPIC_SLOT, gpa);
> + ctrl->updates++;
> + }
> +
> + return NULL;
> +}
> +
> +static void run(void * (*update)(void *), const char *name, bool mmio_ok)
> +{
[ ... ]
> + case KVM_EXIT_MMIO:
> + /*
> + * While the slot is moved away, the GPA under test
> + * is unbacked and the access legitimately bounces
> + * to userspace as MMIO (until the updater moves it
> + * back). madvise() never unbacks the GPA, so there
> + * an MMIO exit means KVM lost the mapping: fail.
> + */
> + TEST_ASSERT(mmio_ok,
> + "%s: unexpected APIC MMIO access at 0x%llx",
> + progname, vcpu->run->mmio.phys_addr);
> + printf
> + ("%s: APIC MMIO access at 0x%llx (memslot move effect)\n",
> + progname, vcpu->run->mmio.phys_addr);
> + break;
[Severity: Medium]
Does checking for the absence of KVM_EXIT_MMIO in run() actually verify that
APIC-access virtualization stayed active across invalidations?
In l2_guest_code(), L2 performs 8 writes to VAPIC_GPA + 0x80 followed by
vmcall(), while l1_guest_code() handles EXIT_REASON_APIC_ACCESS by advancing
GUEST_RIP and executing vmresume() without counting how many
EXIT_REASON_APIC_ACCESS exits occurred.
In the madvise(MADV_DONTNEED) test, VAPIC_SLOT remains registered at
VAPIC_GPA. After madvise() zaps the userspace PTE and EPT02 SPTE, L2's next
write faults in a new anonymous host page in EPT02 rather than triggering
KVM_EXIT_MMIO. If KVM fails to refresh vmcs02.APIC_ACCESS_ADDR to the new
HPA (or fails to re-enable SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES after
update_move_memslot() moves VAPIC_SLOT back to VAPIC_GPA), the EPT02 HPA
will not match vmcs02.APIC_ACCESS_ADDR and L2's writes will silently commit
to normal memory without triggering either EXIT_REASON_APIC_ACCESS or
KVM_EXIT_MMIO.
Should l1_guest_code() and run() count the EXIT_REASON_APIC_ACCESS exits and
assert that all 8 L2 writes are intercepted as EXIT_REASON_APIC_ACCESS in
the madvise case (and as either EXIT_REASON_APIC_ACCESS or KVM_EXIT_MMIO in
the memslot-move case)?
> + default:
> + TEST_FAIL("%s: Unexpected exit reason: %d (flags 0x%x)",
> + progname,
> + vcpu->run->exit_reason, vcpu->run->flags);
> + }
> + }
[Severity: Medium]
As reported by Sashiko on v1 [1], can update_move_memslot() race with
vmlaunch() and vmresume() in l1_guest_code() and cause run() to fail with
KVM_EXIT_INTERNAL_ERROR?
In l1_guest_code(), GUEST_SYNC1(0) starts the updater thread before
vmlaunch(), and each EXIT_REASON_APIC_ACCESS exit into L1 advances GUEST_RIP
and executes vmresume(). Concurrently, update_move_memslot() toggles
VAPIC_SLOT between VAPIC_GPA and VAPIC_GPA + 0x10000.
In KVM's nested VM-entry path:
nested_vmx_run()
nested_vmx_enter_non_root_mode()
nested_get_vmcs12_pages(vcpu, false)
nested_get_vmcs12_pages() is called with lazy == false on both vmlaunch and
vmresume. Unlike lazy revalidation while L2 is already in guest mode, if
VAPIC_GPA is unbacked when L1 executes vmlaunch() or vmresume(),
nested_get_vmcs12_pages() treats the missing gfn as an error and exits to
userspace with KVM_EXIT_INTERNAL_ERROR, hitting the default TEST_FAIL() case
in run().
[ ... ]
> +int main(int argc, char *argv[])
> +{
> + int opt_madvise = 0;
> + int opt_memslot_move = 0;
> +
> + TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX));
> + TEST_REQUIRE(kvm_cpu_has_ept());
[Severity: Medium]
Should main() also check TEST_REQUIRE(kvm_cpu_has_vmx_virtualize_apic_accesses())
here, similar to vmx_apic_access_test.c?
In l1_guest_code(), the test unconditionally sets
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES in SECONDARY_VM_EXEC_CONTROL before
calling vmlaunch(). On hosts or nested environments where virtualize APIC
accesses is unsupported or disabled (for example when flexpriority is
disabled), vmlaunch() will fail with a guest assertion failure instead of
skipping the test.
[1] https://lore.kernel.org/r/20260920211920.928306-13-dwmw2@infradead.org
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261105020104.2507083-1-dwmw2@infradead.org?part=14
^ permalink raw reply [flat|nested] 25+ messages in thread
* [PATCH v2 15/20] KVM: nSVM: Use a gfn_to_pfn_cache for the vmcb12 page
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (13 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 14/20] KVM: selftests: Add nested VMX APIC cache invalidation test David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 16/20] KVM: nSVM: Cache L1's MSR permissions map pages David Woodhouse
` (4 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
Replace the transient map/unmap of the vmcb12 page on every nested
VMRUN (control/save copy-in) and nested VM-exit (state copy-back) with
a gfn_to_pfn_cache keyed on vmcb12_gpa. With unmanaged guest memory
the transient map is a memremap/memunmap cycle on every L1<->L2
transition; the cache persists the kernel mapping for as long as the
gPA and its translation are unchanged. An L1 which multiplexes
several L2s on one vCPU re-keys the cache on each switch, paying a
refresh; the common single-L2 case pays nothing.
No pinning is needed, unlike the nVMX APIC pages: SVM never hands an
L1-owned physical address to the CPU in vmcb02 (the merged MSRPM and
the IOPM are kernel-owned pages), so the vmcb12 page is only ever
accessed by KVM itself, under the SRCU read lock, with the usual
check/refresh protocol.
Dirty marking moves from unconditional-at-unmap to the actual write
paths: the SVM_EXIT_ERR reflection in the VMRUN consistency-check
failure path, and the copy-back at nested VM-exit.
The SMM enter/leave and VMLOAD/VMSAVE emulation paths keep the
transient map: they are cold, and VMLOAD/VMSAVE takes an arbitrary
gPA from RAX which would thrash a single-slot cache.
svm_leave_smm() now validates the vmcb12 GPA it reads from SMRAM, as
VMRUN does for RAX; the copy-back through the cache would otherwise
fail on an unaligned address at the eventual nested VM-exit. The
refused RSM still ends in TRIPLE_FAULT, but at the corrupt-SMRAM
instruction itself, before any L2 state is committed, as for the
other SMRAM consistency checks here.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kvm/svm/nested.c | 27 ++++++++++++++++++++-------
arch/x86/kvm/svm/svm.c | 15 +++++++++++++++
arch/x86/kvm/svm/svm.h | 3 +++
3 files changed, 38 insertions(+), 7 deletions(-)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index ab187abc11b5..32854c8b2bbc 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1098,12 +1098,13 @@ static int nested_svm_copy_vmcb12_to_cache(struct kvm_vcpu *vcpu, u64 vmcb12_gpa
struct vcpu_svm *svm = to_svm(vcpu);
struct vmcb *vmcb12;
int r = 0;
+ int idx;
- CLASS(kvm_vcpu_map_local, m)(vcpu, gpa_to_gfn(vmcb12_gpa));
- if (m.ret)
+ idx = kvm_gpc_lock_page(&svm->nested.vmcb12_cache, vmcb12_gpa);
+ if (idx < 0)
return -EFAULT;
- vmcb12 = m.map.hva;
+ vmcb12 = svm->nested.vmcb12_cache.khva;
nested_copy_vmcb_control_to_cache(svm, &vmcb12->control);
nested_copy_vmcb_save_to_cache(svm, &vmcb12->save);
@@ -1114,9 +1115,11 @@ static int nested_svm_copy_vmcb12_to_cache(struct kvm_vcpu *vcpu, u64 vmcb12_gpa
vmcb12->control.event_inj = 0;
vmcb12->control.event_inj_err = 0;
svm_set_gif(svm, false);
+ kvm_gpc_mark_dirty_in_slot(&svm->nested.vmcb12_cache);
r = -EINVAL;
}
+ kvm_gpc_unlock(&svm->nested.vmcb12_cache, idx);
return r;
}
@@ -1261,12 +1264,14 @@ static int nested_svm_vmexit_update_vmcb12(struct kvm_vcpu *vcpu)
struct vcpu_svm *svm = to_svm(vcpu);
struct vmcb *vmcb02 = svm->nested.vmcb02.ptr;
struct vmcb *vmcb12;
+ int idx;
- CLASS(kvm_vcpu_map_local, m)(vcpu, gpa_to_gfn(svm->nested.vmcb12_gpa));
- if (m.ret)
- return m.ret;
+ idx = kvm_gpc_lock_page(&svm->nested.vmcb12_cache,
+ svm->nested.vmcb12_gpa);
+ if (idx < 0)
+ return idx;
- vmcb12 = m.map.hva;
+ vmcb12 = svm->nested.vmcb12_cache.khva;
vmcb12->save.es = vmcb02->save.es;
vmcb12->save.cs = vmcb02->save.cs;
@@ -1321,6 +1326,9 @@ static int nested_svm_vmexit_update_vmcb12(struct kvm_vcpu *vcpu)
vmcb12->control.exit_int_info_err,
KVM_ISA_SVM);
+ kvm_gpc_mark_dirty_in_slot(&svm->nested.vmcb12_cache);
+ kvm_gpc_unlock(&svm->nested.vmcb12_cache, idx);
+
return 0;
}
@@ -1506,6 +1514,9 @@ int svm_allocate_nested(struct vcpu_svm *svm)
goto err_free_vmcb02;
svm->nested.vmcb02.cpu = -1;
+
+ kvm_gpc_init(&svm->nested.vmcb12_cache, svm->vcpu.kvm);
+
svm->nested.initialized = true;
return 0;
@@ -1525,6 +1536,8 @@ void svm_free_nested(struct vcpu_svm *svm)
svm_vcpu_free_msrpm(svm->nested.msrpm);
svm->nested.msrpm = NULL;
+ kvm_gpc_deactivate(&svm->nested.vmcb12_cache);
+
__free_page(virt_to_page(svm->nested.vmcb02.ptr));
svm->nested.vmcb02.ptr = NULL;
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 136069dfcd44..b78727cca9b1 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5206,6 +5206,21 @@ static int svm_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram)
if (!(smram64->efer & EFER_SVME))
return 1;
+ /*
+ * The vmcb12 GPA read back from SMRAM feeds the vmcb12 cache at
+ * the eventual nested VM-exit, whose copy-back would fail on an
+ * unaligned or illegal address. Refusing the RSM here ends in
+ * TRIPLE_FAULT just the same (em_rsm() shoots a refused RSM),
+ * but it fails before any L2 state has been committed, at the
+ * instruction whose SMRAM contents are corrupt, matching how
+ * this function already treats other inconsistent SMRAM state —
+ * rather than at an unrelated nested VM-exit later. VMRUN
+ * (#GP) and KVM_SET_NESTED_STATE (-EINVAL) police the same
+ * field on their own entry points.
+ */
+ if (!page_address_valid(vcpu, smram64->svm_guest_vmcb_gpa))
+ return 1;
+
CLASS(kvm_vcpu_map_local, m)(vcpu, gpa_to_gfn(smram64->svm_guest_vmcb_gpa));
if (m.ret)
return 1;
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 1b8f3f2057d9..fe964a961305 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -219,6 +219,9 @@ struct svm_nested_state {
u64 last_vmcb12_gpa;
u64 last_bus_lock_rip;
+ /* Cached mapping of the vmcb12 page, keyed on vmcb12_gpa */
+ struct gfn_to_pfn_cache vmcb12_cache;
+
/*
* The MSR permissions map used for vmcb02, which is the merge result
* of vmcb01 and vmcb12
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 16/20] KVM: nSVM: Cache L1's MSR permissions map pages
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (14 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 15/20] KVM: nSVM: Use a gfn_to_pfn_cache for the vmcb12 page David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:00 ` [PATCH v2 17/20] KVM: nSVM: Cache L1's IO " David Woodhouse
` (3 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
Reads of L1's MSRPM go through kvm_vcpu_read_guest() on two paths:
the vmcb02 MSRPM merge on nested VMRUN (~10 reads per entry), and the
per-intercept check on every trapped L2 MSR access. With unmanaged
guest memory each read is a memremap/memunmap cycle.
Cache the two pages of L1's MSRPM in a pair of gfn_to_pfn_caches,
keyed on msrpm_base_pa (re-keying automatically when L1 switches to
a different L2 with a different bitmap). Reads preserve the existing
failure semantics: an unreadable bitmap means NESTED_EXIT_DONE
(reflect to L1) on the intercept path and a failed VMRUN on the merge
path.
An 8-byte merge read is naturally aligned and a one-byte intercept
read cannot cross a page, so each access hits exactly one cached page;
a WARN in the accessor enforces that invariant.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kvm/svm/nested.c | 57 ++++++++++++++++++++++++++++++++-------
arch/x86/kvm/svm/svm.h | 7 +++++
2 files changed, 54 insertions(+), 10 deletions(-)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 32854c8b2bbc..4e22cd7a38e1 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -288,10 +288,35 @@ int __init nested_svm_init_msrpm_merge_offsets(void)
}
/*
- * Merge L0's (KVM) and L1's (Nested VMCB) MSR permission bitmaps. The function
- * is optimized in that it only merges the parts where KVM MSR permission bitmap
- * may contain zero bits.
+ * Read from L1's MSR permissions map through the per-page caches.
+ * @offset is a byte offset into the (two-page) MSRPM; the read must not
+ * cross a page boundary. msrpm_base_pa is page-aligned by construction:
+ * __nested_copy_vmcb_control_to_cache() masks it with PAGE_MASK (hardware
+ * ignores bits 11:0), and nested_vmcb_check_controls() bounds-checks it
+ * via nested_svm_check_bitmap_pa().
*/
+static int nested_svm_read_l1_msrpm(struct vcpu_svm *svm, unsigned long offset,
+ void *val, unsigned int len)
+{
+ gpa_t base = svm->nested.ctl.msrpm_base_pa;
+ struct gfn_to_pfn_cache *gpc;
+ int idx;
+
+ if (WARN_ON_ONCE(offset >= MSRPM_SIZE ||
+ offset_in_page(offset) + len > PAGE_SIZE))
+ return -EINVAL;
+
+ gpc = &svm->nested.msrpm12_cache[offset >> PAGE_SHIFT];
+
+ idx = kvm_gpc_lock_page(gpc, base + (offset & PAGE_MASK));
+ if (idx < 0)
+ return idx;
+
+ memcpy(val, gpc->khva + offset_in_page(offset), len);
+ kvm_gpc_unlock(gpc, idx);
+ return 0;
+}
+
static bool nested_svm_merge_msrpm(struct kvm_vcpu *vcpu)
{
struct vcpu_svm *svm = to_svm(vcpu);
@@ -324,11 +349,9 @@ static bool nested_svm_merge_msrpm(struct kvm_vcpu *vcpu)
for (i = 0; i < nested_svm_nr_msrpm_merge_offsets; i++) {
const int p = nested_svm_msrpm_merge_offsets[i];
nsvm_msrpm_merge_t l1_val;
- gpa_t gpa;
-
- gpa = svm->nested.ctl.msrpm_base_pa + (p * sizeof(l1_val));
- if (kvm_vcpu_read_guest(vcpu, gpa, &l1_val, sizeof(l1_val)))
+ if (nested_svm_read_l1_msrpm(svm, p * sizeof(l1_val),
+ &l1_val, sizeof(l1_val)))
return false;
msrpm02[p] = msrpm01[p] | l1_val;
@@ -1499,6 +1522,7 @@ static void nested_svm_triple_fault(struct kvm_vcpu *vcpu)
int svm_allocate_nested(struct vcpu_svm *svm)
{
struct page *vmcb02_page;
+ int i;
if (svm->nested.initialized)
return 0;
@@ -1517,6 +1541,16 @@ int svm_allocate_nested(struct vcpu_svm *svm)
kvm_gpc_init(&svm->nested.vmcb12_cache, svm->vcpu.kvm);
+ /*
+ * KVM only ever reads the MSR permissions map, which L1 may keep in
+ * memory that its VMM maps read-only: map the cache read-only
+ * (which also means never marking it dirty), matching the read
+ * semantics of the kvm_vcpu_read_guest() calls it replaces.
+ */
+ for (i = 0; i < ARRAY_SIZE(svm->nested.msrpm12_cache); i++)
+ __kvm_gpc_init(&svm->nested.msrpm12_cache[i], svm->vcpu.kvm,
+ false, true, NULL, 0);
+
svm->nested.initialized = true;
return 0;
@@ -1527,6 +1561,8 @@ int svm_allocate_nested(struct vcpu_svm *svm)
void svm_free_nested(struct vcpu_svm *svm)
{
+ int i;
+
if (!svm->nested.initialized)
return;
@@ -1537,6 +1573,8 @@ void svm_free_nested(struct vcpu_svm *svm)
svm->nested.msrpm = NULL;
kvm_gpc_deactivate(&svm->nested.vmcb12_cache);
+ for (i = 0; i < ARRAY_SIZE(svm->nested.msrpm12_cache); i++)
+ kvm_gpc_deactivate(&svm->nested.msrpm12_cache[i]);
__free_page(virt_to_page(svm->nested.vmcb02.ptr));
svm->nested.vmcb02.ptr = NULL;
@@ -1589,7 +1627,6 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
static int nested_svm_exit_handled_msr(struct vcpu_svm *svm)
{
- gpa_t base = svm->nested.ctl.msrpm_base_pa;
int write, bit_nr;
u8 value, mask;
u32 msr;
@@ -1604,8 +1641,8 @@ static int nested_svm_exit_handled_msr(struct vcpu_svm *svm)
if (bit_nr < 0)
return NESTED_EXIT_DONE;
- if (kvm_vcpu_read_guest(&svm->vcpu, base + bit_nr / BITS_PER_BYTE,
- &value, sizeof(value)))
+ if (nested_svm_read_l1_msrpm(svm, bit_nr / BITS_PER_BYTE,
+ &value, sizeof(value)))
return NESTED_EXIT_DONE;
mask = BIT(write) << (bit_nr & (BITS_PER_BYTE - 1));
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index fe964a961305..8a5dff5a9ee3 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -222,6 +222,13 @@ struct svm_nested_state {
/* Cached mapping of the vmcb12 page, keyed on vmcb12_gpa */
struct gfn_to_pfn_cache vmcb12_cache;
+ /*
+ * Cached mappings of the two pages of L1's MSR permissions map,
+ * keyed on ctl.msrpm_base_pa, for the vmcb02 merge and the
+ * per-intercept checks.
+ */
+ struct gfn_to_pfn_cache msrpm12_cache[MSRPM_SIZE / PAGE_SIZE];
+
/*
* The MSR permissions map used for vmcb02, which is the merge result
* of vmcb01 and vmcb12
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 17/20] KVM: nSVM: Cache L1's IO permissions map pages
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (15 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 16/20] KVM: nSVM: Cache L1's MSR permissions map pages David Woodhouse
@ 2026-11-05 2:00 ` David Woodhouse
2026-11-05 2:01 ` [PATCH v2 18/20] KVM: selftests: Add nested transition benchmark David Woodhouse
` (2 subsequent siblings)
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:00 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
The per-intercept IOPM check on every trapped L2 IO instruction reads
L1's IO permissions map with kvm_vcpu_read_guest() — a
memremap/memunmap cycle per access with unmanaged guest memory. Cache
the three IOPM pages in gfn_to_pfn_caches keyed on iopm_base_pa, as
for the MSR permissions map.
Unlike the MSRPM accesses, an IOPM read is not naturally aligned: a
two-byte read for a port range straddling a page boundary (e.g. ports
0x7ff8-0x7fff at bytes 0xfff-0x1000) spans two pages, so read byte by
byte, each from its own page's cache.
Failure semantics are preserved: an unreadable IOPM reflects the
intercept to L1 (NESTED_EXIT_DONE).
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
arch/x86/kvm/svm/nested.c | 71 +++++++++++++++++++++++++++++----------
arch/x86/kvm/svm/svm.c | 2 ++
arch/x86/kvm/svm/svm.h | 7 ++++
3 files changed, 63 insertions(+), 17 deletions(-)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 4e22cd7a38e1..a682b7446c3e 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1519,10 +1519,36 @@ static void nested_svm_triple_fault(struct kvm_vcpu *vcpu)
nested_svm_simple_vmexit(to_svm(vcpu), SVM_EXIT_SHUTDOWN);
}
+/*
+ * Initialize the nested gfn_to_pfn_caches, once per vCPU. Unlike the
+ * allocations in svm_allocate_nested(), the caches persist across
+ * EFER.SVME toggles: svm_free_nested() only deactivates them, and
+ * reuse goes through kvm_gpc_activate() without re-init.
+ */
+void svm_init_nested_caches(struct vcpu_svm *svm)
+{
+ int i;
+
+ kvm_gpc_init(&svm->nested.vmcb12_cache, svm->vcpu.kvm);
+
+ /*
+ * KVM only ever reads the MSR and IO permissions maps, which L1 may
+ * keep in memory that its VMM maps read-only: map the caches
+ * read-only (which also means never marking them dirty), matching
+ * the read semantics of the kvm_vcpu_read_guest() calls they
+ * replace.
+ */
+ for (i = 0; i < ARRAY_SIZE(svm->nested.msrpm12_cache); i++)
+ __kvm_gpc_init(&svm->nested.msrpm12_cache[i], svm->vcpu.kvm,
+ false, true, NULL, 0);
+ for (i = 0; i < ARRAY_SIZE(svm->nested.iopm12_cache); i++)
+ __kvm_gpc_init(&svm->nested.iopm12_cache[i], svm->vcpu.kvm,
+ false, true, NULL, 0);
+}
+
int svm_allocate_nested(struct vcpu_svm *svm)
{
struct page *vmcb02_page;
- int i;
if (svm->nested.initialized)
return 0;
@@ -1539,18 +1565,6 @@ int svm_allocate_nested(struct vcpu_svm *svm)
svm->nested.vmcb02.cpu = -1;
- kvm_gpc_init(&svm->nested.vmcb12_cache, svm->vcpu.kvm);
-
- /*
- * KVM only ever reads the MSR permissions map, which L1 may keep in
- * memory that its VMM maps read-only: map the cache read-only
- * (which also means never marking it dirty), matching the read
- * semantics of the kvm_vcpu_read_guest() calls it replaces.
- */
- for (i = 0; i < ARRAY_SIZE(svm->nested.msrpm12_cache); i++)
- __kvm_gpc_init(&svm->nested.msrpm12_cache[i], svm->vcpu.kvm,
- false, true, NULL, 0);
-
svm->nested.initialized = true;
return 0;
@@ -1575,6 +1589,8 @@ void svm_free_nested(struct vcpu_svm *svm)
kvm_gpc_deactivate(&svm->nested.vmcb12_cache);
for (i = 0; i < ARRAY_SIZE(svm->nested.msrpm12_cache); i++)
kvm_gpc_deactivate(&svm->nested.msrpm12_cache[i]);
+ for (i = 0; i < ARRAY_SIZE(svm->nested.iopm12_cache); i++)
+ kvm_gpc_deactivate(&svm->nested.iopm12_cache[i]);
__free_page(virt_to_page(svm->nested.vmcb02.ptr));
svm->nested.vmcb02.ptr = NULL;
@@ -1654,7 +1670,7 @@ static int nested_svm_intercept_ioio(struct vcpu_svm *svm)
unsigned port, size, iopm_len;
u16 val, mask;
u8 start_bit;
- u64 gpa;
+ int i;
if (!(vmcb12_is_intercept(&svm->nested.ctl, INTERCEPT_IOIO_PROT)))
return NESTED_EXIT_HOST;
@@ -1662,14 +1678,35 @@ static int nested_svm_intercept_ioio(struct vcpu_svm *svm)
port = svm->vmcb->control.exit_info_1 >> 16;
size = (svm->vmcb->control.exit_info_1 & SVM_IOIO_SIZE_MASK) >>
SVM_IOIO_SIZE_SHIFT;
- gpa = svm->nested.ctl.iopm_base_pa + (port / 8);
start_bit = port % 8;
iopm_len = (start_bit + size > 8) ? 2 : 1;
mask = (0xf >> (4 - size)) << start_bit;
val = 0;
- if (kvm_vcpu_read_guest(&svm->vcpu, gpa, &val, iopm_len))
- return NESTED_EXIT_DONE;
+ /*
+ * Read byte by byte: a two-byte read at the end of a page (e.g.
+ * ports 0x7ff8-0x7fff spanning bytes 0xfff-0x1000) crosses into
+ * the next page of the IOPM, i.e. the next cache.
+ */
+ for (i = 0; i < iopm_len; i++) {
+ unsigned long offset = port / 8 + i;
+ struct gfn_to_pfn_cache *gpc;
+ gpa_t gpa;
+ int idx;
+
+ if (WARN_ON_ONCE(offset >= IOPM_SIZE))
+ return NESTED_EXIT_DONE;
+
+ gpc = &svm->nested.iopm12_cache[offset >> PAGE_SHIFT];
+ gpa = svm->nested.ctl.iopm_base_pa + (offset & PAGE_MASK);
+
+ idx = kvm_gpc_lock_page(gpc, gpa);
+ if (idx < 0)
+ return NESTED_EXIT_DONE;
+
+ val |= *(u8 *)(gpc->khva + offset_in_page(offset)) << (i * 8);
+ kvm_gpc_unlock(gpc, idx);
+ }
return (val & mask) ? NESTED_EXIT_DONE : NESTED_EXIT_HOST;
}
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index b78727cca9b1..f601e0650c36 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -1382,6 +1382,8 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
svm->vmcb01.pa = __sme_set(page_to_pfn(vmcb01_page) << PAGE_SHIFT);
svm_switch_vmcb(svm, &svm->vmcb01);
+ svm_init_nested_caches(svm);
+
svm->guest_state_loaded = false;
return 0;
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 8a5dff5a9ee3..256eefb27a83 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -229,6 +229,12 @@ struct svm_nested_state {
*/
struct gfn_to_pfn_cache msrpm12_cache[MSRPM_SIZE / PAGE_SIZE];
+ /*
+ * Cached mappings of the three pages of L1's IO permissions map,
+ * keyed on ctl.iopm_base_pa, for the per-intercept checks.
+ */
+ struct gfn_to_pfn_cache iopm12_cache[IOPM_SIZE / PAGE_SIZE];
+
/*
* The MSR permissions map used for vmcb02, which is the merge result
* of vmcb01 and vmcb12
@@ -904,6 +910,7 @@ int __init nested_svm_init_msrpm_merge_offsets(void);
int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb_gpa, bool from_vmrun);
void svm_leave_nested(struct kvm_vcpu *vcpu);
+void svm_init_nested_caches(struct vcpu_svm *svm);
void svm_free_nested(struct vcpu_svm *svm);
int svm_allocate_nested(struct vcpu_svm *svm);
int nested_svm_vmrun(struct kvm_vcpu *vcpu);
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 18/20] KVM: selftests: Add nested transition benchmark
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (16 preceding siblings ...)
2026-11-05 2:00 ` [PATCH v2 17/20] KVM: nSVM: Cache L1's IO " David Woodhouse
@ 2026-11-05 2:01 ` David Woodhouse
2026-11-05 2:01 ` [PATCH v2 19/20] KVM: selftests: Add Xen evtchn vs memslot-update race test David Woodhouse
2026-11-05 2:01 ` [PATCH v2 20/20] KVM: selftests: Add Xen evtchn vs mmu_notifier race test for HVA caches David Woodhouse
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:01 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
Measure L1<->L2 transition costs from L1's viewpoint, in TSC cycles:
a reflected-exit phase (cpuid in L2, forwarded to L1: the full
emulated vmexit->vmentry round trip, exercising the vmcb12/vmcs12
access paths), and on SVM an MSR-intercept phase (rdmsr in L2 with
INTERCEPT_MSR_PROT: exercises the per-intercept lookup in L1's MSR
permissions map). Reports min/avg/p50/p99/max over 10k iterations.
Works on both VMX and SVM.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Assisted-by: LLM
---
tools/testing/selftests/kvm/Makefile.kvm | 1 +
.../kvm/x86/nested_transition_bench.c | 210 ++++++++++++++++++
2 files changed, 211 insertions(+)
create mode 100644 tools/testing/selftests/kvm/x86/nested_transition_bench.c
diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index dcfe815f4a02..65639e54b554 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -162,6 +162,7 @@ TEST_GEN_PROGS_x86 += x86/recalc_apic_map_test
TEST_GEN_PROGS_x86 += x86/aperfmperf_test
TEST_GEN_PROGS_x86 += x86/guest_memfd_conversions_test
TEST_GEN_PROGS_x86 += x86/vmx_apic_update_test
+TEST_GEN_PROGS_x86 += x86/nested_transition_bench
TEST_GEN_PROGS_x86 += access_tracking_perf_test
TEST_GEN_PROGS_x86 += coalesced_io_test
TEST_GEN_PROGS_x86 += dirty_log_perf_test
diff --git a/tools/testing/selftests/kvm/x86/nested_transition_bench.c b/tools/testing/selftests/kvm/x86/nested_transition_bench.c
new file mode 100644
index 000000000000..ea91ffcc4e0e
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/nested_transition_bench.c
@@ -0,0 +1,210 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * nested_transition_bench
+ *
+ * Measure nested virtualization transition costs:
+ * 1. L2 exit reflected to L1 (cpuid): full emulated vmexit->vmentry
+ * round trip, exercising the vmcb12/vmcs12 access paths.
+ * 2. L2 MSR access with MSR intercepts enabled: exercises the
+ * per-intercept L1 permissions-bitmap lookup on SVM, and the
+ * bitmap merge path when force_msr_bitmap_recalc is set.
+ *
+ * Reports TSC cycles per iteration (min/avg/p99) from L1's viewpoint.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "vmx.h"
+#include "svm_util.h"
+
+/*
+ * Keep the iteration count modest: every timed iteration includes a
+ * ucall round trip to userspace to report its sample, which on a
+ * debug (KASAN) kernel costs milliseconds. 10k samples still give a
+ * stable p99.
+ */
+#define N_WARMUP 1000
+#define N_ITERS 10000
+
+
+static uint64_t results[N_ITERS];
+
+/* L2: exit to L1 in a tight loop. */
+static void l2_guest_code(void)
+{
+ u32 eax, ebx, ecx, edx;
+
+ for (;;)
+ cpuid(0, &eax, &ebx, &ecx, &edx);
+}
+
+static void l2_guest_code_msr(void)
+{
+ for (;;)
+ rdmsr(MSR_IA32_TSC_DEADLINE);
+}
+
+/* SVM L1: run L2, timing VMRUN -> #VMEXIT round trips. */
+static void l1_svm_code(struct svm_test_data *svm)
+{
+ struct vmcb *vmcb = svm->vmcb;
+ uint64_t t0, t1;
+ int i;
+
+ generic_svm_setup(svm, l2_guest_code);
+ /*
+ * Unlike VMX, where CPUID unconditionally exits, SVM only
+ * intercepts CPUID if asked; without it L0 emulates L2's cpuid
+ * and resumes L2 directly, and run_guest() never returns.
+ */
+ vmcb->control.intercept |= BIT_ULL(INTERCEPT_CPUID) |
+ BIT_ULL(INTERCEPT_MSR_PROT);
+
+ /*
+ * Set the read-intercept bit for MSR_IA32_TSC_DEADLINE (0x6e0,
+ * MSRPM range 0: bit 2*msr, i.e. byte 0x1b8 bit 0) in L1's MSRPM,
+ * so that L2's rdmsr in phase 2 reflects to L1. L0's per-exit
+ * lookup in this bitmap is the path being measured.
+ */
+ ((u8 *)svm->msr)[(MSR_IA32_TSC_DEADLINE & 0x1fff) * 2 / 8] |=
+ BIT((MSR_IA32_TSC_DEADLINE & 0x1fff) * 2 % 8);
+
+ for (i = 0; i < N_WARMUP; i++) {
+ run_guest(vmcb, svm->vmcb_gpa);
+ GUEST_ASSERT(vmcb->control.exit_code == SVM_EXIT_CPUID);
+ vmcb->save.rip += 2; /* cpuid */
+ }
+
+ for (i = 0; i < N_ITERS; i++) {
+ t0 = rdtsc();
+ run_guest(vmcb, svm->vmcb_gpa);
+ t1 = rdtsc();
+ GUEST_ASSERT(vmcb->control.exit_code == SVM_EXIT_CPUID);
+ vmcb->save.rip += 2;
+ GUEST_SYNC1(t1 - t0);
+ }
+
+ /* Phase 2: MSR intercept path */
+ vmcb->save.rip = (u64)l2_guest_code_msr;
+ for (i = 0; i < N_WARMUP; i++) {
+ run_guest(vmcb, svm->vmcb_gpa);
+ GUEST_ASSERT(vmcb->control.exit_code == SVM_EXIT_MSR);
+ vmcb->save.rip += 2; /* rdmsr */
+ }
+
+ for (i = 0; i < N_ITERS; i++) {
+ t0 = rdtsc();
+ run_guest(vmcb, svm->vmcb_gpa);
+ t1 = rdtsc();
+ GUEST_ASSERT(vmcb->control.exit_code == SVM_EXIT_MSR);
+ vmcb->save.rip += 2; /* rdmsr */
+ GUEST_SYNC1(t1 - t0);
+ }
+
+ GUEST_DONE();
+}
+
+/* VMX L1: run L2, timing VMRESUME -> vmexit round trips. */
+static void l1_vmx_code(struct vmx_pages *vmx)
+{
+ uint64_t t0, t1;
+ int i;
+
+ prepare_for_vmx_operation(vmx);
+ load_vmcs(vmx);
+ prepare_vmcs(vmx, l2_guest_code);
+
+ vmlaunch();
+ /* first exit is here; loop resumes */
+ for (i = 0; i < N_WARMUP; i++) {
+ GUEST_ASSERT(vmread(VM_EXIT_REASON) == EXIT_REASON_CPUID);
+ vmwrite(GUEST_RIP, vmread(GUEST_RIP) +
+ vmread(VM_EXIT_INSTRUCTION_LEN));
+ vmresume();
+ }
+
+ for (i = 0; i < N_ITERS; i++) {
+ GUEST_ASSERT(vmread(VM_EXIT_REASON) == EXIT_REASON_CPUID);
+ vmwrite(GUEST_RIP, vmread(GUEST_RIP) +
+ vmread(VM_EXIT_INSTRUCTION_LEN));
+ t0 = rdtsc();
+ vmresume();
+ t1 = rdtsc();
+ GUEST_SYNC1(t1 - t0);
+ }
+
+ GUEST_DONE();
+}
+
+static int cmp_u64(const void *a, const void *b)
+{
+ uint64_t x = *(const uint64_t *)a, y = *(const uint64_t *)b;
+
+ return x < y ? -1 : x > y;
+}
+
+static void report(const char *phase, int n)
+{
+ uint64_t sum = 0;
+ int i;
+
+ qsort(results, n, sizeof(results[0]), cmp_u64);
+ for (i = 0; i < n; i++)
+ sum += results[i];
+
+ printf("%-24s n=%d min=%" PRIu64 " avg=%" PRIu64 " p50=%" PRIu64
+ " p99=%" PRIu64 " max=%" PRIu64 " cycles\n",
+ phase, n, results[0], sum / n, results[n / 2],
+ results[(int)(n * 0.99)], results[n - 1]);
+}
+
+int main(int argc, char *argv[])
+{
+ struct kvm_vcpu *vcpu;
+ struct kvm_vm *vm;
+ struct ucall uc;
+ gva_t nested_gva = 0;
+ bool is_svm = kvm_cpu_has(X86_FEATURE_SVM);
+ int n = 0, phase = 0;
+ bool done = false;
+
+ TEST_REQUIRE(is_svm || kvm_cpu_has(X86_FEATURE_VMX));
+
+ vm = vm_create_with_one_vcpu(&vcpu, is_svm ? (void *)l1_svm_code
+ : (void *)l1_vmx_code);
+ if (is_svm)
+ vcpu_alloc_svm(vm, &nested_gva);
+ else
+ vcpu_alloc_vmx(vm, &nested_gva);
+ vcpu_args_set(vcpu, 1, nested_gva);
+
+ while (!done) {
+ vcpu_run(vcpu);
+ switch (get_ucall(vcpu, &uc)) {
+ case UCALL_SYNC:
+ results[n++] = uc.args[0];
+ if (n == N_ITERS) {
+ report(phase == 0 ? "reflected-exit" :
+ "msr-intercept", n);
+ n = 0;
+ phase++;
+ }
+ break;
+ case UCALL_ABORT:
+ REPORT_GUEST_ASSERT(uc);
+ case UCALL_DONE:
+ if (n)
+ report("partial", n);
+ done = true;
+ break;
+ default:
+ TEST_FAIL("Unknown ucall %lu", uc.cmd);
+ }
+ }
+
+ kvm_vm_free(vm);
+ return 0;
+}
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 19/20] KVM: selftests: Add Xen evtchn vs memslot-update race test
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (17 preceding siblings ...)
2026-11-05 2:01 ` [PATCH v2 18/20] KVM: selftests: Add nested transition benchmark David Woodhouse
@ 2026-11-05 2:01 ` David Woodhouse
2026-11-05 2:01 ` [PATCH v2 20/20] KVM: selftests: Add Xen evtchn vs mmu_notifier race test for HVA caches David Woodhouse
19 siblings, 0 replies; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:01 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
Race a guest EVTCHNOP_send loop against a host thread which deletes
and recreates the shared_info memslot. The delivery path runs in the
vCPU's exit-handling context, under vcpu_run's kvm->srcu read
section; if it ever waited there for the memslot update to complete,
it would deadlock against kvm_swap_active_memslots() waiting out that
very read section in synchronize_srcu(), and this test livelocks
within seconds. The deadlock is the one claim the test pins: a send
which races the update legitimately fails (the hypercall bounces to
userspace, which completes it with an error), so the test makes no
assertion about delivery counts beyond "the churn did not starve
delivery entirely".
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
tools/testing/selftests/kvm/Makefile.kvm | 1 +
.../kvm/x86/xen_evtchn_memslot_race_test.c | 232 ++++++++++++++++++
2 files changed, 233 insertions(+)
create mode 100644 tools/testing/selftests/kvm/x86/xen_evtchn_memslot_race_test.c
diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 65639e54b554..c17408f8d61f 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -149,6 +149,7 @@ TEST_GEN_PROGS_x86 += x86/xss_msr_test
TEST_GEN_PROGS_x86 += x86/debug_regs
TEST_GEN_PROGS_x86 += x86/tsc_msrs_test
TEST_GEN_PROGS_x86 += x86/vmx_pmu_caps_test
+TEST_GEN_PROGS_x86 += x86/xen_evtchn_memslot_race_test
TEST_GEN_PROGS_x86 += x86/xen_shinfo_test
TEST_GEN_PROGS_x86 += x86/xen_vmcall_test
TEST_GEN_PROGS_x86 += x86/sev_dbg_test
diff --git a/tools/testing/selftests/kvm/x86/xen_evtchn_memslot_race_test.c b/tools/testing/selftests/kvm/x86/xen_evtchn_memslot_race_test.c
new file mode 100644
index 000000000000..c9448df52a6e
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/xen_evtchn_memslot_race_test.c
@@ -0,0 +1,232 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * xen_evtchn_memslot_race_test
+ *
+ * A guest EVTCHNOP_send whose delivery races a DELETE/re-CREATE of the
+ * memslot holding the shared_info page. The delivery path runs in the
+ * vCPU's exit-handling context, under vcpu_run's kvm->srcu read
+ * section; if it waits there for the memslot update to complete, it
+ * deadlocks against kvm_swap_active_memslots()'s
+ * synchronize_srcu_expedited() waiting out that very read section.
+ * Regression test for that deadlock: KVM must leave the hypercall
+ * uncompleted so the guest re-executes it, rather than wait in exit
+ * context.
+ *
+ * The guest sends events in a tight loop; a host thread deletes and
+ * recreates the shared_info slot in a tight loop. On a broken host
+ * this livelocks within a second or two (the test then dies by
+ * timeout); on a fixed host the guest completes all N
+ * iterations (sends which hit the no-slot gap between DELETE and
+ * re-CREATE legitimately fail; most deliver).
+ */
+#include <pthread.h>
+#include <stdatomic.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+
+#include <stdint.h>
+
+#define SHINFO_REGION_GPA 0xc0000000ULL
+#define SHINFO_REGION_SLOT 10
+
+#define EVTCHN_PORT 15
+#define TEST_ITERATIONS 4000
+
+#define EVTCHNOP_send 4
+#define __HYPERVISOR_event_channel_op 32
+
+#define XEN_HYPERCALL_MSR 0x40000000
+
+#define EVTCHNSTAT_ipi 5
+
+struct evtchn_send {
+ uint32_t port;
+};
+
+static void guest_code(void)
+{
+ struct evtchn_send s = { .port = EVTCHN_PORT };
+ int delivered = 0;
+ int i;
+
+ for (i = 0; i < TEST_ITERATIONS; i++) {
+ /*
+ * The delivery may legitimately fail while the shared_info
+ * slot is mid-churn (between DELETE and re-CREATE there is
+ * no slot at all, which is -EFAULT, not a transient). The
+ * failure mode under test is the vCPU never coming back at
+ * all (deadlock -> timeout), not the hypercall rc.
+ */
+ if (!__xen_hypercall(__HYPERVISOR_event_channel_op,
+ EVTCHNOP_send, &s))
+ delivered++;
+ GUEST_SYNC(i);
+ }
+
+ /*
+ * The churn must not have starved delivery entirely. Every
+ * iteration exits to userspace for the GUEST_SYNC, so the slot
+ * spends real wall time present between churn cycles; empirically
+ * a few hundred of the sends deliver.
+ */
+ GUEST_ASSERT(delivered);
+
+ /*
+ * Report the delivered count. The value is always at least
+ * TEST_ITERATIONS + 1, so the host can distinguish it from the
+ * in-loop syncs, whose values are below TEST_ITERATIONS.
+ */
+ GUEST_SYNC(TEST_ITERATIONS + delivered);
+
+ GUEST_DONE();
+}
+
+static atomic_bool churn_stop;
+
+static void *memslot_churn(void *arg)
+{
+ struct kvm_vm *vm = arg;
+
+ while (!atomic_load(&churn_stop)) {
+ vm_mem_region_delete(vm, SHINFO_REGION_SLOT);
+ vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,
+ SHINFO_REGION_GPA,
+ SHINFO_REGION_SLOT, 2, 0);
+ }
+ return NULL;
+}
+
+int main(int argc, char *argv[])
+{
+ struct kvm_vcpu *vcpu;
+ struct kvm_vm *vm;
+ pthread_t thread;
+ struct ucall uc;
+ int bounced = 0;
+ int done = 0;
+
+ int xen_caps = kvm_check_cap(KVM_CAP_XEN_HVM);
+
+ TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_SHARED_INFO);
+ TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_EVTCHN_SEND);
+
+ vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+
+ vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,
+ SHINFO_REGION_GPA, SHINFO_REGION_SLOT, 2, 0);
+
+ struct kvm_xen_hvm_config hvmc = {
+ .flags = KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL |
+ KVM_XEN_HVM_CONFIG_EVTCHN_SEND,
+ .msr = XEN_HYPERCALL_MSR,
+ };
+ vm_ioctl(vm, KVM_XEN_HVM_CONFIG, &hvmc);
+
+ struct kvm_xen_hvm_attr lm = {
+ .type = KVM_XEN_ATTR_TYPE_LONG_MODE,
+ .u.long_mode = 1,
+ };
+ vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &lm);
+
+ struct kvm_xen_hvm_attr ha = {
+ .type = KVM_XEN_ATTR_TYPE_SHARED_INFO,
+ .u.shared_info.gfn = SHINFO_REGION_GPA / PAGE_SIZE,
+ };
+ vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &ha);
+
+ struct kvm_xen_vcpu_attr vi = {
+ .type = KVM_XEN_VCPU_ATTR_TYPE_VCPU_INFO,
+ .u.gpa = SHINFO_REGION_GPA + 0x40,
+ };
+ vcpu_ioctl(vcpu, KVM_XEN_VCPU_SET_ATTR, &vi);
+
+ /* Loop the port back to this vCPU as an IPI. */
+ struct kvm_xen_hvm_attr evt = {
+ .type = KVM_XEN_ATTR_TYPE_EVTCHN,
+ .u.evtchn.send_port = EVTCHN_PORT,
+ .u.evtchn.type = EVTCHNSTAT_ipi,
+ .u.evtchn.flags = 0,
+ .u.evtchn.deliver.port.port = EVTCHN_PORT,
+ .u.evtchn.deliver.port.vcpu = vcpu->id,
+ .u.evtchn.deliver.port.priority = KVM_IRQ_ROUTING_XEN_EVTCHN_PRIO_2LEVEL,
+ };
+ vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &evt);
+
+ kvm_pthread_create(&thread, NULL, memslot_churn, vm);
+
+ /*
+ * Self-guard: on a kernel with the deadlock under test, the vCPU
+ * never returns from KVM_RUN. Die with SIGALRM rather than hang
+ * when run standalone. Deliberately no SIGALRM handler: in the
+ * deadlock, no thread ever returns to userspace to run one, and a
+ * handled signal is no longer fatal, so only the default
+ * disposition can kill the process (exit by SIGALRM is the
+ * failure signature). One budget for the whole run — a healthy
+ * kernel finishes in seconds, and a per-iteration bound would
+ * impose a KVM_RUN latency claim this test doesn't make.
+ * (On a CONFIG_DEBUG_LOCK_ALLOC kernel which has regressed, the
+ * wait degrades to a transient failure instead of deadlocking,
+ * and this test cannot see that; the WARN it fires can.)
+ */
+ alarm(90);
+
+ while (!done) {
+ vcpu_run(vcpu);
+
+ /*
+ * In the window where the shared_info slot is fully deleted
+ * (between the churn thread's DELETE and re-CREATE), the
+ * delivery fails with -EFAULT and the hypercall bounces to
+ * userspace as KVM_EXIT_XEN. Complete it with an error
+ * result and keep going; the failure mode under test is
+ * the vCPU never coming back (deadlock -> timeout), not
+ * the hypercall rc.
+ */
+ if (vcpu->run->exit_reason == KVM_EXIT_XEN &&
+ vcpu->run->xen.type == KVM_EXIT_XEN_HCALL) {
+ TEST_ASSERT(vcpu->run->xen.u.hcall.input ==
+ __HYPERVISOR_event_channel_op &&
+ vcpu->run->xen.u.hcall.params[0] ==
+ EVTCHNOP_send,
+ "Unexpected hypercall bounce: input %llu params[0] %llu",
+ vcpu->run->xen.u.hcall.input,
+ vcpu->run->xen.u.hcall.params[0]);
+ vcpu->run->xen.u.hcall.result = -EFAULT;
+ bounced++;
+ continue;
+ }
+
+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+
+ switch (get_ucall(vcpu, &uc)) {
+ case UCALL_SYNC:
+ /*
+ * The final sync reports how many of the sends were
+ * delivered: purely a diagnostic for how much of the
+ * churn the guest actually raced with.
+ */
+ if (uc.args[1] >= TEST_ITERATIONS)
+ pr_info("%lu of %u sends delivered, %d bounced\n",
+ uc.args[1] - TEST_ITERATIONS,
+ TEST_ITERATIONS, bounced);
+ break;
+ case UCALL_DONE:
+ done = 1;
+ break;
+ case UCALL_ABORT:
+ REPORT_GUEST_ASSERT(uc);
+ /* NOT REACHED */
+ default:
+ TEST_FAIL("Unexpected ucall %lu", uc.cmd);
+ }
+ }
+
+ atomic_store(&churn_stop, true);
+ kvm_pthread_join(thread, NULL);
+ alarm(0);
+
+ kvm_vm_free(vm);
+ return 0;
+}
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH v2 20/20] KVM: selftests: Add Xen evtchn vs mmu_notifier race test for HVA caches
2026-11-05 2:00 [PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting David Woodhouse
` (18 preceding siblings ...)
2026-11-05 2:01 ` [PATCH v2 19/20] KVM: selftests: Add Xen evtchn vs memslot-update race test David Woodhouse
@ 2026-11-05 2:01 ` David Woodhouse
2026-11-05 2:28 ` sashiko-bot
19 siblings, 1 reply; 25+ messages in thread
From: David Woodhouse @ 2026-11-05 2:01 UTC (permalink / raw)
To: kvm
Cc: linux-kernel, linux-kselftest, rcu, linux-rt-devel, Paolo Bonzini,
Sean Christopherson, Paul Durrant, Vitaly Kuznetsov,
Fred Griffoul, Paul E . McKenney, Kunwu Chan, Kunwu Chan,
syzbot+208f7f3e5f59c11aeb90, David Woodhouse
From: David Woodhouse <dwmw@amazon.co.uk>
Race a guest EVTCHNOP_send loop against a host thread hammering
MADV_DONTNEED on the userspace page backing HVA-based shared_info and
vcpu_info caches. Every madvise zaps the PTE and invalidates both
gfn_to_pfn_caches; every delivery must refresh through the window
(hva_to_pfn_retry() retries until the invalidation has finished, then
faults the page back in) and must succeed. The caches are HVA-based,
so no memslot semantics excuse a drop: unlike
xen_evtchn_memslot_race_test, which only pins the no-deadlock claim,
this test asserts every one of the 20000 sends delivers, and any
hypercall bounce to userspace fails the test.
This exercises the pfncache invalidation protocol itself — the
invalidate_start walk, the reader draining for a cache mid-teardown,
and the handshake between a refresh publishing a new mapping and a
concurrent invalidation vetoing it — which is where a lifetime bug
in the cache would live; on a debug kernel one is expected to
surface as a splat rather than a dropped event.
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
tools/testing/selftests/kvm/Makefile.kvm | 1 +
.../kvm/x86/xen_evtchn_hva_notify_race_test.c | 188 ++++++++++++++++++
2 files changed, 189 insertions(+)
create mode 100644 tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_test.c
diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index c17408f8d61f..19dc0744f25f 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -150,6 +150,7 @@ TEST_GEN_PROGS_x86 += x86/debug_regs
TEST_GEN_PROGS_x86 += x86/tsc_msrs_test
TEST_GEN_PROGS_x86 += x86/vmx_pmu_caps_test
TEST_GEN_PROGS_x86 += x86/xen_evtchn_memslot_race_test
+TEST_GEN_PROGS_x86 += x86/xen_evtchn_hva_notify_race_test
TEST_GEN_PROGS_x86 += x86/xen_shinfo_test
TEST_GEN_PROGS_x86 += x86/xen_vmcall_test
TEST_GEN_PROGS_x86 += x86/sev_dbg_test
diff --git a/tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_test.c b/tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_test.c
new file mode 100644
index 000000000000..c105435f6357
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_test.c
@@ -0,0 +1,188 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * xen_evtchn_hva_notify_race_test
+ *
+ * A guest EVTCHNOP_send loop racing continuous mmu_notifier
+ * invalidation of the HVA-based shared_info and vcpu_info caches: a
+ * host thread hammers MADV_DONTNEED on the userspace page backing
+ * both, zapping the PTE and invalidating the gfn_to_pfn_caches on
+ * every pass. Delivery must refresh through each invalidation window
+ * (hva_to_pfn_retry() retries until the notifier has finished, then
+ * faults the page back in) and must never fail: the caches are
+ * HVA-based, so no memslot semantics excuse a drop.
+ *
+ * This is the stress for the pfncache invalidation protocol itself —
+ * the invalidate_start walk, the GPC_INVALIDATING grace-period
+ * draining, and the announce/veto handshake between a refresh's
+ * publish and a concurrent invalidation. Unlike the sibling
+ * xen_evtchn_memslot_race_test, every send is asserted to succeed;
+ * any -EFAULT bounce to userspace, or any guest-visible failure, is
+ * a test failure, and a lifetime bug is expected to show up as a
+ * crash or splat on a debug kernel.
+ */
+#include <pthread.h>
+#include <stdatomic.h>
+#include <sys/mman.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+
+#include <stdint.h>
+
+#define EVTCHN_PORT 15
+#define TEST_ITERATIONS 5000
+
+#define EVTCHNOP_send 4
+#define __HYPERVISOR_event_channel_op 32
+
+#define XEN_HYPERCALL_MSR 0x40000000
+
+#define EVTCHNSTAT_ipi 5
+
+struct evtchn_send {
+ uint32_t port;
+};
+
+static void guest_code(void)
+{
+ struct evtchn_send s = { .port = EVTCHN_PORT };
+ uint64_t rc;
+ int i;
+
+ for (i = 0; i < TEST_ITERATIONS; i++) {
+ rc = __xen_hypercall(__HYPERVISOR_event_channel_op,
+ EVTCHNOP_send, &s);
+ __GUEST_ASSERT(!rc,
+ "EVTCHNOP_send failed with %ld on iteration %d",
+ (long)rc, i);
+ GUEST_SYNC(i);
+ }
+
+ GUEST_DONE();
+}
+
+static atomic_bool churn_stop;
+
+static void *backing_churn(void *arg)
+{
+ void *backing = arg;
+
+ while (!atomic_load(&churn_stop))
+ madvise(backing, PAGE_SIZE, MADV_DONTNEED);
+
+ return NULL;
+}
+
+int main(int argc, char *argv[])
+{
+ struct kvm_vcpu *vcpu;
+ struct kvm_vm *vm;
+ pthread_t thread;
+ struct ucall uc;
+ void *backing;
+ int done = 0;
+
+ int xen_caps = kvm_check_cap(KVM_CAP_XEN_HVM);
+
+ TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA);
+ TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_EVTCHN_SEND);
+
+ vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+
+ /*
+ * The shared_info page and the vcpu_info within it live in plain
+ * anonymous VMM memory, not in any memslot: nothing but the
+ * mmu_notifier churn below can perturb the caches.
+ */
+ backing = mmap(NULL, PAGE_SIZE, PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ TEST_ASSERT(backing != MAP_FAILED, "mmap of backing page failed");
+
+ struct kvm_xen_hvm_config hvmc = {
+ .flags = KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL |
+ KVM_XEN_HVM_CONFIG_EVTCHN_SEND,
+ .msr = XEN_HYPERCALL_MSR,
+ };
+ vm_ioctl(vm, KVM_XEN_HVM_CONFIG, &hvmc);
+
+ struct kvm_xen_hvm_attr lm = {
+ .type = KVM_XEN_ATTR_TYPE_LONG_MODE,
+ .u.long_mode = 1,
+ };
+ vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &lm);
+
+ struct kvm_xen_hvm_attr ha = {
+ .type = KVM_XEN_ATTR_TYPE_SHARED_INFO_HVA,
+ .u.shared_info.hva = (unsigned long)backing,
+ };
+ vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &ha);
+
+ struct kvm_xen_vcpu_attr vi = {
+ .type = KVM_XEN_VCPU_ATTR_TYPE_VCPU_INFO_HVA,
+ .u.hva = (unsigned long)backing + 0x40,
+ };
+ vcpu_ioctl(vcpu, KVM_XEN_VCPU_SET_ATTR, &vi);
+
+ /* Loop the port back to this vCPU as an IPI. */
+ struct kvm_xen_hvm_attr evt = {
+ .type = KVM_XEN_ATTR_TYPE_EVTCHN,
+ .u.evtchn.send_port = EVTCHN_PORT,
+ .u.evtchn.type = EVTCHNSTAT_ipi,
+ .u.evtchn.flags = 0,
+ .u.evtchn.deliver.port.port = EVTCHN_PORT,
+ .u.evtchn.deliver.port.vcpu = vcpu->id,
+ .u.evtchn.deliver.port.priority = KVM_IRQ_ROUTING_XEN_EVTCHN_PRIO_2LEVEL,
+ };
+ vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &evt);
+
+ kvm_pthread_create(&thread, NULL, backing_churn, backing);
+
+ /*
+ * Self-guard against a kernel which livelocks a refresh under
+ * continuous invalidation. Deliberately no SIGALRM handler, as in
+ * xen_evtchn_memslot_race_test: only the default disposition can
+ * kill a process whose vCPU thread never returns to userspace.
+ * Empirically ~100-170 iterations/sec on a c6a.metal host: nearly
+ * every send pays a full refresh (fault the page back in and remap
+ * it), so budget generously; the healthy runtime is ~30-50s.
+ */
+ alarm(240);
+
+ while (!done) {
+ vcpu_run(vcpu);
+
+ /*
+ * An HVA-based cache has no memslot to be mid-update:
+ * there is no legitimate reason for delivery to fail, so
+ * unlike the memslot race test, a hypercall bouncing to
+ * userspace is a straight failure.
+ */
+ TEST_ASSERT(vcpu->run->exit_reason != KVM_EXIT_XEN,
+ "EVTCHNOP_send bounced to userspace: delivery failed");
+
+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+
+ switch (get_ucall(vcpu, &uc)) {
+ case UCALL_SYNC:
+ if (uc.args[1] && !(uc.args[1] % 1000))
+ pr_info("progress: %lu iterations\n", uc.args[1]);
+ break;
+ case UCALL_DONE:
+ done = 1;
+ break;
+ case UCALL_ABORT:
+ REPORT_GUEST_ASSERT(uc);
+ /* NOT REACHED */
+ default:
+ TEST_FAIL("Unexpected ucall %lu", uc.cmd);
+ }
+ }
+
+ atomic_store(&churn_stop, true);
+ kvm_pthread_join(thread, NULL);
+ alarm(0);
+
+ kvm_vm_free(vm);
+ return 0;
+}
--
2.43.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* Re: [PATCH v2 20/20] KVM: selftests: Add Xen evtchn vs mmu_notifier race test for HVA caches
2026-11-05 2:01 ` [PATCH v2 20/20] KVM: selftests: Add Xen evtchn vs mmu_notifier race test for HVA caches David Woodhouse
@ 2026-11-05 2:28 ` sashiko-bot
0 siblings, 0 replies; 25+ messages in thread
From: sashiko-bot @ 2026-11-05 2:28 UTC (permalink / raw)
To: David Woodhouse, kvm; +Cc: linux-rt-devel, linux-kernel
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Low] KVM: selftests: commit message claims 20000 sends but TEST_ITERATIONS is 5000
- [Low] KVM: selftests: xen_evtchn_hva_notify_race_test duplicates memslot race test
- [Low] KVM: selftests: unchecked madvise() and raw mmap() in HVA race test
--
commit cf7aec0532723ffed105931f7f5b74967ac26723
Author: David Woodhouse <dwmw@amazon.co.uk>
KVM: selftests: Add Xen evtchn vs mmu_notifier race test for HVA caches
This commit adds a KVM selftest racing a guest EVTCHNOP_send loop against a
host thread invoking MADV_DONTNEED on the userspace page backing the
HVA-based shared_info and vcpu_info caches.
> KVM: selftests: Add Xen evtchn vs mmu_notifier race test for HVA caches
>
> Race a guest EVTCHNOP_send loop against a host thread hammering
> MADV_DONTNEED on the userspace page backing HVA-based shared_info and
> vcpu_info caches. Every madvise zaps the PTE and invalidates both
> gfn_to_pfn_caches; every delivery must refresh through the window
> (hva_to_pfn_retry() retries until the invalidation has finished, then
> faults the page back in) and must succeed. The caches are HVA-based,
> so no memslot semantics excuse a drop: unlike
> xen_evtchn_memslot_race_test, which only pins the no-deadlock claim,
> this test asserts every one of the 20000 sends delivers, and any
> hypercall bounce to userspace fails the test.
>
> This exercises the pfncache invalidation protocol itself — the
> invalidate_start walk, the reader draining for a cache mid-teardown,
> and the handshake between a refresh publishing a new mapping and a
> concurrent invalidation vetoing it — which is where a lifetime bug
> in the cache would live; on a debug kernel one is expected to
> surface as a splat rather than a dropped event.
>
> Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
[Severity: Low]
The commit message states that this test asserts every one of the 20000 sends
delivers, but in tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_test.c
TEST_ITERATIONS is defined as 5000 and guest_code() runs TEST_ITERATIONS
hypercalls:
#define TEST_ITERATIONS 5000
...
for (i = 0; i < TEST_ITERATIONS; i++) {
rc = __xen_hypercall(__HYPERVISOR_event_channel_op,
EVTCHNOP_send, &s);
Should the commit message say 5000 sends to match TEST_ITERATIONS and the
runtime estimate comment in main()?
> diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
> index c17408f8d61f5..19dc0744f25f8 100644
> --- a/tools/testing/selftests/kvm/Makefile.kvm
> +++ b/tools/testing/selftests/kvm/Makefile.kvm
> @@ -150,6 +150,7 @@ TEST_GEN_PROGS_x86 += x86/debug_regs
> TEST_GEN_PROGS_x86 += x86/tsc_msrs_test
> TEST_GEN_PROGS_x86 += x86/vmx_pmu_caps_test
> TEST_GEN_PROGS_x86 += x86/xen_evtchn_memslot_race_test
> +TEST_GEN_PROGS_x86 += x86/xen_evtchn_hva_notify_race_test
[Severity: Low]
This isn't a bug, but tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_test.c
duplicates most of tools/testing/selftests/kvm/x86/xen_evtchn_memslot_race_test.c,
including the hypercall definitions, struct evtchn_send, VM and vCPU creation,
KVM_XEN_HVM_CONFIG setup, KVM_XEN_ATTR_TYPE_LONG_MODE setup,
KVM_XEN_ATTR_TYPE_EVTCHN IPI loopback routing, worker thread lifecycle,
alarm() watchdog setup, and the vcpu_run() / get_ucall() loop in main().
Since only the backing memory setup (HVA vs GPA/memslot), the churn thread
body (madvise MADV_DONTNEED vs memslot update), and whether KVM_EXIT_XEN
hypercall bounces are tolerated differ between the two tests, could this HVA
mmu_notifier variant be folded into xen_evtchn_memslot_race_test.c as a test
mode or parameter rather than adding a separate standalone binary?
> diff --git a/tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_test.c b/tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_test.c
> new file mode 100644
> index 0000000000000..c105435f63577
> --- /dev/null
> +++ b/tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_test.c
[ ... ]
> +static void *backing_churn(void *arg)
> +{
> + void *backing = arg;
> +
> + while (!atomic_load(&churn_stop))
> + madvise(backing, PAGE_SIZE, MADV_DONTNEED);
> +
> + return NULL;
> +}
> +
> +int main(int argc, char *argv[])
> +{
> + struct kvm_vcpu *vcpu;
> + struct kvm_vm *vm;
> + pthread_t thread;
> + struct ucall uc;
> + void *backing;
> + int done = 0;
> +
> + int xen_caps = kvm_check_cap(KVM_CAP_XEN_HVM);
> +
> + TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA);
> + TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_EVTCHN_SEND);
> +
> + vm = vm_create_with_one_vcpu(&vcpu, guest_code);
> +
> + /*
> + * The shared_info page and the vcpu_info within it live in plain
> + * anonymous VMM memory, not in any memslot: nothing but the
> + * mmu_notifier churn below can perturb the caches.
> + */
> + backing = mmap(NULL, PAGE_SIZE, PROT_READ | PROT_WRITE,
> + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> + TEST_ASSERT(backing != MAP_FAILED, "mmap of backing page failed");
[ ... ]
> + atomic_store(&churn_stop, true);
> + kvm_pthread_join(thread, NULL);
> + alarm(0);
> +
> + kvm_vm_free(vm);
> + return 0;
> +}
[Severity: Low]
In backing_churn(), madvise(backing, PAGE_SIZE, MADV_DONTNEED) does not check
for errors; if madvise() fails, the churn loop would spin without zapping PTEs
or triggering mmu_notifier invalidations, allowing the test to pass without
exercising the race.
In addition, main() open-codes mmap() + TEST_ASSERT() and exits after
kvm_vm_free(vm) without unmapping backing.
Should backing_churn() and main() use the kvm_madvise(), kvm_mmap(), and
kvm_munmap() helpers from kvm_syscalls.h instead?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261105020104.2507083-1-dwmw2@infradead.org?part=20
^ permalink raw reply [flat|nested] 25+ messages in thread