All of lore.kernel.org
 help / color / mirror / Atom feed
From: Dan Carpenter <dan@pereq.eu>
To: Marc Zyngier <maz@kernel.org>
Cc: Oliver Upton <oupton@kernel.org>,
	kvmarm@lists.linux.dev, sashiko-reviews@lists.linux.dev
Subject: Re: [PATCH v2 1/2] KVM: arm64: Load the vcpu in KVM_SET_VCPU_EVENTS
Date: Thu, 19 Nov 2026 11:18:18 +0300	[thread overview]
Message-ID: <av6wypzwAJxIyKOU@stanley.mountain> (raw)
In-Reply-To: <86zev6yqzd.wl-maz@kernel.org>

[-- Attachment #1: Type: text/plain, Size: 4648 bytes --]

On Wed, Nov 18, 2026 at 05:44:06PM +0000, Marc Zyngier wrote:
> On Wed, 18 Nov 2026 13:45:34 +0000,
> sashiko-bot@kernel.org wrote:
> > 
> > Thank you for your contribution! Sashiko AI review found 1 potential
> > issue to consider:
> > - [Medium] KVM: arm64: stale mmio_needed after kvm_inject_nested() in
> >   KVM_SET_VCPU_EVENTS
> > --
> > 
> > commit 178d0de7451612b8ddf14a943cb82094efbd77d1
> > Author: Dan Carpenter <dan@pereq.eu>
> > 
> > KVM: arm64: Load the vcpu in KVM_SET_VCPU_EVENTS
> > 
> > This commit wraps __kvm_arm_vcpu_set_events() in vcpu_load() and vcpu_put()
> > so that kvm_inject_nested() operates on a loaded vCPU during
> > KVM_SET_VCPU_EVENTS.
> > 
> > > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> > > index 84a52e27ed859..a148e80d0aef7 100644
> > > --- a/arch/arm64/kvm/arm.c
> > > +++ b/arch/arm64/kvm/arm.c
> > > @@ -2002,7 +2002,7 @@ static int kvm_arm_vcpu_get_events(struct kvm_vcpu *vcpu,
> > >  static int kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,
> > >  				   struct kvm_vcpu_events *events)
> > >  {
> > > -	int i;
> > > +	int i, ret;
> > >  
> > >  	/* check whether the reserved field is zero */
> > >  	for (i = 0; i < ARRAY_SIZE(events->reserved); i++)
> > > @@ -2014,7 +2014,11 @@ static int kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,
> > >  		if (events->exception.pad[i])
> > >  			return -EINVAL;
> > >  
> > > -	return __kvm_arm_vcpu_set_events(vcpu, events);
> > > +	vcpu_load(vcpu);
> > > +	ret = __kvm_arm_vcpu_set_events(vcpu, events);
> > > +	vcpu_put(vcpu);
> > > +
> > > +	return ret;
> > >  }
> > 
> > [Severity: Medium]
> > Does vcpu->mmio_needed remain set when KVM_SET_VCPU_EVENTS injects an
> > exception from L2 that routes to L1 EL2?
> > 
> > When an L2 guest exits with KVM_EXIT_MMIO and userspace aborts the access via
> > KVM_SET_VCPU_EVENTS (for example with ext_dabt_pending = 1 while HCR_EL2.TEA
> > is set), kvm_inject_sea_dabt() -> kvm_inject_nested() commits the EL2
> > exception entry directly via __kvm_adjust_pc(), which clears
> > PENDING_EXCEPTION.  When __kvm_arm_vcpu_set_events() then calls
> > commit_pending_events(), PENDING_EXCEPTION is already clear, so it returns
> > early without clearing vcpu->mmio_needed:
> > 
> > arch/arm64/kvm/guest.c:commit_pending_events() {
> > 	if (!vcpu_get_flag(vcpu, PENDING_EXCEPTION))
> > 		return;
> > 
> > 	vcpu->mmio_needed = false;
> > 	kvm_call_hyp(__kvm_adjust_pc, vcpu);
> > }
> > 
> > On the next KVM_RUN, kvm_handle_mmio_return() sees vcpu->mmio_needed still
> > set, writes the MMIO read value into L1 EL2's destination register and
> > advances L1 EL2's exception vector PC by 4 bytes via kvm_incr_pc().
> 
> I think this is a separate issue, which doesn't intersect with the
> current patch.

Yep.

> The problem is that we have multiple ways to "commit'
> the state on exception delivery, and that we miss MMIO handling in the
> EL1->EL2 path.
> 
> Oliver, WDYT?
> 

I have a reproducer for the bug Sashiko found and this patch doesn't
work.  Attached.  Claude wanted to just set vcpu->mmio_needed = false;
but I told it to call commit_pending_events() instead.

I can send that.

regards,
dan carpenter

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 0614c3b3a40dfd..c75b0c62382444 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -1354,6 +1354,7 @@ int __kvm_arm_vcpu_get_events(struct kvm_vcpu *vcpu,
 
 int __kvm_arm_vcpu_set_events(struct kvm_vcpu *vcpu,
 			      struct kvm_vcpu_events *events);
+void commit_pending_events(struct kvm_vcpu *vcpu);
 
 void kvm_arm_halt_guest(struct kvm *kvm);
 void kvm_arm_resume_guest(struct kvm *kvm);
diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index 39136a89d74bb6..5a7019bb719b2a 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -2921,7 +2921,7 @@ static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2,
 	 * Make sure the exception actually took place before we load
 	 * the new context.
 	 */
-	__kvm_adjust_pc(vcpu);
+	commit_pending_events(vcpu);
 
 	kvm_arch_vcpu_load(vcpu, smp_processor_id());
 	vcpu_clear_flag(vcpu, IN_NESTED_EXCEPTION);
diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c
index ba4570c52a9162..b7d36c2abfdf9b 100644
--- a/arch/arm64/kvm/guest.c
+++ b/arch/arm64/kvm/guest.c
@@ -770,7 +770,7 @@ int __kvm_arm_vcpu_get_events(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
-static void commit_pending_events(struct kvm_vcpu *vcpu)
+void commit_pending_events(struct kvm_vcpu *vcpu)
 {
 	if (!vcpu_get_flag(vcpu, PENDING_EXCEPTION))
 		return;
-- 
2.53.0

[-- Attachment #2: build.sh --]
[-- Type: application/x-sh, Size: 383 bytes --]

[-- Attachment #3: stale_mmio.c --]
[-- Type: text/x-csrc, Size: 9414 bytes --]

/*
 * stale_mmio: after an L2 MMIO exit, the KVM_SET_VCPU_EVENTS ioctl with
 * serror_pending routes the SError to L1 (L1's HCR_EL2.AMO is set), and
 * kvm_inject_nested() performs the exception entry straight away.  That
 * must also drop the pending MMIO emulation: otherwise the next KVM_RUN
 * completes the L2 access in L1's context, skips the first instruction of
 * L1's SError vector and, for a read, writes the MMIO data to an L1
 * register.
 *
 * The vcpu is put in L2 context with KVM_SET_ONE_REG (HCR_EL2 = E2H | RW |
 * AMO, PSTATE = EL1h) and run once: L2 does one access to an address with
 * no memslot, which exits to userspace as MMIO.  Then the SError is set
 * pending, the vcpu must be at L1's SError vector (VBAR_EL2 + 0x580), and
 * it is run again.  L1's vector is
 *
 *	+0x580	str	x0, [x3]	// MMIO at L1_MMIO: the vector ran from its start
 *	+0x584	str	x0, [x4]	// MMIO at SKIP_MMIO: its first instruction was skipped
 *
 * Usage: stale_mmio write|read
 *   write: L2 does str x0, [x2].
 *   read:  L2 does ldr x5, [x2]; userspace answers with MMIO_DATA, and x5
 *          must keep its value, as the load never completed.
 *
 * Exit 0 when L1's vector runs from its start (and x5 is untouched), 1
 * when the stale MMIO emulation shows, 2 on a set-up error or another
 * result.  Kernels without "KVM: arm64: Load the vcpu in
 * KVM_SET_VCPU_EVENTS" oops in the ioctl instead (see
 * nv-set-vcpu-events-put-load-unloaded).
 */
#include <errno.h>
#include <fcntl.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <unistd.h>
#include <linux/kvm.h>

#ifndef KVM_ARM_VCPU_HAS_EL2
#define KVM_ARM_VCPU_HAS_EL2	7
#endif
#ifndef KVM_CAP_ARM_EL2
#define KVM_CAP_ARM_EL2		240
#endif

#define MEM_BASE	0x40000000UL		/* the memslot: 64K of guest memory */
#define MEM_SIZE	0x10000UL
#define CODE		(MEM_BASE + 0x0)	/* L2 code */
#define VECTORS		(MEM_BASE + 0x800)	/* L1's EL2 vectors, 2K aligned */
#define MMIO		0x10000000UL		/* no memslot here */
#define L1_MMIO		0x10001000UL
#define SKIP_MMIO	0x10002000UL
#define MMIO_DATA	0x1234567876543210ULL
#define X5_INIT		0x5555555555555555ULL

#define GICD_BASE	0x08000000UL
#define GICR_BASE	0x080a0000UL

#define HCR_AMO		(1ULL << 5)
#define HCR_RW		(1ULL << 31)
#define HCR_E2H		(1ULL << 34)

/* PSR_MODE_MASK, PSR_MODE_EL1h and PSR_MODE_EL2h come from asm/ptrace.h. */
#define PSR_EL1h_DAIF	0x3c5ULL		/* EL1h, D, A, I and F masked */

#define SERROR_LOWER_A64	0x580		/* lower EL using AArch64, SError */

#define CORE_REG(name)	(KVM_REG_ARM64 | KVM_REG_SIZE_U64 | KVM_REG_ARM_CORE | \
			 KVM_REG_ARM_CORE_REG(name))
#define REG_HCR_EL2	ARM64_SYS_REG(3, 4, 1, 1, 0)
#define REG_VBAR_EL2	ARM64_SYS_REG(3, 4, 12, 0, 0)
#define REG_ESR_EL2	ARM64_SYS_REG(3, 4, 5, 2, 0)

static int fail(const char *what)
{
	fprintf(stderr, "%s: %s\n", what, strerror(errno));
	return 2;
}

static int set_reg(int vcpu, uint64_t id, uint64_t val)
{
	struct kvm_one_reg reg = { .id = id, .addr = (uint64_t)(uintptr_t)&val };

	return ioctl(vcpu, KVM_SET_ONE_REG, &reg);
}

static int get_reg(int vcpu, uint64_t id, uint64_t *val)
{
	struct kvm_one_reg reg = { .id = id, .addr = (uint64_t)(uintptr_t)val };

	return ioctl(vcpu, KVM_GET_ONE_REG, &reg);
}

static int dev_attr(int fd, uint32_t group, uint64_t attr, void *val)
{
	struct kvm_device_attr a = {
		.group = group, .attr = attr, .addr = (uint64_t)(uintptr_t)val,
	};

	return ioctl(fd, KVM_SET_DEVICE_ATTR, &a);
}

int main(int argc, char **argv)
{
	struct kvm_userspace_memory_region region;
	struct kvm_create_device gic = { .type = KVM_DEV_TYPE_ARM_VGIC_V3 };
	struct kvm_vcpu_events events;
	struct kvm_vcpu_init init;
	struct kvm_run *run;
	uint64_t gicd = GICD_BASE, gicr = GICR_BASE, pc, pstate, esr, x5;
	int kvm, vm, vcpu, run_size, i, read;
	uint32_t *code;
	uint8_t *mem;

	if (argc != 2 || (strcmp(argv[1], "write") && strcmp(argv[1], "read"))) {
		fprintf(stderr, "usage: %s write|read\n", argv[0]);
		return 2;
	}
	read = !strcmp(argv[1], "read");

	kvm = open("/dev/kvm", O_RDWR);
	if (kvm < 0)
		return fail("/dev/kvm");
	if (ioctl(kvm, KVM_CHECK_EXTENSION, KVM_CAP_ARM_EL2) <= 0) {
		fprintf(stderr, "KVM_CAP_ARM_EL2 missing: boot with kvm-arm.mode=nested\n");
		return 2;
	}
	vm = ioctl(kvm, KVM_CREATE_VM, 0);
	if (vm < 0)
		return fail("KVM_CREATE_VM");

	mem = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
	if (mem == MAP_FAILED)
		return fail("mmap");
	region = (struct kvm_userspace_memory_region) {
		.slot = 0, .guest_phys_addr = MEM_BASE, .memory_size = MEM_SIZE,
		.userspace_addr = (uint64_t)(uintptr_t)mem,
	};
	if (ioctl(vm, KVM_SET_USER_MEMORY_REGION, &region))
		return fail("KVM_SET_USER_MEMORY_REGION");

	/* L2: one access that exits as MMIO.  L1's vectors: spin, except SError. */
	code = (uint32_t *)(mem + (CODE - MEM_BASE));
	code[0] = read ? 0xf9400045 :	/* ldr x5, [x2] */
			 0xf9000040;	/* str x0, [x2] */
	code[1] = 0x14000000;		/* b . */
	for (i = 0; i < 0x800 / 4; i++)
		((uint32_t *)(mem + (VECTORS - MEM_BASE)))[i] = 0x14000000;
	code = (uint32_t *)(mem + (VECTORS + SERROR_LOWER_A64 - MEM_BASE));
	code[0] = 0xf9000060;		/* str x0, [x3] */
	code[1] = 0xf9000080;		/* str x0, [x4] */

	if (ioctl(vm, KVM_CREATE_DEVICE, &gic))
		return fail("KVM_CREATE_DEVICE (vGICv3)");
	if (dev_attr(gic.fd, KVM_DEV_ARM_VGIC_GRP_ADDR, KVM_VGIC_V3_ADDR_TYPE_DIST, &gicd) ||
	    dev_attr(gic.fd, KVM_DEV_ARM_VGIC_GRP_ADDR, KVM_VGIC_V3_ADDR_TYPE_REDIST, &gicr))
		return fail("setting the vGIC addresses");

	vcpu = ioctl(vm, KVM_CREATE_VCPU, 0);
	if (vcpu < 0)
		return fail("KVM_CREATE_VCPU");
	if (ioctl(vm, KVM_ARM_PREFERRED_TARGET, &init))
		return fail("KVM_ARM_PREFERRED_TARGET");
	init.features[0] |= 1U << KVM_ARM_VCPU_HAS_EL2;
	if (ioctl(vcpu, KVM_ARM_VCPU_INIT, &init))
		return fail("KVM_ARM_VCPU_INIT with KVM_ARM_VCPU_HAS_EL2");
	if (dev_attr(gic.fd, KVM_DEV_ARM_VGIC_GRP_CTRL, KVM_DEV_ARM_VGIC_CTRL_INIT, NULL))
		return fail("initialising the vGIC");

	/* L2 context, with L1 asking for SErrors (AMO).  No stage 2 (VM=0). */
	if (set_reg(vcpu, REG_HCR_EL2, HCR_E2H | HCR_RW | HCR_AMO) ||
	    set_reg(vcpu, REG_VBAR_EL2, VECTORS) ||
	    set_reg(vcpu, CORE_REG(regs.regs[2]), MMIO) ||
	    set_reg(vcpu, CORE_REG(regs.regs[3]), L1_MMIO) ||
	    set_reg(vcpu, CORE_REG(regs.regs[4]), SKIP_MMIO) ||
	    set_reg(vcpu, CORE_REG(regs.regs[5]), X5_INIT) ||
	    set_reg(vcpu, CORE_REG(regs.pc), CODE) ||
	    set_reg(vcpu, CORE_REG(regs.pstate), PSR_EL1h_DAIF))
		return fail("KVM_SET_ONE_REG");

	run_size = ioctl(kvm, KVM_GET_VCPU_MMAP_SIZE, 0);
	run = mmap(NULL, run_size, PROT_READ | PROT_WRITE, MAP_SHARED, vcpu, 0);
	if (run == MAP_FAILED)
		return fail("mmap kvm_run");

	if (ioctl(vcpu, KVM_RUN, 0))
		return fail("KVM_RUN");
	if (run->exit_reason != KVM_EXIT_MMIO || run->mmio.phys_addr != MMIO ||
	    run->mmio.is_write == read) {
		fprintf(stderr, "unexpected exit: reason %u (expected an MMIO %s at 0x%lx)\n",
			run->exit_reason, argv[1], MMIO);
		return 2;
	}
	if (get_reg(vcpu, CORE_REG(regs.pstate), &pstate))
		return fail("KVM_GET_ONE_REG PSTATE");
	if ((pstate & PSR_MODE_MASK) != PSR_MODE_EL1h) {
		fprintf(stderr, "after the run the vCPU is not in L2's EL1 (PSTATE 0x%llx)\n",
			(unsigned long long)pstate);
		return 2;
	}
	printf("%s: ran L2 and exited to userspace (MMIO %s at 0x%llx); now KVM_SET_VCPU_EVENTS serror_pending\n",
	       argv[1], argv[1], (unsigned long long)run->mmio.phys_addr);
	fflush(stdout);

	memset(&events, 0, sizeof(events));
	events.exception.serror_pending = 1;
	if (ioctl(vcpu, KVM_SET_VCPU_EVENTS, &events))
		return fail("KVM_SET_VCPU_EVENTS");

	if (get_reg(vcpu, CORE_REG(regs.pc), &pc) ||
	    get_reg(vcpu, CORE_REG(regs.pstate), &pstate) ||
	    get_reg(vcpu, REG_ESR_EL2, &esr))
		return fail("KVM_GET_ONE_REG after the injection");
	printf("KVM_SET_VCPU_EVENTS returned: PC 0x%llx, PSTATE 0x%llx, ESR_EL2 0x%llx\n",
	       (unsigned long long)pc, (unsigned long long)pstate, (unsigned long long)esr);

	if (pc != VECTORS + SERROR_LOWER_A64 || (pstate & PSR_MODE_MASK) != PSR_MODE_EL2h) {
		fprintf(stderr, "the ioctl returned, but the vCPU is not at L1's SError vector\n");
		return 2;
	}
	printf("the SError was delivered to L1's EL2; running again\n");

	/* What a VMM would hand back for the read, had it not aborted it. */
	if (read)
		memcpy(run->mmio.data, &(uint64_t){ MMIO_DATA }, sizeof(uint64_t));

	if (ioctl(vcpu, KVM_RUN, 0))
		return fail("second KVM_RUN");
	if (run->exit_reason != KVM_EXIT_MMIO) {
		fprintf(stderr, "second run: unexpected exit: reason %u (expected MMIO)\n",
			run->exit_reason);
		return 2;
	}
	if (get_reg(vcpu, CORE_REG(regs.regs[5]), &x5))
		return fail("KVM_GET_ONE_REG x5");
	printf("%s: second run: MMIO at 0x%llx, x5 0x%llx\n", argv[1],
	       (unsigned long long)run->mmio.phys_addr, (unsigned long long)x5);

	if (run->mmio.phys_addr == L1_MMIO && x5 == X5_INIT) {
		printf("%s: L1's SError vector ran from its first instruction\n", argv[1]);
		return 0;
	}
	if (run->mmio.phys_addr == SKIP_MMIO || (read && x5 == MMIO_DATA)) {
		printf("%s: stale MMIO emulation: %s%s\n", argv[1],
		       run->mmio.phys_addr == SKIP_MMIO ? "L1's first vector instruction was skipped" : "",
		       read && x5 == MMIO_DATA ? ", x5 got the aborted read's data" : "");
		return 1;
	}
	fprintf(stderr, "%s: unexpected result\n", argv[1]);
	return 2;
}

[-- Attachment #4: run_on_L1.sh --]
[-- Type: application/x-sh, Size: 1050 bytes --]

  reply	other threads:[~2026-11-19  8:18 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 13:05 [PATCH v2 0/2] KVM: arm64: nv: Fix a couple NULL dereferences in kvm_inject_nested() Dan Carpenter
2026-11-18 13:05 ` [PATCH v2 1/2] KVM: arm64: Load the vcpu in KVM_SET_VCPU_EVENTS Dan Carpenter
2026-11-18 13:45   ` sashiko-bot
2026-11-18 17:44     ` Marc Zyngier
2026-11-19  8:18       ` Dan Carpenter [this message]
2026-11-19  9:53         ` Marc Zyngier
2026-11-18 13:05 ` [PATCH v2 2/2] KVM: arm64: nv: Fix NULL dereference in kvm_pmu_nested_transition() Dan Carpenter
2026-11-19 12:43 ` [PATCH v2 0/2] KVM: arm64: nv: Fix a couple NULL dereferences in kvm_inject_nested() Marc Zyngier

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av6wypzwAJxIyKOU@stanley.mountain \
    --to=dan@pereq.eu \
    --cc=kvmarm@lists.linux.dev \
    --cc=maz@kernel.org \
    --cc=oupton@kernel.org \
    --cc=sashiko-reviews@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.