* [PATCH kvmtool v2 2/2] cpu: vmexit: Handle KVM_EXIT_MEMORY_FAULT in KVM_RUN ioctl return
2025-02-24 9:09 [PATCH kvmtool v2 1/2] cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly Aneesh Kumar K.V (Arm)
@ 2025-02-24 9:10 ` Aneesh Kumar K.V (Arm)
2025-03-13 15:37 ` Alexandru Elisei
2025-03-13 15:37 ` [PATCH kvmtool v2 1/2] cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly Alexandru Elisei
2025-04-17 12:07 ` Will Deacon
2 siblings, 1 reply; 7+ messages in thread
From: Aneesh Kumar K.V (Arm) @ 2025-02-24 9:10 UTC (permalink / raw)
To: kvm
Cc: Suzuki K Poulose, Steven Price, Will Deacon, Julien Thierry,
Alexandru Elisei, Aneesh Kumar K.V (Arm)
Linux kernel documentation states:
"Note! KVM_EXIT_MEMORY_FAULT is unique among all KVM exit reasons in
that it accompanies a return code of '-1', not '0'! errno will always be
set to EFAULT or EHWPOISON when KVM exits with KVM_EXIT_MEMORY_FAULT,
userspace should assume kvm_run.exit_reason is stale/undefined for all
other error numbers." "
Update KVM_RUN ioctl error handling to correctly handle
KVM_EXIT_MEMORY_FAULT. This enables the memory fault exit handlers in
the kernel to return -EFAULT as the return value. VMM support is
still required to handle these memory fault exits, but that is not
included in this change
Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
kvm-cpu.c | 15 +++++++++++++--
kvm.c | 1 +
2 files changed, 14 insertions(+), 2 deletions(-)
diff --git a/kvm-cpu.c b/kvm-cpu.c
index 7c62bfc56679..c0b10b1534ab 100644
--- a/kvm-cpu.c
+++ b/kvm-cpu.c
@@ -41,8 +41,19 @@ void kvm_cpu__run(struct kvm_cpu *vcpu)
return;
err = ioctl(vcpu->vcpu_fd, KVM_RUN, 0);
- if (err < 0 && (errno != EINTR && errno != EAGAIN))
- die_perror("KVM_RUN failed");
+ if (err < 0) {
+ switch (errno) {
+ case EINTR:
+ case EAGAIN:
+ return;
+ case EFAULT:
+ if (vcpu->kvm_run->exit_reason == KVM_EXIT_MEMORY_FAULT)
+ return;
+ /* faullthrough */
+ default:
+ die_perror("KVM_RUN failed");
+ }
+ }
}
static void kvm_cpu_signal_handler(int signum)
diff --git a/kvm.c b/kvm.c
index 42b881217df6..172d951bfe4e 100644
--- a/kvm.c
+++ b/kvm.c
@@ -55,6 +55,7 @@ const char *kvm_exit_reasons[] = {
#ifdef CONFIG_PPC64
DEFINE_KVM_EXIT_REASON(KVM_EXIT_PAPR_HCALL),
#endif
+ DEFINE_KVM_EXIT_REASON(KVM_EXIT_MEMORY_FAULT),
};
static int pause_event;
--
2.43.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH kvmtool v2 2/2] cpu: vmexit: Handle KVM_EXIT_MEMORY_FAULT in KVM_RUN ioctl return
2025-02-24 9:10 ` [PATCH kvmtool v2 2/2] cpu: vmexit: Handle KVM_EXIT_MEMORY_FAULT in KVM_RUN ioctl return Aneesh Kumar K.V (Arm)
@ 2025-03-13 15:37 ` Alexandru Elisei
0 siblings, 0 replies; 7+ messages in thread
From: Alexandru Elisei @ 2025-03-13 15:37 UTC (permalink / raw)
To: Aneesh Kumar K.V (Arm)
Cc: kvm, Suzuki K Poulose, Steven Price, Will Deacon, Julien Thierry
Hi Aneesh,
On Mon, Feb 24, 2025 at 02:40:00PM +0530, Aneesh Kumar K.V (Arm) wrote:
> Linux kernel documentation states:
>
> "Note! KVM_EXIT_MEMORY_FAULT is unique among all KVM exit reasons in
> that it accompanies a return code of '-1', not '0'! errno will always be
> set to EFAULT or EHWPOISON when KVM exits with KVM_EXIT_MEMORY_FAULT,
> userspace should assume kvm_run.exit_reason is stale/undefined for all
> other error numbers." "
>
> Update KVM_RUN ioctl error handling to correctly handle
> KVM_EXIT_MEMORY_FAULT. This enables the memory fault exit handlers in
> the kernel to return -EFAULT as the return value. VMM support is
I think this is too much information. Are all architectures that kvmtool
supports going to/have implemented forwarding memory faults to userspace
this same way? I would think not.
The first paragraph should be enough - kvmtool implements the KVM ABI, it
shouldn't care about how that is implemented by KVM.
Other than that, looks correct to me, and more readable (but that might
just be my personal bias):
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Thanks,
Alex
> still required to handle these memory fault exits, but that is not
> included in this change
>
> Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
> ---
> kvm-cpu.c | 15 +++++++++++++--
> kvm.c | 1 +
> 2 files changed, 14 insertions(+), 2 deletions(-)
>
> diff --git a/kvm-cpu.c b/kvm-cpu.c
> index 7c62bfc56679..c0b10b1534ab 100644
> --- a/kvm-cpu.c
> +++ b/kvm-cpu.c
> @@ -41,8 +41,19 @@ void kvm_cpu__run(struct kvm_cpu *vcpu)
> return;
>
> err = ioctl(vcpu->vcpu_fd, KVM_RUN, 0);
> - if (err < 0 && (errno != EINTR && errno != EAGAIN))
> - die_perror("KVM_RUN failed");
> + if (err < 0) {
> + switch (errno) {
> + case EINTR:
> + case EAGAIN:
> + return;
> + case EFAULT:
> + if (vcpu->kvm_run->exit_reason == KVM_EXIT_MEMORY_FAULT)
> + return;
> + /* faullthrough */
> + default:
> + die_perror("KVM_RUN failed");
> + }
> + }
> }
>
> static void kvm_cpu_signal_handler(int signum)
> diff --git a/kvm.c b/kvm.c
> index 42b881217df6..172d951bfe4e 100644
> --- a/kvm.c
> +++ b/kvm.c
> @@ -55,6 +55,7 @@ const char *kvm_exit_reasons[] = {
> #ifdef CONFIG_PPC64
> DEFINE_KVM_EXIT_REASON(KVM_EXIT_PAPR_HCALL),
> #endif
> + DEFINE_KVM_EXIT_REASON(KVM_EXIT_MEMORY_FAULT),
> };
>
> static int pause_event;
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH kvmtool v2 1/2] cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly
2025-02-24 9:09 [PATCH kvmtool v2 1/2] cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly Aneesh Kumar K.V (Arm)
2025-02-24 9:10 ` [PATCH kvmtool v2 2/2] cpu: vmexit: Handle KVM_EXIT_MEMORY_FAULT in KVM_RUN ioctl return Aneesh Kumar K.V (Arm)
@ 2025-03-13 15:37 ` Alexandru Elisei
2025-04-17 12:07 ` Will Deacon
2 siblings, 0 replies; 7+ messages in thread
From: Alexandru Elisei @ 2025-03-13 15:37 UTC (permalink / raw)
To: Aneesh Kumar K.V (Arm)
Cc: kvm, Suzuki K Poulose, Steven Price, Will Deacon, Julien Thierry
Hi Aneesh,
On Mon, Feb 24, 2025 at 02:39:59PM +0530, Aneesh Kumar K.V (Arm) wrote:
> The return value for kernel VM exit handlers is confusing and has led to
> errors in different kernel exit handlers. A return value of 0 indicates
> a return to the VMM, whereas a return value of 1 indicates resuming
> execution in the guest. Some handlers mistakenly return 0 to force a
> return to the guest.
>
> This worked in kvmtool because the exit_reason defaulted to
> 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit
> reason. However, forcing a VMM exit with error on KVM_EXIT_UNKNOWN
> exit_reson would help catch these bugs early.
I think I understand what you're saying - if there's a bug in handle_exit()
in KVM that triggers an erroneous exit to userspace, exit_reason and the
exit information struct could still be at their default values, which are 0
from when kvm_run was allocated (in kvm_vm_ioctl_create_vcpu()).
An exit_reason of 0 is interpreted by userspace as KVM_EXIT_UNKNOWN, but
kvmtool on KVM_EXIT_UNKNOWN resumes the guest instead of signalling the
error, thus masking the buggy KVM behaviour.
The patch looks good to me:
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Thanks,
Alex
>
> Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
> ---
> kvm-cpu.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kvm-cpu.c b/kvm-cpu.c
> index f66dcd07220c..7c62bfc56679 100644
> --- a/kvm-cpu.c
> +++ b/kvm-cpu.c
> @@ -170,7 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)
>
> switch (cpu->kvm_run->exit_reason) {
> case KVM_EXIT_UNKNOWN:
> - break;
> + goto panic_kvm;
> case KVM_EXIT_DEBUG:
> kvm_cpu__show_registers(cpu);
> kvm_cpu__show_code(cpu);
> --
> 2.43.0
>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH kvmtool v2 1/2] cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly
2025-02-24 9:09 [PATCH kvmtool v2 1/2] cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly Aneesh Kumar K.V (Arm)
2025-02-24 9:10 ` [PATCH kvmtool v2 2/2] cpu: vmexit: Handle KVM_EXIT_MEMORY_FAULT in KVM_RUN ioctl return Aneesh Kumar K.V (Arm)
2025-03-13 15:37 ` [PATCH kvmtool v2 1/2] cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly Alexandru Elisei
@ 2025-04-17 12:07 ` Will Deacon
2025-04-20 14:25 ` Aneesh Kumar K.V
2025-04-23 10:53 ` Aneesh Kumar K.V
2 siblings, 2 replies; 7+ messages in thread
From: Will Deacon @ 2025-04-17 12:07 UTC (permalink / raw)
To: Aneesh Kumar K.V (Arm)
Cc: kvm, Suzuki K Poulose, Steven Price, Julien Thierry,
Alexandru Elisei
On Mon, Feb 24, 2025 at 02:39:59PM +0530, Aneesh Kumar K.V (Arm) wrote:
> The return value for kernel VM exit handlers is confusing and has led to
> errors in different kernel exit handlers. A return value of 0 indicates
> a return to the VMM, whereas a return value of 1 indicates resuming
> execution in the guest. Some handlers mistakenly return 0 to force a
> return to the guest.
>
> This worked in kvmtool because the exit_reason defaulted to
> 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit
> reason. However, forcing a VMM exit with error on KVM_EXIT_UNKNOWN
> exit_reson would help catch these bugs early.
>
> Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
> ---
> kvm-cpu.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kvm-cpu.c b/kvm-cpu.c
> index f66dcd07220c..7c62bfc56679 100644
> --- a/kvm-cpu.c
> +++ b/kvm-cpu.c
> @@ -170,7 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)
>
> switch (cpu->kvm_run->exit_reason) {
> case KVM_EXIT_UNKNOWN:
> - break;
> + goto panic_kvm;
> case KVM_EXIT_DEBUG:
> kvm_cpu__show_registers(cpu);
> kvm_cpu__show_code(cpu);
> --
> 2.43.0
This breaks SMP boot on my x86 machine:
# ./lkvm run
...
[ 0.628472] smp: Bringing up secondary CPUs ...
[ 0.630401] smpboot: x86: Booting SMP configuration:
Error: KVM exit reason: 0 ("KVM_EXIT_UNKNOWN")
Error: KVM exit code: 0
Registers:
----------
rip: 0000000000000000 rsp: 0000000000000000 flags: 0000000000000002
rax: 0000000000000000 rbx: 0000000000000000 rcx: 0000000000000000
rdx: 0000000000050654 rsi: 0000000000000000 rdi: 0000000000000000
rbp: 0000000000000000 r8: 0000000000000000 r9: 0000000000000000
r10: 0000000000000000 r11: 0000000000000000 r12: 0000000000000000
r13: 0000000000000000 r14: 0000000000000000 r15: 0000000000000000
cr0: 0000000060000010 cr2: 0000000000000000 cr3: 0000000000000000
cr4: 0000000000000000 cr8: 0000000000000000
Segment registers:
------------------
register selector base limit type p dpl db s l g avl
cs 9900 0000000000099000 0000ffff 0b 1 0 0 1 0 0 0
ss 0000 0000000000000000 0000ffff 03 1 0 0 1 0 0 0
ds 0000 0000000000000000 0000ffff 03 1 0 0 1 0 0 0
es 0000 0000000000000000 0000ffff 03 1 0 0 1 0 0 0
fs 0000 0000000000000000 0000ffff 03 1 0 0 1 0 0 0
gs 0000 0000000000000000 0000ffff 03 1 0 0 1 0 0 0
tr 0000 0000000000000000 0000ffff 0b 1 0 0 0 0 0 0
ldt 0000 0000000000000000 0000ffff 02 1 0 0 0 0 0 0
gdt 0000000000000000 0000ffff
idt 0000000000000000 0000ffff
APIC:
-----
efer: 0000000000000000 apic base: 00000000fee00800 nmi: enabled
Interrupt bitmap:
-----------------
0000000000000000 0000000000000000 0000000000000000 0000000000000000
Code:
-----
Warning: symbol_lookup() failed to find symbol with error: -2
rip: [<0000000000000000>] <unknown>
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <fa> 0f 09 ea 08 10 00 98 8c c8 8e d8 8e c0 8e d0 66 f0 0f ba 2e
Stack:
------
rsp: [<0000000000000000>]
0x00000000: 30 00 00 f0 30 00 00 f0
0x00000008: 30 00 00 f0 30 00 00 f0
0x00000010: 30 00 00 f0 30 00 00 f0
0x00000018: 30 00 00 f0 30 00 00 f0
Page Tables:
------
Not in protected mode
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH kvmtool v2 1/2] cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly
2025-04-17 12:07 ` Will Deacon
@ 2025-04-20 14:25 ` Aneesh Kumar K.V
2025-04-23 10:53 ` Aneesh Kumar K.V
1 sibling, 0 replies; 7+ messages in thread
From: Aneesh Kumar K.V @ 2025-04-20 14:25 UTC (permalink / raw)
To: Will Deacon
Cc: kvm, Suzuki K Poulose, Steven Price, Julien Thierry,
Alexandru Elisei
Will Deacon <will@kernel.org> writes:
> On Mon, Feb 24, 2025 at 02:39:59PM +0530, Aneesh Kumar K.V (Arm) wrote:
>> The return value for kernel VM exit handlers is confusing and has led to
>> errors in different kernel exit handlers. A return value of 0 indicates
>> a return to the VMM, whereas a return value of 1 indicates resuming
>> execution in the guest. Some handlers mistakenly return 0 to force a
>> return to the guest.
>>
>> This worked in kvmtool because the exit_reason defaulted to
>> 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit
>> reason. However, forcing a VMM exit with error on KVM_EXIT_UNKNOWN
>> exit_reson would help catch these bugs early.
>>
>> Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
>> ---
>> kvm-cpu.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/kvm-cpu.c b/kvm-cpu.c
>> index f66dcd07220c..7c62bfc56679 100644
>> --- a/kvm-cpu.c
>> +++ b/kvm-cpu.c
>> @@ -170,7 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)
>>
>> switch (cpu->kvm_run->exit_reason) {
>> case KVM_EXIT_UNKNOWN:
>> - break;
>> + goto panic_kvm;
>> case KVM_EXIT_DEBUG:
>> kvm_cpu__show_registers(cpu);
>> kvm_cpu__show_code(cpu);
>> --
>> 2.43.0
>
> This breaks SMP boot on my x86 machine:
>
> # ./lkvm run
> ...
> [ 0.628472] smp: Bringing up secondary CPUs ...
> [ 0.630401] smpboot: x86: Booting SMP configuration:
> Error: KVM exit reason: 0 ("KVM_EXIT_UNKNOWN")
> Error: KVM exit code: 0
>
I can recreate this with distro kernel as the guest kernel but not with
CONFIG_SMP enabled with kvm_guest.config. I will have to find out what
is causing that KVM_EXIT_UNKNOWN on the secondary cpu boot.
-aneesh
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH kvmtool v2 1/2] cpu: vmexit: Handle KVM_EXIT_UNKNOWN exit reason correctly
2025-04-17 12:07 ` Will Deacon
2025-04-20 14:25 ` Aneesh Kumar K.V
@ 2025-04-23 10:53 ` Aneesh Kumar K.V
1 sibling, 0 replies; 7+ messages in thread
From: Aneesh Kumar K.V @ 2025-04-23 10:53 UTC (permalink / raw)
To: Will Deacon
Cc: kvm, Suzuki K Poulose, Steven Price, Julien Thierry,
Alexandru Elisei
Will Deacon <will@kernel.org> writes:
> On Mon, Feb 24, 2025 at 02:39:59PM +0530, Aneesh Kumar K.V (Arm) wrote:
>> The return value for kernel VM exit handlers is confusing and has led to
>> errors in different kernel exit handlers. A return value of 0 indicates
>> a return to the VMM, whereas a return value of 1 indicates resuming
>> execution in the guest. Some handlers mistakenly return 0 to force a
>> return to the guest.
>>
>> This worked in kvmtool because the exit_reason defaulted to
>> 0 (KVM_EXIT_UNKNOWN), and kvmtool did not error out on an unknown exit
>> reason. However, forcing a VMM exit with error on KVM_EXIT_UNKNOWN
>> exit_reson would help catch these bugs early.
>>
>> Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
>> ---
>> kvm-cpu.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/kvm-cpu.c b/kvm-cpu.c
>> index f66dcd07220c..7c62bfc56679 100644
>> --- a/kvm-cpu.c
>> +++ b/kvm-cpu.c
>> @@ -170,7 +170,7 @@ int kvm_cpu__start(struct kvm_cpu *cpu)
>>
>> switch (cpu->kvm_run->exit_reason) {
>> case KVM_EXIT_UNKNOWN:
>> - break;
>> + goto panic_kvm;
>> case KVM_EXIT_DEBUG:
>> kvm_cpu__show_registers(cpu);
>> kvm_cpu__show_code(cpu);
>> --
>> 2.43.0
>
> This breaks SMP boot on my x86 machine:
>
> # ./lkvm run
> ...
> [ 0.628472] smp: Bringing up secondary CPUs ...
> [ 0.630401] smpboot: x86: Booting SMP configuration:
> Error: KVM exit reason: 0 ("KVM_EXIT_UNKNOWN")
> Error: KVM exit code: 0
>
Turns out we should handle EINTR and EAGAIN as special such that we do
an retry of KVM_RUN ioctl without checking the exit_reason.
I can send a v3 if you are ok with the change below.
@@ -16,7 +16,7 @@ void kvm_cpu__delete(struct kvm_cpu *vcpu);
void kvm_cpu__reset_vcpu(struct kvm_cpu *vcpu);
void kvm_cpu__setup_cpuid(struct kvm_cpu *vcpu);
void kvm_cpu__enable_singlestep(struct kvm_cpu *vcpu);
-void kvm_cpu__run(struct kvm_cpu *vcpu);
+int kvm_cpu__run(struct kvm_cpu *vcpu);
int kvm_cpu__start(struct kvm_cpu *cpu);
bool kvm_cpu__handle_exit(struct kvm_cpu *vcpu);
int kvm_cpu__get_endianness(struct kvm_cpu *vcpu);
modified kvm-cpu.c
@@ -35,27 +35,32 @@ void kvm_cpu__enable_singlestep(struct kvm_cpu *vcpu)
pr_warning("KVM_SET_GUEST_DEBUG failed");
}
-void kvm_cpu__run(struct kvm_cpu *vcpu)
+/*
+ * return value -1 if we need to call the kvm_cpu__run again without checking
+ * exit_reason. return value 0 results in taking action based on exit_reason.
+ */
+int kvm_cpu__run(struct kvm_cpu *vcpu)
{
int err;
if (!vcpu->is_running)
- return;
+ return -1;
err = ioctl(vcpu->vcpu_fd, KVM_RUN, 0);
if (err < 0) {
switch (errno) {
case EINTR:
case EAGAIN:
- return;
+ return -1;
case EFAULT:
if (vcpu->kvm_run->exit_reason == KVM_EXIT_MEMORY_FAULT)
- return;
+ return 0;
/* faullthrough */
default:
die_perror("KVM_RUN failed");
}
}
+ return 0;
}
static void kvm_cpu_signal_handler(int signum)
@@ -179,11 +184,13 @@ int kvm_cpu__start(struct kvm_cpu *cpu)
if (cpu->task)
kvm_cpu__run_task(cpu);
- kvm_cpu__run(cpu);
+ if (kvm_cpu__run(cpu) == -1)
+ /* retry without an exit_reason check */
+ continue;
-aneesh
^ permalink raw reply [flat|nested] 7+ messages in thread