Re: Deadlock due to EPT_VIOLATION

[Sean Christopherson <seanjc@google.com>]

On Thu, Aug 17, 2023, Eric Wheeler wrote:
> On Thu, 17 Aug 2023, Sean Christopherson wrote:
> > > > kprobe:handle_ept_violation
> > > > {
> > > > 	printf("vcpu = %lx pid = %u MMU seq = %lx, in-prog = %lx, start = %lx, end = %lx\n",
> > > > 	       arg0, ((struct kvm_vcpu *)arg0)->pid->numbers[0].nr,
> > > > 	       ((struct kvm_vcpu *)arg0)->kvm->mmu_invalidate_seq,
> > > > 	       ((struct kvm_vcpu *)arg0)->kvm->mmu_invalidate_in_progress,
> > > > 	       ((struct kvm_vcpu *)arg0)->kvm->mmu_invalidate_range_start,
> > > > 	       ((struct kvm_vcpu *)arg0)->kvm->mmu_invalidate_range_end);
> > > > }
> > > > 
> > > > If you don't have BTF info, we can still use a bpf program, but to get at the
> > > > fields of interested, I think we'd have to resort to pointer arithmetic with struct
> > > > offsets grab from your build.
> > > 
> > > We have BTF, so hurray for not needing struct offsets!
> 
> Well, I was part right: not all hosts have BTF.
> 
> What is involved in doing this with struct offsets for Linux v6.1.x?

Unless you are up for a challenge, I'd drop the PID entirely, getting that will
be ugly.

For the KVM info, you need the offset of "kvm" within struct kvm_vcpu (more than
likely it's '0'), and then the offset of each of the mmu_invaliate_* fields within
struct kvm.  These need to come from the exact kernel you're running, though unless
a field is added/removed to/from struct kvm between kernel versions, the offsets
should be stable.

A cheesy/easy way to get the offsets is to feed offsetof() into __aligned and
then compile.  So long as the offset doesn't happen to be a power-of-2, the
compiler will yell.  E.g. with this

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 92c50dc159e8..04ec37f7374a 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -543,7 +543,13 @@ struct kvm_hva_range {
  */
 static void kvm_null_fn(void)
 {
+       int v __aligned(offsetof(struct kvm_vcpu, kvm));
+       int w __aligned(offsetof(struct kvm, mmu_invalidate_seq));
+       int x __aligned(offsetof(struct kvm, mmu_invalidate_in_progress));
+       int y __aligned(offsetof(struct kvm, mmu_invalidate_range_start));
+       int z __aligned(offsetof(struct kvm, mmu_invalidate_range_end));
 
+       v = w = x = y = z = 0;
 }
 #define IS_KVM_NULL_FN(fn) ((fn) == (void *)kvm_null_fn)

I get yelled at with (trimmed):

arch/x86/kvm/../../../virt/kvm/kvm_main.c:546:34: error: requested alignment ‘0’ is not a positive power of 2 [-Werror=attributes]
arch/x86/kvm/../../../virt/kvm/kvm_main.c:547:20: error: requested alignment ‘36960’ is not a positive power of 2
arch/x86/kvm/../../../virt/kvm/kvm_main.c:549:20: error: requested alignment ‘36968’ is not a positive power of 2
arch/x86/kvm/../../../virt/kvm/kvm_main.c:551:20: error: requested alignment ‘36976’ is not a positive power of 2
arch/x86/kvm/../../../virt/kvm/kvm_main.c:553:20: error: requested alignment ‘36984’ is not a positive power of 2

Then take those offsets and do math.  For me, this provides the same output as
the above pretty version.  Just use common sense and verify you're getting sane
data.

kprobe:handle_ept_violation
{
	$kvm = *((uint64 *)((uint64)arg0 + 0));

	printf("vcpu = %lx MMU seq = %lx, in-prog = %lx, start = %lx, end = %lx\n",
	       arg0,
               *((uint64 *)($kvm + 36960)),
               *((uint64 *)($kvm + 36968)),
               *((uint64 *)($kvm + 36976)),
               *((uint64 *)($kvm + 36984)));
}