Re: kernel BUG at arch/x86/kvm/x86.c:361! on AMD CPU

Dmitry Vyukov
2019-06-18T18:01:21Z

On Tue, Jun 18, 2019 at 7:52 PM Borislav Petkov <bp@alien8.de> wrote:
>
> On Mon, Jun 17, 2019 at 11:13:22AM -0400, George Kennedy wrote:
> > 319f3: 0f 01 da vmload <--- svm_vcpu_run+0xa83
>
> Hmm, so VMLOAD can fault for a bunch of reasons if you look at its
> description in the APM.
>
> Looking at your Code: section and building with your config, rIP and the
> insns around it point to:
>
> All code
> ========
> 0: 00 55 89 add %dl,-0x77(%rbp)
> 3: d2 45 89 rolb %cl,-0x77(%rbp)
> 6: c9 leaveq
> 7: 48 89 e5 mov %rsp,%rbp
> a: 8b 45 18 mov 0x18(%rbp),%eax
> d: 50 push %rax
> e: 8b 45 10 mov 0x10(%rbp),%eax
> 11: 50 push %rax
> 12: e8 8a 42 6b 00 callq 0x6b42a1
> 17: 58 pop %rax
> 18: 5a pop %rdx
> 19: c9 leaveq
> 1a: c3 retq
> 1b: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
> 21: 66 66 66 66 90 data16 data16 data16 xchg %ax,%ax
> 26: 55 push %rbp
> 27: 48 89 e5 mov %rsp,%rbp
> 2a:* 0f 0b ud2 <-- trapping instruction
> 2c: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
> 31: 66 66 66 66 90 data16 data16 data16 xchg %ax,%ax
> 36: 55 push %rbp
> 37: 48 89 e5 mov %rsp,%rbp
> 3a: 41 55 push %r13
> 3c: 41 54 push %r12
> 3e: 41 rex.B
> 3f: 89 .byte 0x89
>
> 0000000000017f30 <__bpf_trace_kvm_nested_vmexit>:
> 17f30: 55 push %rbp
> 17f31: 89 d2 mov %edx,%edx
> 17f33: 45 89 c9 mov %r9d,%r9d
> 17f36: 48 89 e5 mov %rsp,%rbp
> 17f39: 8b 45 18 mov 0x18(%rbp),%eax
> 17f3c: 50 push %rax
> 17f3d: 8b 45 10 mov 0x10(%rbp),%eax
> 17f40: 50 push %rax
> 17f41: e8 00 00 00 00 callq 17f46 <__bpf_trace_kvm_nested_vmexit+0x16>
> 17f46: 58 pop %rax
> 17f47: 5a pop %rdx
> 17f48: c9 leaveq
> 17f49: c3 retq
> 17f4a: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
>
> 0000000000017f50 <kvm_spurious_fault>:
> 17f50: e8 00 00 00 00 callq 17f55 <kvm_spurious_fault+0x5>
> 17f55: 55 push %rbp
> 17f56: 48 89 e5 mov %rsp,%rbp
> 17f59: 0f 0b ud2
> 17f5b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
>
> so the invalid opcode splat is dumping the insn bytes around rIP the
> UD2 happens but in order to see which of the VMLOAD conditions causes
> the fault, you'd probably need to intercept the fault handler and dump
> registers in the hypervisor to check.
>
> There's something else that's bothering me though: your splat is from
> the guest yet there is svm_vcpu_run() mentioned there which should be
> called by the hypervisor and not by the guest. Unless you're doing
> nested virt stuff...
>
> Anyway, sorry that I cannot be of more help - I'm sure KVM guys would
> make much more sense of it.

I am not a KVM folk either, but FWIW syzkaller is capable of creating
a double-nested VM. The code is somewhat VMX-specific, but it should
be capable at least executing some SVM instructions inside of guest.
This code setups VM to run a given instruction sequences (should be generic):
https://github.com/google/syzkaller/blob/34bf9440bd06034f86b5d9ac8afbf078129cbdae/executor/common_kvm_amd64.h
The instruction generator is based on Intel XED so it may be somewhat
Intel-biased, but at least I see some mentions of SVM there:
https://raw.githubusercontent.com/google/syzkaller/34bf9440bd06034f86b5d9ac8afbf078129cbdae/pkg/ifuzz/gen/all-enc-instructions.txt
And this code can setup a double-nested VM, but it's VMX-specific:
https://github.com/google/syzkaller/blob/34bf9440bd06034f86b5d9ac8afbf078129cbdae/executor/kvm.S#L125

If there is specific interested in testing SVM, it can make sense to
improve SVM support in syzkaller to at least match VMX.