* KVM: unknown exit, hardware reason 31 @ 2015-07-24 19:25 Pavel Shirshov 2015-07-27 3:23 ` Xiao Guangrong 0 siblings, 1 reply; 4+ messages in thread From: Pavel Shirshov @ 2015-07-24 19:25 UTC (permalink / raw) To: kvm Hello, I'm running a lot of identical VMs under KVM. Sometimes (one time per 2000-3000 runs) I got following: 1. VM is paused in libvirt. It can't be just resumed. I can just reset it and resume. 2. In VM log file I see following: "KVM: unknown exit, hardware reason 31" with a CPU dump. 3. In dmesg I see following: [84245.284948] EPT: Misconfiguration. [84245.285056] EPT: GPA: 0xfeda848 [84245.285154] ept_misconfig_inspect_spte: spte 0x5eaef50107 level 4 [84245.285344] ept_misconfig_inspect_spte: spte 0x5f5fadc107 level 3 [84245.285532] ept_misconfig_inspect_spte: spte 0x5141d18107 level 2 [84245.285723] ept_misconfig_inspect_spte: spte 0x52e40dad77 level 1 OS. 3.16.0-44-generic #59~14.04.1-Ubuntu SMP QEMU: QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.14), Copyright (c) 2003-2008 Fabrice Bellard Is it linux kvm bug or CPU bug? How can I fix that? I can reproduce the bug in one-two days. Is it possible to enable deeper debug for the issue? Thanks -- With best regards, Pavel S. Shirshov ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: KVM: unknown exit, hardware reason 31 2015-07-24 19:25 KVM: unknown exit, hardware reason 31 Pavel Shirshov @ 2015-07-27 3:23 ` Xiao Guangrong 2015-07-30 22:51 ` Pavel Shirshov 0 siblings, 1 reply; 4+ messages in thread From: Xiao Guangrong @ 2015-07-27 3:23 UTC (permalink / raw) To: Pavel Shirshov, kvm I guess it happened on this scenario: 1. QEMU drops mmio region 2. invalidate all mmio sptes 3. VCPU 0 KVM VCPU 1 access the invalid mmio spte page reclaim zap shadow page access the region originally was MMIO before set the spte to the normal ram map mmio #PF check the spte and see it becomes normal ram mapping !!! The issue is caused by fast invalidate mmio sptes which increases generation number instead of zapping mmio sptes (SRCU can ensure the vcpu either see mmio spte or being zapped / zapped sptes.). The simple fix is just drop the check_direct_spte_mmio_pf(), let VCPU access again as follows: diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c index 4417146..299a5da 100644 --- a/arch/x86/kvm/mmu.c +++ b/arch/x86/kvm/mmu.c @@ -3299,21 +3299,6 @@ static bool quickly_check_mmio_pf(struct kvm_vcpu *vcpu, u64 addr, bool direct) return vcpu_match_mmio_gva(vcpu, addr); } - -/* - * On direct hosts, the last spte is only allows two states - * for mmio page fault: - * - It is the mmio spte - * - It is zapped or it is being zapped. - * - * This function completely checks the spte when the last spte - * is not the mmio spte. - */ -static bool check_direct_spte_mmio_pf(u64 spte) -{ - return __check_direct_spte_mmio_pf(spte); -} - static u64 walk_shadow_page_get_mmio_spte(struct kvm_vcpu *vcpu, u64 addr) { struct kvm_shadow_walk_iterator iterator; @@ -3356,13 +3341,6 @@ int handle_mmio_page_fault_common(struct kvm_vcpu *vcpu, u64 addr, bool direct) } /* - * It's ok if the gva is remapped by other cpus on shadow guest, - * it's a BUG if the gfn is not a mmio page. - */ - if (direct && !check_direct_spte_mmio_pf(spte)) - return RET_MMIO_PF_BUG; - - /* * If the page table is zapped by other cpus, let CPU fault again on * the address. */ Pavel, could you please check if it works for you? I will fully consider the case and post the right fix out... On 07/25/2015 03:25 AM, Pavel Shirshov wrote: > Hello, > > I'm running a lot of identical VMs under KVM. Sometimes (one time per > 2000-3000 runs) I got following: > > 1. VM is paused in libvirt. It can't be just resumed. I can just reset > it and resume. > 2. In VM log file I see following: "KVM: unknown exit, hardware reason > 31" with a CPU dump. > 3. In dmesg I see following: > [84245.284948] EPT: Misconfiguration. > [84245.285056] EPT: GPA: 0xfeda848 > [84245.285154] ept_misconfig_inspect_spte: spte 0x5eaef50107 level 4 > [84245.285344] ept_misconfig_inspect_spte: spte 0x5f5fadc107 level 3 > [84245.285532] ept_misconfig_inspect_spte: spte 0x5141d18107 level 2 > [84245.285723] ept_misconfig_inspect_spte: spte 0x52e40dad77 level 1 > > OS. 3.16.0-44-generic #59~14.04.1-Ubuntu SMP > QEMU: QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.14), > Copyright (c) 2003-2008 Fabrice Bellard > > Is it linux kvm bug or CPU bug? How can I fix that? > > I can reproduce the bug in one-two days. Is it possible to enable > deeper debug for the issue? > > Thanks > ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: KVM: unknown exit, hardware reason 31 2015-07-27 3:23 ` Xiao Guangrong @ 2015-07-30 22:51 ` Pavel Shirshov 2015-07-31 1:42 ` Xiao Guangrong 0 siblings, 1 reply; 4+ messages in thread From: Pavel Shirshov @ 2015-07-30 22:51 UTC (permalink / raw) To: Xiao Guangrong; +Cc: kvm Hi Xiao, I've been testing your patch since Jul, 27. Everything works perfectly with it. There are no EPT messages in dmesg output anymore. Also all VMs work as expected: no paused VMs, no "code 31" error. Let me know if you need any additional information. Thanks On Sun, Jul 26, 2015 at 8:23 PM, Xiao Guangrong <guangrong.xiao@linux.intel.com> wrote: > > I guess it happened on this scenario: > > > 1. QEMU drops mmio region > 2. invalidate all mmio sptes > 3. > > VCPU 0 KVM VCPU 1 > access the invalid mmio spte > page reclaim > zap shadow page > > access the region originally > was MMIO before > set the spte to the normal > ram map > > mmio #PF > check the spte and see it becomes normal ram mapping !!! > > > The issue is caused by fast invalidate mmio sptes which increases > generation number instead of zapping mmio sptes (SRCU can ensure the vcpu > either see mmio spte or being zapped / zapped sptes.). > > The simple fix is just drop the check_direct_spte_mmio_pf(), let VCPU access > again as follows: > > diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c > index 4417146..299a5da 100644 > --- a/arch/x86/kvm/mmu.c > +++ b/arch/x86/kvm/mmu.c > @@ -3299,21 +3299,6 @@ static bool quickly_check_mmio_pf(struct kvm_vcpu > *vcpu, u64 addr, bool direct) > return vcpu_match_mmio_gva(vcpu, addr); > } > > - > -/* > - * On direct hosts, the last spte is only allows two states > - * for mmio page fault: > - * - It is the mmio spte > - * - It is zapped or it is being zapped. > - * > - * This function completely checks the spte when the last spte > - * is not the mmio spte. > - */ > -static bool check_direct_spte_mmio_pf(u64 spte) > -{ > - return __check_direct_spte_mmio_pf(spte); > -} > - > static u64 walk_shadow_page_get_mmio_spte(struct kvm_vcpu *vcpu, u64 addr) > { > struct kvm_shadow_walk_iterator iterator; > @@ -3356,13 +3341,6 @@ int handle_mmio_page_fault_common(struct kvm_vcpu > *vcpu, u64 addr, bool direct) > } > > /* > - * It's ok if the gva is remapped by other cpus on shadow guest, > - * it's a BUG if the gfn is not a mmio page. > - */ > - if (direct && !check_direct_spte_mmio_pf(spte)) > - return RET_MMIO_PF_BUG; > - > - /* > * If the page table is zapped by other cpus, let CPU fault again on > * the address. > */ > > Pavel, could you please check if it works for you? > > I will fully consider the case and post the right fix out... > > On 07/25/2015 03:25 AM, Pavel Shirshov wrote: >> >> Hello, >> >> I'm running a lot of identical VMs under KVM. Sometimes (one time per >> 2000-3000 runs) I got following: >> >> 1. VM is paused in libvirt. It can't be just resumed. I can just reset >> it and resume. >> 2. In VM log file I see following: "KVM: unknown exit, hardware reason >> 31" with a CPU dump. >> 3. In dmesg I see following: >> [84245.284948] EPT: Misconfiguration. >> [84245.285056] EPT: GPA: 0xfeda848 >> [84245.285154] ept_misconfig_inspect_spte: spte 0x5eaef50107 level 4 >> [84245.285344] ept_misconfig_inspect_spte: spte 0x5f5fadc107 level 3 >> [84245.285532] ept_misconfig_inspect_spte: spte 0x5141d18107 level 2 >> [84245.285723] ept_misconfig_inspect_spte: spte 0x52e40dad77 level 1 >> >> OS. 3.16.0-44-generic #59~14.04.1-Ubuntu SMP >> QEMU: QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.14), >> Copyright (c) 2003-2008 Fabrice Bellard >> >> Is it linux kvm bug or CPU bug? How can I fix that? >> >> I can reproduce the bug in one-two days. Is it possible to enable >> deeper debug for the issue? >> >> Thanks >> > -- With best regards, Pavel S. Shirshov ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: KVM: unknown exit, hardware reason 31 2015-07-30 22:51 ` Pavel Shirshov @ 2015-07-31 1:42 ` Xiao Guangrong 0 siblings, 0 replies; 4+ messages in thread From: Xiao Guangrong @ 2015-07-31 1:42 UTC (permalink / raw) To: Pavel Shirshov; +Cc: kvm On 07/31/2015 06:51 AM, Pavel Shirshov wrote: > Hi Xiao, > > I've been testing your patch since Jul, 27. Everything works perfectly > with it. There are no EPT messages in dmesg output anymore. Also all > VMs work as expected: no paused VMs, no "code 31" error. Let me know > if you need any additional information. Thanks for your test, Pavel. I will post the fix out asap. ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2015-07-31 1:48 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-07-24 19:25 KVM: unknown exit, hardware reason 31 Pavel Shirshov 2015-07-27 3:23 ` Xiao Guangrong 2015-07-30 22:51 ` Pavel Shirshov 2015-07-31 1:42 ` Xiao Guangrong
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.