From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Kiszka Subject: Re: Nested VMX - L1 hangs on running L2 Date: Wed, 20 Jul 2011 09:58:47 +0200 Message-ID: <4E268AB7.3010205@web.de> References: <20110708184053.GA23958@stratus.com> <20110718182628.GB5324@amt.cnet> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig5D2D60E7AA731CAC36232872" Cc: Bandan Das , KVM Mailing List , Nadav Har'El , Zachary Amsden To: Marcelo Tosatti Return-path: Received: from fmmailgate02.web.de ([217.72.192.227]:46884 "EHLO fmmailgate02.web.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750846Ab1GTH6x (ORCPT ); Wed, 20 Jul 2011 03:58:53 -0400 In-Reply-To: <20110718182628.GB5324@amt.cnet> Sender: kvm-owner@vger.kernel.org List-ID: This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig5D2D60E7AA731CAC36232872 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 2011-07-18 20:26, Marcelo Tosatti wrote: >=20 > On Fri, Jul 08, 2011 at 02:40:53PM -0400, Bandan Das wrote: >> I have already discussed this a bit with Nadav but hoping someone=20 >> else has any other ideas/clues/suggestions/comments. With recent versi= ons of the=20 >> kernel (The last I tried is 3.0-rc5 with nVMX patches already merged),= my L1 guest=20 >> always hangs when I start L2.=20 >> >> My setup : The host, L1 and L2 all are FC15 with the host running 3.0-= rc5. When L1 is up=20 >> and running, I start L2 from L1. Within a minute or two, both L1 and L= 2 hang. Although, if >> if I run tracing on the host, I see : >> >> ... >> qemu-kvm-19756 [013] 153774.856178: kvm_exit: reason APIC_ACCESS rip 0= xffffffff81025098 info 1380 0 >> qemu-kvm-19756 [013] 153774.856189: kvm_exit: reason VMREAD rip 0xffff= ffffa00d5127 info 0 0 >> qemu-kvm-19756 [013] 153774.856191: kvm_exit: reason VMREAD rip 0xffff= ffffa00d5127 info 0 0 >> ... >> >> My point being that I only see kvm_exit messages but no kvm_entry. Doe= s this mean that the VCPUs >> are somehow stuck in L2 ? >> >> Anyway, since this setup was running fine for me on older kernels, and= I couldn't >> identify any significant changes in nVMX, I sifted through the other K= VM changes and found this : >> >> -- >> commit 1aa8ceef0312a6aae7dd863a120a55f1637b361d >> Author: Nikola Ciprich >> Date: Wed Mar 9 23:36:51 2011 +0100 >> >> KVM: fix kvmclock regression due to missing clock update >> =20 >> commit 387b9f97750444728962b236987fbe8ee8cc4f8c moved kvm_request_= guest_time_update(vcpu), >> breaking 32bit SMP guests using kvm-clock. Fix this by moving (new= ) clock update function >> to proper place. >> =20 >> Signed-off-by: Nikola Ciprich >> Acked-by: Zachary Amsden >> Signed-off-by: Avi Kivity >> >> index 01f08a6..f1e4025 100644 (file) >> --- a/arch/x86/kvm/x86.c >> +++ b/arch/x86/kvm/x86.c >> @@ -2127,8 +2127,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, i= nt cpu) >> if (check_tsc_unstable()) { >> kvm_x86_ops->adjust_tsc_offset(vcpu, -tsc_delt= a); >> vcpu->arch.tsc_catchup =3D 1; >> - kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); >> } >> + kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); >> if (vcpu->cpu !=3D cpu) >> kvm_migrate_timers(vcpu); >> vcpu->cpu =3D cpu; >> -- >> >> If I revert this change, my L1/L2 guests run fine. This ofcourse, just= hides the bug >> because on my machine, check_tsc_unstable() returns false. >> >> I found out from Nadav that when KVM decides to run L2, it will write = >> vmcs01->tsc_offset + vmcs12->tsc_offset to the active TSC_OFFSET which= seems right. >> But I verified that, if instead, I just write=20 >> vmcs01->tsc_offset to TSC_OFFSET in prepare_vmcs02(), I don't see the = bug anymore. >> >> Not sure where to go from here. I would appreciate if any one has any = ideas. >> >> >> Bandan >=20 > Using guests TSC value when performing TSC adjustments is wrong. Can > you please try the following patch, which skips TSC adjustments if > vcpu is in guest mode. >=20 > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c > index 2b76ae3..44c90d1 100644 > --- a/arch/x86/kvm/x86.c > +++ b/arch/x86/kvm/x86.c > @@ -1096,6 +1096,9 @@ static int kvm_guest_time_update(struct kvm_vcpu = *v) > s64 kernel_ns, max_kernel_ns; > u64 tsc_timestamp; > =20 > + if (is_guest_mode(v)) > + return 0; > + > /* Keep irq disabled to prevent changes to the clock */ > local_irq_save(flags); > kvm_get_msr(v, MSR_IA32_TSC, &tsc_timestamp); > @@ -2214,6 +2217,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, in= t cpu) > tsc_delta =3D !vcpu->arch.last_guest_tsc ? 0 : > tsc - vcpu->arch.last_guest_tsc; > =20 > + if (is_guest_mode(vcpu)) > + tsc_delta =3D 0; > + > if (tsc_delta < 0) > mark_tsc_unstable("KVM discovered backwards TSC"); > if (check_tsc_unstable()) { > @@ -2234,7 +2240,8 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) > { > kvm_x86_ops->vcpu_put(vcpu); > kvm_put_guest_fpu(vcpu); > - kvm_get_msr(vcpu, MSR_IA32_TSC, &vcpu->arch.last_guest_tsc); > + if (!is_guest_mode(vcpu)) > + kvm_get_msr(vcpu, MSR_IA32_TSC, &vcpu->arch.last_guest_tsc); > } > =20 > static int is_efer_nx(void) > @@ -5717,7 +5724,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu= ) > if (hw_breakpoint_active()) > hw_breakpoint_restore(); > =20 > - kvm_get_msr(vcpu, MSR_IA32_TSC, &vcpu->arch.last_guest_tsc); > + if (!is_guest_mode(vcpu)) > + kvm_get_msr(vcpu, MSR_IA32_TSC, &vcpu->arch.last_guest_tsc); > =20 > vcpu->mode =3D OUTSIDE_GUEST_MODE; > smp_wmb(); That unfortunately does not fix the L1 lockups I get here - unless I confine L1 to a single CPU. It looks like (don't have all symbols for the guest kernel ATM) that we are stuck in processing a timer IRQ. Jan --------------enig5D2D60E7AA731CAC36232872 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4miroACgkQitSsb3rl5xT3xgCgoNJU1PV01aLI4hmAFRbF78Zu 9IAAn2tVFo9LezELBi2oQeKZjpqpt7f8 =fKMK -----END PGP SIGNATURE----- --------------enig5D2D60E7AA731CAC36232872--