From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Kiszka Subject: Re: Nested VMX - L1 hangs on running L2 Date: Wed, 20 Jul 2011 18:19:19 +0200 Message-ID: <4E270007.4050900@web.de> References: <20110708184053.GA23958@stratus.com> <20110718182628.GB5324@amt.cnet> <4E268AB7.3010205@web.de> <20110720161235.GA17055@amt.cnet> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigB934A2621B151DF007C1EB8F" Cc: Bandan Das , KVM Mailing List , Nadav Har'El , Zachary Amsden To: Marcelo Tosatti Return-path: Received: from fmmailgate03.web.de ([217.72.192.234]:43363 "EHLO fmmailgate03.web.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752346Ab1GTQT0 (ORCPT ); Wed, 20 Jul 2011 12:19:26 -0400 In-Reply-To: <20110720161235.GA17055@amt.cnet> Sender: kvm-owner@vger.kernel.org List-ID: This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigB934A2621B151DF007C1EB8F Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 2011-07-20 18:12, Marcelo Tosatti wrote: > On Wed, Jul 20, 2011 at 09:58:47AM +0200, Jan Kiszka wrote: >> On 2011-07-18 20:26, Marcelo Tosatti wrote: >>> >>> On Fri, Jul 08, 2011 at 02:40:53PM -0400, Bandan Das wrote: >>>> I have already discussed this a bit with Nadav but hoping someone=20 >>>> else has any other ideas/clues/suggestions/comments. With recent ver= sions of the=20 >>>> kernel (The last I tried is 3.0-rc5 with nVMX patches already merged= ), my L1 guest=20 >>>> always hangs when I start L2.=20 >>>> >>>> My setup : The host, L1 and L2 all are FC15 with the host running 3.= 0-rc5. When L1 is up=20 >>>> and running, I start L2 from L1. Within a minute or two, both L1 and= L2 hang. Although, if >>>> if I run tracing on the host, I see : >>>> >>>> ... >>>> qemu-kvm-19756 [013] 153774.856178: kvm_exit: reason APIC_ACCESS rip= 0xffffffff81025098 info 1380 0 >>>> qemu-kvm-19756 [013] 153774.856189: kvm_exit: reason VMREAD rip 0xff= ffffffa00d5127 info 0 0 >>>> qemu-kvm-19756 [013] 153774.856191: kvm_exit: reason VMREAD rip 0xff= ffffffa00d5127 info 0 0 >>>> ... >>>> >>>> My point being that I only see kvm_exit messages but no kvm_entry. D= oes this mean that the VCPUs >>>> are somehow stuck in L2 ? >>>> >>>> Anyway, since this setup was running fine for me on older kernels, a= nd I couldn't >>>> identify any significant changes in nVMX, I sifted through the other= KVM changes and found this : >>>> >>>> -- >>>> commit 1aa8ceef0312a6aae7dd863a120a55f1637b361d >>>> Author: Nikola Ciprich >>>> Date: Wed Mar 9 23:36:51 2011 +0100 >>>> >>>> KVM: fix kvmclock regression due to missing clock update >>>> =20 >>>> commit 387b9f97750444728962b236987fbe8ee8cc4f8c moved kvm_reques= t_guest_time_update(vcpu), >>>> breaking 32bit SMP guests using kvm-clock. Fix this by moving (n= ew) clock update function >>>> to proper place. >>>> =20 >>>> Signed-off-by: Nikola Ciprich >>>> Acked-by: Zachary Amsden >>>> Signed-off-by: Avi Kivity >>>> >>>> index 01f08a6..f1e4025 100644 (file) >>>> --- a/arch/x86/kvm/x86.c >>>> +++ b/arch/x86/kvm/x86.c >>>> @@ -2127,8 +2127,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu,= int cpu) >>>> if (check_tsc_unstable()) { >>>> kvm_x86_ops->adjust_tsc_offset(vcpu, -tsc_de= lta); >>>> vcpu->arch.tsc_catchup =3D 1; >>>> - kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu)= ; >>>> } >>>> + kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); >>>> if (vcpu->cpu !=3D cpu) >>>> kvm_migrate_timers(vcpu); >>>> vcpu->cpu =3D cpu; >>>> -- >>>> >>>> If I revert this change, my L1/L2 guests run fine. This ofcourse, ju= st hides the bug >>>> because on my machine, check_tsc_unstable() returns false. >>>> >>>> I found out from Nadav that when KVM decides to run L2, it will writ= e=20 >>>> vmcs01->tsc_offset + vmcs12->tsc_offset to the active TSC_OFFSET whi= ch seems right. >>>> But I verified that, if instead, I just write=20 >>>> vmcs01->tsc_offset to TSC_OFFSET in prepare_vmcs02(), I don't see th= e bug anymore. >>>> >>>> Not sure where to go from here. I would appreciate if any one has an= y ideas. >>>> >>>> >>>> Bandan >>> >>> Using guests TSC value when performing TSC adjustments is wrong. Can >>> you please try the following patch, which skips TSC adjustments if >>> vcpu is in guest mode. >>> >>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c >>> index 2b76ae3..44c90d1 100644 >>> --- a/arch/x86/kvm/x86.c >>> +++ b/arch/x86/kvm/x86.c >>> @@ -1096,6 +1096,9 @@ static int kvm_guest_time_update(struct kvm_vcp= u *v) >>> s64 kernel_ns, max_kernel_ns; >>> u64 tsc_timestamp; >>> =20 >>> + if (is_guest_mode(v)) >>> + return 0; >>> + >>> /* Keep irq disabled to prevent changes to the clock */ >>> local_irq_save(flags); >>> kvm_get_msr(v, MSR_IA32_TSC, &tsc_timestamp); >>> @@ -2214,6 +2217,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, = int cpu) >>> tsc_delta =3D !vcpu->arch.last_guest_tsc ? 0 : >>> tsc - vcpu->arch.last_guest_tsc; >>> =20 >>> + if (is_guest_mode(vcpu)) >>> + tsc_delta =3D 0; >>> + >>> if (tsc_delta < 0) >>> mark_tsc_unstable("KVM discovered backwards TSC"); >>> if (check_tsc_unstable()) { >>> @@ -2234,7 +2240,8 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) >>> { >>> kvm_x86_ops->vcpu_put(vcpu); >>> kvm_put_guest_fpu(vcpu); >>> - kvm_get_msr(vcpu, MSR_IA32_TSC, &vcpu->arch.last_guest_tsc); >>> + if (!is_guest_mode(vcpu)) >>> + kvm_get_msr(vcpu, MSR_IA32_TSC, &vcpu->arch.last_guest_tsc); >>> } >>> =20 >>> static int is_efer_nx(void) >>> @@ -5717,7 +5724,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vc= pu) >>> if (hw_breakpoint_active()) >>> hw_breakpoint_restore(); >>> =20 >>> - kvm_get_msr(vcpu, MSR_IA32_TSC, &vcpu->arch.last_guest_tsc); >>> + if (!is_guest_mode(vcpu)) >>> + kvm_get_msr(vcpu, MSR_IA32_TSC, &vcpu->arch.last_guest_tsc); >>> =20 >>> vcpu->mode =3D OUTSIDE_GUEST_MODE; >>> smp_wmb(); >> >> That unfortunately does not fix the L1 lockups I get here - unless I >> confine L1 to a single CPU. It looks like (don't have all symbols for >> the guest kernel ATM) that we are stuck in processing a timer IRQ. >> >> Jan >=20 > Is L1 using kvmclock?=20 Yes, it's a standard 3.0-rc7 SUSE kernel. Disabling it seems to help on first glance. Jan --------------enigB934A2621B151DF007C1EB8F Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4nAAkACgkQitSsb3rl5xR74ACgnxsO2LLLqaza8Vi04hS+6xHi vo0AoMYAzQQD5Qut3ZI1fXPi18TrBKJk =7Upe -----END PGP SIGNATURE----- --------------enigB934A2621B151DF007C1EB8F--