kvm-27 kernel oops

kvm-27 kernel oops

[Jon]

Hi,

I just tried kvm-27 with the following result when starting a vm. 
-no-kvm works, as does kvm-26. No other changes to my set up:

BUG: unable to handle kernel NULL pointer dereference at virtual address 00000024
 printing eip:
f8d16ee7
*pde = 00000000
Oops: 0002 [#1]
PREEMPT SMP 
Modules linked in: kvm_intel kvm i915 drm acpi_cpufreq freq_table rfcomm l2cap bluetooth button ac battery iptable_raw xt_comment xt_policy xt_multiport ipt_TTL ipt_ttl ipt_TOS ipt_tos ipt_REJECT ipt_recent ipt_owner ipt_LOG ipt_iprange ipt_ECN ipt_ecn ipt_ah ipt_addrtype xt_tcpmss xt_pkttype xt_NFQUEUE xt_NFLOG xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_hashlimit xt_dccp xt_conntrack xt_CONNMARK xt_connmark xt_CLASSIFY xt_tcpudp xt_state nf_conntrack_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables tun bridge llc dock thinkpad_acpi hwmon backlight loop mmc_block snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_timer sdhci snd mmc_core psmouse yenta_socket rsrc_nonstatic i2c_i801 pcmcia_core i2c_core intel_agp agpgart soundcore snd_page_alloc evdev sha256 usbhid hid ehci_hcd usbcore e1000 sd_mod thermal processor fan
CPU:    0
EIP:    0060:[<f8d16ee7>]    Not tainted VLI
EFLAGS: 00010206   (2.6.22-rc3-20070531-1 #1)
EIP is at mmu_free_roots+0x47/0x90 [kvm]
eax: 00000000   ebx: 00000000   ecx: 32560001   edx: 0064ac00
esi: 00000008   edi: f229109c   ebp: f2290000   esp: f2527e98
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Process qemu (pid: 6147, ti=f2526000 task=f2874000 task.ti=f2526000)
Stack: f229109c fffffff5 00000000 f8d16fb1 f229109c f8d18128 f229109c f8d14d78 
       00000000 00000000 f2527ed8 c034ea98 00000020 00100000 00000010 00000020 
       00000002 f2290068 00000020 000fffe0 00000080 f2290040 00000002 c17f18e0 
Call Trace:
 [<f8d16fb1>] destroy_kvm_mmu+0x21/0x40 [kvm]
 [<f8d18128>] kvm_mmu_reset_context+0x8/0x30 [kvm]
 [<f8d14d78>] kvm_vm_ioctl+0x758/0x830 [kvm]
 [<c01359e6>] __atomic_notifier_call_chain+0x26/0x50
 [<f8d14620>] kvm_vm_ioctl+0x0/0x830 [kvm]
 [<c018233b>] do_ioctl+0x2b/0x90
 [<c011cf33>] do_page_fault+0x333/0x620
 [<c01823fc>] vfs_ioctl+0x5c/0x290
 [<c018266d>] sys_ioctl+0x3d/0x70
 [<c0104182>] sysenter_past_esp+0x5f/0x85
 [<c02d0000>] packet_setsockopt+0x2b0/0x3b0
 =======================
Code: 30 89 da 09 ca 74 2d 89 c8 89 da 81 e2 ff ff 0f 00 25 00 f0 ff ff 0f ac d0 0c c1 ea 0c 89 c2 a1 00 b9 3c c0 c1 e2 05 8b 44 02 0c <ff> 48 24 8b 87 10 01 00 00 b9 ff ff ff ff c7 04 30 ff ff ff ff 
EIP: [<f8d16ee7>] mmu_free_roots+0x47/0x90 [kvm] SS:ESP 0068:f2527e98
note: qemu[6147] exited with preempt_count 1
BUG: scheduling while atomic: qemu/0x00000001/6147
 [<c02d1990>] __sched_text_start+0x4f0/0x950
 [<c015a36d>] __pagevec_free+0x1d/0x30
 [<c015cd27>] release_pages+0x137/0x160
 [<c02d2ad2>] __mutex_lock_slowpath+0x52/0x90
 [<c02d292a>] mutex_lock+0xa/0x10
 [<f8d13e0b>] vcpu_load+0xb/0x20 [kvm]
 [<f8d14481>] kvm_free_vcpu+0x11/0x60 [kvm]
 [<f8d14e9a>] kvm_destroy_vm+0x4a/0x80 [kvm]
 [<f8d15138>] kvm_vm_release+0x8/0x10 [kvm]
 [<c01779c7>] __fput+0x97/0x160
 [<f8d13df0>] kvm_vcpu_release+0x10/0x20 [kvm]
 [<c01779c7>] __fput+0x97/0x160
 [<c0174e09>] filp_close+0x49/0x80
 [<c012ab5c>] put_files_struct+0x9c/0xc0
 [<c012bde9>] do_exit+0x129/0x7b0
 [<c0105840>] die+0x250/0x260
 [<c011cee6>] do_page_fault+0x2e6/0x620
 [<c011cc00>] do_page_fault+0x0/0x620
 [<c02d3c72>] error_code+0x72/0x78
 [<f8d16ee7>] mmu_free_roots+0x47/0x90 [kvm]
 [<f8d16fb1>] destroy_kvm_mmu+0x21/0x40 [kvm]
 [<f8d18128>] kvm_mmu_reset_context+0x8/0x30 [kvm]
 [<f8d14d78>] kvm_vm_ioctl+0x758/0x830 [kvm]
 [<c01359e6>] __atomic_notifier_call_chain+0x26/0x50
 [<f8d14620>] kvm_vm_ioctl+0x0/0x830 [kvm]
 [<c018233b>] do_ioctl+0x2b/0x90
 [<c011cf33>] do_page_fault+0x333/0x620
 [<c01823fc>] vfs_ioctl+0x5c/0x290
 [<c018266d>] sys_ioctl+0x3d/0x70
 [<c0104182>] sysenter_past_esp+0x5f/0x85
 [<c02d0000>] packet_setsockopt+0x2b0/0x3b0
 =======================

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: kvm-27 kernel oops

[Avi Kivity]

[-- Attachment #1: Type: text/plain, Size: 339 bytes --]

Jon wrote:
> Hi,
>
> I just tried kvm-27 with the following result when starting a vm. 
> -no-kvm works, as does kvm-26. No other changes to my set up:
>
>   

This was fixed in kvm.git, use the attached patch (cd kernel/, and apply
with patch -p3).

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


[-- Attachment #2: lazy-cr3.patch --]
[-- Type: text/x-patch, Size: 4973 bytes --]

commit 4b82b37a35a085a07d9ed84efee06c69655fd3d1
Author: Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>
Date:   Mon Jun 4 15:58:30 2007 +0300

    KVM: Lazy guest cr3 switching
    
    Switch guest paging context may require us to allocate memory, which
    might fail.  Instead of wiring up error paths everywhere, make context
    switching lazy and actually do the switch before the next guest entry,
    where we can return an error if allocation fails.
    
    Signed-off-by: Avi Kivity <avi-atKUWr5tajBWk0Htik3J/w@public.gmane.org>

diff --git a/drivers/kvm/kvm.h b/drivers/kvm/kvm.h
index 0632d0b..0fdd5a6 100644
--- a/drivers/kvm/kvm.h
+++ b/drivers/kvm/kvm.h
@@ -543,6 +543,8 @@ void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa,
 		       const u8 *old, const u8 *new, int bytes);
 int kvm_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva);
 void kvm_mmu_free_some_pages(struct kvm_vcpu *vcpu);
+int kvm_mmu_load(struct kvm_vcpu *vcpu);
+void kvm_mmu_unload(struct kvm_vcpu *vcpu);
 
 int kvm_hypercall(struct kvm_vcpu *vcpu, struct kvm_run *run);
 
@@ -554,6 +556,14 @@ static inline int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gva_t gva,
 	return vcpu->mmu.page_fault(vcpu, gva, error_code);
 }
 
+static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu)
+{
+	if (likely(vcpu->mmu.root_hpa != INVALID_PAGE))
+		return 0;
+
+	return kvm_mmu_load(vcpu);
+}
+
 static inline int is_long_mode(struct kvm_vcpu *vcpu)
 {
 #ifdef CONFIG_X86_64
diff --git a/drivers/kvm/mmu.c b/drivers/kvm/mmu.c
index 283df03..5915d7a 100644
--- a/drivers/kvm/mmu.c
+++ b/drivers/kvm/mmu.c
@@ -949,9 +949,7 @@ static int nonpaging_init_context(struct kvm_vcpu *vcpu)
 	context->free = nonpaging_free;
 	context->root_level = 0;
 	context->shadow_root_level = PT32E_ROOT_LEVEL;
-	mmu_alloc_roots(vcpu);
-	ASSERT(VALID_PAGE(context->root_hpa));
-	kvm_arch_ops->set_cr3(vcpu, context->root_hpa);
+	context->root_hpa = INVALID_PAGE;
 	return 0;
 }
 
@@ -965,11 +963,6 @@ static void paging_new_cr3(struct kvm_vcpu *vcpu)
 {
 	pgprintk("%s: cr3 %lx\n", __FUNCTION__, vcpu->cr3);
 	mmu_free_roots(vcpu);
-	if (unlikely(vcpu->kvm->n_free_mmu_pages < KVM_MIN_FREE_MMU_PAGES))
-		kvm_mmu_free_some_pages(vcpu);
-	mmu_alloc_roots(vcpu);
-	kvm_mmu_flush_tlb(vcpu);
-	kvm_arch_ops->set_cr3(vcpu, vcpu->mmu.root_hpa);
 }
 
 static void inject_page_fault(struct kvm_vcpu *vcpu,
@@ -1003,10 +996,7 @@ static int paging64_init_context_common(struct kvm_vcpu *vcpu, int level)
 	context->free = paging_free;
 	context->root_level = level;
 	context->shadow_root_level = level;
-	mmu_alloc_roots(vcpu);
-	ASSERT(VALID_PAGE(context->root_hpa));
-	kvm_arch_ops->set_cr3(vcpu, context->root_hpa |
-		    (vcpu->cr3 & (CR3_PCD_MASK | CR3_WPT_MASK)));
+	context->root_hpa = INVALID_PAGE;
 	return 0;
 }
 
@@ -1025,10 +1015,7 @@ static int paging32_init_context(struct kvm_vcpu *vcpu)
 	context->free = paging_free;
 	context->root_level = PT32_ROOT_LEVEL;
 	context->shadow_root_level = PT32E_ROOT_LEVEL;
-	mmu_alloc_roots(vcpu);
-	ASSERT(VALID_PAGE(context->root_hpa));
-	kvm_arch_ops->set_cr3(vcpu, context->root_hpa |
-		    (vcpu->cr3 & (CR3_PCD_MASK | CR3_WPT_MASK)));
+	context->root_hpa = INVALID_PAGE;
 	return 0;
 }
 
@@ -1042,7 +1029,6 @@ static int init_kvm_mmu(struct kvm_vcpu *vcpu)
 	ASSERT(vcpu);
 	ASSERT(!VALID_PAGE(vcpu->mmu.root_hpa));
 
-	mmu_topup_memory_caches(vcpu);
 	if (!is_paging(vcpu))
 		return nonpaging_init_context(vcpu);
 	else if (is_long_mode(vcpu))
@@ -1064,16 +1050,31 @@ static void destroy_kvm_mmu(struct kvm_vcpu *vcpu)
 
 int kvm_mmu_reset_context(struct kvm_vcpu *vcpu)
 {
+	destroy_kvm_mmu(vcpu);
+	return init_kvm_mmu(vcpu);
+}
+
+int kvm_mmu_load(struct kvm_vcpu *vcpu)
+{
 	int r;
 
-	destroy_kvm_mmu(vcpu);
-	r = init_kvm_mmu(vcpu);
-	if (r < 0)
-		goto out;
+	spin_lock(&vcpu->kvm->lock);
 	r = mmu_topup_memory_caches(vcpu);
+	if (r)
+		goto out;
+	mmu_alloc_roots(vcpu);
+	kvm_arch_ops->set_cr3(vcpu, vcpu->mmu.root_hpa);
+	kvm_mmu_flush_tlb(vcpu);
 out:
+	spin_unlock(&vcpu->kvm->lock);
 	return r;
 }
+EXPORT_SYMBOL_GPL(kvm_mmu_load);
+
+void kvm_mmu_unload(struct kvm_vcpu *vcpu)
+{
+	mmu_free_roots(vcpu);
+}
 
 static void mmu_pte_write_zap_pte(struct kvm_vcpu *vcpu,
 				  struct kvm_mmu_page *page,
diff --git a/drivers/kvm/svm.c b/drivers/kvm/svm.c
index b621403..ed33f59 100644
--- a/drivers/kvm/svm.c
+++ b/drivers/kvm/svm.c
@@ -1482,6 +1482,10 @@ static int svm_vcpu_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 	int r;
 
 again:
+	r = kvm_mmu_reload(vcpu);
+	if (unlikely(r))
+		return r;
+
 	if (!vcpu->mmio_read_completed)
 		do_interrupt_requests(vcpu, kvm_run);
 
diff --git a/drivers/kvm/vmx.c b/drivers/kvm/vmx.c
index 09e3609..a499989 100644
--- a/drivers/kvm/vmx.c
+++ b/drivers/kvm/vmx.c
@@ -1987,6 +1987,10 @@ again:
 	vmx_save_host_state(vcpu);
 	kvm_load_guest_fpu(vcpu);
 
+	r = kvm_mmu_reload(vcpu);
+	if (unlikely(r))
+		goto out;
+
 	/*
 	 * Loading guest fpu may have cleared host cr0.ts
 	 */

[-- Attachment #3: Type: text/plain, Size: 286 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[-- Attachment #4: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: kvm-27 kernel oops

[Jon]

Hi Avi,

On Thu, Jun 07, 2007 at 08:15:29AM +0300, Avi Kivity wrote:
> This was fixed in kvm.git, use the attached patch (cd kernel/, and apply
> with patch -p3).

Applied, but no go. Managed to save this before things started dying:

BUG: unable to handle kernel paging request at virtual address 30312030
 printing eip:
c01712e9
*pde = 00000000
Oops: 0000 [#1]
SMP 
Modules linked in: kvm_intel kvm i915 drm acpi_cpufreq freq_table rfcomm l2cap bluetooth button ac battery iptable_raw xt_comment xt_policy xt_multiport ipt_TTL ipt_ttl ipt_TOS ipt_tos ipt_REJECT ipt_recent ipt_owner ipt_LOG ipt_iprange ipt_ECN ipt_ecn ipt_ah ipt_addrtype xt_tcpmss xt_pkttype xt_NFQUEUE xt_NFLOG xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_hashlimit xt_dccp xt_conntrack xt_CONNMARK xt_connmark xt_CLASSIFY xt_tcpudp xt_state nf_conntrack_ipv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables tun bridge llc dock thinkpad_acpi hwmon backlight loop mmc_block snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd sdhci i2c_i801 i2c_core yenta_socket rsrc_nonstatic pcmcia_core mmc_core intel_agp agpgart soundcore snd_page_alloc psmouse evdev sha256 sg sr_mod cdrom usb_storage usbhid hid ehci_hcd usbcore e1000 sd_mod thermal processor fan
CPU:    0
EIP:    0060:[<c01712e9>]    Not tainted VLI
EFLAGS: 00010006   (2.6.22-rc4-20070607-1 #1)
EIP is at kmem_cache_alloc+0x39/0x70
eax: 00000000   ebx: 00000282   ecx: c159e080   edx: 30312030
esi: c03655a0   edi: 000000d0   ebp: f7530000   esp: f7531f64
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Process powersaved (pid: 4809, ti=f7530000 task=f74db900 task.ti=f7530000)
Stack: 00005973 00000000 080cf840 ffffff9c fffffff4 c017cdd8 00000000 ffffff9c 
       00000000 f7530000 c017342e 0e6dc36e 46679a17 0e6dc36e 080cf840 00001000 
       00000000 f7530000 c017352c 0e6dc36e c01040ba 080cf840 00000000 0e6dc36e 
Call Trace:
 [<c017cdd8>] getname+0x28/0xe0
 [<c017342e>] do_sys_open+0x1e/0xe0
 [<c017352c>] sys_open+0x1c/0x20
 [<c01040ba>] sysenter_past_esp+0x5f/0x85
 =======================
Code: d7 8b 54 24 14 89 5c 24 08 9c 5b fa 64 a1 08 c0 39 c0 8b 8c 86 90 00 00 00 85 c9 74 28 8b 41 0c 85 c0 74 21 8b 51 0c 0f b7 41 0a <8b> 04 82 89 41 0c 53 9d 8b 5c 24 08 89 d0 8b 74 24 0c 8b 7c 24 
EIP: [<c01712e9>] kmem_cache_alloc+0x39/0x70 SS:ESP 0068:f7531f64

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: kvm-27 kernel oops

[Avi Kivity]

Jon wrote:
> Hi Avi,
>
> On Thu, Jun 07, 2007 at 08:15:29AM +0300, Avi Kivity wrote:
>   
>> This was fixed in kvm.git, use the attached patch (cd kernel/, and apply
>> with patch -p3).
>>     
>
> Applied, but no go. Managed to save this before things started dying:
>
> BUG: unable to handle kernel paging request at virtual address 30312030
>   

This looks unrelated.  Does it happen immediately after starting qemu, 
or during, or afterwards, or ...?

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: kvm-27 kernel oops

[Jon]

On Thu, Jun 07, 2007 at 11:25:07AM +0300, Avi Kivity wrote:
> Jon wrote:
> >Hi Avi,
> >
> >On Thu, Jun 07, 2007 at 08:15:29AM +0300, Avi Kivity wrote:
> >  
> >>This was fixed in kvm.git, use the attached patch (cd kernel/, and apply
> >>with patch -p3).
> >>    
> >
> >Applied, but no go. Managed to save this before things started dying:
> >
> >BUG: unable to handle kernel paging request at virtual address 30312030
> >  
> 
> This looks unrelated.  Does it happen immediately after starting qemu, 
> or during, or afterwards, or ...?

It happens immediately. The SDL window shows but is unresponsive, the 
CPU is pegged to 100%, and everything starts dying.

However, I just tried it with the regular Debian 2.6.18 kernel and 
kvm-27 works fine with that. I had been messing with a 2.6.22-rc when I 
got the oops...

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: kvm-27 kernel oops

[Avi Kivity]

Jon wrote:
> On Thu, Jun 07, 2007 at 11:25:07AM +0300, Avi Kivity wrote:
>   
>> Jon wrote:
>>     
>>> Hi Avi,
>>>
>>> On Thu, Jun 07, 2007 at 08:15:29AM +0300, Avi Kivity wrote:
>>>  
>>>       
>>>> This was fixed in kvm.git, use the attached patch (cd kernel/, and apply
>>>> with patch -p3).
>>>>    
>>>>         
>>> Applied, but no go. Managed to save this before things started dying:
>>>
>>> BUG: unable to handle kernel paging request at virtual address 30312030
>>>  
>>>       
>> This looks unrelated.  Does it happen immediately after starting qemu, 
>> or during, or afterwards, or ...?
>>     
>
> It happens immediately. The SDL window shows but is unresponsive, the 
> CPU is pegged to 100%, and everything starts dying.
>
> However, I just tried it with the regular Debian 2.6.18 kernel and 
> kvm-27 works fine with that. I had been messing with a 2.6.22-rc when I 
> got the oops...
>   

Wait.  Does the problem occur with a clean 2.6.22-rc?  We don't want 
2.6.22 to have such bugs when it is release, do we?

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/