Oops on 2.6.23.11 with KVM 57

Oops on 2.6.23.11 with KVM 57

[Guido Guenther]

Hi,
I got the Oops below on an Dual CPU Dual-Core AMD Opteron(tm) Processor
2220 running a 32bit kernel. Kernel is 2.6.23.11, kvm version 57.
Doesn't happen all the time but usually right at VM startup. Seems to be
easier to trigger when starting up several VMs in row. Does it help to
post more of these as they show up?
Cheers,
 -- Guido

BUG: unable to handle kernel paging request<1>BUG: unable to handle kernel paging request at virtual address 52004605 
 printing eip: 
52004605 
*pdpt = 000000002f35d001 
*pde = 0000000000000000 
Oops: 0010 [#1] 
SMP  
Modules linked in: tun bitrev nfs nfsd exportfs lockd sunrpc ac lock_nolock lock_dlm gfs2 dlm configfs bridge llc kvm_amd kvm bonding ipmi_si ipmi_msghandler i2c_piix4 container dm_mirror dm_snapshot raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear md_mod dm_round_robin dm_emc dm_multipath dm_mod sd_mod qla2xxx firmware_class scsi_transport_fc scsi_mod bnx2 crc32 serverworks usbhid thermal processor fan 
BUG: unable to handle kernel paging request at virtual address 3131ce82 
 printing eip: 
3131ce82 
*pdpt = 000000002fcfa001 
*pde = 0000000000000000 
CPU:    3 
EIP:    0060:[<52004605>]    Not tainted VLI 
EFLAGS: 00010046   (2.6.23.11-aa-smp-64gb #1) 
EIP is at 0x52004605 
eax: 45811b60   ebx: c0f602c1   ecx: 00000003   edx: 00000000 
esi: c0f602c1   edi: ede20000   ebp: 00000000   esp: f08e9d70 
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068 
Process kvm (pid: 10396, ti=f08e8000 task=ef53d000 task.ti=efaea000) 
Stack: f8d498d8 00000090 f8d49f8e ef64a000 eff3c020 f8cb5be6 00000000 eff3c020  
       f8d4121e 00000000 00000097 10002800 00000000 f5611140 eff3c020 0000ae80  
       f8d3da32 00000002 c01759cd 00000000 f655d7d0 dfc213c0 00000000 df803a80  
Call Trace: 
 [<f8d498d8>] pic_update_irq+0x68/0x80 [kvm] 
 [<f8d49f8e>] kvm_pic_read_irq+0x6e/0xe0 [kvm] 
 [<f8cb5be6>] svm_intr_assist+0x66/0x100 [kvm_amd] 
 [<f8d4121e>] kvm_arch_vcpu_ioctl_run+0x8e/0x470 [kvm] 
 [<f8d3da32>] kvm_vcpu_ioctl+0x2a2/0x2f0 [kvm] 
 [<c01759cd>] flush_old_exec+0x43d/0x890 
 [<c011e5dc>] kmap_atomic+0x1c/0x30 
 [<c012b523>] current_fs_time+0x13/0x20 
 [<c0184dc7>] touch_atime+0x87/0x130 
 [<c01514f5>] do_generic_mapping_read+0x3c5/0x490 
 [<c0152d90>] generic_file_aio_read+0xd0/0x1e0 
 [<c0150a00>] file_read_actor+0x0/0x100 
 [<c0170ed7>] do_sync_read+0xc7/0x110 
 [<c013a180>] autoremove_wake_function+0x0/0x50 
 [<c015bfba>] kmap_high+0x1a/0x1b0 
 [<c015be5b>] page_address+0xcb/0xd0 
 [<f8d3d790>] kvm_vcpu_ioctl+0x0/0x2f0 [kvm] 
 [<c017c9fb>] do_ioctl+0x2b/0x90 
 [<c017cabc>] vfs_ioctl+0x5c/0x290 
 [<c0170e10>] do_sync_read+0x0/0x110 
 [<c017cd2d>] sys_ioctl+0x3d/0x70 
 [<c010416e>] sysenter_past_esp+0x5f/0x85 
 ======================= 
Code:  Bad EIP value. 
EIP: [<52004605>] 0x52004605 SS:ESP 0068:f08e9d70 
kernel tried to execute NX-protected page - exploit attempt? (uid: 65534) 
BUG: unable to handle kernel paging request at virtual address f08e9b00 
 printing eip: 
f08e9b00 
*pdpt = 0000000000004001 
*pde = 80000000308001e3 
*pte = 0000000000000000 
BUG: unable to handle kernel paging request at virtual address 01f56124 
 printing eip: 
c0172142 
*pdpt = 000000000044e001 
*pde = 0000000000000000 

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Oops on 2.6.23.11 with KVM 57

[Avi Kivity]

Guido Guenther wrote:
> Hi,
> I got the Oops below on an Dual CPU Dual-Core AMD Opteron(tm) Processor
> 2220 running a 32bit kernel. Kernel is 2.6.23.11, kvm version 57.
> Doesn't happen all the time but usually right at VM startup. Seems to be
> easier to trigger when starting up several VMs in row. 

What guest are you running?  smp or uni?

> Call Trace: 
>  [<f8d498d8>] pic_update_irq+0x68/0x80 [kvm] 
>  [<f8d49f8e>] kvm_pic_read_irq+0x6e/0xe0 [kvm] 
>   

Please do

  gdb /path/to/kvm.ko
  disassemble pic_update_irq

and post the results.  If you recompiled the module since getting the
oops, please generate a fresh oops.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Oops on 2.6.23.11 with KVM 57

[Guido Guenther]

On Sat, Dec 22, 2007 at 09:59:43PM +0200, Avi Kivity wrote:
> Guido Guenther wrote:
> > Hi,
> > I got the Oops below on an Dual CPU Dual-Core AMD Opteron(tm) Processor
> > 2220 running a 32bit kernel. Kernel is 2.6.23.11, kvm version 57.
> > Doesn't happen all the time but usually right at VM startup. Seems to be
> > easier to trigger when starting up several VMs in row. 
> 
> What guest are you running?  smp or uni?
Uniprocessor Linux, standard etch 2.6.18 kernel.

kvm commandline was:

-M pc -m 512 -smp 1 -monitor pty -boot c -hda /dev/mapper/foo -hdb
/dev/mapper/bar -net nic,macaddr=54:52:00:1b:9b:10,vlan=0 -net
tap,fd=11,script=,vlan=0 -usb -usbdevice tablet -vnc 127.0.0.1:0


> > Call Trace: 
> >  [<f8d498d8>] pic_update_irq+0x68/0x80 [kvm] 
> >  [<f8d49f8e>] kvm_pic_read_irq+0x6e/0xe0 [kvm] 
> >   
> 
> Please do
> 
>   gdb /path/to/kvm.ko
>   disassemble pic_update_irq
> 
> and post the results.  If you recompiled the module since getting the
> oops, please generate a fresh oops.

(gdb) disassemble pic_update_irq
Dump of assembler code for function pic_update_irq:
0x0000d870 <pic_update_irq+0>:  push   %ebx
0x0000d871 <pic_update_irq+1>:  mov    %eax,%ebx
0x0000d873 <pic_update_irq+3>:  lea    0x14(%eax),%eax
0x0000d876 <pic_update_irq+6>:  call   0xd7a0 <pic_get_irq>
0x0000d87b <pic_update_irq+11>: test   %eax,%eax
0x0000d87d <pic_update_irq+13>: js     0xd899 <pic_update_irq+41>
0x0000d87f <pic_update_irq+15>: testb  $0x4,0xe(%ebx)
0x0000d883 <pic_update_irq+19>: je     0xd8b1 <pic_update_irq+65>
0x0000d885 <pic_update_irq+21>: orb    $0x4,0x1(%ebx)
0x0000d889 <pic_update_irq+25>: orb    $0x4,(%ebx)
0x0000d88c <pic_update_irq+28>: testb  $0x4,0xe(%ebx)
0x0000d890 <pic_update_irq+32>: je     0xd8c2 <pic_update_irq+82>
0x0000d892 <pic_update_irq+34>: andb   $0xfb,0x1(%ebx)
0x0000d896 <pic_update_irq+38>: andb   $0xfb,(%ebx)
0x0000d899 <pic_update_irq+41>: mov    %ebx,%eax
0x0000d89b <pic_update_irq+43>: call   0xd7a0 <pic_get_irq>
0x0000d8a0 <pic_update_irq+48>: test   %eax,%eax
0x0000d8a2 <pic_update_irq+50>: js     0xd8d0 <pic_update_irq+96>
0x0000d8a4 <pic_update_irq+52>: mov    0x2c(%ebx),%eax
0x0000d8a7 <pic_update_irq+55>: mov    $0x1,%edx
0x0000d8ac <pic_update_irq+60>: call   *0x28(%ebx)
0x0000d8af <pic_update_irq+63>: pop    %ebx
0x0000d8b0 <pic_update_irq+64>: ret
0x0000d8b1 <pic_update_irq+65>: testb  $0x4,(%ebx)
0x0000d8b4 <pic_update_irq+68>: je     0xd8e0 <pic_update_irq+112>
0x0000d8b6 <pic_update_irq+70>: orb    $0x4,(%ebx)
0x0000d8b9 <pic_update_irq+73>: testb  $0x4,0xe(%ebx)
0x0000d8bd <pic_update_irq+77>: lea    0x0(%esi),%esi
0x0000d8c0 <pic_update_irq+80>: jne    0xd892 <pic_update_irq+34>
0x0000d8c2 <pic_update_irq+82>: andb   $0xfb,(%ebx)
0x0000d8c5 <pic_update_irq+85>: mov    %ebx,%eax
0x0000d8c7 <pic_update_irq+87>: call   0xd7a0 <pic_get_irq>
0x0000d8cc <pic_update_irq+92>: test   %eax,%eax
0x0000d8ce <pic_update_irq+94>: jns    0xd8a4 <pic_update_irq+52>
0x0000d8d0 <pic_update_irq+96>: mov    0x2c(%ebx),%eax
0x0000d8d3 <pic_update_irq+99>: xor    %edx,%edx
0x0000d8d5 <pic_update_irq+101>:        call   *0x28(%ebx)
0x0000d8d8 <pic_update_irq+104>:        pop    %ebx
0x0000d8d9 <pic_update_irq+105>:        ret
0x0000d8da <pic_update_irq+106>:        lea    0x0(%esi),%esi
0x0000d8e0 <pic_update_irq+112>:        orb    $0x4,0x1(%ebx)
0x0000d8e4 <pic_update_irq+116>:        orb    $0x4,(%ebx)
0x0000d8e7 <pic_update_irq+119>:        jmp    0xd8b9 <pic_update_irq+73>
0x0000d8e9 <pic_update_irq+121>:        lea    0x0(%esi),%esi
End of assembler dump.

Cheers,
 -- Guido

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Oops on 2.6.23.11 with KVM 57

[Avi Kivity]

Guido Guenther wrote:
> On Sat, Dec 22, 2007 at 09:59:43PM +0200, Avi Kivity wrote:
>   
>> Guido Guenther wrote:
>>     
>>> Hi,
>>> I got the Oops below on an Dual CPU Dual-Core AMD Opteron(tm) Processor
>>> 2220 running a 32bit kernel. Kernel is 2.6.23.11, kvm version 57.
>>> Doesn't happen all the time but usually right at VM startup. Seems to be
>>> easier to trigger when starting up several VMs in row. 
>>>       
>> What guest are you running?  smp or uni?
>>     
> Uniprocessor Linux, standard etch 2.6.18 kernel.
>
> kvm commandline was:
>
> -M pc -m 512 -smp 1 -monitor pty -boot c -hda /dev/mapper/foo -hdb
> /dev/mapper/bar -net nic,macaddr=54:52:00:1b:9b:10,vlan=0 -net
> tap,fd=11,script=,vlan=0 -usb -usbdevice tablet -vnc 127.0.0.1:0
>
>   

Looks like memory corruption; this was a call through a function pointer 
that pointed into userspace.

Please try kvm-58, that has a fix for preemption notifiers on AMD; this 
may solve the problem.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Oops on 2.6.23.11 with KVM 57

[Guido Guenther]

On Thu, Dec 27, 2007 at 12:34:58PM +0200, Avi Kivity wrote:
> Looks like memory corruption; this was a call through a function pointer 
> that pointed into userspace.
>
> Please try kvm-58, that has a fix for preemption notifiers on AMD; this may 
> solve the problem.
kvm-58 doesn't show this problem, thanks! The host runs stable now. 

I'm still seeing network stalls on large transfers in the guest though.
Ifup/ifdown'ing the interface in the guest gets things going again until
it stops again. There's nothing in dmesg of either host or guest. I still
have to check with different guest kernels and qemu network adapter
emulations though (current one is ne2k using a bridge on the host). The
problem has at least been around since kvm-52.

Cheers,
 -- Guido

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Oops on 2.6.23.11 with KVM 57

[Izik Eidus]

Guido Guenther wrote:
> On Thu, Dec 27, 2007 at 12:34:58PM +0200, Avi Kivity wrote:
>   
>> Looks like memory corruption; this was a call through a function pointer 
>> that pointed into userspace.
>>
>> Please try kvm-58, that has a fix for preemption notifiers on AMD; this may 
>> solve the problem.
>>     
> kvm-58 doesn't show this problem, thanks! The host runs stable now. 
>
> I'm still seeing network stalls on large transfers in the guest though.
> Ifup/ifdown'ing the interface in the guest gets things going again until
> it stops again. There's nothing in dmesg of either host or guest. I still
> have to check with different guest kernels and qemu network adapter
> emulations though (current one is ne2k using a bridge on the host). The
>
>   
there are known problems with ne2k, please rtl8139 and report if it work

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[PATCH] use rtl8139 as default network adapter

[Guido Guenther]

On Fri, Dec 28, 2007 at 10:53:53PM +0200, Izik Eidus wrote:
> Guido Guenther wrote:
[..snip..]
>> kvm-58 doesn't show this problem, thanks! The host runs stable now. 
>> I'm still seeing network stalls on large transfers in the guest though.
>> Ifup/ifdown'ing the interface in the guest gets things going again until
>> it stops again. There's nothing in dmesg of either host or guest. I still
>> have to check with different guest kernels and qemu network adapter
>> emulations though (current one is ne2k using a bridge on the host). The
>>
>>   
> there are known problems with ne2k, please rtl8139 and report if it work
rtl8139 works stable with the standard etch kernel I tested as guest,
thanks!  Given the fact that ne2k has known problems, wouldn't using
rtl8139 as default adaptor make sense:

diff --git a/qemu/hw/pc.c b/qemu/hw/pc.c
index 6c0a360..0d697cc 100644
--- a/qemu/hw/pc.c
+++ b/qemu/hw/pc.c
@@ -1005,7 +1005,7 @@ static void pc_init1(ram_addr_t ram_size, int vga_ram_size,
         nd = &nd_table[i];
         if (!nd->model) {
             if (pci_enabled) {
-                nd->model = "ne2k_pci";
+                nd->model = "rtl8139";
             } else {
                 nd->model = "ne2k_isa";
             }

Cheers,
 -- Guido

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH] use rtl8139 as default network adapter

[Avi Kivity]

Guido Guenther wrote:
> On Fri, Dec 28, 2007 at 10:53:53PM +0200, Izik Eidus wrote:
>   
>> Guido Guenther wrote:
>>     
> [..snip..]
>   
>>> kvm-58 doesn't show this problem, thanks! The host runs stable now. 
>>> I'm still seeing network stalls on large transfers in the guest though.
>>> Ifup/ifdown'ing the interface in the guest gets things going again until
>>> it stops again. There's nothing in dmesg of either host or guest. I still
>>> have to check with different guest kernels and qemu network adapter
>>> emulations though (current one is ne2k using a bridge on the host). The
>>>
>>>   
>>>       
>> there are known problems with ne2k, please rtl8139 and report if it work
>>     
> rtl8139 works stable with the standard etch kernel I tested as guest,
> thanks!  Given the fact that ne2k has known problems, wouldn't using
> rtl8139 as default adaptor make sense:
>
>   

Yes.  Applied, thanks.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[PATCH] use rtl8139 for bootp too and adjust docs accordingly

[Guido Guenther]

If we change the default nic, we should do so in all places. Without
this PXE booting is broken since we select the wrong pxe etherboot
image.
Cheers,
 -- Guido

diff --git a/qemu/qemu-doc.texi b/qemu/qemu-doc.texi
index 04529a1..accf360 100644
--- a/qemu/qemu-doc.texi
+++ b/qemu/qemu-doc.texi
@@ -498,7 +498,7 @@ Network options:
 
 @item -net nic[,vlan=@var{n}][,macaddr=@var{addr}][,model=@var{type}]
 Create a new Network Interface Card and connect it to VLAN @var{n} (@var{n}
-= 0 is the default). The NIC is an ne2k_pci by default on the PC
+= 0 is the default). The NIC is an rtl8139 by default on the PC
 target. Optionally, the MAC address can be changed. If no
 @option{-net} option is specified, a single NIC is created.
 Qemu can emulate several different models of network card.
diff --git a/qemu/vl.c b/qemu/vl.c
index b84ddd5..64f2709 100644
--- a/qemu/vl.c
+++ b/qemu/vl.c
@@ -9217,7 +9217,7 @@ int main(int argc, char **argv)
 	    char buf[1024];
             if (net_boot & (1 << i)) {
                 if (model == NULL)
-                    model = "ne2k_pci";
+                    model = "rtl8139";
                 snprintf(buf, sizeof(buf), "%s/pxe-%s.bin", bios_dir, model);
                 if (get_image_size(buf) > 0) {
                     if (nb_option_roms >= MAX_OPTION_ROMS) {

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Re: [PATCH] use rtl8139 for bootp too and adjust docs accordingly

[Avi Kivity]

Guido Guenther wrote:
> If we change the default nic, we should do so in all places. Without
> this PXE booting is broken since we select the wrong pxe etherboot
> image.
>   

Applied, thanks.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Re: Oops on 2.6.23.11 with KVM 57

[Yves Dorfsman]

On Fri, 28 Dec 2007, Guido Guenther wrote:

> I'm still seeing network stalls on large transfers in the guest though.
> Ifup/ifdown'ing the interface in the guest gets things going again until
> it stops again. There's nothing in dmesg of either host or guest. I still
> have to check with different guest kernels and qemu network adapter
> emulations though (current one is ne2k using a bridge on the host). The
> problem has at least been around since kvm-52.

Is this the same problem I was getting (mvstat shows 100% wait time) ? If 
so, moving to RTL8139 fixed it. If you use Linux, make sure you use 8139cp 
which is still part of the experimental drivers, 8139too (the currently 
default driver for Linux for the 8139) showed the same problem.


Yves.
----
Yves Dorfsman                                             yves-Qq95MN99UUsAvxtiuMwx3w@public.gmane.org
                                                    http://www.SollerS.ca


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/