kvm problems on new hardware

kvm problems on new hardware

[Danny ter Haar]

Hello,
I have a KVM virtualization problem.
I've put together new hardware (supermicro) server with 2 E5530 cpu's
and memory & disk to start experimenting with virtualization.

I intend to use the www.proxmox.com system/setup.
I installed proxmox and started stress testing the hardware: 
parallel kernel compiles in a loop (concurrency_level=32) &
memtest86+ during the night etc.
The hardware/os performs rocksolid when i stress test it, but the moment
i start a virtual guest (eg debian netinstall) i get the first screen of the
installation procedure in a vnc screen. I choose either normal install or 
expert install , the guest screen goes blank with only a cursor and the 
kvm process prints an error on the console and starts to eat cpu cycles. 
So the host OS is not barfing, only the kvm process is giving problems and the 
guest is frozen.

To see if it was/is related to the "older" proxmox kernel setup i installed
ubuntu karmic with libvirt on another harddrive for comparison: 
Same error happens when i start a guest.

Oct 23 09:34:14 ubuntu kernel: [  416.226550] device vnet0 left promiscuous mode
Oct 23 09:34:14 ubuntu kernel: [  416.226554] br0: port 2(vnet0) entering disabled state
Oct 23 09:34:57 ubuntu kernel: [  459.544150] type=1505 audit(1256290497.414:17): operation="profile_load" pid=1676 name=libvirt-2ae923e6-f06d-9f0d-d072-c2067b7cbee4
Oct 23 09:34:57 ubuntu kernel: [  459.550725] device vnet0 entered promiscuous mode
Oct 23 09:34:57 ubuntu kernel: [  459.551888] br0: port 2(vnet0) entering learning state
Oct 23 09:34:57 ubuntu kernel: [  459.557989] type=1503 audit(1256290497.429:18): operation="open" pid=1679 parent=1 profile="libvirt-2ae923e6-f06d-9f0d-d072-c2067b7cbee4" requested_mask="rw::" denied_mask="w::" fsuid=0 ouid=0 name="/var/tmp/debian-503-amd64-netinst.iso"
Oct 23 09:35:05 ubuntu kernel: [  468.066681] handle_exception: unexpected, vectoring info 0x80000010 intr info 0x80000b0d
Oct 23 09:35:05 ubuntu kernel: [  468.066760] handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
Oct 23 09:35:05 ubuntu kernel: [  468.066836] handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d


In the bios there are settings for vt-d etc, i tried (imho) all combinations
but i am not able to start a guest. I spoke to supermicro support and even 
got a more recent (yet unpublished) bios. All without success.

No i am up to the point where i dont know if i have a hardware or a software
problem. I installed the intel-microcode package to see if that maybe fixed 
something: It didn't


References:
chassis: 
http://supermicro.com/products/system/2U/6026/SYS-6026TT-BIBQRF.cfm?INF=
motherboard:
http://supermicro.com/products/motherboard/QPI/5500/X8DTT-F.cfm
7 hourmemtest:
http://dth.net/supermicro/memtest86_completed_7hr.jpg

More output (dmesg and other stuff) in the same dir:
http://dth.net/supermicro/

Am i trying to run this on hardware that is to recent and not yet tested ?

I hope somebody has some ideas/hints about this.


Danny

Re: kvm problems on new hardware

[Avi Kivity]

On 10/26/2009 12:06 PM, Danny ter Haar wrote:
> Hello,
> I have a KVM virtualization problem.
> I've put together new hardware (supermicro) server with 2 E5530 cpu's
> and memory&  disk to start experimenting with virtualization.
>
> I intend to use the www.proxmox.com system/setup.
> I installed proxmox and started stress testing the hardware:
> parallel kernel compiles in a loop (concurrency_level=32)&
> memtest86+ during the night etc.
> The hardware/os performs rocksolid when i stress test it, but the moment
> i start a virtual guest (eg debian netinstall) i get the first screen of the
> installation procedure in a vnc screen. I choose either normal install or
> expert install , the guest screen goes blank with only a cursor and the
> kvm process prints an error on the console and starts to eat cpu cycles.
> So the host OS is not barfing, only the kvm process is giving problems and the
> guest is frozen.
>
>    

Does this happen for all guests (different OSes), or just this one?

Please provide a link to the install media.

-- 
error compiling committee.c: too many arguments to function

Re: kvm problems on new hardware

[Danny ter Haar]

On Tue, 2009-10-27 at 11:45 +0200, Avi Kivity wrote:
> Does this happen for all guests (different OSes), or just this one?
I tried an iso of both X86_64 and i386 of debian.
I even burned the iso image to a real cd to try if it would boot (it
did)
Then i tried a rescuecd image: same behaviour.

> Please provide a link to the install media.

It's really a standard cd image i got from 

http://cdimage.debian.org/debian-cd/5.0.3/amd64/iso-cd/debian-503-amd64-netinst.iso

I opened a question at the proxmox forum (first):

http://www.proxmox.com/forum/showthread.php?t=2384

I just followed up with this posting:

http://www.proxmox.com/forum/showthread.php?p=13192#post13192

which may give more detailed/useful information.

Re: kvm problems on new hardware

[Danny ter Haar]

On Tue, 2009-10-27 at 11:45 +0200, Avi Kivity wrote:
> Does this happen for all guests (different OSes), or just this one?

I just tried a windows7 cd image 
Same error:

vmbr0: port 2(vmtab105i0) entering learning state
vmbr0: topology change detected, propagating
vmbr0: port 2(vmtab105i0) entering forwarding state
handle_exception: unexpected, vectoring info 0x80000410 intr info
0x80000b0d

I am thinking it's either a bios setting (i tried (afaik) all possible
combinations). I even have a not-yet-released version of the bios.
All still won't let me use virtualization.
Could it be i have faulty cpu's ?
Is there a way to debug what happens when ik start the kvm guest
process ?

Re: kvm problems on new hardware

[Avi Kivity]

On 10/27/2009 12:27 PM, Danny ter Haar wrote:
> On Tue, 2009-10-27 at 11:45 +0200, Avi Kivity wrote:
>    
>> Does this happen for all guests (different OSes), or just this one?
>>      
> I just tried a windows7 cd image
> Same error:
>
> vmbr0: port 2(vmtab105i0) entering learning state
> vmbr0: topology change detected, propagating
> vmbr0: port 2(vmtab105i0) entering forwarding state
> handle_exception: unexpected, vectoring info 0x80000410 intr info
> 0x80000b0d
>    

I'm not able to reproduce this on a similar processor.  Can you post 
your qemu command line?

-- 
error compiling committee.c: too many arguments to function

Re: kvm problems on new hardware

[Danny ter Haar]

On Tue, 2009-10-27 at 12:34 +0200, Avi Kivity wrote:
> I'm not able to reproduce this on a similar processor.  Can you post 
> your qemu command line?

vhost1:/var/tmp# ps axuw |grep kvm
root      5843 42.7  0.0 621048 13560 ?        Sl   11:35   0:06 /usr/bin/kvm -monitor unix:/var/run/qemu-server/105.mon,server,nowait -vnc unix:/var/run/qemu-server/105.vnc,password -pidfile /var/run/qemu-server/105.pid -daemonize -usbdevice tablet -name win7 -smp sockets=1,cores=1 -vga cirrus -tdf -localtime -rtc-td-hack -k en-us -drive file=/var/lib/vz/images/105/vm-105-disk-1.raw,if=ide,index=0,boot=on -drive file=/var/lib/vz/template/iso/windows7.iso,if=ide,index=2,media=cdrom -m 512 -net tap,vlan=0,ifname=vmtab105i0,script=/var/lib/qemu-server/bridge-vlan -net nic,vlan=0,model=e1000,macaddr=BA:29:E3:CC:95:F6 -id 105 -cpuunits 1000


vhost1:/etc/qemu-server# cat 105.conf 

name: win7
ide2: local:iso/windows.iso,media=cdrom
vlan0: e1000=BA:29:E3:CC:95:F6
bootdisk: ide0
ostype: w2k8
ide0: local:105/vm-105-disk-1.raw
memory: 512
sockets: 1

I started the debian install again,
attached a strace and then hit enter in the vnc screen.
The output of the strace can be found here:
http://dth.net/supermicro/strace_debian_install

Don't know if it helps, _i_ don't know what i'm looking at ;-)

Re: kvm problems on new hardware

[Danny ter Haar]

Update:
I compiled/installed 2.6.32-rc5-git3 on this machine

I manually start kvm:

vhost1:~# kvm -m 512 -cdrom  /vz/template/iso/debian-503-amd64-netinst.iso

The bootscreen comes up, i hit enter to install and i get these messages
(copied from dmesg)

handle_exception: unexpected, vectoring info 0x80000010 intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
[this line is repeated many times: 
dmesg |grep 0x80000b0d | wc -l 
570 ]
and finally before ending the kvm session i get:
vmx_handle_exit: unexpected, valid vectoring info (0x8000000d) and exit reason is 0x80000021


Can somebody make a slightly related guess what this means ?

Bad hardware ? 
cpu ? mem ?

Bueller, anyone ?

Re: kvm problems on new hardware

[Ryan Harper]

* Danny ter Haar <dth@dth.net> [2009-10-29 13:38]:
> Update:
> I compiled/installed 2.6.32-rc5-git3 on this machine
> 
> I manually start kvm:
> 
> vhost1:~# kvm -m 512 -cdrom  /vz/template/iso/debian-503-amd64-netinst.iso
> 
> The bootscreen comes up, i hit enter to install and i get these messages
> (copied from dmesg)
> 
> handle_exception: unexpected, vectoring info 0x80000010 intr info 0x80000b0d
> handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
> [this line is repeated many times: 
> dmesg |grep 0x80000b0d | wc -l 
> 570 ]
> and finally before ending the kvm session i get:
> vmx_handle_exit: unexpected, valid vectoring info (0x8000000d) and exit reason is 0x80000021

I've seen that in a couple places.  I don't think we have root cause,
but in at least one situation (running win2k3 with > 4G of ram) the
work around was to use:

   -cpu pentium3

This relates to what cpu features are exported in the virtual processor,
so if this helps it's masking the real issue, but hopefully can
help narrow down what we have to look at.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com

Re: kvm problems on new hardware

[Danny ter Haar]

On Thu, 2009-10-29 at 17:07 -0500, Ryan Harper wrote:
> I've seen that in a couple places.  I don't think we have root cause,
> but in at least one situation (running win2k3 with > 4G of ram) the
> work around was to use:
> 
>    -cpu pentium3

Thanks for your response.

When i add this option it segv's on bootup, it doesn't even make it to the
bootscreen of debian:


vhost1:~# kvm -m 512 -cdrom  /vz/template/iso/debian-503-amd64-netinst.iso -cpu pentium3

kvm[4829]: segfault at c ip 00000000004674ac sp 0000000042323fd0 error 4 in kvm[400000+21b000]

Does this provide more hints where to look ?

Re: kvm problems on new hardware

[Chris Lalancette]

Ryan Harper wrote:
> * Danny ter Haar <dth@dth.net> [2009-10-29 13:38]:
>> Update:
>> I compiled/installed 2.6.32-rc5-git3 on this machine
>>
>> I manually start kvm:
>>
>> vhost1:~# kvm -m 512 -cdrom  /vz/template/iso/debian-503-amd64-netinst.iso
>>
>> The bootscreen comes up, i hit enter to install and i get these messages
>> (copied from dmesg)
>>
>> handle_exception: unexpected, vectoring info 0x80000010 intr info 0x80000b0d
>> handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
>> [this line is repeated many times: 
>> dmesg |grep 0x80000b0d | wc -l 
>> 570 ]
>> and finally before ending the kvm session i get:
>> vmx_handle_exit: unexpected, valid vectoring info (0x8000000d) and exit reason is 0x80000021
> 
> I've seen that in a couple places.  I don't think we have root cause,
> but in at least one situation (running win2k3 with > 4G of ram) the
> work around was to use:

Actually, I'm not 100% sure, but I might have root-caused this.  At least, the
exit reason is the same exit reason I fixed.  That exit reason essentially means
"vmenter failed because of invalid guest fields".  In the case I tracked down,
it was because we were wrongly sign-extending the segment fields (instead of
zero-extending them).  Can either you or Danny:

1)  Get the output from stderr of qemu when this happens?  I believe qemu dumps
the state of all of the guest fields when it's going to crash like this, and
that can tell us if the GUEST_STATE is wrong.

2)  Try the patch posted here: http://lkml.org/lkml/2009/10/28/201, and see if
it helps?

-- 
Chris Lalancette

Re: kvm problems on new hardware

[Danny ter Haar]

On Fri, 2009-10-30 at 09:10 +0100, Chris Lalancette wrote:
>  Can either you or Danny:
> 
> 1)  Get the output from stderr of qemu when this happens?  I believe qemu dumps
> the state of all of the guest fields when it's going to crash like this, and
> that can tell us if the GUEST_STATE is wrong.

<blush>
Any pointer how to produce/where to find the dump ?
</blush>


> 2)  Try the patch posted here: http://lkml.org/lkml/2009/10/28/201, and see if
> it helps?
> 

[oneliner patch manually applied]

vhost1:/usr/src/linux-2.6.32-rc5-git4# time make-kpkg kernel-image --initrd

[SNIP]

dpkg-deb: building package `linux-image-2.6.32-rc5-git4' in `../linux-image-2.6.32-rc5-git4_2.6.32-rc5-git4-10.00.Custom_amd64.deb'.
make[2]: Leaving directory `/usr/src/linux-2.6.32-rc5-git4'
make[1]: Leaving directory `/usr/src/linux-2.6.32-rc5-git4'

real    4m50.866s
user    30m3.549s
sys     2m41.270s

[REBOOT]


vhost1:~# kvm -m 512 -cdrom  /vz/template/iso/debian-503-amd64-netinst.iso -cpu pentium3
Segmentation fault
vhost1:~# dmesg| tail
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
vmx_handle_exit: unexpected, valid vectoring info (0x8000000d) and exit reason is 0x80000021
kvm[5138]: segfault at c ip 00000000004674ac sp 0000000041cebfd0 error 4 in kvm[400000+21b000]

In my case it doesn't seem to help.

I straced the process in case that should provide some more help:
http://dth.net/supermicro/strace_2.6.32-rc5-git4_p3_cpu

Thanks for the feedback!

Re: kvm problems on new hardware

[Avi Kivity]

On 10/29/2009 08:37 PM, Danny ter Haar wrote:
> Update:
> I compiled/installed 2.6.32-rc5-git3 on this machine
>
> I manually start kvm:
>
> vhost1:~# kvm -m 512 -cdrom  /vz/template/iso/debian-503-amd64-netinst.iso
>
> The bootscreen comes up, i hit enter to install and i get these messages
> (copied from dmesg)
>
> handle_exception: unexpected, vectoring info 0x80000010 intr info 0x80000b0d
> handle_exception: unexpected, vectoring info 0x8000000d intr info 0x80000b0d
> [this line is repeated many times:
> dmesg |grep 0x80000b0d | wc -l
> 570 ]
> and finally before ending the kvm session i get:
> vmx_handle_exit: unexpected, valid vectoring info (0x8000000d) and exit reason is 0x80000021
>
>
> Can somebody make a slightly related guess what this means ?
>    

Can't guess, so a debugging patch.

Please install a kvm.git kernel and qemu.git qemu, both from the 
simultaenous-exceptions branch.  kvm.git is very similar to 2.5.32-rc5, 
except for kvm changes.  Please enable ftrace in the kernel.

For starters, run your guest; instead of dmesg errors it should print 
something on stderr and freeze.  Please report those messages, as well 
as the output of the following monitor commands:

   (qemu) info registers
   (qemu) x/30i $eip-20

You can find kvm.git and qemu-kvm.git in

   git://git.kernel.org/pub/scm/virt/kvm/kvm.git
   git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git

remember, the simultaneous-exceptions branch, or we don't get any output.

-- 
error compiling committee.c: too many arguments to function