Re: Nested virtualization on Intel does not work - second level freezes when third level is starting

[Orit Wasserman <owasserm@redhat.com>]

On 04/11/2012 03:44 PM, Guido Winkelmann wrote:
> Hi,
> 
> Nested virtualization on Intel does not work for me with qemu-kvm. As soon as 
> the third layer OS (second virtualised) is starting the Linux kernel, the 
> entire second layer freezes up. The last thing I can see console of the third 
> layer system before it freezes is "Decompressing Linux... ". (no "done", 
> though). When starting without nofb option, the kernel still manages to set 
> the screen resolution before freezing.
> 
> Grub/Syslinux still work, but are extremely slow.
> 
> Both the first layer OS (i.e. the one running on bare metal) and the second 
> layer OS are 64-bit-Fedora 16 with Kernel 3.3.1-3.fc16.x86_64. On both the 
> first and second layer OS, the kvm_intel modules are loaded with nested=Y 
> parameter. (I've also tried with nested=N in the second layer. Didn't change 
> anything.)
> Qemu-kvm was originally the Fedora-shipped 0.14, but I have since upgraded to 
> 1.0. (Using rpmbuild with the specfile and patches from 
> http://pkgs.fedoraproject.org/gitweb/?p=qemu.git;a=blob;f=qemu.spec;hb=HEAD)
> 
> The second layer machine has this CPU specification in libvirt on the first 
> layer OS:
> 
>   <cpu mode='custom' match='exact'>
>     <model fallback='allow'>Nehalem</model>
>     <feature policy='require' name='vmx'/>
>   </cpu>
> 
> which results in this qemu commandline (from libvirt's logs):
> 
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
> kvm -S -M pc-0.15 -cpu kvm64,+lahf_lm,+popcnt,+sse4.2,+sse4.1,+ssse3,+vmx -
> enable-kvm -m 8192 -smp 8,sockets=8,cores=1,threads=1 -name vshost1 -uuid 
> 192b8c4b-0ded-07aa-2545-d7fef4cd897f -nodefconfig -nodefaults -chardev 
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/vshost1.monitor,server,nowait 
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -
> no-acpi -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
> file=/data/vshost1.img,if=none,id=drive-virtio-disk0,format=qcow2 -device 
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-
> disk0,bootindex=1 -drive file=/data/Fedora-16-x86_64-
> netinst.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -
> device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev 
> tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-
> pci,netdev=hostnet0,id=net0,mac=52:54:00:84:7d:46,bus=pci.0,addr=0x3 -netdev 
> tap,fd=23,id=hostnet1,vhost=on,vhostfd=24 -device virtio-net-
> pci,netdev=hostnet1,id=net1,mac=52:54:00:84:8d:46,bus=pci.0,addr=0x4 -vnc 
> 127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
> pci,id=balloon0,bus=pci.0,addr=0x6
> 
> I have also tried some other combinations for the cpu element, like changing 
> the model to core2duo and/or including all the features reported by libvirt's 
> capabalities command.
> 
> The third level machine does not have a cpu element in libvirt, and its 
> commandline looks like this:
> 
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
> kvm -S -M pc-0.14 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -name 
> gentoo -uuid 3cdcc902-4520-df25-92ac-31ca5c707a50 -nodefconfig -nodefaults -
> chardev 
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/gentoo.monitor,server,nowait 
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-acpi -drive 
> file=/data/gentoo.img,if=none,id=drive-virtio-disk0,format=qcow2 -device 
> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -
> drive file=/data/install-amd64-
> minimal-20120223.iso,if=none,media=cdrom,id=drive-
> ide0-1-0,readonly=on,format=raw -device ide-
> drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev 
> tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-
> pci,netdev=hostnet0,id=net0,mac=52:54:00:84:6d:46,bus=pci.0,addr=0x3 -usb -vnc 
> 127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
> pci,id=balloon0,bus=pci.0,addr=0x5
> 
> The third layer OS is a recent Gentoo minimal install (amd64), but somehow I 
> don't think that matters at this point...
> 
> The metal is a Dell PowerEdge R710 server with two Xeon E5520 CPUs. I've tried 
> updating the machine's BIOS and other firmware to the latest version. That 
> took a lot of time and a lot of searching on Dell websites, but didn't change 
> anything.
> 
> Does anyone have any idea what might be going wrong here or how I could debug 
> this further?

I'm not sure if this is the problem but I noticed that the second layer and the third layer have 
the same memory size (8G), how about trying to reduce the memory for the third layer ?

Orit

> 
> 	Guido
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html