Nested virtualization on Intel does not work - second level freezes when third level is starting

[Guido Winkelmann <guido-kvml@thisisnotatest.de>]

Hi,

Nested virtualization on Intel does not work for me with qemu-kvm. As soon as 
the third layer OS (second virtualised) is starting the Linux kernel, the 
entire second layer freezes up. The last thing I can see console of the third 
layer system before it freezes is "Decompressing Linux... ". (no "done", 
though). When starting without nofb option, the kernel still manages to set 
the screen resolution before freezing.

Grub/Syslinux still work, but are extremely slow.

Both the first layer OS (i.e. the one running on bare metal) and the second 
layer OS are 64-bit-Fedora 16 with Kernel 3.3.1-3.fc16.x86_64. On both the 
first and second layer OS, the kvm_intel modules are loaded with nested=Y 
parameter. (I've also tried with nested=N in the second layer. Didn't change 
anything.)
Qemu-kvm was originally the Fedora-shipped 0.14, but I have since upgraded to 
1.0. (Using rpmbuild with the specfile and patches from 
http://pkgs.fedoraproject.org/gitweb/?p=qemu.git;a=blob;f=qemu.spec;hb=HEAD)

The second layer machine has this CPU specification in libvirt on the first 
layer OS:

  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Nehalem</model>
    <feature policy='require' name='vmx'/>
  </cpu>

which results in this qemu commandline (from libvirt's logs):

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
kvm -S -M pc-0.15 -cpu kvm64,+lahf_lm,+popcnt,+sse4.2,+sse4.1,+ssse3,+vmx -
enable-kvm -m 8192 -smp 8,sockets=8,cores=1,threads=1 -name vshost1 -uuid 
192b8c4b-0ded-07aa-2545-d7fef4cd897f -nodefconfig -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/vshost1.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -
no-acpi -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
file=/data/vshost1.img,if=none,id=drive-virtio-disk0,format=qcow2 -device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-
disk0,bootindex=1 -drive file=/data/Fedora-16-x86_64-
netinst.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -
device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev 
tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-
pci,netdev=hostnet0,id=net0,mac=52:54:00:84:7d:46,bus=pci.0,addr=0x3 -netdev 
tap,fd=23,id=hostnet1,vhost=on,vhostfd=24 -device virtio-net-
pci,netdev=hostnet1,id=net1,mac=52:54:00:84:8d:46,bus=pci.0,addr=0x4 -vnc 
127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
pci,id=balloon0,bus=pci.0,addr=0x6

I have also tried some other combinations for the cpu element, like changing 
the model to core2duo and/or including all the features reported by libvirt's 
capabalities command.

The third level machine does not have a cpu element in libvirt, and its 
commandline looks like this:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-
kvm -S -M pc-0.14 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -name 
gentoo -uuid 3cdcc902-4520-df25-92ac-31ca5c707a50 -nodefconfig -nodefaults -
chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/gentoo.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-acpi -drive 
file=/data/gentoo.img,if=none,id=drive-virtio-disk0,format=qcow2 -device 
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -
drive file=/data/install-amd64-
minimal-20120223.iso,if=none,media=cdrom,id=drive-
ide0-1-0,readonly=on,format=raw -device ide-
drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev 
tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-
pci,netdev=hostnet0,id=net0,mac=52:54:00:84:6d:46,bus=pci.0,addr=0x3 -usb -vnc 
127.0.0.1:0,password -k de -vga cirrus -device virtio-balloon-
pci,id=balloon0,bus=pci.0,addr=0x5

The third layer OS is a recent Gentoo minimal install (amd64), but somehow I 
don't think that matters at this point...

The metal is a Dell PowerEdge R710 server with two Xeon E5520 CPUs. I've tried 
updating the machine's BIOS and other firmware to the latest version. That 
took a lot of time and a lot of searching on Dell websites, but didn't change 
anything.

Does anyone have any idea what might be going wrong here or how I could debug 
this further?

	Guido