Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Gleb Natapov <gleb@redhat.com>]

On Fri, Oct 04, 2013 at 06:03:33PM +0530, Kashyap Chamarthy wrote:
> On Fri, Oct 4, 2013 at 3:09 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > On Fri, Oct 04, 2013 at 03:03:17PM +0530, Kashyap Chamarthy wrote:
> >> Discussed with Gleb on IRC: I'll have to try with kvm.git
> 
> I just compiled kernel from kvm.git queue on both L0 and L1.
> 
> For completeness' sake, this how I did:
> 
>     $ git describe
>     for-linus-12017-g8a3c1a33
>     $ git log | head -1
>     commit 8a3c1a33476f6bfebd07954e2277dbc88003bd37
> 
>     $ git branch -d test_ept1 origin/queue
>     $ make -j6 && make bzImage && make modules
>     $ sudo make modules_install && make install
> 
> (Rebooted both L0 and L1 into the new kernel)
> 
> > and a simple
> >> qemu-kvm command line.
> >>
> > And try to disable shadow vmcs on the host just to rule it out.
> 
>     $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
>     N
> 
> The above didn't have any effect.
> 
> No, booting L2 is still stuck.
> 
> Not sure what's broken (if at all) here. Gleb, you mentioned you were
> able to test the MMU combination 'shadow on EPT' just fine with an F19
> liveCD without libvirt in-place. If you have still have that test
> environment, can you please post your qemu-kvm command line?
> 
Something as basic as "qemu -m 2G -cdrom /path/to/image -cpu host

> Thanks.
> 
> >
> >> Thanks, Gleb.
> >>
> >> On Fri, Oct 4, 2013 at 2:01 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
> >> > I have EPT enabled on L0, disabled on L1, and when attempted to boot
> >> > an L2 guest, it results in cursor just blinking on the serial console.
> >> >  (All of them are Fedora 19 minimal guests.)
> >> >
> >> > To be clear, by "disable" I mean, the below KVM parameter is N
> >> >
> >> >         $ cat /sys/module/kvm_intel/parameters/ept
> >> >         N
> >> >
> >> > I don't see any boot messages on serial console either. Running
> >> >
> >> >         $ virsh start nguest-01 --console
> >> >
> >> > results in cursor just waiting there and hung.
> >> >
> >> > I thought I'll see what's going on in L2 logs by mounting the disk
> >> > image read-only via guestfish
> >> >
> >> >     $ guestfish --ro -d nguest-01 -i
> >> >
> >> > This is hung too.
> >> >
> >> > NOTE: If I enable EPT in L1, I'm able to boot L2 guests just fine and
> >> > can see all the boot messages.
> >> >
> >> > Any hints on what might be going on here?
> >> >
> >> > Details about test environment below.
> >> >
> >> > Setup info:
> >> > ----------------
> >> >
> >> > L0 (Intel Haswell):
> >> >     - 4 pCPU 16G pMEM.
> >> >
> >> >     - Version (same for both L0 and L1):
> >> >         $ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
> >> >         3.12.0-0.rc3.git1.2.fc21.x86_64
> >> >         qemu-kvm-1.4.2-7.fc19.x86_64
> >> >         libvirt-daemon-kvm-1.0.5.5-1.fc19.x86_64
> >> >         libguestfs-1.22.5-1.fc19.x86_64
> >> >
> >> > KVM Parameters in L0:
> >> >
> >> >         $ cat /sys/module/kvm_intel/parameters/nested
> >> >         Y
> >> >         $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> >> >         Y
> >> >         $ cat /sys/module/kvm_intel/parameters/ept
> >> >         Y
> >> >
> >> >
> >> > KVM Parameters in L1 (Note - EPT is disabled below):
> >> >
> >> >         $ cat /sys/module/kvm_intel/parameters/nested
> >> >         Y
> >> >         $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> >> >         N
> >> >         $ cat /sys/module/kvm_intel/parameters/ept
> >> >         N
> >> >
> >> >
> >> > L1 qemu-kvm command line:
> >> > ----------------------------------------
> >> >
> >> > $ ps -ef | grep qemu
> >> > qemu      4329     1 30 03:35 ?        00:13:47
> >> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S
> >> > -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu host -m 10240 -smp
> >> > 4,sockets=4,cores=1,threads=1 -uuid
> >> > 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config
> >> > -nodefaults -chardev
> >> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
> >> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> >> > -no-shutdown -boot c -usb -drive
> >> > file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> >> > -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> >> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> >> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
> >> > -chardev pty,id=charserial0 -device
> >> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> >> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >> >
> >> >
> >> > L2 qemu-kvm command line:
> >> > ----------------------------------------
> >> > $ ps -ef | grep qemu
> >> > qemu      1174     1 99 04:13 ?        00:06:12
> >> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name nguest-01 -S
> >> > -machine pc-i440fx-1.4,accel=kvm,usb=off -m 2048 -smp
> >> > 2,sockets=2,cores=1,threads=1 -uuid
> >> > b47c5cbb-b320-ce9d-c595-4e083b0e465d -nographic -no-user-config
> >> > -nodefaults -chardev
> >> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/nguest-01.monitor,server,nowait
> >> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> >> > -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> >> > -drive file=/home/test/vmimages/nguest-01.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> >> > -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> >> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> >> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:be:d5:8e,bus=pci.0,addr=0x3
> >> > -chardev pty,id=charserial0 -device
> >> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> >> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >> >
> >> >
> >> > /kashyap
> >
> > --
> >                         Gleb.

--
			Gleb.