[nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Kashyap Chamarthy]

I have EPT enabled on L0, disabled on L1, and when attempted to boot
an L2 guest, it results in cursor just blinking on the serial console.
 (All of them are Fedora 19 minimal guests.)

To be clear, by "disable" I mean, the below KVM parameter is N

        $ cat /sys/module/kvm_intel/parameters/ept
        N

I don't see any boot messages on serial console either. Running

        $ virsh start nguest-01 --console

results in cursor just waiting there and hung.

I thought I'll see what's going on in L2 logs by mounting the disk
image read-only via guestfish

    $ guestfish --ro -d nguest-01 -i

This is hung too.

NOTE: If I enable EPT in L1, I'm able to boot L2 guests just fine and
can see all the boot messages.

Any hints on what might be going on here?

Details about test environment below.

Setup info:
----------------

L0 (Intel Haswell):
    - 4 pCPU 16G pMEM.

    - Version (same for both L0 and L1):
        $ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
        3.12.0-0.rc3.git1.2.fc21.x86_64
        qemu-kvm-1.4.2-7.fc19.x86_64
        libvirt-daemon-kvm-1.0.5.5-1.fc19.x86_64
        libguestfs-1.22.5-1.fc19.x86_64

KVM Parameters in L0:

        $ cat /sys/module/kvm_intel/parameters/nested
        Y
        $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
        Y
        $ cat /sys/module/kvm_intel/parameters/ept
        Y


KVM Parameters in L1 (Note - EPT is disabled below):

        $ cat /sys/module/kvm_intel/parameters/nested
        Y
        $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
        N
        $ cat /sys/module/kvm_intel/parameters/ept
        N


L1 qemu-kvm command line:
----------------------------------------

$ ps -ef | grep qemu
qemu      4329     1 30 03:35 ?        00:13:47
/usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S
-machine pc-i440fx-1.4,accel=kvm,usb=off -cpu host -m 10240 -smp
4,sockets=4,cores=1,threads=1 -uuid
4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config
-nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -boot c -usb -drive
file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5


L2 qemu-kvm command line:
----------------------------------------
$ ps -ef | grep qemu
qemu      1174     1 99 04:13 ?        00:06:12
/usr/bin/qemu-system-x86_64 -machine accel=kvm -name nguest-01 -S
-machine pc-i440fx-1.4,accel=kvm,usb=off -m 2048 -smp
2,sockets=2,cores=1,threads=1 -uuid
b47c5cbb-b320-ce9d-c595-4e083b0e465d -nographic -no-user-config
-nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/nguest-01.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
-drive file=/home/test/vmimages/nguest-01.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:be:d5:8e,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5


/kashyap

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Kashyap Chamarthy]

Discussed with Gleb on IRC: I'll have to try with kvm.git and a simple
qemu-kvm command line.

Thanks, Gleb.

On Fri, Oct 4, 2013 at 2:01 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
> I have EPT enabled on L0, disabled on L1, and when attempted to boot
> an L2 guest, it results in cursor just blinking on the serial console.
>  (All of them are Fedora 19 minimal guests.)
>
> To be clear, by "disable" I mean, the below KVM parameter is N
>
>         $ cat /sys/module/kvm_intel/parameters/ept
>         N
>
> I don't see any boot messages on serial console either. Running
>
>         $ virsh start nguest-01 --console
>
> results in cursor just waiting there and hung.
>
> I thought I'll see what's going on in L2 logs by mounting the disk
> image read-only via guestfish
>
>     $ guestfish --ro -d nguest-01 -i
>
> This is hung too.
>
> NOTE: If I enable EPT in L1, I'm able to boot L2 guests just fine and
> can see all the boot messages.
>
> Any hints on what might be going on here?
>
> Details about test environment below.
>
> Setup info:
> ----------------
>
> L0 (Intel Haswell):
>     - 4 pCPU 16G pMEM.
>
>     - Version (same for both L0 and L1):
>         $ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
>         3.12.0-0.rc3.git1.2.fc21.x86_64
>         qemu-kvm-1.4.2-7.fc19.x86_64
>         libvirt-daemon-kvm-1.0.5.5-1.fc19.x86_64
>         libguestfs-1.22.5-1.fc19.x86_64
>
> KVM Parameters in L0:
>
>         $ cat /sys/module/kvm_intel/parameters/nested
>         Y
>         $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
>         Y
>         $ cat /sys/module/kvm_intel/parameters/ept
>         Y
>
>
> KVM Parameters in L1 (Note - EPT is disabled below):
>
>         $ cat /sys/module/kvm_intel/parameters/nested
>         Y
>         $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
>         N
>         $ cat /sys/module/kvm_intel/parameters/ept
>         N
>
>
> L1 qemu-kvm command line:
> ----------------------------------------
>
> $ ps -ef | grep qemu
> qemu      4329     1 30 03:35 ?        00:13:47
> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S
> -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu host -m 10240 -smp
> 4,sockets=4,cores=1,threads=1 -uuid
> 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config
> -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-shutdown -boot c -usb -drive
> file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>
>
> L2 qemu-kvm command line:
> ----------------------------------------
> $ ps -ef | grep qemu
> qemu      1174     1 99 04:13 ?        00:06:12
> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name nguest-01 -S
> -machine pc-i440fx-1.4,accel=kvm,usb=off -m 2048 -smp
> 2,sockets=2,cores=1,threads=1 -uuid
> b47c5cbb-b320-ce9d-c595-4e083b0e465d -nographic -no-user-config
> -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/nguest-01.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> -drive file=/home/test/vmimages/nguest-01.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:be:d5:8e,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>
>
> /kashyap

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Gleb Natapov]

On Fri, Oct 04, 2013 at 03:03:17PM +0530, Kashyap Chamarthy wrote:
> Discussed with Gleb on IRC: I'll have to try with kvm.git and a simple
> qemu-kvm command line.
> 
And try to disable shadow vmcs on the host just to rule it out.

> Thanks, Gleb.
> 
> On Fri, Oct 4, 2013 at 2:01 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
> > I have EPT enabled on L0, disabled on L1, and when attempted to boot
> > an L2 guest, it results in cursor just blinking on the serial console.
> >  (All of them are Fedora 19 minimal guests.)
> >
> > To be clear, by "disable" I mean, the below KVM parameter is N
> >
> >         $ cat /sys/module/kvm_intel/parameters/ept
> >         N
> >
> > I don't see any boot messages on serial console either. Running
> >
> >         $ virsh start nguest-01 --console
> >
> > results in cursor just waiting there and hung.
> >
> > I thought I'll see what's going on in L2 logs by mounting the disk
> > image read-only via guestfish
> >
> >     $ guestfish --ro -d nguest-01 -i
> >
> > This is hung too.
> >
> > NOTE: If I enable EPT in L1, I'm able to boot L2 guests just fine and
> > can see all the boot messages.
> >
> > Any hints on what might be going on here?
> >
> > Details about test environment below.
> >
> > Setup info:
> > ----------------
> >
> > L0 (Intel Haswell):
> >     - 4 pCPU 16G pMEM.
> >
> >     - Version (same for both L0 and L1):
> >         $ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
> >         3.12.0-0.rc3.git1.2.fc21.x86_64
> >         qemu-kvm-1.4.2-7.fc19.x86_64
> >         libvirt-daemon-kvm-1.0.5.5-1.fc19.x86_64
> >         libguestfs-1.22.5-1.fc19.x86_64
> >
> > KVM Parameters in L0:
> >
> >         $ cat /sys/module/kvm_intel/parameters/nested
> >         Y
> >         $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> >         Y
> >         $ cat /sys/module/kvm_intel/parameters/ept
> >         Y
> >
> >
> > KVM Parameters in L1 (Note - EPT is disabled below):
> >
> >         $ cat /sys/module/kvm_intel/parameters/nested
> >         Y
> >         $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> >         N
> >         $ cat /sys/module/kvm_intel/parameters/ept
> >         N
> >
> >
> > L1 qemu-kvm command line:
> > ----------------------------------------
> >
> > $ ps -ef | grep qemu
> > qemu      4329     1 30 03:35 ?        00:13:47
> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S
> > -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu host -m 10240 -smp
> > 4,sockets=4,cores=1,threads=1 -uuid
> > 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config
> > -nodefaults -chardev
> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> > -no-shutdown -boot c -usb -drive
> > file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> > -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
> > -chardev pty,id=charserial0 -device
> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >
> >
> > L2 qemu-kvm command line:
> > ----------------------------------------
> > $ ps -ef | grep qemu
> > qemu      1174     1 99 04:13 ?        00:06:12
> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name nguest-01 -S
> > -machine pc-i440fx-1.4,accel=kvm,usb=off -m 2048 -smp
> > 2,sockets=2,cores=1,threads=1 -uuid
> > b47c5cbb-b320-ce9d-c595-4e083b0e465d -nographic -no-user-config
> > -nodefaults -chardev
> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/nguest-01.monitor,server,nowait
> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> > -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> > -drive file=/home/test/vmimages/nguest-01.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> > -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:be:d5:8e,bus=pci.0,addr=0x3
> > -chardev pty,id=charserial0 -device
> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >
> >
> > /kashyap

--
			Gleb.

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Kashyap Chamarthy]

On Fri, Oct 4, 2013 at 3:09 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Fri, Oct 04, 2013 at 03:03:17PM +0530, Kashyap Chamarthy wrote:
>> Discussed with Gleb on IRC: I'll have to try with kvm.git

I just compiled kernel from kvm.git queue on both L0 and L1.

For completeness' sake, this how I did:

    $ git describe
    for-linus-12017-g8a3c1a33
    $ git log | head -1
    commit 8a3c1a33476f6bfebd07954e2277dbc88003bd37

    $ git branch -d test_ept1 origin/queue
    $ make -j6 && make bzImage && make modules
    $ sudo make modules_install && make install

(Rebooted both L0 and L1 into the new kernel)

> and a simple
>> qemu-kvm command line.
>>
> And try to disable shadow vmcs on the host just to rule it out.

    $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
    N

The above didn't have any effect.

No, booting L2 is still stuck.

Not sure what's broken (if at all) here. Gleb, you mentioned you were
able to test the MMU combination 'shadow on EPT' just fine with an F19
liveCD without libvirt in-place. If you have still have that test
environment, can you please post your qemu-kvm command line?

Thanks.

>
>> Thanks, Gleb.
>>
>> On Fri, Oct 4, 2013 at 2:01 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
>> > I have EPT enabled on L0, disabled on L1, and when attempted to boot
>> > an L2 guest, it results in cursor just blinking on the serial console.
>> >  (All of them are Fedora 19 minimal guests.)
>> >
>> > To be clear, by "disable" I mean, the below KVM parameter is N
>> >
>> >         $ cat /sys/module/kvm_intel/parameters/ept
>> >         N
>> >
>> > I don't see any boot messages on serial console either. Running
>> >
>> >         $ virsh start nguest-01 --console
>> >
>> > results in cursor just waiting there and hung.
>> >
>> > I thought I'll see what's going on in L2 logs by mounting the disk
>> > image read-only via guestfish
>> >
>> >     $ guestfish --ro -d nguest-01 -i
>> >
>> > This is hung too.
>> >
>> > NOTE: If I enable EPT in L1, I'm able to boot L2 guests just fine and
>> > can see all the boot messages.
>> >
>> > Any hints on what might be going on here?
>> >
>> > Details about test environment below.
>> >
>> > Setup info:
>> > ----------------
>> >
>> > L0 (Intel Haswell):
>> >     - 4 pCPU 16G pMEM.
>> >
>> >     - Version (same for both L0 and L1):
>> >         $ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
>> >         3.12.0-0.rc3.git1.2.fc21.x86_64
>> >         qemu-kvm-1.4.2-7.fc19.x86_64
>> >         libvirt-daemon-kvm-1.0.5.5-1.fc19.x86_64
>> >         libguestfs-1.22.5-1.fc19.x86_64
>> >
>> > KVM Parameters in L0:
>> >
>> >         $ cat /sys/module/kvm_intel/parameters/nested
>> >         Y
>> >         $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
>> >         Y
>> >         $ cat /sys/module/kvm_intel/parameters/ept
>> >         Y
>> >
>> >
>> > KVM Parameters in L1 (Note - EPT is disabled below):
>> >
>> >         $ cat /sys/module/kvm_intel/parameters/nested
>> >         Y
>> >         $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
>> >         N
>> >         $ cat /sys/module/kvm_intel/parameters/ept
>> >         N
>> >
>> >
>> > L1 qemu-kvm command line:
>> > ----------------------------------------
>> >
>> > $ ps -ef | grep qemu
>> > qemu      4329     1 30 03:35 ?        00:13:47
>> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S
>> > -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu host -m 10240 -smp
>> > 4,sockets=4,cores=1,threads=1 -uuid
>> > 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config
>> > -nodefaults -chardev
>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
>> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
>> > -no-shutdown -boot c -usb -drive
>> > file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
>> > -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
>> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
>> > -chardev pty,id=charserial0 -device
>> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
>> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>> >
>> >
>> > L2 qemu-kvm command line:
>> > ----------------------------------------
>> > $ ps -ef | grep qemu
>> > qemu      1174     1 99 04:13 ?        00:06:12
>> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name nguest-01 -S
>> > -machine pc-i440fx-1.4,accel=kvm,usb=off -m 2048 -smp
>> > 2,sockets=2,cores=1,threads=1 -uuid
>> > b47c5cbb-b320-ce9d-c595-4e083b0e465d -nographic -no-user-config
>> > -nodefaults -chardev
>> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/nguest-01.monitor,server,nowait
>> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
>> > -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
>> > -drive file=/home/test/vmimages/nguest-01.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
>> > -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:be:d5:8e,bus=pci.0,addr=0x3
>> > -chardev pty,id=charserial0 -device
>> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
>> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>> >
>> >
>> > /kashyap
>
> --
>                         Gleb.

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Gleb Natapov]

On Fri, Oct 04, 2013 at 06:03:33PM +0530, Kashyap Chamarthy wrote:
> On Fri, Oct 4, 2013 at 3:09 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > On Fri, Oct 04, 2013 at 03:03:17PM +0530, Kashyap Chamarthy wrote:
> >> Discussed with Gleb on IRC: I'll have to try with kvm.git
> 
> I just compiled kernel from kvm.git queue on both L0 and L1.
> 
> For completeness' sake, this how I did:
> 
>     $ git describe
>     for-linus-12017-g8a3c1a33
>     $ git log | head -1
>     commit 8a3c1a33476f6bfebd07954e2277dbc88003bd37
> 
>     $ git branch -d test_ept1 origin/queue
>     $ make -j6 && make bzImage && make modules
>     $ sudo make modules_install && make install
> 
> (Rebooted both L0 and L1 into the new kernel)
> 
> > and a simple
> >> qemu-kvm command line.
> >>
> > And try to disable shadow vmcs on the host just to rule it out.
> 
>     $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
>     N
> 
> The above didn't have any effect.
> 
> No, booting L2 is still stuck.
> 
> Not sure what's broken (if at all) here. Gleb, you mentioned you were
> able to test the MMU combination 'shadow on EPT' just fine with an F19
> liveCD without libvirt in-place. If you have still have that test
> environment, can you please post your qemu-kvm command line?
> 
Something as basic as "qemu -m 2G -cdrom /path/to/image -cpu host

> Thanks.
> 
> >
> >> Thanks, Gleb.
> >>
> >> On Fri, Oct 4, 2013 at 2:01 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
> >> > I have EPT enabled on L0, disabled on L1, and when attempted to boot
> >> > an L2 guest, it results in cursor just blinking on the serial console.
> >> >  (All of them are Fedora 19 minimal guests.)
> >> >
> >> > To be clear, by "disable" I mean, the below KVM parameter is N
> >> >
> >> >         $ cat /sys/module/kvm_intel/parameters/ept
> >> >         N
> >> >
> >> > I don't see any boot messages on serial console either. Running
> >> >
> >> >         $ virsh start nguest-01 --console
> >> >
> >> > results in cursor just waiting there and hung.
> >> >
> >> > I thought I'll see what's going on in L2 logs by mounting the disk
> >> > image read-only via guestfish
> >> >
> >> >     $ guestfish --ro -d nguest-01 -i
> >> >
> >> > This is hung too.
> >> >
> >> > NOTE: If I enable EPT in L1, I'm able to boot L2 guests just fine and
> >> > can see all the boot messages.
> >> >
> >> > Any hints on what might be going on here?
> >> >
> >> > Details about test environment below.
> >> >
> >> > Setup info:
> >> > ----------------
> >> >
> >> > L0 (Intel Haswell):
> >> >     - 4 pCPU 16G pMEM.
> >> >
> >> >     - Version (same for both L0 and L1):
> >> >         $ uname -r; rpm -q qemu-kvm libvirt-daemon-kvm libguestfs
> >> >         3.12.0-0.rc3.git1.2.fc21.x86_64
> >> >         qemu-kvm-1.4.2-7.fc19.x86_64
> >> >         libvirt-daemon-kvm-1.0.5.5-1.fc19.x86_64
> >> >         libguestfs-1.22.5-1.fc19.x86_64
> >> >
> >> > KVM Parameters in L0:
> >> >
> >> >         $ cat /sys/module/kvm_intel/parameters/nested
> >> >         Y
> >> >         $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> >> >         Y
> >> >         $ cat /sys/module/kvm_intel/parameters/ept
> >> >         Y
> >> >
> >> >
> >> > KVM Parameters in L1 (Note - EPT is disabled below):
> >> >
> >> >         $ cat /sys/module/kvm_intel/parameters/nested
> >> >         Y
> >> >         $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> >> >         N
> >> >         $ cat /sys/module/kvm_intel/parameters/ept
> >> >         N
> >> >
> >> >
> >> > L1 qemu-kvm command line:
> >> > ----------------------------------------
> >> >
> >> > $ ps -ef | grep qemu
> >> > qemu      4329     1 30 03:35 ?        00:13:47
> >> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name regular-guest -S
> >> > -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu host -m 10240 -smp
> >> > 4,sockets=4,cores=1,threads=1 -uuid
> >> > 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config
> >> > -nodefaults -chardev
> >> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
> >> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> >> > -no-shutdown -boot c -usb -drive
> >> > file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> >> > -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
> >> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> >> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
> >> > -chardev pty,id=charserial0 -device
> >> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> >> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >> >
> >> >
> >> > L2 qemu-kvm command line:
> >> > ----------------------------------------
> >> > $ ps -ef | grep qemu
> >> > qemu      1174     1 99 04:13 ?        00:06:12
> >> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name nguest-01 -S
> >> > -machine pc-i440fx-1.4,accel=kvm,usb=off -m 2048 -smp
> >> > 2,sockets=2,cores=1,threads=1 -uuid
> >> > b47c5cbb-b320-ce9d-c595-4e083b0e465d -nographic -no-user-config
> >> > -nodefaults -chardev
> >> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/nguest-01.monitor,server,nowait
> >> > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> >> > -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> >> > -drive file=/home/test/vmimages/nguest-01.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
> >> > -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> >> > -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
> >> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:be:d5:8e,bus=pci.0,addr=0x3
> >> > -chardev pty,id=charserial0 -device
> >> > isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0
> >> > -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >> >
> >> >
> >> > /kashyap
> >
> > --
> >                         Gleb.

--
			Gleb.

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Gleb Natapov]

On Fri, Oct 04, 2013 at 04:05:50PM +0300, Gleb Natapov wrote:
> On Fri, Oct 04, 2013 at 06:03:33PM +0530, Kashyap Chamarthy wrote:
> > On Fri, Oct 4, 2013 at 3:09 PM, Gleb Natapov <gleb@redhat.com> wrote:
> > > On Fri, Oct 04, 2013 at 03:03:17PM +0530, Kashyap Chamarthy wrote:
> > >> Discussed with Gleb on IRC: I'll have to try with kvm.git
> > 
> > I just compiled kernel from kvm.git queue on both L0 and L1.
> > 
> > For completeness' sake, this how I did:
> > 
> >     $ git describe
> >     for-linus-12017-g8a3c1a33
> >     $ git log | head -1
> >     commit 8a3c1a33476f6bfebd07954e2277dbc88003bd37
> > 
> >     $ git branch -d test_ept1 origin/queue
> >     $ make -j6 && make bzImage && make modules
> >     $ sudo make modules_install && make install
> > 
> > (Rebooted both L0 and L1 into the new kernel)
> > 
> > > and a simple
> > >> qemu-kvm command line.
> > >>
> > > And try to disable shadow vmcs on the host just to rule it out.
> > 
> >     $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
> >     N
> > 
> > The above didn't have any effect.
> > 
> > No, booting L2 is still stuck.
> > 
> > Not sure what's broken (if at all) here. Gleb, you mentioned you were
> > able to test the MMU combination 'shadow on EPT' just fine with an F19
> > liveCD without libvirt in-place. If you have still have that test
> > environment, can you please post your qemu-kvm command line?
> > 
> Something as basic as "qemu -m 2G -cdrom /path/to/image -cpu host
> 
And Live CD I use is XFCE based, not sure if it matters.

--
			Gleb.

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Kashyap Chamarthy]

On Fri, Oct 4, 2013 at 6:38 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Fri, Oct 04, 2013 at 04:05:50PM +0300, Gleb Natapov wrote:
>> On Fri, Oct 04, 2013 at 06:03:33PM +0530, Kashyap Chamarthy wrote:
>> > On Fri, Oct 4, 2013 at 3:09 PM, Gleb Natapov <gleb@redhat.com> wrote:
>> > > On Fri, Oct 04, 2013 at 03:03:17PM +0530, Kashyap Chamarthy wrote:
>> > >> Discussed with Gleb on IRC: I'll have to try with kvm.git
>> >
>> > I just compiled kernel from kvm.git queue on both L0 and L1.
>> >
>> > For completeness' sake, this how I did:
>> >
>> >     $ git describe
>> >     for-linus-12017-g8a3c1a33
>> >     $ git log | head -1
>> >     commit 8a3c1a33476f6bfebd07954e2277dbc88003bd37
>> >
>> >     $ git branch -d test_ept1 origin/queue
>> >     $ make -j6 && make bzImage && make modules
>> >     $ sudo make modules_install && make install
>> >
>> > (Rebooted both L0 and L1 into the new kernel)
>> >
>> > > and a simple
>> > >> qemu-kvm command line.
>> > >>
>> > > And try to disable shadow vmcs on the host just to rule it out.
>> >
>> >     $ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs
>> >     N
>> >
>> > The above didn't have any effect.
>> >
>> > No, booting L2 is still stuck.
>> >
>> > Not sure what's broken (if at all) here. Gleb, you mentioned you were
>> > able to test the MMU combination 'shadow on EPT' just fine with an F19
>> > liveCD without libvirt in-place. If you have still have that test
>> > environment, can you please post your qemu-kvm command line?
>> >
>> Something as basic as "qemu -m 2G -cdrom /path/to/image -cpu host

Thanks. I'll see if I can figure further what's going on here,
otherwise I'll just write a bug for now with all the details, just so
it's tracked. If it turns out to be a false alarm, I can just close
it.

>>
> And Live CD I use is XFCE based, not sure if it matters.

I'm using no X. So, it shouldn't matter :-)

>
> --
>                         Gleb.

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Kashyap Chamarthy]

Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
stuck with shadow on EPT

Ensure, EPT is enabled on L0 & disabled on L1

On L0:
---------
  $ cat /sys/module/kvm_intel/parameters/ept
  Y

On L1
---------
  $ cat /sys/module/kvm_intel/parameters/ept
  N

Build and install trace-cmd on L1:

  $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
trace-cmd
  $ make && sudo make-install

Run trace command (on L1) to capture kvmmmu event:

  $ trace-cmd record -b 2000 -e kvmmmu
  /sys/kernel/debug/tracing/events/kvmmmu/filter
  /sys/kernel/debug/tracing/events/*/kvmmmu/filter
  Hit Ctrl^C to stop recording

Boot L2 guest:

  $ virsh start nguest-01 --console

After a minute or two, ctl-C the trace-cmd, generate the report:

  $ trace-cmd report
  version = 6
  trace-cmd: No such file or directory
    function is_writable_pte not defined
  CPU 1 is empty
  CPU 2 is empty
  CPU 3 is empty
  cpus=4
   qemu-system-x86-2554  [000] 90667.995345:
kvm_mmu_invalidate_zap_all_pages: kvm-mmu-valid-gen 0 used_pages 0
   qemu-system-x86-2557  [000] 90668.078978: kvm_mmu_get_page:     new
sp gfn 0 0/2 q0 direct --- !pge !nxe root 0 sync
   qemu-system-x86-2557  [000] 90668.078980: kvm_mmu_get_page:     new
sp gfn 40000 0/2 q0 direct --- !pge !nxe root 0 sync
   qemu-system-x86-2557  [000] 90668.078980: kvm_mmu_get_page:     new
sp gfn 80000 0/2 q0 direct --- !pge !nxe root 0 sync
   qemu-system-x86-2557  [000] 90668.078981: kvm_mmu_get_page:     new
sp gfn c0000 0/2 q0 direct --- !pge !nxe root 0 sync
   qemu-system-x86-2557  [000] 90668.079025: kvm_mmu_get_page:     new
sp gfn 0 0/1 q0 direct --- !pge !nxe root 0 sync

Side question:  I wonder why would trace-cmd say "trace-cmd: No such
file or directory"?

/kashyap

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Gleb Natapov]

On Mon, Oct 07, 2013 at 06:29:30PM +0530, Kashyap Chamarthy wrote:
> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
> stuck with shadow on EPT
> 
> Ensure, EPT is enabled on L0 & disabled on L1
> 
> On L0:
> ---------
>   $ cat /sys/module/kvm_intel/parameters/ept
>   Y
> 
> On L1
> ---------
>   $ cat /sys/module/kvm_intel/parameters/ept
>   N
> 
> Build and install trace-cmd on L1:
> 
>   $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
> trace-cmd
>   $ make && sudo make-install
> 
> Run trace command (on L1) to capture kvmmmu event:
> 
>   $ trace-cmd record -b 2000 -e kvmmmu
You need to trace kvm events too and make buffer bigger
trace-cmd record -b 100000 -e kvmmmu -e kvm.

>   /sys/kernel/debug/tracing/events/kvmmmu/filter
>   /sys/kernel/debug/tracing/events/*/kvmmmu/filter
>   Hit Ctrl^C to stop recording
> 
> Boot L2 guest:
> 
>   $ virsh start nguest-01 --console
> 
> After a minute or two, ctl-C the trace-cmd, generate the report:
> 
>   $ trace-cmd report
>   version = 6
>   trace-cmd: No such file or directory
>     function is_writable_pte not defined
>   CPU 1 is empty
>   CPU 2 is empty
>   CPU 3 is empty
>   cpus=4
>    qemu-system-x86-2554  [000] 90667.995345:
> kvm_mmu_invalidate_zap_all_pages: kvm-mmu-valid-gen 0 used_pages 0
>    qemu-system-x86-2557  [000] 90668.078978: kvm_mmu_get_page:     new
> sp gfn 0 0/2 q0 direct --- !pge !nxe root 0 sync
>    qemu-system-x86-2557  [000] 90668.078980: kvm_mmu_get_page:     new
> sp gfn 40000 0/2 q0 direct --- !pge !nxe root 0 sync
>    qemu-system-x86-2557  [000] 90668.078980: kvm_mmu_get_page:     new
> sp gfn 80000 0/2 q0 direct --- !pge !nxe root 0 sync
>    qemu-system-x86-2557  [000] 90668.078981: kvm_mmu_get_page:     new
> sp gfn c0000 0/2 q0 direct --- !pge !nxe root 0 sync
>    qemu-system-x86-2557  [000] 90668.079025: kvm_mmu_get_page:     new
> sp gfn 0 0/1 q0 direct --- !pge !nxe root 0 sync
Is this all?

> 
> Side question:  I wonder why would trace-cmd say "trace-cmd: No such
> file or directory"?
> 
Ignore it.

--
			Gleb.

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Kashyap Chamarthy]

On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
> stuck with shadow on EPT

Paolo, were you able to reproduce this again? Yesterday, on #qemu you
mentioned you'll test it again :-)

I was using kvm.git  queue on both L0 and L1:

  $ git describe ; git log | head -1
  for-linus-11227-g8a3c1a33
  commit 8a3c1a33476f6bfebd07954e2277dbc88003bd37

/kashyap

>
> Ensure, EPT is enabled on L0 & disabled on L1
>
> On L0:
> ---------
>   $ cat /sys/module/kvm_intel/parameters/ept
>   Y
>
> On L1
> ---------
>   $ cat /sys/module/kvm_intel/parameters/ept
>   N
>
> Build and install trace-cmd on L1:
>
>   $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
> trace-cmd
>   $ make && sudo make-install
>
> Run trace command (on L1) to capture kvmmmu event:
>
>   $ trace-cmd record -b 2000 -e kvmmmu
>   /sys/kernel/debug/tracing/events/kvmmmu/filter
>   /sys/kernel/debug/tracing/events/*/kvmmmu/filter
>   Hit Ctrl^C to stop recording
>
> Boot L2 guest:
>
>   $ virsh start nguest-01 --console
>
> After a minute or two, ctl-C the trace-cmd, generate the report:
>
>   $ trace-cmd report
>   version = 6
>   trace-cmd: No such file or directory
>     function is_writable_pte not defined
>   CPU 1 is empty
>   CPU 2 is empty
>   CPU 3 is empty
>   cpus=4
>    qemu-system-x86-2554  [000] 90667.995345:
> kvm_mmu_invalidate_zap_all_pages: kvm-mmu-valid-gen 0 used_pages 0
>    qemu-system-x86-2557  [000] 90668.078978: kvm_mmu_get_page:     new
> sp gfn 0 0/2 q0 direct --- !pge !nxe root 0 sync
>    qemu-system-x86-2557  [000] 90668.078980: kvm_mmu_get_page:     new
> sp gfn 40000 0/2 q0 direct --- !pge !nxe root 0 sync
>    qemu-system-x86-2557  [000] 90668.078980: kvm_mmu_get_page:     new
> sp gfn 80000 0/2 q0 direct --- !pge !nxe root 0 sync
>    qemu-system-x86-2557  [000] 90668.078981: kvm_mmu_get_page:     new
> sp gfn c0000 0/2 q0 direct --- !pge !nxe root 0 sync
>    qemu-system-x86-2557  [000] 90668.079025: kvm_mmu_get_page:     new
> sp gfn 0 0/1 q0 direct --- !pge !nxe root 0 sync
>
> Side question:  I wonder why would trace-cmd say "trace-cmd: No such
> file or directory"?
>
> /kashyap

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Paolo Bonzini]

Il 08/10/2013 07:38, Kashyap Chamarthy ha scritto:
> On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
>> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
>> stuck with shadow on EPT
> 
> Paolo, were you able to reproduce this again? Yesterday, on #qemu you
> mentioned you'll test it again :-)

Yes, I could reproduce it too.

>> Boot L2 guest:

Here L2 doesn't go past the second instruction.  It gets a page fault
even though the spte is present, and KVM then loops on a page fault
for 0xfe05b.

Here is an annotated function_graph trace of L1.

It's possible that L0 is injecting the same fault repeatedly, i.e.
they are not different faults from the processor.  I'll get an L0
trace next.

---- KVM executes at 0xfffffff0 via emulation

kvm_cpu_has_pending_timer [kvm]() {
  apic_has_pending_timer [kvm]();
}
kvm_check_async_pf_completion [kvm]();
vmx_save_host_state [kvm_intel]();
__srcu_read_unlock();
rcu_note_context_switch();
/* kvm_entry: vcpu 0 */
vmx_vcpu_run [kvm_intel]();
vmx_read_l1_tsc [kvm_intel]();
vmx_handle_external_intr [kvm_intel]() {
}
__srcu_read_lock();
vmx_handle_exit [kvm_intel]() {
  guest_state_valid.part.27 [kvm_intel]() {
    rmode_segment_valid [kvm_intel]() {
      vmx_get_segment [kvm_intel]() {
        vmx_read_guest_seg_selector [kvm_intel]();
      }
      vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
    }
  }
  vmx_interrupt_allowed [kvm_intel]();
  x86_emulate_instruction [kvm]() {
    init_emulate_ctxt [kvm]() {
      vmx_get_cs_db_l_bits [kvm_intel]() {
        vmx_read_guest_seg_ar [kvm_intel]();
      }
      vmx_get_rflags [kvm_intel]();
    }
    x86_decode_insn [kvm]() {
      do_insn_fetch [kvm]() {
        ...
        kvm_fetch_guest_virt [kvm]() {
          vmx_get_cpl [kvm_intel]();
          kvm_read_guest_virt_helper [kvm]() {
            nonpaging_gva_to_gpa [kvm]();
            kvm_read_guest [kvm]() {
              kvm_read_guest_page [kvm]() {
                gfn_to_hva_prot [kvm]() {
                  __gfn_to_hva_many [kvm]();
                }
              }
            }
          } /* kvm_read_guest_virt_helper [kvm] */
        }
      }
      ...
    }
    x86_emulate_insn [kvm]() {
      /* kvm_emulate_insn: ffff0000:fff0:ea 5b e0 00 f0 (real) */
      em_jmp_far [kvm]() {
        load_segment_descriptor [kvm]() {
          emulator_get_segment [kvm]() {
            vmx_get_segment [kvm_intel]() {
              vmx_read_guest_seg_selector [kvm_intel]();
            }
          }
          emulator_set_segment [kvm]() {
            vmx_set_segment [kvm_intel]() {
              fix_rmode_seg [kvm_intel]() {
                vmcs_writel [kvm_intel]();
                vmcs_writel [kvm_intel]();
                vmcs_writel [kvm_intel]();
                vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
                vmcs_writel [kvm_intel]();
              }
              emulation_required [kvm_intel]() {
                guest_state_valid.part.27 [kvm_intel]() {
                  rmode_segment_valid [kvm_intel]() {
                    vmx_get_segment [kvm_intel]() {
                      vmx_read_guest_seg_selector [kvm_intel]();
                    }
                    vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
                  }
                  rmode_segment_valid [kvm_intel]() {
                    vmx_get_segment [kvm_intel]() {
                      vmx_read_guest_seg_selector [kvm_intel]();
                    }
                    vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
                  }
                  rmode_segment_valid [kvm_intel]() {
                    vmx_get_segment [kvm_intel]() {
                      vmx_read_guest_seg_selector [kvm_intel]();
                    }
                    vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
                  }
                  rmode_segment_valid [kvm_intel]() {
                    vmx_get_segment [kvm_intel]() {
                      vmx_read_guest_seg_selector [kvm_intel]();
                    }
                    vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
                  }
                  rmode_segment_valid [kvm_intel]() {
                    vmx_get_segment [kvm_intel]() {
                      vmx_read_guest_seg_selector [kvm_intel]();
                    }
                    vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
                  } /* rmode_segment_valid [kvm_intel] */
                  rmode_segment_valid [kvm_intel]() {
                    vmx_get_segment [kvm_intel]() {
                      vmx_read_guest_seg_selector [kvm_intel]();
                    }
                    vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
                  }
                }
              }
            }
          }
        }
      }
      writeback [kvm]();
      writeback_registers [kvm]();
    }
    vmx_get_interrupt_shadow [kvm_intel]();
    vmx_set_interrupt_shadow [kvm_intel]();
    vmx_get_rflags [kvm_intel]();
    kvm_set_rflags [kvm]() {
      vmx_set_rflags [kvm_intel]() {
        vmcs_writel [kvm_intel]();
      }
    }
  }
  guest_state_valid.part.27 [kvm_intel]() {
    rmode_segment_valid [kvm_intel]() {
      vmx_get_segment [kvm_intel]() {
        vmx_read_guest_seg_selector [kvm_intel]();
      }
      vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
    }
    rmode_segment_valid [kvm_intel]() {
      vmx_get_segment [kvm_intel]() {
        vmx_read_guest_seg_selector [kvm_intel]();
      }
      vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
    }
    rmode_segment_valid [kvm_intel]() {
      vmx_get_segment [kvm_intel]() {
        vmx_read_guest_seg_selector [kvm_intel]();
      }
      vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
    }
    rmode_segment_valid [kvm_intel]() {
      vmx_get_segment [kvm_intel]() {
        vmx_read_guest_seg_selector [kvm_intel]();
      }
      vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
    }
    rmode_segment_valid [kvm_intel]() {
      vmx_get_segment [kvm_intel]() {
        vmx_read_guest_seg_selector [kvm_intel]();
      }
      vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
    }
    rmode_segment_valid [kvm_intel]() {
      vmx_get_segment [kvm_intel]() {
        vmx_read_guest_seg_selector [kvm_intel]();
      }
      vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
    }
  }
  emulation_required [kvm_intel]() {
    guest_state_valid.part.27 [kvm_intel]() {
      rmode_segment_valid [kvm_intel]() {
        vmx_get_segment [kvm_intel]() {
          vmx_read_guest_seg_selector [kvm_intel]();
        }
        vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
      }
      rmode_segment_valid [kvm_intel]() {
        vmx_get_segment [kvm_intel]() {
          vmx_read_guest_seg_selector [kvm_intel]();
        }
        vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
      }
      rmode_segment_valid [kvm_intel]() {
        vmx_get_segment [kvm_intel]() {
          vmx_read_guest_seg_selector [kvm_intel]();
        }
        vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
      }
      rmode_segment_valid [kvm_intel]() {
        vmx_get_segment [kvm_intel]() {
          vmx_read_guest_seg_selector [kvm_intel]();
        }
        vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
      }
      rmode_segment_valid [kvm_intel]() {
        vmx_get_segment [kvm_intel]() {
          vmx_read_guest_seg_selector [kvm_intel]();
        }
        vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
      }
      rmode_segment_valid [kvm_intel]() {
        vmx_get_segment [kvm_intel]() {
          vmx_read_guest_seg_selector [kvm_intel]();
        }
        vmx_segment_access_rights.isra.25.part.26 [kvm_intel]();
      }
    }
  }
}

---- KVM executes at f000:e05b and gets a page fault


kvm_cpu_has_pending_timer [kvm]() {
  apic_has_pending_timer [kvm]();
}
kvm_check_async_pf_completion [kvm]();
kvm_apic_accept_events [kvm]();
kvm_cpu_has_injectable_intr [kvm]() {
  kvm_apic_accept_pic_intr [kvm]();
}
vmx_interrupt_allowed [kvm_intel]();
kvm_cpu_has_injectable_intr [kvm]() {
  kvm_apic_accept_pic_intr [kvm]();
}
enable_irq_window [kvm_intel]() {
  vmcs_writel [kvm_intel]();
}
vmx_save_host_state [kvm_intel]();
__srcu_read_unlock();
rcu_note_context_switch();
/* kvm_entry: vcpu 0 */
vmx_vcpu_run [kvm_intel]() {
  vmcs_writel [kvm_intel]();
  vmcs_writel [kvm_intel]();
  perf_guest_get_msrs();
  /* kvm_exit: reason EXCEPTION_NMI rip 0xe05b info fe05b 80000b0e */
}
vmx_read_l1_tsc [kvm_intel]();
vmx_handle_external_intr [kvm_intel]();
__srcu_read_lock();
vmx_handle_exit [kvm_intel]() {
  handle_exception [kvm_intel]() {
    /* kvm_page_fault: address fe05b error_code 14 */
    kvm_mmu_page_fault [kvm]() {
      nonpaging_page_fault [kvm]() {
        mmu_topup_memory_caches [kvm]();
        gfn_to_memslot_dirty_bitmap.isra.67 [kvm]() {
          gfn_to_memslot [kvm]();
        }
        mapping_level.isra.86 [kvm]() {
          kvm_host_page_size [kvm]() {
            gfn_to_hva [kvm]() {
              __gfn_to_hva_many [kvm]();
            }
            down_read() {
              _cond_resched();
            }
            find_vma();
            vma_kernel_pagesize();
            up_read();
          }
        }
        try_async_pf [kvm]() {
          gfn_to_pfn_async [kvm]() {
            __gfn_to_pfn [kvm]() {
              __gfn_to_pfn_memslot [kvm]() {
                __gfn_to_hva_many [kvm]();
                __get_user_pages_fast() {
                  gup_pud_range() {
                    gup_pte_range();
                  }
                }
              }
            }
          }
        }
        handle_abnormal_pfn [kvm]();
        _raw_spin_lock();
        make_mmu_pages_available.isra.80 [kvm]();
        transparent_hugepage_adjust.isra.91 [kvm]() {
          kvm_is_mmio_pfn [kvm]();
        }
        __direct_map.isra.104 [kvm]() {
          shadow_walk_init [kvm]();
          kvm_mmu_get_page [kvm]() {
            pte_list_add [kvm]();
            /* kvm_mmu_get_page: sp gen 2 gfn 0 1 q0 direct wux !nxe root 0 sync new */
          }
          link_shadow_page.isra.63 [kvm]() {
            mmu_spte_set [kvm]();
          }
          mmu_set_spte [kvm]() {
            set_spte [kvm]() {
              mark_page_dirty [kvm]();
              mmu_spte_update [kvm]() {
                mmu_spte_set [kvm]();
              }
            }
            kvm_mmu_page_get_gfn [kvm]();
            gfn_to_rmap [kvm]() {
              gfn_to_memslot [kvm]();
            }
            pte_list_add [kvm]();
            kvm_release_pfn_clean [kvm]() {
              kvm_is_mmio_pfn [kvm]();
              put_page();
            }
          }
          __direct_pte_prefetch [kvm]() {
            kvm_mmu_page_get_gfn [kvm]();
            gfn_to_memslot_dirty_bitmap.isra.67 [kvm]() {
              gfn_to_memslot [kvm]();
            }
            gfn_to_page_many_atomic [kvm]() {
              __gfn_to_hva_many [kvm]();
              __get_user_pages_fast() {
                gup_pud_range() {
                  gup_pte_range();
                }
              }
            }
            mmu_set_spte [kvm]() {
              set_spte [kvm]() {
                mark_page_dirty [kvm]();
                mmu_spte_update [kvm]() {
                  mmu_spte_set [kvm]();
                }
              }
              kvm_mmu_page_get_gfn [kvm]();
              gfn_to_rmap [kvm]() {
                gfn_to_memslot [kvm]();
              }
              pte_list_add [kvm]();
              kvm_release_pfn_clean [kvm]() {
                kvm_is_mmio_pfn [kvm]();
                put_page();
              }
            }
            mmu_set_spte [kvm]() {
              set_spte [kvm]() {
                mark_page_dirty [kvm]();
                mmu_spte_update [kvm]() {
                  mmu_spte_set [kvm]();
                }
              }
              kvm_mmu_page_get_gfn [kvm]();
              gfn_to_rmap [kvm]() {
                gfn_to_memslot [kvm]();
              }
              pte_list_add [kvm]();
              kvm_release_pfn_clean [kvm]() {
                kvm_is_mmio_pfn [kvm]();
                put_page();
              }
            }
            mmu_set_spte [kvm]() {
              set_spte [kvm]() {
                mark_page_dirty [kvm]();
                mmu_spte_update [kvm]() {
                  mmu_spte_set [kvm]();
                }
              }
              kvm_mmu_page_get_gfn [kvm]();
              gfn_to_rmap [kvm]() {
                gfn_to_memslot [kvm]();
              }
              pte_list_add [kvm]();
              kvm_release_pfn_clean [kvm]() {
                kvm_is_mmio_pfn [kvm]();
                put_page();
              }
            }
            mmu_set_spte [kvm]() {
              set_spte [kvm]() {
                mark_page_dirty [kvm]();
                mmu_spte_update [kvm]() {
                  mmu_spte_set [kvm]();
                }
              }
              kvm_mmu_page_get_gfn [kvm]();
              gfn_to_rmap [kvm]() {
                gfn_to_memslot [kvm]();
              }
              pte_list_add [kvm]();
              kvm_release_pfn_clean [kvm]() {
                kvm_is_mmio_pfn [kvm]();
                put_page();
              }
            }
            mmu_set_spte [kvm]() {
              set_spte [kvm]() {
                mark_page_dirty [kvm]();
                mmu_spte_update [kvm]() {
                  mmu_spte_set [kvm]();
                }
              }
              kvm_mmu_page_get_gfn [kvm]();
              gfn_to_rmap [kvm]() {
                gfn_to_memslot [kvm]();
              }
              pte_list_add [kvm]();
              kvm_release_pfn_clean [kvm]() {
                kvm_is_mmio_pfn [kvm]();
                put_page();
              }
            }
            mmu_set_spte [kvm]() {
              set_spte [kvm]() {
                mark_page_dirty [kvm]();
                mmu_spte_update [kvm]() {
                  mmu_spte_set [kvm]();
                }
              }
              kvm_mmu_page_get_gfn [kvm]();
              gfn_to_rmap [kvm]() {
                gfn_to_memslot [kvm]();
              }
              pte_list_add [kvm]();
              kvm_release_pfn_clean [kvm]() {
                kvm_is_mmio_pfn [kvm]();
                put_page();
              }
            }
          }
        }
      }
    }
  }
}

---- up to this point the trace is identical for shadow-on-shadow (working)
---- and shadow-on-EPT (not working)
----
---- for shadow-on-EPT L1 gets another page fault even though the spte is 
---- present.  The error code is identical.

kvm_cpu_has_pending_timer [kvm]() {
  apic_has_pending_timer [kvm]();
}
kvm_check_async_pf_completion [kvm]();
vmx_save_host_state [kvm_intel]();
__srcu_read_unlock();
rcu_note_context_switch();
/* kvm_entry: vcpu 0 */
vmx_vcpu_run [kvm_intel]() {
  vmcs_writel [kvm_intel]();
  vmcs_writel [kvm_intel]();
  perf_guest_get_msrs();
  /* kvm_exit: reason EXCEPTION_NMI rip 0xe05b info fe05b 80000b0e */
}
vmx_read_l1_tsc [kvm_intel]();
vmx_handle_external_intr [kvm_intel]();
__srcu_read_lock();
vmx_handle_exit [kvm_intel]() {
  handle_exception [kvm_intel]() {
    /* kvm_page_fault: address fe05b error_code 14 */
    kvm_mmu_page_fault [kvm]() {
      nonpaging_page_fault [kvm]() {
        mmu_topup_memory_caches [kvm]();
        gfn_to_memslot_dirty_bitmap.isra.67 [kvm]() {
          gfn_to_memslot [kvm]();
        }
        mapping_level.isra.86 [kvm]() {
          kvm_host_page_size [kvm]() {
            gfn_to_hva [kvm]() {
              __gfn_to_hva_many [kvm]();
            }
            down_read() {
              _cond_resched();
            }
            find_vma();
            vma_kernel_pagesize();
            up_read();
          }
        }
        try_async_pf [kvm]() {
          gfn_to_pfn_async [kvm]() {
            __gfn_to_pfn [kvm]() {
              __gfn_to_pfn_memslot [kvm]() {
                __gfn_to_hva_many [kvm]();
                __get_user_pages_fast() {
                  gup_pud_range() {
                    gup_pte_range();
                  }
                }
              }
            }
          }
        }
        handle_abnormal_pfn [kvm]();
        _raw_spin_lock();
        make_mmu_pages_available.isra.80 [kvm]();
        transparent_hugepage_adjust.isra.91 [kvm]() {
          kvm_is_mmio_pfn [kvm]();
        }
        __direct_map.isra.104 [kvm]() {
          shadow_walk_init [kvm]();

          >>>>>> here starts the difference with the second part,
          >>>>>> this shows that the spte is present

          mmu_set_spte [kvm]() {
            set_spte [kvm]() {
              mark_page_dirty [kvm]();
              mmu_spte_update [kvm]() {
                spte_has_volatile_bits [kvm]();
              }
            }
            kvm_release_pfn_clean [kvm]() {
              kvm_is_mmio_pfn [kvm]();
              put_page();
            }
          }
          __direct_pte_prefetch [kvm]();
        }
      }
    }
  }
}

---- this part of the trace then loops endlessly

Paolo

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Paolo Bonzini]

Il 08/10/2013 15:03, Paolo Bonzini ha scritto:
> Il 08/10/2013 07:38, Kashyap Chamarthy ha scritto:
>> On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
>>> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
>>> stuck with shadow on EPT
>>
>> Paolo, were you able to reproduce this again? Yesterday, on #qemu you
>> mentioned you'll test it again :-)
> 
> Yes, I could reproduce it too.
> 
>>> Boot L2 guest:
> 
> Here L2 doesn't go past the second instruction.  It gets a page fault
> even though the spte is present, and KVM then loops on a page fault
> for 0xfe05b.
> 
> Here is an annotated function_graph trace of L1.
> 
> It's possible that L0 is injecting the same fault repeatedly, i.e.
> they are not different faults from the processor.  I'll get an L0
> trace next.
> 

The L0 trace is not particularly helpful (and probably would not be
particularly helpful even if there were a specific tracepoint for
VMREAD):

287.534156: kvm_exit:             reason VMRESUME rip 0xffffffffa021f8d1 info 0 0
287.534160: kvm_mmu_get_page:     sp gfn 0 0/4 q0 direct --- !pge !nxe root 0sync
287.534161: kvm_entry:            vcpu 0
287.534162: kvm_exit:             reason EXCEPTION_NMI rip 0xe05b info fe05b 80000b0e
287.534170: kvm_mmu_get_page:     sp gfn 0 0/4 q0 direct --- !pge !nxe root 0sync
287.534171: kvm_entry:            vcpu 0
287.534172: kvm_exit:             reason VMREAD rip 0xffffffffa021f97d info 0 0
287.534173: kvm_entry:            vcpu 0
287.534174: kvm_exit:             reason VMREAD rip 0xffffffffa021f996 info 0 0
287.534174: kvm_entry:            vcpu 0
287.534175: kvm_exit:             reason VMREAD rip 0xffffffffa021f9b5 info 0 0
287.534175: kvm_entry:            vcpu 0
287.534177: kvm_exit:             reason VMREAD rip 0xffffffffa021b377 info 0 0
287.534177: kvm_entry:            vcpu 0
287.534178: kvm_exit:             reason VMREAD rip 0xffffffffa021b5ce info 0 0
287.534179: kvm_entry:            vcpu 0
287.534180: kvm_exit:             reason VMREAD rip 0xffffffffa0222c95 info 0 0
287.534180: kvm_entry:            vcpu 0
287.534181: kvm_exit:             reason VMREAD rip 0xffffffffa0222e1c info 0 0
287.534182: kvm_entry:            vcpu 0
287.534185: kvm_exit:             reason MSR_READ rip 0xffffffff8104c2b6 info 0 0
287.534185: kvm_msr:              msr_read 1d9 = 0x0
287.534185: kvm_entry:            vcpu 0

And then it repeats:

287.534186: kvm_exit:             reason VMRESUME rip 0xffffffffa021f8d1 info 0 0
287.534191: kvm_mmu_get_page:     sp gfn 0 0/4 q0 direct --- !pge !nxe root 0sync
287.534192: kvm_entry:            vcpu 0

Trying to add function_graph loses a lot of events.

Paolo

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Kashyap Chamarthy]

On Tue, Oct 8, 2013 at 8:18 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 08/10/2013 15:03, Paolo Bonzini ha scritto:
>> Il 08/10/2013 07:38, Kashyap Chamarthy ha scritto:
>>> On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
>>>> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
>>>> stuck with shadow on EPT
>>>
>>> Paolo, were you able to reproduce this again? Yesterday, on #qemu you
>>> mentioned you'll test it again :-)
>>
>> Yes, I could reproduce it too.
>>
>>>> Boot L2 guest:
>>
>> Here L2 doesn't go past the second instruction.  It gets a page fault
>> even though the spte is present, and KVM then loops on a page fault
>> for 0xfe05b.
>>
>> Here is an annotated function_graph trace of L1.
>>
>> It's possible that L0 is injecting the same fault repeatedly, i.e.
>> they are not different faults from the processor.  I'll get an L0
>> trace next.
>>
>
> The L0 trace is not particularly helpful (and probably would not be
> particularly helpful even if there were a specific tracepoint for
> VMREAD):
>
> 287.534156: kvm_exit:             reason VMRESUME rip 0xffffffffa021f8d1 info 0 0
> 287.534160: kvm_mmu_get_page:     sp gfn 0 0/4 q0 direct --- !pge !nxe root 0sync
> 287.534161: kvm_entry:            vcpu 0
> 287.534162: kvm_exit:             reason EXCEPTION_NMI rip 0xe05b info fe05b 80000b0e
> 287.534170: kvm_mmu_get_page:     sp gfn 0 0/4 q0 direct --- !pge !nxe root 0sync
> 287.534171: kvm_entry:            vcpu 0
> 287.534172: kvm_exit:             reason VMREAD rip 0xffffffffa021f97d info 0 0
> 287.534173: kvm_entry:            vcpu 0
> 287.534174: kvm_exit:             reason VMREAD rip 0xffffffffa021f996 info 0 0
> 287.534174: kvm_entry:            vcpu 0
> 287.534175: kvm_exit:             reason VMREAD rip 0xffffffffa021f9b5 info 0 0
> 287.534175: kvm_entry:            vcpu 0
> 287.534177: kvm_exit:             reason VMREAD rip 0xffffffffa021b377 info 0 0
> 287.534177: kvm_entry:            vcpu 0
> 287.534178: kvm_exit:             reason VMREAD rip 0xffffffffa021b5ce info 0 0
> 287.534179: kvm_entry:            vcpu 0
> 287.534180: kvm_exit:             reason VMREAD rip 0xffffffffa0222c95 info 0 0
> 287.534180: kvm_entry:            vcpu 0
> 287.534181: kvm_exit:             reason VMREAD rip 0xffffffffa0222e1c info 0 0
> 287.534182: kvm_entry:            vcpu 0
> 287.534185: kvm_exit:             reason MSR_READ rip 0xffffffff8104c2b6 info 0 0
> 287.534185: kvm_msr:              msr_read 1d9 = 0x0
> 287.534185: kvm_entry:            vcpu 0
>
> And then it repeats:
>
> 287.534186: kvm_exit:             reason VMRESUME rip 0xffffffffa021f8d1 info 0 0
> 287.534191: kvm_mmu_get_page:     sp gfn 0 0/4 q0 direct --- !pge !nxe root 0sync
> 287.534192: kvm_entry:            vcpu 0
>
> Trying to add function_graph loses a lot of events.
>
> Paolo


Paolo, here's L0's trace with shadow-on-EPT.

I invoked with the below command line on L0

 $ /usr/bin/trace-cmd record -b 10000 -e kvmmmu

then, started the L2 guest.

Report:

 $ trace-cmd report 2>&1 | less


The trace data file was large (150MB), just pasting the repeating message here:
---------------------
.
.
.
 qemu-system-x86-2006  [000] 57931.272466: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
 qemu-system-x86-2006  [000] 57931.272468: kvm_mmu_paging_element: pte
1c0f067 level 4
 qemu-system-x86-2006  [000] 57931.272469: kvm_mmu_paging_element: pte
1c10063 level 3
 qemu-system-x86-2006  [000] 57931.272469: kvm_mmu_paging_element: pte
12001e1 level 2
 qemu-system-x86-2006  [000] 57931.272497: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
 qemu-system-x86-2006  [000] 57931.272497: kvm_mmu_paging_element: pte
1c0f067 level 4
 qemu-system-x86-2006  [000] 57931.272497: kvm_mmu_paging_element: pte
1c10063 level 3
 qemu-system-x86-2006  [000] 57931.272497: kvm_mmu_paging_element: pte
12001e1 level 2
 qemu-system-x86-2006  [000] 57931.272516: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
 qemu-system-x86-2006  [000] 57931.272516: kvm_mmu_paging_element: pte
1c0f067 level 4
 qemu-system-x86-2006  [000] 57931.272516: kvm_mmu_paging_element: pte
1c10063 level 3
 qemu-system-x86-2006  [000] 57931.272517: kvm_mmu_paging_element: pte
12001e1 level 2
 qemu-system-x86-2006  [000] 57931.272525: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
 qemu-system-x86-2006  [000] 57931.272525: kvm_mmu_paging_element: pte
1c0f067 level 4
 qemu-system-x86-2006  [000] 57931.272525: kvm_mmu_paging_element: pte
1c10063 level 3
 qemu-system-x86-2006  [000] 57931.272525: kvm_mmu_paging_element: pte
12001e1 level 2
 qemu-system-x86-2006  [000] 57931.272540: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
 qemu-system-x86-2006  [000] 57931.272540: kvm_mmu_paging_element: pte
1c0f067 level 4
 qemu-system-x86-2006  [000] 57931.272540: kvm_mmu_paging_element: pte
1c10063 level 3
 qemu-system-x86-2006  [000] 57931.272541: kvm_mmu_paging_element: pte
12001e1 level 2
 qemu-system-x86-2006  [000] 57931.272580: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
 qemu-system-x86-2006  [000] 57931.272581: kvm_mmu_paging_element: pte
1c0f067 level 4
 qemu-system-x86-2006  [000] 57931.272581: kvm_mmu_paging_element: pte
1c10063 level 3
 qemu-system-x86-2006  [000] 57931.272581: kvm_mmu_paging_element: pte
12001e1 level 2
 qemu-system-x86-2006  [000] 57931.272593: kvm_mmu_pagetable_walk:
addr ffffffff813d8404 pferr 10 F
 qemu-system-x86-2006  [000] 57931.272594: kvm_mmu_paging_element: pte
1c0f067 level 4
 qemu-system-x86-2006  [000] 57931.272594: kvm_mmu_paging_element: pte
1c10063 level 3
 qemu-system-x86-2006  [000] 57931.272594: kvm_mmu_paging_element: pte
12001e1 level 2
.
.
.
.
qemu-system-x86-2007  [000] 57967.047692: kvm_mmu_get_page:     sp gfn
0 0/4 q0 direct --- !pge !nxe root 3sync
 qemu-system-x86-2007  [000] 57967.047698: kvm_mmu_get_page:     sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
 qemu-system-x86-2007  [000] 57967.047701: kvm_mmu_get_page:     sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
 qemu-system-x86-2007  [000] 57967.047706: kvm_mmu_get_page:     sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
 qemu-system-x86-2007  [000] 57967.047710: kvm_mmu_get_page:     sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
 qemu-system-x86-2007  [000] 57967.047724: kvm_mmu_get_page:     sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
 qemu-system-x86-2007  [000] 57967.047727: kvm_mmu_get_page:     sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
 qemu-system-x86-2007  [000] 57967.047732: kvm_mmu_get_page:     sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
 qemu-system-x86-2007  [000] 57967.047736: kvm_mmu_get_page:     sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3sync
 qemu-system-x86-2007  [000] 57967.047741: kvm_mmu_get_page:     sp
gfn 0 0/4 q0 direct --- !pge !nxe root 3s
---------------------

/kashyap

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Gleb Natapov]

On Wed, Oct 09, 2013 at 11:52:29AM +0530, Kashyap Chamarthy wrote:
> On Tue, Oct 8, 2013 at 8:18 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > Il 08/10/2013 15:03, Paolo Bonzini ha scritto:
> >> Il 08/10/2013 07:38, Kashyap Chamarthy ha scritto:
> >>> On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
> >>>> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
> >>>> stuck with shadow on EPT
> >>>
> >>> Paolo, were you able to reproduce this again? Yesterday, on #qemu you
> >>> mentioned you'll test it again :-)
> >>
> >> Yes, I could reproduce it too.

72f857950f6f19cba42a9ded078bbc99f10aa667 causes it and I ran kernel
without this commit. Will look into it.

--
			Gleb.

Re: [nvmx testing, linux on linux] Disabling EPT in L1 renders L2 stuck on boot

[Kashyap Chamarthy]

On Wed, Oct 9, 2013 at 1:46 PM, Gleb Natapov <gleb@redhat.com> wrote:
> On Wed, Oct 09, 2013 at 11:52:29AM +0530, Kashyap Chamarthy wrote:
>> On Tue, Oct 8, 2013 at 8:18 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> > Il 08/10/2013 15:03, Paolo Bonzini ha scritto:
>> >> Il 08/10/2013 07:38, Kashyap Chamarthy ha scritto:
>> >>> On Mon, Oct 7, 2013 at 6:29 PM, Kashyap Chamarthy <kashyap.cv@gmail.com> wrote:
>> >>>> Gleb, so I just did a trace of KVM MMU to try to understand why L2 is
>> >>>> stuck with shadow on EPT
>> >>>
>> >>> Paolo, were you able to reproduce this again? Yesterday, on #qemu you
>> >>> mentioned you'll test it again :-)
>> >>
>> >> Yes, I could reproduce it too.
>
> 72f857950f6f19cba42a9ded078bbc99f10aa667 causes it and I ran kernel
> without this commit. Will look into it.

Thank you, this fixed it. I tested by applying your patch from the
other email.[1]

I'm now able to boot L2 guest with shadow-on-EPT with the above fix.

Thank you.

/kashyap

[1] http://www.mail-archive.com/kvm@vger.kernel.org/msg96484.html
>
> --
>                         Gleb.