[Qemu-devel] Nested KVM is weird?

[Qemu-devel] Nested KVM is weird?

[Jun Koi]

[-- Attachment #1: Type: text/plain, Size: 833 bytes --]

Hi,

I managed to run VMWare ESXi hypervisor (type 1) inside KVM. From there, I
can install & run nested guest Linux on this ESXi.

However, I find this very strange: I put some printk() at the top of
function nested_vmx_run(), which should run to handle nested KVM. Below is
the definition of nested_vmx_run() in vmx.c


/*
 * nested_vmx_run() handles a nested entry, i.e., a VMLAUNCH or VMRESUME on
L1
 * for running an L2 nested guest.
 */
static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch)


What I cannot understand is that this function is never hit, because my
printk() never prints out anything to dmesg output.

So this means ESXi never uses VMResume/VMLaunch? How is this possible,
because it uses VMX for its implementation?

I am banging my head into keyboard to figure out why. Any idea, please?

Thanks,
Jun

[-- Attachment #2: Type: text/html, Size: 1100 bytes --]

Re: [Qemu-devel] Nested KVM is weird?

[Muli Ben-Yehuda]

On Sun, Jun 01, 2014 at 05:54:25PM +0700, Jun Koi wrote:

> So this means ESXi never uses VMResume/VMLaunch? How is this
> possible, because it uses VMX for its implementation?

ESXi will fall back to binary translation if it decides that it cannot
use VMX for some reason. Look at the L2's log file inside ESXi and you
will likely find some errors related to running nested, MSRs KVM does
not emulate properly, or something else that causes ESXi to use binary
translation for its L2.

Cheers,
Muli

Re: [Qemu-devel] Nested KVM is weird?

[Jun Koi]

[-- Attachment #1: Type: text/plain, Size: 935 bytes --]

On Sun, Jun 1, 2014 at 5:59 PM, Muli Ben-Yehuda <mulix@mulix.org> wrote:

> On Sun, Jun 01, 2014 at 05:54:25PM +0700, Jun Koi wrote:
>
> > So this means ESXi never uses VMResume/VMLaunch? How is this
> > possible, because it uses VMX for its implementation?
>
> ESXi will fall back to binary translation if it decides that it cannot
> use VMX for some reason. Look at the L2's log file inside ESXi and you
> will likely find some errors related to running nested, MSRs KVM does
> not emulate properly, or something else that causes ESXi to use binary
> translation for its L2.
>

now i noticed in dmesg output, there is one message with content:

    " VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL does not work properly. Using
workaround"

(1) do you think this VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL is the reason why
ESXi falls back to binary translation?

(3) how hard it is to fix this issue, so we have complete support for ESXi?

thanks,
Jun

[-- Attachment #2: Type: text/html, Size: 1421 bytes --]

Re: [Qemu-devel] Nested KVM is weird?

[Muli Ben-Yehuda]

On Sun, Jun 01, 2014 at 11:30:02PM +0700, Jun Koi wrote:

> (1) do you think this VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL is the
>     reason why ESXi falls back to binary translation?

It might be, its been a while since I got ESXi to use VMX on KVM. Take
a look at the VMware log file for the L2, it should have a lot more
information.

Cheers,
Muli