ESXi under KVM

ESXi under KVM

[Oleg Ilyin]

Hi,
 could you help me please to solve my issue with run VM on ESXi which
running on KVM?


I have   Ubuntu 14.04.4 LTS
             ESX  6.0   vmware-201601001-3380124-iso
             Running hypervisor: QEMU 2.4.0

I try to run VM on EXSi  which running under KVM.
So, ESXi was successfull installed,  the command in ps :

qemu-system-x86_64 -enable-kvm -name esx -S -machine
pc-i440fx-2.4,accel=kvm,usb=off -cpu Nehalem,+invpcid,+vmx -m 8096
-realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid
b2c3b37b-9596-98ab-8e0d-f05df89d2920 -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/esx.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/var/lib/libvirt/images/esx-20.img,if=none,id=drive-ide0-0-0,format=raw
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
tap,fd=26,id=hostnet0 -device
e1000,netdev=hostnet0,id=net0,mac=52:54:00:96:74:1c,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -vnc 0.0.0.0:0 -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4

But I always get error like :
Virtualized INtel VT-x/EPT is not supported on this platform.
Continure without virtualized msg.intel.hvhwmmu? during starting VM on
ESXi

cat /proc/cpuinfo  on 0 host machine
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq
dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid
dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer xsave avx
f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm
tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2
smep bmi2 erms invpcid

I added these options in  /etc/vmware/config

vhv.allow = "TRUE"
hv.assumeEnabled="TRUE"
vhv.enable = "TRUE"
vmx.allowNested = "TRUE"

But, it didn't help me.  ( I rebooted ESXi also)

It affected only for 32-bit VM, which failed with new error .


cat /sys/module/kvm_intel/parameters/nested
Y
cat /sys/module/kvm/parameters/ignore_msrs
Y
cat /sys/module/kvm_intel/parameters/ept
Y

lsmod |grep kvm
kvm_intel             143148  6
kvm                   451729  1 kvm_intel


my xml:

<domain type='kvm' id='41'>
  <name>esx</name>
  <uuid>b2c3b37b-9596-98ab-8e0d-f05df89d2920</uuid>
  <memory unit='KiB'>8290304</memory>
  <currentMemory unit='KiB'>8290304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Nehalem</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='lm'/>
    <feature policy='require' name='lahf_lm'/>
    <feature policy='require' name='nx'/>
    <feature policy='require' name='fxsr'/>
    <feature policy='require' name='invpcid'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/esx-20.img'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:96:74:1c'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none'/>
</domain>

I also try to run it on 2.5.1 and 2.6, but it didn't help me.


please, what is wrong ?  how to be able to run VM on ESXi which runs on the KVM?

Re: ESXi under KVM

[Mihai Donțu]

On Tue, 17 May 2016 16:42:05 +0300 Oleg Ilyin wrote:
>   <cpu mode='custom' match='exact'>
>     <model fallback='allow'>Nehalem</model>
>     <vendor>Intel</vendor>
>     <feature policy='require' name='vmx'/>
>     <feature policy='require' name='lm'/>
>     <feature policy='require' name='lahf_lm'/>
>     <feature policy='require' name='nx'/>
>     <feature policy='require' name='fxsr'/>
>     <feature policy='require' name='invpcid'/>
>   </cpu>

I use the following CPU configuration with xen & Linux (nested kvm):

<cpu mode='custom' match='exact'>
	<model fallback='allow'>Haswell-noTSX</model>
	<vendor>Intel</vendor>
	<topology sockets='1' cores='4' threads='1'/>
	<feature policy='require' name='abm'/>
	<feature policy='require' name='pdpe1gb'/>
	<feature policy='require' name='rdrand'/>
	<feature policy='require' name='f16c'/>
	<feature policy='require' name='osxsave'/>
	<feature policy='require' name='pdcm'/>
	<feature policy='require' name='xtpr'/>
	<feature policy='require' name='tm2'/>
	<feature policy='require' name='est'/>
	<feature policy='require' name='smx'/>
	<feature policy='require' name='vmx'/>
	<feature policy='require' name='ds_cpl'/>
	<feature policy='require' name='monitor'/>
	<feature policy='require' name='dtes64'/>
	<feature policy='require' name='pbe'/>
	<feature policy='require' name='tm'/>
	<feature policy='require' name='ht'/>
	<feature policy='require' name='ss'/>
	<feature policy='require' name='acpi'/>
	<feature policy='require' name='ds'/>
	<feature policy='require' name='vme'/>
</cpu>

I've put it together from various bits I found all over the Internet.

The CPU model should be whatever is closest to your host. I would also
recommend a newer kernel (Ubuntu 16.04?).

-- 
Mihai Donțu

Re: ESXi under KVM

[Bandan Das]

Oleg Ilyin <on.ilyin@gmail.com> writes:

> Hi,
>  could you help me please to solve my issue with run VM on ESXi which
> running on KVM?
>
>
> I have   Ubuntu 14.04.4 LTS
>              ESX  6.0   vmware-201601001-3380124-iso
>              Running hypervisor: QEMU 2.4.0
>
> I try to run VM on EXSi  which running under KVM.
> So, ESXi was successfull installed,  the command in ps :
>
> qemu-system-x86_64 -enable-kvm -name esx -S -machine
> pc-i440fx-2.4,accel=kvm,usb=off -cpu Nehalem,+invpcid,+vmx -m 8096
> -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid
> b2c3b37b-9596-98ab-8e0d-f05df89d2920 -no-user-config -nodefaults
> -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/esx.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
> -no-shutdown -boot strict=on -device
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=/var/lib/libvirt/images/esx-20.img,if=none,id=drive-ide0-0-0,format=raw
> -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
> -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device
> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
> tap,fd=26,id=hostnet0 -device
> e1000,netdev=hostnet0,id=net0,mac=52:54:00:96:74:1c,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -vnc 0.0.0.0:0 -device
> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
>
> But I always get error like :
> Virtualized INtel VT-x/EPT is not supported on this platform.
> Continure without virtualized msg.intel.hvhwmmu? during starting VM on
> ESXi

I haven't tried to figure out that virtual hardware mode ESX has but with testing help
from infernix on #qemu, I know these are the things ESX looks for:

invept global

invept single context (easy to advertise a dummy, kvm actually used to but we removed
it sometime in the past)

ept execute only bit (on my todo list)

vpid:
global/single context/single context retaining globals
I first two are advertised, the last one isn't and just advertising this bit seems to work
atleast for the boot part.

Also, you probably need kvm=off.

If you get to the log when esx tries to launch the guest, we will
have a clearer picture of what's actually failing but I suspect what
I mentioned above is what you are hitting.

> cat /proc/cpuinfo  on 0 host machine
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
> rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq
> dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid
> dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer xsave avx
> f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm
> tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2
> smep bmi2 erms invpcid
>
> I added these options in  /etc/vmware/config
>
> vhv.allow = "TRUE"
> hv.assumeEnabled="TRUE"
> vhv.enable = "TRUE"

As far as I know, this won't work. You have to trick esx into thinking
it's running on real hardware. vhv has esx-isms we probably don't know
of yet.

Bandan

> vmx.allowNested = "TRUE"
>
> But, it didn't help me.  ( I rebooted ESXi also)
>
> It affected only for 32-bit VM, which failed with new error .
>
>
> cat /sys/module/kvm_intel/parameters/nested
> Y
> cat /sys/module/kvm/parameters/ignore_msrs
> Y
> cat /sys/module/kvm_intel/parameters/ept
> Y
>
> lsmod |grep kvm
> kvm_intel             143148  6
> kvm                   451729  1 kvm_intel
>
>
> my xml:
>
> <domain type='kvm' id='41'>
>   <name>esx</name>
>   <uuid>b2c3b37b-9596-98ab-8e0d-f05df89d2920</uuid>
>   <memory unit='KiB'>8290304</memory>
>   <currentMemory unit='KiB'>8290304</currentMemory>
>   <vcpu placement='static'>4</vcpu>
>   <resource>
>     <partition>/machine</partition>
>   </resource>
>   <os>
>     <type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>     <pae/>
>   </features>
>   <cpu mode='custom' match='exact'>
>     <model fallback='allow'>Nehalem</model>
>     <vendor>Intel</vendor>
>     <feature policy='require' name='vmx'/>
>     <feature policy='require' name='lm'/>
>     <feature policy='require' name='lahf_lm'/>
>     <feature policy='require' name='nx'/>
>     <feature policy='require' name='fxsr'/>
>     <feature policy='require' name='invpcid'/>
>   </cpu>
>   <clock offset='utc'/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>restart</on_crash>
>   <devices>
>     <emulator>/usr/bin/kvm-spice</emulator>
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='raw'/>
>       <source file='/var/lib/libvirt/images/esx-20.img'/>
>       <target dev='hda' bus='ide'/>
>       <alias name='ide0-0-0'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
>     <disk type='block' device='cdrom'>
>       <driver name='qemu' type='raw'/>
>       <target dev='hdc' bus='ide'/>
>       <readonly/>
>       <alias name='ide0-1-0'/>
>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>     </disk>
>     <controller type='usb' index='0'>
>       <alias name='usb0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x2'/>
>     </controller>
>     <controller type='pci' index='0' model='pci-root'>
>       <alias name='pci.0'/>
>     </controller>
>     <controller type='ide' index='0'>
>       <alias name='ide0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x1'/>
>     </controller>
>     <interface type='bridge'>
>       <mac address='52:54:00:96:74:1c'/>
>       <source bridge='virbr0'/>
>       <target dev='vnet0'/>
>       <model type='e1000'/>
>       <alias name='net0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </interface>
>     <serial type='pty'>
>       <source path='/dev/pts/2'/>
>       <target port='0'/>
>       <alias name='serial0'/>
>     </serial>
>     <console type='pty' tty='/dev/pts/2'>
>       <source path='/dev/pts/2'/>
>       <target type='serial' port='0'/>
>       <alias name='serial0'/>
>     </console>
>     <input type='mouse' bus='ps2'/>
>     <input type='keyboard' bus='ps2'/>
>     <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
>       <listen type='address' address='0.0.0.0'/>
>     </graphics>
>     <video>
>       <model type='cirrus' vram='9216' heads='1'/>
>       <alias name='video0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
>     </video>
>     <memballoon model='virtio'>
>       <alias name='balloon0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
>     </memballoon>
>   </devices>
>   <seclabel type='none'/>
> </domain>
>
> I also try to run it on 2.5.1 and 2.6, but it didn't help me.
>
>
> please, what is wrong ?  how to be able to run VM on ESXi which runs on the KVM?
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html