ESX on KVM requirements

ESX on KVM requirements

[Ben Sanders]

I'm looking to reproduce the results of Alexander Graf earlier this
year when he said that he was able to get ESX to run a ReactOS guest
all on top of KVM.  More information on that can be found here:
http://kerneltrap.org/mailarchive/linux-kvm/2009/1/5/4600354/thread

So far, I can only boot esx into service console mode, not regular
mode or debug mode without a kernel panic (in ESX).  I've enabled
nesting, both in the module and in the kvm command on v86, using the
-cpu phenom option as well.

First question is about cpu type.  I've been using some dual core AMD
cpu's with svm enabled, but I'm wondering if I actually need a phenom
cpu for all of this to work.  I was under the impression that KVM/QEMU
could emulate a different CPU if the features didn't exist.

Second question is about the host OS.  Are the nesting features in KVM
only supported in an x86 OS? or should x86_64 work as well?  I've
tried each (in varying degrees), but if one implementation wouldn't
work (like emulating a phenom, an x86_64 processor on an x86 host OS),
I'd like to know so I can stay away from that.  Also, I've been using
Ubuntu 8.10 and 9.04, but if there's some reason I should switch to a
different distribution, I'd like to know that as well.

Thanks for your help

Re: ESX on KVM requirements

[Avi Kivity]

Ben Sanders wrote:
> I'm looking to reproduce the results of Alexander Graf earlier this
> year when he said that he was able to get ESX to run a ReactOS guest
> all on top of KVM.  More information on that can be found here:
> http://kerneltrap.org/mailarchive/linux-kvm/2009/1/5/4600354/thread
>
> So far, I can only boot esx into service console mode, not regular
> mode or debug mode without a kernel panic (in ESX).  I've enabled
> nesting, both in the module and in the kvm command on v86, using the
> -cpu phenom option as well.
>
> First question is about cpu type.  I've been using some dual core AMD
> cpu's with svm enabled, but I'm wondering if I actually need a phenom
> cpu for all of this to work.  I was under the impression that KVM/QEMU
> could emulate a different CPU if the features didn't exist.
>
>   

No, kvm (mostly) doesn't emulate cpu features.  I don't know why -cpu 
phenom is required.  Copying Alex.

> Second question is about the host OS.  Are the nesting features in KVM
> only supported in an x86 OS? or should x86_64 work as well?  I've
> tried each (in varying degrees), but if one implementation wouldn't
> work (like emulating a phenom, an x86_64 processor on an x86 host OS),
> I'd like to know so I can stay away from that.  Also, I've been using
> Ubuntu 8.10 and 9.04, but if there's some reason I should switch to a
> different distribution, I'd like to know that as well.
>   

x86 generally means i386 and x86_64.  Both should work, and x86_64 is 
recommended.


-- 
error compiling committee.c: too many arguments to function

Re: ESX on KVM requirements

[Alexander Graf]

On 14.06.2009, at 13:57, Avi Kivity <avi@redhat.com> wrote:

> Ben Sanders wrote:
>> I'm looking to reproduce the results of Alexander Graf earlier this
>> year when he said that he was able to get ESX to run a ReactOS guest
>> all on top of KVM.  More information on that can be found here:
>> http://kerneltrap.org/mailarchive/linux-kvm/2009/1/5/4600354/thread
>>
>> So far, I can only boot esx into service console mode, not regular
>> mode or debug mode without a kernel panic (in ESX).  I've enabled
>> nesting, both in the module and in the kvm command on v86, using the
>> -cpu phenom option as well.
>>
>> First question is about cpu type.  I've been using some dual core AMD
>> cpu's with svm enabled, but I'm wondering if I actually need a phenom
>> cpu for all of this to work.  I was under the impression that KVM/ 
>> QEMU
>> could emulate a different CPU if the features didn't exist.
>>
>>
>
> No, kvm (mostly) doesn't emulate cpu features.  I don't know why - 
> cpu phenom is required.  Copying Alex.

I needed nested paging to have things work, so phenom was mandatory.

Also, because I was running things on a Phenom, I figured the safest  
bet is to be truthful in telling the guest what it's running on. Also  
ESX checked for specific cpuid revisions and I knew the CPU I was on  
is safe.

If you get it running in debug mode, do -serial stdio and send me the  
output so I can see if I remember the cure ;)

Alex


>
>
>> Second question is about the host OS.  Are the nesting features in  
>> KVM
>> only supported in an x86 OS? or should x86_64 work as well?  I've
>> tried each (in varying degrees), but if one implementation wouldn't
>> work (like emulating a phenom, an x86_64 processor on an x86 host  
>> OS),
>> I'd like to know so I can stay away from that.  Also, I've been using
>> Ubuntu 8.10 and 9.04, but if there's some reason I should switch to a
>> different distribution, I'd like to know that as well.
>>
>
> x86 generally means i386 and x86_64.  Both should work, and x86_64  
> is recommended.
>
>
> -- 
> error compiling committee.c: too many arguments to function
>

Re: ESX on KVM requirements

[Alexander Graf]

On 19.06.2009, at 03:19, Ben Sanders wrote:

>> I needed nested paging to have things work, so phenom was mandatory.
>>
>> Also, because I was running things on a Phenom, I figured the  
>> safest bet is
>> to be truthful in telling the guest what it's running on. Also ESX  
>> checked
>> for specific cpuid revisions and I knew the CPU I was on is safe.
>>
>> If you get it running in debug mode, do -serial stdio and send me  
>> the output
>> so I can see if I remember the cure ;)
>>
>> Alex
>
>
> I cut out the mailing list since people probably don't want my  
> attachment.
>
> I switched to a phenom 9950 to see if i'd have some better luck.
>
> I'm runing kvm v86 with nested=1 and -enable nesting on Ubuntu 9.04
>
> Here's the command i'm running.  Setting the nic model was necessary
> to get it to install.
> qemu-system-x86_64 -cpu phenom -enable-nesting -hda ~/esx.img -m 1024
> -net nic,model=e1000 -net user
>
> Attached is the debug output.  Thanks for your help.

Looks like you're missing my VMware interface extension patches that  
implement TSC fake value passing to the guest? The series was called  
"VMware ESX guest bringup (partial)".

Also, try to keep the ML in CC and simply pastebin instead of  
attaching the log file. There are probably more people out there who  
have the same question ;-).

Alex

Re: ESX on KVM requirements

[Ben Sanders]

I applied the patch I found here:
http://thread.gmane.org/gmane.comp.emulators.qemu/35419
and here are my new debug results: http://pastebin.com/m60d53e9d

It looks like it is still crashing in the same place, so I double
checked that the patch was applied correctly.  I also didn't have the
"npt=1" originally on in the module, so I added that as well.

Any ideas?

Thanks,

Ben

On Fri, Jun 19, 2009 at 1:30 AM, Alexander Graf<agraf@suse.de> wrote:
>
> Looks like you're missing my VMware interface extension patches that
> implement TSC fake value passing to the guest? The series was called "VMware
> ESX guest bringup (partial)".
>
> Also, try to keep the ML in CC and simply pastebin instead of attaching the
> log file. There are probably more people out there who have the same
> question ;-).
>
> Alex
>
>

Re: ESX on KVM requirements

[Alexander Graf]

On 25.06.2009, at 04:04, Ben Sanders wrote:

> I applied the patch I found here:
> http://thread.gmane.org/gmane.comp.emulators.qemu/35419
> and here are my new debug results: http://pastebin.com/m60d53e9d
>
> It looks like it is still crashing in the same place, so I double
> checked that the patch was applied correctly.  I also didn't have the
> "npt=1" originally on in the module, so I added that as well.

Well, your machine can do npt, right?

> Any ideas?

Hm. ESX breaks because of

ASSERT /build/mts/release/bora-123630/bora/vmkernel/sched/ 
cpusched_alloc.c:3399

which is just after the TSC check. So I'm pretty sure it's the "Make  
vmport report the processor	speed" patch that should make your  
scenario work.

I can't really tell why it doesn't work for you.

Alex

>
> Thanks,
>
> Ben

Re: ESX on KVM requirements

[Ben Sanders]

Finally got it to work on a 32 bit OS (Ubuntu 9.04), both on the
phenom 9950 and another machine.  I haven't tried running any guests
yet.

I suppose the TSC patch doesn't work on 64 bit hosts.

Thanks for all your help,

Ben

On Fri, Jun 26, 2009 at 1:38 AM, Alexander Graf<agraf@suse.de> wrote:
>
> Well, your machine can do npt, right?
>
>> Any ideas?
>
> Hm. ESX breaks because of
>
> ASSERT
> /build/mts/release/bora-123630/bora/vmkernel/sched/cpusched_alloc.c:3399
>
> which is just after the TSC check. So I'm pretty sure it's the "Make vmport
> report the processor     speed" patch that should make your scenario work.
>
> I can't really tell why it doesn't work for you.
>
> Alex
>

Re: ESX on KVM requirements

[Ben Sanders]

Maybe my first announcement of it working was a bit premature.  ESX is
indeed running on KVM, but is somewhat useless as I can't seem to add
a datastore (where ESX puts the virtual machine disks).  I tried
adding a SCSI drive with "-drive file=datastore.img,if=scsi" but to no
avail.  It seems that ESX doesn't have the drivers for the type of
SCSI drive that KVM emulates.

To Alexander Graf: Is there anything special you did when you got
ReactOS running on ESX?
To everyone else: I've never used SCSI drives before (qemu or
otherwise), is there anything more I have to do than creating a
rawdisk image and using the command shown above?

Thanks,

Ben

On Wed, Jul 1, 2009 at 10:49 AM, Ben Sanders<ben.m.sanders+kvm@gmail.com> wrote:
> Finally got it to work on a 32 bit OS (Ubuntu 9.04), both on the
> phenom 9950 and another machine.  I haven't tried running any guests
> yet.
>
> I suppose the TSC patch doesn't work on 64 bit hosts.
>
> Thanks for all your help,
>
> Ben

Re: ESX on KVM requirements

[Alexander Graf]

On 28.07.2009, at 23:24, Ben Sanders <ben.m.sanders+kvm@gmail.com>  
wrote:

> Maybe my first announcement of it working was a bit premature.  ESX is
> indeed running on KVM, but is somewhat useless as I can't seem to add
> a datastore (where ESX puts the virtual machine disks).  I tried
> adding a SCSI drive with "-drive file=datastore.img,if=scsi" but to no
> avail.  It seems that ESX doesn't have the drivers for the type of
> SCSI drive that KVM emulates.
>
> To Alexander Graf: Is there anything special you did when you got
> ReactOS running on ESX?
> To everyone else: I've never used SCSI drives before (qemu or
> otherwise), is there anything more I have to do than creating a
> rawdisk image and using the command shown above?

I think I used an NFS backed datastore back then.

Alex

>
> Thanks,
>
> Ben
>
> On Wed, Jul 1, 2009 at 10:49 AM, Ben Sanders<ben.m.sanders+kvm@gmail.com 
> > wrote:
>> Finally got it to work on a 32 bit OS (Ubuntu 9.04), both on the
>> phenom 9950 and another machine.  I haven't tried running any guests
>> yet.
>>
>> I suppose the TSC patch doesn't work on 64 bit hosts.
>>
>> Thanks for all your help,
>>
>> Ben