[Qemu-devel] Illumos/SmarOS support

[Qemu-devel] Illumos/SmarOS support

[Lee Essen]

[-- Attachment #1: Type: text/plain, Size: 1408 bytes --]

Resend from different email -- not sure what happened to my first 2 attempts!

Hi,

I've been working on getting the mainline qemu tree working on SmartOS (illumos), the current qemu support seems to be a branch of qemu-kvm and we've been having some problems getting spice working, so moving to the main qemu tree seemed sensible.

Anyway, couple of questions…

1. Are you happy to accept patches that enable this?   
	- There are a few build related things that don't work … assumption of sh being bash, missing -64 argument to dtrace etc. 
	- There are also some dtrace language problems with tracetool (no support for bool, and self being reserved.) 
	- More fundamentally some object linking issues with dtrace which I'll need some help to find the best way to resolve.
	- Specific #ifdef CONFIG_SOLARIS/__sun__ stuff to support differences
	- ultimately the support for vnic's … but I haven't even looked at that yet.

	The current state is that it's working ok for Windows guests, and non-KVM for Linux, but kvm & linux seems to fail, hence question number 2.

2. The last thing that happens when I try to boot linux with kvm enabled is some apic mmio accesses, then it stops at a hlt instruction and never gets going again. Is there something I can enable to help debug this? Or has anyone else seen similar behaviour? I've run out of places to look.

Many thanks,

Lee.

[-- Attachment #2.1: Type: text/html, Size: 1459 bytes --]

Re: [Qemu-devel] Illumos/SmarOS support

[Stefan Hajnoczi]

On Thu, Mar 15, 2012 at 11:19 AM, Lee Essen <lee.essen@me.com> wrote:
> I've been working on getting the mainline qemu tree working on SmartOS
> (illumos), the current qemu support seems to be a branch of qemu-kvm and
> we've been having some problems getting spice working, so moving to the main
> qemu tree seemed sensible.

qemu-kvm.git is a fork of qemu.git.  qemu-kvm.git periodically pulls
in qemu.git updates, but patches are also sent back to mainline
qemu.git so the delta between the two has been decreasing.  Perhaps
the two can merge soon.

> Anyway, couple of questions…
>
> 1. Are you happy to accept patches that enable this?
> 	- There are a few build related things that don't work … assumption of sh
> being bash, missing -64 argument to dtrace etc.

Sounds good, but have you checked the Joyent qemu-kvm Illumos port to
see what fixes they already have?

https://github.com/joyent/illumos-kvm-cmd

> 	- There are also some dtrace language problems with tracetool (no support
> for bool, and self being reserved.)

If there is a reserved keyword then we need to update ./trace-events
to avoid using it, and we need to add a check to scripts/tracetool so
that it refuses such trace event definitions.  There was discussion
about reserved keywords recently here:
http://patchwork.ozlabs.org/patch/146054/

> 	- More fundamentally some object linking issues with dtrace which I'll need
> some help to find the best way to resolve.

I think Joyent has already done this, they said they use QEMU tracing
with DTrace as the backend.

> 	- Specific #ifdef CONFIG_SOLARIS/__sun__ stuff to support differences
> 	- ultimately the support for vnic's … but I haven't even looked at that
> yet.
>
> 	The current state is that it's working ok for Windows guests, and non-KVM
> for Linux, but kvm & linux seems to fail, hence question number 2.
>
> 2. The last thing that happens when I try to boot linux with kvm enabled is
> some apic mmio accesses, then it stops at a hlt instruction and never gets
> going again. Is there something I can enable to help debug this? Or has
> anyone else seen similar behaviour? I've run out of places to look.

So you are using qemu.git/master with the IllumOS KVM port?

I think what you are trying to do makes sense - IllumOS as a host OS
and QEMU platform should be actively involved in the upstream QEMU
community.  But you may want to first check the Joyent port to see
what can easily be upstreamed and also try talking with them if you
haven't already.  I'm not sure why things aren't being pushed upstream
but perhaps the process can be started.

Stefan

Re: [Qemu-devel] Illumos/SmarOS support

[Andreas Färber]

Hi,

Am 15.03.2012 12:19, schrieb Lee Essen:
> I've been working on getting the mainline qemu tree working on SmartOS (illumos), the current qemu support seems to be a branch of qemu-kvm and we've been having some problems getting spice working, so moving to the main qemu tree seemed sensible.
> 
> Anyway, couple of questions…
> 
> 1. Are you happy to accept patches that enable this?

In the past I've kept QEMU running on OpenSolaris. Avi had one
outstanding patch to fix C99 compilation, and I have some to start
tackling builds without --disable-guest-agent on OpenIndiana in a local
queue that I need to revisit. Never tested our DTrace support so far.

But sure, host support patches for qemu.git are welcome.

See http://wiki.qemu.org/Contribute/StartHere for info on expected style
and workflows. Please cc me on patches.

Andreas

Re: [Qemu-devel] Illumos/SmarOS support

[lee.essen]

[-- Attachment #1: Type: text/plain, Size: 3755 bytes --]

On 15/03/2012 11:56, Stefan Hajnoczi wrote:
>> Anyway, couple of questions…
>>
>> 1. Are you happy to accept patches that enable this?
>> 	- There are a few build related things that don't work … assumption of sh
>> being bash, missing -64 argument to dtrace etc.
> Sounds good, but have you checked the Joyent qemu-kvm Illumos port to
> see what fixes they already have?
>
> https://github.com/joyent/illumos-kvm-cmd
 
Yes, they have solved the same problems .. however the build mechanism seems to be sufficiently different that things need to be done slightly differently for qemu.git.

These aren't complex changes and amount to generally only a couple of lines.

The illumos-kvm-cmd tree seems to be only updated to address joyent live-image build requirements, and hasn't really moved in about 7 months. I've been posting profusely on the smartos mailing list and they also don't seem that concerned about this work.

However, there are a few of us that would like to get spice up and running and the illumos fork doesn't seem like a sensible place to start when they are so far behind.

They have said that they will incorporate the spice work once it's complete, so I'm pretty confident that we can push them to the main qemu tree at that point.
>> 	- More fundamentally some object linking issues with dtrace which I'll need
>> some help to find the best way to resolve.
> I think Joyent has already done this, they said they use QEMU tracing
> with DTrace as the backend.
 
Yes they have, but again the build mechanism has changed quite significantly.

The basic issue here is that dtrace on Illumos seems to need all objects to be provided to it when you create the dtrace provider object, I'm guessing it uses the objects to work out which probes to actually include ... if you don't provide any objects you get an output object with none of the symbols you need in it.

So for each binary created you need to do something like...

dtrace -o <binary>-dtrace.o -G -s <dtrace-source> <all-the-objects-used-by-the-binary>

... then include the specific <binary>-drace.o when linking the main binary.
> So you are using qemu.git/master with the IllumOS KVM port?
>
> I think what you are trying to do makes sense - IllumOS as a host OS
> and QEMU platform should be actively involved in the upstream QEMU
> community.  But you may want to first check the Joyent port to see
> what can easily be upstreamed and also try talking with them if you
> haven't already.  I'm not sure why things aren't being pushed upstream
> but perhaps the process can be started.
 
Yes, and (other than my Linux issue) it's working well.

The changes needed to get it working are pretty minimal, most of them are #ifdef's around KVM_CAP's that are assumed to be defined (at least in some of the code.)

The specific code changes boil down to:
- a different mechanism for creating a vcpu filehandle (which is cloned from /dev/kvm rather than the return code of KVM_CREATE_VCPU)
- some of the kvm ioctls using different filehandles.
- the requirement to lock guest memory
- a KVM_EXIT_INTR case with ret=0 from the ioctl (exernal interrupt)

... which amounts to probably less than 50 lines of changes, and again the code structure is pretty different to their port, so I think it's easier to apply new rather than push upstream.

I would have thought that applying this (minimal) stuff to the qemu.git tree and using that for illumos would remove the need for a specific joyent version, and hence be a good thing.

The joyent guys are very focused (understandably) on their specific needs, however if someone else is prepared to do the work they do seem pretty willing to adopt it.

Regards,

Lee.


[-- Attachment #2.1: Type: text/html, Size: 4462 bytes --]

Re: [Qemu-devel] Illumos/SmarOS support

[Stefan Hajnoczi]

On Thu, Mar 15, 2012 at 12:39 PM,  <lee.essen@me.com> wrote:
>> On 15/03/2012 11:56, Stefan Hajnoczi wrote:
>> I think what you are trying to do makes sense - IllumOS as a host OS
>> and QEMU platform should be actively involved in the upstream QEMU
>> community.  But you may want to first check the Joyent port to see
>> what can easily be upstreamed and also try talking with them if you
>> haven't already.  I'm not sure why things aren't being pushed upstream
>> but perhaps the process can be started.
>
>
> Yes, and (other than my Linux issue) it's working well.
>
> The changes needed to get it working are pretty minimal, most of them are
> #ifdef's around KVM_CAP's that are assumed to be defined (at least in some
> of the code.)
>
> The specific code changes boil down to:
> - a different mechanism for creating a vcpu filehandle (which is cloned from
> /dev/kvm rather than the return code of KVM_CREATE_VCPU)
> - some of the kvm ioctls using different filehandles.
> - the requirement to lock guest memory
> - a KVM_EXIT_INTR case with ret=0 from the ioctl (exernal interrupt)
>
> ... which amounts to probably less than 50 lines of changes, and again the
> code structure is pretty different to their port, so I think it's easier to
> apply new rather than push upstream.
>
> I would have thought that applying this (minimal) stuff to the qemu.git tree
> and using that for illumos would remove the need for a specific joyent
> version, and hence be a good thing.
>
> The joyent guys are very focused (understandably) on their specific needs,
> however if someone else is prepared to do the work they do seem pretty
> willing to adopt it.

Okay, that's important because doing a porting/upstreaming effort like
this also has a big human or political element.  If they are
supportive then it's more likely everyone involved will end up with a
qemu.git tree they like and can use.

My main concern with the KVM functionality is that Joyent is leading
the kernel side and you are leading the upstream qemu.git side - this
means there needs to be collaboration.  Perhaps you can CC your
qemu-devel patches to an illumos/joyent mailing list so that it's
always clear where you're headed.  That way, folks experienced with
the IllumOS KVM kernel interface can review and give advice.  The
scenario to avoid would be for upstream qemu.git to have IllumOS
support merged but it breaks because the KVM kernel side goes in a
different direction.

Anway, your plan to get patches upstream sounds good to me.

Stefan

Re: [Qemu-devel] Illumos/SmarOS support

[Andreas Färber]

Something went wrong with your reply... While it contains a quote it
does not reference the mail you replied to, that way messages get lost
among the constant chatter on qemu-devel.

Am 15.03.2012 13:39, schrieb lee.essen@me.com:
> On 15/03/2012 11:56, Stefan Hajnoczi wrote:
>>> Anyway, couple of questions…
>>>
>>> 1. Are you happy to accept patches that enable this?
>>> 	- There are a few build related things that don't work … assumption of sh
>>> being bash, missing -64 argument to dtrace etc.
>> Sounds good, but have you checked the Joyent qemu-kvm Illumos port to
>> see what fixes they already have?
>>
>> https://github.com/joyent/illumos-kvm-cmd
>  
> Yes, they have solved the same problems .. however the build mechanism
> seems to be sufficiently different that things need to be done slightly
> differently for qemu.git.
> 
> These aren't complex changes and amount to generally only a couple of lines.
> 
> The illumos-kvm-cmd tree seems to be only updated to address joyent
> live-image build requirements, and hasn't really moved in about 7
> months. I've been posting profusely on the smartos mailing list and they
> also don't seem that concerned about this work.

At FOSDEM, I talked to a Bryant from illumos who wanted to forward my
business card to their KVM/QEMU guys re upstreaming, but I haven't heard
back from anyone so far.

> So for each binary created you need to do something like...
> 
> dtrace -o <binary>-dtrace.o -G -s <dtrace-source>
> <all-the-objects-used-by-the-binary>
> 
> ... then include the specific <binary>-drace.o when linking the main
> binary.

Same as on Solaris 10. There was some alternative trick to "glom"
objects together to avoid linking in that extra object, don't remember
exactly. Hopefully there's an archive of the old Sun blogs somewhere.


Concerning KVM, there was some resentment on the Red Hat side againt the
SmartOS KVM port due to the legal issue of whether GPL'ed KVM may be
combined with CDDL. But that is independent of currently broken QEMU
host support in userland.

Andreas

Re: [Qemu-devel] Illumos/SmarOS support

[Lee Essen]

On 15/03/2012 13:12, Andreas Färber wrote:
> Something went wrong with your reply... While it contains a quote it
> does not reference the mail you replied to, that way messages get lost
> among the constant chatter on qemu-devel.
>
Hmmm ... sorry -- I'm having to cut and paste emails between mail client 
and web browser because I can't get my normal email (this one) to 
deliver to the list. Any ideas who I talk to to resolve it? I've tried 
qemu-devel-owner but has no response.

Regards,

Lee.

Re: [Qemu-devel] Illumos/SmarOS support

[Andreas Färber]

Am 15.03.2012 16:48, schrieb Lee Essen:
> On 15/03/2012 13:12, Andreas Färber wrote:
>> Something went wrong with your reply... While it contains a quote it
>> does not reference the mail you replied to, that way messages get lost
>> among the constant chatter on qemu-devel.
>>
> Hmmm ... sorry -- I'm having to cut and paste emails between mail client
> and web browser because I can't get my normal email (this one) to
> deliver to the list. [...]

This mail arrived fine! The Savannah server is sometimes lagging lately.

Andreas