[Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Gerd Hoffmann]

  Hi folks,

Here are a bunch of patches which start adding xen support to qemu.
Overview (individual patches have longer descriptions):

  #1 -- groundwork: build system, cmd line options, ...
  #2 -- xen backend driver infrastructrure
  #3 -- xen console backend driver
  #4 -- xen framebuffer backend driver
  #5 -- xen block backend driver
  #6 -- xen nic backend driver
  #7 -- allow xen disks and nics being configured via qemu command
        line options.

With the first four patches in place upstream qemu can replace xen's
qemu-dm for paravirtual domains.  The block and nic backend drivers are
full userspace implementations using the grant table device (gntdev).

xen support is implemented using another machine type.  xen's qemu-dm
already uses the machine type to switch between paravirtualized and
fully virtualized machines, so this was the natural choice.  qemu has
gets a new "xenpv" machine type additionally to the "pc" and "isapc"
ones.

I've placed the new files into the hw/ directory.  With all my xen bits
(partly not submitted yet with these patches) those add up to 18 new
files, all prefixed with xen.  Hmm, maybe I should better place them
somewhere else, so the already quite crowded hw/ directory doesn't
become even more cluttered?  If so, any suggestions?  New toplevel
directory?  New subdirectory below hw/?  Something else?

Comments?

cheers,
  Gerd

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Ian Jackson]

Gerd Hoffmann writes ("[Qemu-devel] [PATCH 0/7] merge some xen bits into qemu"):
> Here are a bunch of patches which start adding xen support to qemu.
> Overview (individual patches have longer descriptions):

Just to clarify: as far as I can tell from the description,
this code has scant relationship with Xen upstream.

I'm generally in favour of pushing functionality out of the Xen branch
of qemu into upstream.  But going by Gerd Hoffman's description I
don't think that's what his branch is.  His code doesn't appear to be
related to the Xen branch of qemu that we are using.

For example,

> With the first four patches in place upstream qemu can replace xen's
> qemu-dm for paravirtual domains.  The block and nic backend drivers are
> full userspace implementations using the grant table device (gntdev).

we only use qemu-dm in paravirtualised domains for certain marginal
functions - in many cases it is not used at all.  Certainly the
functionality Gerd describes, such as xen backend block and network
drivers, are not in our qemu tree and we do not intend for them to be
there.

In a Xen installation this functionality is in the dom0 (host) kernel.


As far as I can see much of Gerd Hoffman's intended submission is
effectively an _emulator_ for Xen guests.  That is, it emulates a Xen
host without being one, so that a Xen domU can be run without the Xen
hypervisor.  Am I right, Gerd ?

I don't think there's anything wrong with that from a qemu point of
view.  Obviously we think that the genuine Xen hypervisor is much
better :-) but it is right for people to have a choice, and having
qemu emulate more environments is generally good.

But if this functionality is to go into qemu upstream perhaps it
should be distinguished from `real Xen' functions, because otherwise
users are going to become very very confused.

Ian.

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Anthony Liguori]

Ian Jackson wrote:
> Gerd Hoffmann writes ("[Qemu-devel] [PATCH 0/7] merge some xen bits into qemu"):
>   
>> Here are a bunch of patches which start adding xen support to qemu.
>> Overview (individual patches have longer descriptions):
>>     
>
> Just to clarify: as far as I can tell from the description,
> this code has scant relationship with Xen upstream.
>
> I'm generally in favour of pushing functionality out of the Xen branch
> of qemu into upstream.  But going by Gerd Hoffman's description I
> don't think that's what his branch is.  His code doesn't appear to be
> related to the Xen branch of qemu that we are using.
>   

I think it's more closely related to Xenite and Xenner.  Gerd: are you 
planning on folding in domain creation?  Right now it appears to be a 
helper launched after the domain creation.

> For example,
>
>   
>> With the first four patches in place upstream qemu can replace xen's
>> qemu-dm for paravirtual domains.  The block and nic backend drivers are
>> full userspace implementations using the grant table device (gntdev).
>>     
>
> we only use qemu-dm in paravirtualised domains for certain marginal
> functions - in many cases it is not used at all.  Certainly the
> functionality Gerd describes, such as xen backend block and network
> drivers, are not in our qemu tree and we do not intend for them to be
> there.
>   

There's no reason they couldn't be though.  It's just like blktap.

> In a Xen installation this functionality is in the dom0 (host) kernel.
>
>
> As far as I can see much of Gerd Hoffman's intended submission is
> effectively an _emulator_ for Xen guests.  That is, it emulates a Xen
> host without being one, so that a Xen domU can be run without the Xen
> hypervisor.  Am I right, Gerd ?
>   

No, it's definitely for use with Xen (hypervisor).  But it's different 
architecturally from how Xen uses QEMU in xen-unstable.

Personally, I really like the way Gerd has the code structured.  It 
seems like it could be used as almost a drop-in replacement for qemu-dm 
for PV guests.  HVM guests would require more work.  Of course, the 
net/disk support can be completely optionally.

With that said, it would be silly to have two Xen implementations within 
QEMU so there has to be some general agreement in the Xen community 
about how QEMU is going to be used before merging would make sense.  
That probably requires some discussion about how the HVM support would 
fit into this architecture.

Regards,

Anthony Liguori



> I don't think there's anything wrong with that from a qemu point of
> view.  Obviously we think that the genuine Xen hypervisor is much
> better :-) but it is right for people to have a choice, and having
> qemu emulate more environments is generally good.
>
> But if this functionality is to go into qemu upstream perhaps it
> should be distinguished from `real Xen' functions, because otherwise
> users are going to become very very confused.
>
> Ian.
>
>
>   

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Gerd Hoffmann]

Ian Jackson wrote:
> Gerd Hoffmann writes ("[Qemu-devel] [PATCH 0/7] merge some xen bits into qemu"):
>> Here are a bunch of patches which start adding xen support to qemu.
>> Overview (individual patches have longer descriptions):
> 
> Just to clarify: as far as I can tell from the description,
> this code has scant relationship with Xen upstream.

Patches #3 + #4 are largely based on qemu-dm code (as noted in the
individual patch  descriptions).

> I'm generally in favour of pushing functionality out of the Xen branch
> of qemu into upstream.  But going by Gerd Hoffman's description I
> don't think that's what his branch is.  His code doesn't appear to be
> related to the Xen branch of qemu that we are using.

I want to merge the *functionality*.  IMHO that doesn't mean that it
must be the *code* used by Xen at the moment.

> For example,
> 
>> With the first four patches in place upstream qemu can replace xen's
>> qemu-dm for paravirtual domains.  The block and nic backend drivers are
>> full userspace implementations using the grant table device (gntdev).
> 
> we only use qemu-dm in paravirtualised domains for certain marginal
> functions - in many cases it is not used at all.

It's used for xen console (optionally, can also be handled by
xenconsoled) and the virtual framebuffer.

> Certainly the
> functionality Gerd describes, such as xen backend block and network
> drivers, are not in our qemu tree and we do not intend for them to be
> there.

I want them be there though.  You can use them or not, that is your call.

> In a Xen installation this functionality is in the dom0 (host) kernel.

That is only half the story.  The block backend can run in userspace too
(when using blktap).  The block backend driver should be pretty much
identical to blktap functionality-wise.  The implementation is quite
different though.

> As far as I can see much of Gerd Hoffman's intended submission is
> effectively an _emulator_ for Xen guests.  That is, it emulates a Xen
> host without being one, so that a Xen domU can be run without the Xen
> hypervisor.  Am I right, Gerd ?

That is part of the longterm plan, yes.  I want qemu being able to do
both, run as device model for xen and also to boot xen guest images
without xen, using emulation.  If the intention would have been
emulation only I wouldn't have Cc'ed xen-devel for patch review.

The emulation bits are not in that patchset btw.

> But if this functionality is to go into qemu upstream perhaps it
> should be distinguished from `real Xen' functions, because otherwise
> users are going to become very very confused.

Huh?

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Ian Jackson]

Anthony Liguori writes ("Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu"):
> I think it's more closely related to Xenite and Xenner.  Gerd: are you 
> planning on folding in domain creation?  Right now it appears to be a 
> helper launched after the domain creation.
...
> No, it's definitely for use with Xen (hypervisor).  But it's different 
> architecturally from how Xen uses QEMU in xen-unstable.

Xenner is an emulator for allowing Xen domUs to be booted without the
Xen hypervisor.

Xennite is an experimental replacement for the Xen userland management
stack in dom0: it moves more functionality from the Xen tools in dom0
into the qemu-dm process.  This is moving in almost the opposite
direction to Xen upstream is moving: we are moving qemu-dm into its
own tiny domain, so that the qemu code doesn't need to run as a
process in dom0; this has important security and scalability
advantages.

Ian.

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Gerd Hoffmann]

Anthony Liguori wrote:
> I think it's more closely related to Xenite and Xenner.  Gerd: are you
> planning on folding in domain creation?  Right now it appears to be a
> helper launched after the domain creation.

Not sure yet.  Given that xen upstream most likely wouldn't follow a
xenite-like model I'm not sure how useful it would be.  Shouldn't be
that much extra code though, and as long as it is an opt-in upstream xen
should be happy too ...

> It
> seems like it could be used as almost a drop-in replacement for qemu-dm
> for PV guests.

Yes, that is the intention.

> HVM guests would require more work.

Yes.  Not addressed (yet).  hvm is more code, more invasive and I also
hope that Glauber Costa's work on a cpu abstraction layer (for
emulation/kqemu/kvm) makes it easier to add xen hvm to the picture.

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Anthony Liguori]

Ian Jackson wrote:
> Anthony Liguori writes ("Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu"):
>   
>> I think it's more closely related to Xenite and Xenner.  Gerd: are you 
>> planning on folding in domain creation?  Right now it appears to be a 
>> helper launched after the domain creation.
>>     
> ...
>   
>> No, it's definitely for use with Xen (hypervisor).  But it's different 
>> architecturally from how Xen uses QEMU in xen-unstable.
>>     
>
> Xenner is an emulator for allowing Xen domUs to be booted without the
> Xen hypervisor.
>   

Or "shim".  It's almost architecturally identical to the XenSource 
developed "shim" for Hyper-V.  It seems like a popular thing to do these 
days :-)

> Xennite is an experimental replacement for the Xen userland management
> stack in dom0: it moves more functionality from the Xen tools in dom0
> into the qemu-dm process.  This is moving in almost the opposite
> direction to Xen upstream is moving: we are moving qemu-dm into its
> own tiny domain, so that the qemu code doesn't need to run as a
> process in dom0; this has important security and scalability
> advantages.
>   

Are there separate requirements for stub domains verses Xennite?

Gerd's code is different than what's upstream in QEMU but the question 
is whether it's reconcilable with what's upstream.  If not, what makes 
it that way?

Regards,

Anthony Liguori

> Ian.
>   

Re: [Xen-devel] Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Samuel Thibault]

Gerd Hoffmann, le Mon 28 Jul 2008 16:43:54 +0200, a écrit :
> Ian Jackson wrote:
> > I'm generally in favour of pushing functionality out of the Xen branch
> > of qemu into upstream.  But going by Gerd Hoffman's description I
> > don't think that's what his branch is.  His code doesn't appear to be
> > related to the Xen branch of qemu that we are using.
> 
> I want to merge the *functionality*.  IMHO that doesn't mean that it
> must be the *code* used by Xen at the moment.

Well, the patch comments weren't so clear about that.

Samuel

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Daniel P. Berrange]

On Mon, Jul 28, 2008 at 03:58:15PM +0100, Ian Jackson wrote:
> Anthony Liguori writes ("Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu"):
> > I think it's more closely related to Xenite and Xenner.  Gerd: are you 
> > planning on folding in domain creation?  Right now it appears to be a 
> > helper launched after the domain creation.
> ...
> > No, it's definitely for use with Xen (hypervisor).  But it's different 
> > architecturally from how Xen uses QEMU in xen-unstable.
> 
> Xenner is an emulator for allowing Xen domUs to be booted without the
> Xen hypervisor.
> 
> Xennite is an experimental replacement for the Xen userland management
> stack in dom0: it moves more functionality from the Xen tools in dom0
> into the qemu-dm process.  This is moving in almost the opposite
> direction to Xen upstream is moving: we are moving qemu-dm into its
> own tiny domain, so that the qemu code doesn't need to run as a
> process in dom0; this has important security and scalability
> advantages.

Yes, to be clear that the Xennite code is *not* an official Xen project. 

It is my experimental work and I don't expect anyone to use it for real
in the near future. It does however align very well with Gerd's serious 
Xenner project (which is incredibly useful tool allowing admins to mix
and match Xen and KVM) since both Xenner & Xennite have much same needs
in  terms of QEMU support code

The motiviation with 'Xennite' is to make the process of launching
and managing an instance of a Xen guest, as close as possible to that
of KVM. Start QEMU to launch the Xen guest, interact with the monitor
to control it, and kill QEMU to destroy the Xen guest. I'd class it as
a proof-of-concept since I never got around to making stuff like save
and restore work - I postponed further work until Xen re-synced with a
new QEMU codebase.  We could have a long discussion about the pros/cons
of this idea vs the upstream Xen move to QEMU in stub domains, but that's
off topic for this list, so I'll leave it :-)  

Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Gerd Hoffmann]

  Hi folks,

This is the second version of the xen support for qemu patch series,
addressing review comments.  Lots of little changes, the largest
change probably is the switch-over to the esisting sys-queue.h list
implementation instead of bringing in a copy of the linux kernel
list.h header.

The console and framebuffer backend drivers are largely based on the
xen code, the other bits are rewritten from scratch.  Nevertheless
the code should be functionally identical.

Overview (individual patches have longer descriptions):

  #1 -- groundwork: build system, cmd line options, ...
  #2 -- xen backend driver infrastructrure
  #3 -- xen console backend driver
  #4 -- xen framebuffer backend driver
  #5 -- xen block backend driver
  #6 -- xen nic backend driver
  #7 -- allow xen disks and nics being configured via qemu command
        line options.

With the first four patches in place upstream qemu can replace xen's
qemu-dm for paravirtual domains.  The block and nic backend drivers
are full userspace implementations using the grant table device
(gntdev).

xen support is implemented using another machine type.  xen's qemu-dm
already uses the machine type to switch between paravirtualized and
fully virtualized machines, so this was the natural choice.  qemu has
gets a new "xenpv" machine type additionally to the "pc" and "isapc"
ones.

Comments?

cheers,
  Gerd

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Anthony Liguori]

Gerd Hoffmann wrote:
>   Hi folks,
>
> xen support is implemented using another machine type.  xen's qemu-dm
> already uses the machine type to switch between paravirtualized and
> fully virtualized machines, so this was the natural choice.  qemu has
> gets a new "xenpv" machine type additionally to the "pc" and "isapc"
> ones.
>
> Comments?
>   

Modulo some of the stylistic feedback that needs to be addressed, I 
think these series looks pretty good.  It fits pretty well into QEMU.  
Unless there are major objections, I'll apply an updated series once 
people have had some time to look through it.

However, I'd like to see agreement within the Xen community that this is 
the right approach first.  In particular, I would like to see these 
patches either merged with upstream Xen or for upstream Xen to plan to 
rebase against QEMU to pick them up and use them for PV support.  This 
doesn't mean they have to use the block and net backends of course.  I 
really don't want to support two implementations of Xen support in QEMU.

Regards,

Anthony Liguori

> cheers,
>   Gerd
>
>
>
>   

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Gerd Hoffmann]

Anthony Liguori wrote:
> However, I'd like to see agreement within the Xen community that this
>  is the right approach first.  In particular, I would like to see 
> these patches either merged with upstream Xen or for upstream Xen to 
> plan to rebase against QEMU to pick them up and use them for PV 
> support.

Ian Jackson has a qemu git tree at
  http://www.chiark.greenend.org.uk/~ijackson/qemu-xen.git

As far I know it is rebased now and then and the long-term plan is to
get the delta to upstream qemu at least smaller.  Dunno whenever it ever
will be zero due to stuff like mapcache.

There are a few compatibility issues to be solved somehow before
upstream qemu is able to completely replace xen's qemu-dm:

(1) my patches use -domid to specify the domain id.  xen uses (used?) -d
instead.  Which isn't going to work because upstream qemu uses that
switch for something else.  Ian's tree supports both -d and -domid, with
a comment saying -d is deprecated.  So I think we have an agreement to
use -domid here ;)

(2) xen added a -domain-name switch.  upstream qemu has a -name switch.
my patches do not implement -domain-name, I use the name set via -name
instead.  IMHO xen should pick up the upstream way to do things here,
i.e. switch over to -name.

(3) vnc configuration is different.  This one is not so easy due to
xenstore being involved.  So the way things work are pretty xen specific
and don't make that much sense for upstream qemu.  The differences are:

Setting the vnc password:  qemu uses a monitor command.  xen reads it
from xenstore (set by xend).  I think we can just support both ways here.

xen has a -vncunused switch which makes qemu scan for a free vnc
display.  The tcp port actually used is written to xenstore.  Dunno how
to handle that one best.  First, I think should be a -vnc switch option
instead of a separate command line switch.  Second, propagating the used
port via xenstore doesn't make sense at all for upstream qemu ...

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Samuel Thibault]

Gerd Hoffmann, le Tue 05 Aug 2008 11:58:55 +0200, a écrit :
> Anthony Liguori wrote:
> > However, I'd like to see agreement within the Xen community that this
> >  is the right approach first.  In particular, I would like to see 
> > these patches either merged with upstream Xen or for upstream Xen to 
> > plan to rebase against QEMU to pick them up and use them for PV 
> > support.
> 
> Ian Jackson has a qemu git tree at
>   http://www.chiark.greenend.org.uk/~ijackson/qemu-xen.git
> 
> As far I know it is rebased now and then and the long-term plan is to
> get the delta to upstream qemu at least smaller.  Dunno whenever it ever
> will be zero due to stuff like mapcache.

There are two branches there: the xen branch is stuff that Ian indeed
doesn't intend to push.  The qemu branch is stuff that he wishes to
push.

Samuel

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Samuel Thibault]

Anthony Liguori, le Mon 04 Aug 2008 12:42:38 -0500, a écrit :
> However, I'd like to see agreement within the Xen community that this is 
> the right approach first.

Unfortunately, Ian Jackson (who will continue to maintain his qemu
tree) is away this week.

> In particular, I would like to see these 
> patches either merged with upstream Xen or for upstream Xen to plan to 
> rebase against QEMU to pick them up and use them for PV support.

I have not had a precise look at e.g. the xenfb.c code, but if what Gerd
has submitted is exactly the same as xen-unstable's, then that part is
already merged :)

As for the rest, as Gerd said Ian is maintaining a rebased tree, which
is currently used by default in the upcoming xen 4.0 (aka 3.3).  He
won't rebase it until that release, but after that he will probably
regularly, with much less pain than previously anyway since we now have
a fine git tree.

Samuel

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Gerd Hoffmann]

> I have not had a precise look at e.g. the xenfb.c code, but if what Gerd
> has submitted is exactly the same as xen-unstable's, then that part is
> already merged :)

It isn't exactly the same code.  For example the xenbus state engine has
been factored out into generic code, shared by all backend drivers.  The
behaviour should be identical though.

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Samuel Thibault]

Gerd Hoffmann, le Tue 05 Aug 2008 13:12:35 +0200, a écrit :
> > I have not had a precise look at e.g. the xenfb.c code, but if what Gerd
> > has submitted is exactly the same as xen-unstable's, then that part is
> > already merged :)
> 
> It isn't exactly the same code.  For example the xenbus state engine has
> been factored out into generic code, shared by all backend drivers.

Then it'd probably be good to push that upstream Xen.

Samuel

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Gerd Hoffmann]

Samuel Thibault wrote:
> Gerd Hoffmann, le Tue 05 Aug 2008 13:12:35 +0200, a écrit :
>>> I have not had a precise look at e.g. the xenfb.c code, but if what Gerd
>>> has submitted is exactly the same as xen-unstable's, then that part is
>>> already merged :)
>> It isn't exactly the same code.  For example the xenbus state engine has
>> been factored out into generic code, shared by all backend drivers.
> 
> Then it'd probably be good to push that upstream Xen.

I'm cross-posting the patches to xen-devel for a reason.

Whenever you want to pick them up now or just wait for them appear in
upstream qemu and handle it with the next rebase is your call.  But at
least reviewing and commenting now would be very helpful, so I don't
break things for xen by mistake and we can proceed with merging those bits.

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Samuel Thibault]

Gerd Hoffmann, le Tue 05 Aug 2008 15:18:46 +0200, a écrit :
> Samuel Thibault wrote:
> > Then it'd probably be good to push that upstream Xen.
> 
> I'm cross-posting the patches to xen-devel for a reason.

Well, cross-posting qemu patches to xen-devel is not the same as posting
xen patches to xen-devel...

Samuel

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Samuel Thibault]

Samuel Thibault, le Tue 05 Aug 2008 16:03:28 +0100, a écrit :
> Gerd Hoffmann, le Tue 05 Aug 2008 15:18:46 +0200, a écrit :
> > Samuel Thibault wrote:
> > > Then it'd probably be good to push that upstream Xen.
> > 
> > I'm cross-posting the patches to xen-devel for a reason.
> 
> Well, cross-posting qemu patches to xen-devel is not the same as posting
> xen patches to xen-devel...

Just to explain a bit more, now that I have read the patches a bit: what
is there is somehow far from the xen unstable tree, at least because
of the backend driver core, and because of other 'details' (renaming,
moving).  I doubt Ian will be happy to have to make the effort to merge
that in his tree and I guess he will probably just drop everything and
keep the xen-unstable version.

Samuel

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Anthony Liguori]

Samuel Thibault wrote:
> Samuel Thibault, le Tue 05 Aug 2008 16:03:28 +0100, a écrit :
>   
>> Gerd Hoffmann, le Tue 05 Aug 2008 15:18:46 +0200, a écrit :
>>     
>> Well, cross-posting qemu patches to xen-devel is not the same as posting
>> xen patches to xen-devel...
>>     
>
> Just to explain a bit more, now that I have read the patches a bit: what
> is there is somehow far from the xen unstable tree, at least because
> of the backend driver core, and because of other 'details' (renaming,
> moving).  I doubt Ian will be happy to have to make the effort to merge
> that in his tree and I guess he will probably just drop everything and
> keep the xen-unstable version.
>   

Which is why ya'll need to work something out before we apply these 
patches to QEMU :-)

Regards,

Anthony Liguori

> Samuel
>   

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Samuel Thibault]

Samuel Thibault, le Tue 05 Aug 2008 16:41:40 +0100, a écrit :
> Samuel Thibault, le Tue 05 Aug 2008 16:03:28 +0100, a écrit :
> > Gerd Hoffmann, le Tue 05 Aug 2008 15:18:46 +0200, a écrit :
> > > Samuel Thibault wrote:
> > > > Then it'd probably be good to push that upstream Xen.
> > > 
> > > I'm cross-posting the patches to xen-devel for a reason.
> > 
> > Well, cross-posting qemu patches to xen-devel is not the same as posting
> > xen patches to xen-devel...
> 
> Just to explain a bit more, now that I have read the patches a bit:

And to be even more explicit, by "I have read the patches", I mean: I
have applied them to upstream qemu, them vimdiffed the files there with
the ones from Ian's tree (sometimes with different names), in order
to have an idea of what the merge would be (i.e. the "xen patches" I
mentioned above, which is what we care about from the point of view of
xen-unstable).

Samuel

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Blue Swirl]

On 8/5/08, Anthony Liguori <anthony@codemonkey.ws> wrote:
> Samuel Thibault wrote:
>
> > Samuel Thibault, le Tue 05 Aug 2008 16:03:28 +0100, a écrit :
> >
> >
> > > Gerd Hoffmann, le Tue 05 Aug 2008 15:18:46 +0200, a écrit :
> > >    Well, cross-posting qemu patches to xen-devel is not the same as
> posting
> > > xen patches to xen-devel...
> > >
> > >
> >
> > Just to explain a bit more, now that I have read the patches a bit: what
> > is there is somehow far from the xen unstable tree, at least because
> > of the backend driver core, and because of other 'details' (renaming,
> > moving).  I doubt Ian will be happy to have to make the effort to merge
> > that in his tree and I guess he will probably just drop everything and
> > keep the xen-unstable version.
> >
> >
>
>  Which is why ya'll need to work something out before we apply these patches
> to QEMU :-)

I'd salvage nodisk feature (useful for Sparc and embedded systems) and
if ever produced, patch that introduces qemu_realloc and does
s/realloc/qemu_realloc/g.

For global -Wall -Wstrict-prototypes flags probably the project
leaders should agree that it will be useful.

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Gerd Hoffmann]

Samuel Thibault wrote:
> Samuel Thibault, le Tue 05 Aug 2008 16:03:28 +0100, a écrit :
>> Gerd Hoffmann, le Tue 05 Aug 2008 15:18:46 +0200, a écrit :
>>> Samuel Thibault wrote:
>>>> Then it'd probably be good to push that upstream Xen.
>>> I'm cross-posting the patches to xen-devel for a reason.
>> Well, cross-posting qemu patches to xen-devel is not the same as posting
>> xen patches to xen-devel...
> 
> Just to explain a bit more, now that I have read the patches a bit: what
> is there is somehow far from the xen unstable tree, at least because
> of the backend driver core, and because of other 'details' (renaming,
> moving).  I doubt Ian will be happy to have to make the effort to merge
> that in his tree and I guess he will probably just drop everything and
> keep the xen-unstable version.

Sure, merging stuff upstream involves alot of work short-term, you'll
get the benefits long-term.

pv domain support is easy, as this is largely self-contained.  hvm will
be even more work, because hvm support is much more invasive and on top
of that the qemu interfaces will change to nicely support the various
ways to run code natively instead of emulated (kqemu, kvm, xen).

The rough way to merge would look like this:

  - drop xen_console.[ch]
  - drop xenfb.[ch]
  - drop xen_machine_pv.c

  - add xen.h
  - add xen-machine.c
  - add xen-backend.[ch]
  - add xen-console.c
  - add xen-framebuffer.c

  - wind up stuff in the Makefiles.
  - some global renames (domid -> xen_domid for example) as I took care
    to prefix global xen variables & functions with xen_.
  - probably some small fixups are needed ...

You probably wouldn't do that before the 3.3 (4.0?) release.  If you
keep the xenish vl.c version the differences between upstream qemu and
qemu-dm command line options and vnc handling go away.  The you can
gradually move qemu-dm to be more upstream-ish while also updating xend
accordingly.

Note that qemu-dm has a few optimizations in the qemu display code to
avoid unneeded work.  I'd suggest to submit them upstream.  The we can
enable the bits in xen-framebuffer.c which make use of them.
DisplayState->idle for example.

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Samuel Thibault]

Gerd Hoffmann, le Wed 06 Aug 2008 12:14:25 +0200, a écrit :
> The rough way to merge would look like this:
> 
>   - drop xen_console.[ch]
>   - drop xenfb.[ch]
>   - drop xen_machine_pv.c
> 
>   - add xen.h
>   - add xen-machine.c
>   - add xen-backend.[ch]
>   - add xen-console.c
>   - add xen-framebuffer.c
> 
>   - wind up stuff in the Makefiles.
>   - some global renames (domid -> xen_domid for example) as I took care
>     to prefix global xen variables & functions with xen_.
>   - probably some small fixups are needed ...

You forgot the _test_ stage.  You are basically asking us to replace our
well-tested implementation with your implementation, that's quite a
move.  You are not even providing a patch for us to check that nothing
has been left behind...

> Note that qemu-dm has a few optimizations in the qemu display code to
> avoid unneeded work.  I'd suggest to submit them upstream.

That's planned, yes.  But you too could do the same for the device
backend core for instance...

> The you can gradually move qemu-dm to be more upstream-ish while also
> updating xend accordingly.

That's what we are already doing in Ian's tree.

Samuel

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Gerd Hoffmann]

  Hi,

> You forgot the _test_ stage.  You are basically asking us to replace our
> well-tested implementation with your implementation, that's quite a
> move.

Sure, and more will follow.  I'd expect the merge involves rewriting
large parts of the code anyway, especially the hvm bits due to the
nessesary interface cleanups in qemu, but also because the code must
pass the review process before it can be merged.  Pretty much like
upstreaming the xen support into the linux kernel.  All that also needs
testing, no doubt.

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Samuel Thibault]

Gerd Hoffmann, le Wed 06 Aug 2008 15:24:39 +0200, a écrit :
> > You forgot the _test_ stage.  You are basically asking us to replace our
> > well-tested implementation with your implementation, that's quite a
> > move.
> 
> Sure, and more will follow.

But by having it merged into just upstream qemu you won't magically get
Xen people actually test it, so it would be mostly dead code.

> Pretty much like upstreaming the xen support into the linux kernel.

There's a big difference here.  The upstream linux kernel merge won't be
merged back to the linux-2.6.18 tree.  And people do test upstream xen
support.

Samuel

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Gerd Hoffmann]

Samuel Thibault wrote:
> Gerd Hoffmann, le Wed 06 Aug 2008 15:24:39 +0200, a écrit :
>>> You forgot the _test_ stage.  You are basically asking us to replace our
>>> well-tested implementation with your implementation, that's quite a
>>> move.
>> Sure, and more will follow.
> 
> But by having it merged into just upstream qemu you won't magically get
> Xen people actually test it, so it would be mostly dead code.
> 
>> Pretty much like upstreaming the xen support into the linux kernel.
> 
> There's a big difference here.  The upstream linux kernel merge won't be
> merged back to the linux-2.6.18 tree.

Oh, we can also arrange things without merging back.  Stop rebasing
qemu-dm (pretty much like 2.6.18), additionally hook upstream qemu as-is
into testing (like pv_ops kernels).  Then start submitting patches to
qemu.  When done, zap old qemu-dm.

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Samuel Thibault]

Gerd Hoffmann, le Wed 06 Aug 2008 16:18:51 +0200, a écrit :
> Samuel Thibault wrote:
> > But by having it merged into just upstream qemu you won't magically get
> > Xen people actually test it, so it would be mostly dead code.
> > 
> >> Pretty much like upstreaming the xen support into the linux kernel.
> > 
> > There's a big difference here.  The upstream linux kernel merge won't be
> > merged back to the linux-2.6.18 tree.
> 
> Oh, we can also arrange things without merging back.  Stop rebasing
> qemu-dm (pretty much like 2.6.18), additionally hook upstream qemu as-is
> into testing (like pv_ops kernels).  Then start submitting patches to
> qemu.  When done, zap old qemu-dm.

Again, there's a difference.  The interface between Linux and Xen has
always been meant to be a "public" thing.  The interface between ioemu
and the other xen components has not, and changing it is considered fine
(we still do it).  Sure that could be cleaned up, but that's probably
not something that should happen on the upstream qemu side.

Samuel

Re: [Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Gerd Hoffmann]

Samuel Thibault wrote:
>> Oh, we can also arrange things without merging back.  Stop rebasing
>> qemu-dm (pretty much like 2.6.18), additionally hook upstream qemu as-is
>> into testing (like pv_ops kernels).  Then start submitting patches to
>> qemu.  When done, zap old qemu-dm.
> 
> Again, there's a difference.  The interface between Linux and Xen has
> always been meant to be a "public" thing.  The interface between ioemu
> and the other xen components has not, and changing it is considered fine
> (we still do it).  Sure that could be cleaned up, but that's probably
> not something that should happen on the upstream qemu side.

Hmm.  That makes me think that starting over from scratch really is the
best option.  Drop the old cruft, design a clean interface, code it up,
discuss it in public and submit to upstream qemu.

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

[Qemu-devel] [PATCH 0/7] merge some xen bits into qemu

[Gerd Hoffmann]

  Hi folks,

This is version #5 of the xen support for qemu patch series.

It has been quiet for a while, mostly due to qemu-xen lagging behind
qemu-upstream.  Now it catched up with upstream, so it is time to make
some progress here.  Not many changes since the last time I posted the
patches.  Some prelimary stuff has been merged and thus is gone, some
bugs have been fixed, some adaptions to recent changes happend.

The console and framebuffer backend drivers are largely based on the
xen code, the other bits are rewritten from scratch.  Nevertheless
the code should be functionally identical.

Overview (individual patches have longer descriptions):

  #1 -- groundwork: build system, cmd line options, ...
  #2 -- xen backend driver infrastructrure
  #3 -- xen console backend driver
  #4 -- xen framebuffer backend driver

  #5 -- xen block backend driver
  #6 -- xen nic backend driver
  #7 -- allow xen disks and nics being configured via qemu command
        line options.

With first next four patches in place upstream qemu can replace xen's
qemu-dm for paravirtual domains.  Due to some small differences in
the command line syntax it can't work as drop-in replacement though.

The last three patches add full userspace implementations of block and
nic backend drivers using the grant table device (gntdev) and command
line support config support for setting up these backends.

xen support is implemented using another machine type.  xen's qemu-dm
already uses the machine type to switch between paravirtualized and
fully virtualized machines, so this was the natural choice.  qemu
gets a new "xenpv" machine type additionally to the "pc" and "isapc"
ones.

The patches are also available here (Patches 0002 -> 0008):
	http://kraxel.fedorapeople.org/patches/qemu-upstream/


There is also a patch queue for qemu-xen, located here:
	http://kraxel.fedorapeople.org/patches/qemu-xen/

There also a '-b' (no whitespace changes) version for easier review:
	http://kraxel.fedorapeople.org/patches/qemu-xen/no-ws/

The qemu-xen patch series has a slightly different ordering to make the
patch queue more bisect-friendly.  Also the command line syntax is
unchanged, so we don't break qemu-dm <=> xend interaction.  It should
work just fine as drop-in replacement, without regressions, with the
additional bonus that the framebuffer handles a restart cycle correctly
(i.e. pvgrub in graphical mode actually works).

Comments?

cheers,
  Gerd