Rumprun based upstream QEMU stubdom status update

Rumprun based upstream QEMU stubdom status update

[Wei Liu]

Hi all

I've been working on and off on upstream QEMU stubdom. Unfortunately in
the last few months I didn't make much progress as I had expected. Some
folks expressed interests during Xen developer summit on this project
and I think it would be better I post what I have as soon as possible.

In short, it is now possible to compile and run QEMU as a PV guest,
but I haven't finished the plumbing of toolstack code, hence I don't
know whether it functions as a device model.

Note that there are quite a bit of hacks inside those scripts to work
around problems in build systems of different piece of software.

There is a small program to configure network topology inside the
guest. It creates a bridge, then adds a bunch of tap interfaces and
existing Xen vif interface(s) to the bridge.

What are missing:

1. Inter-domain communication mechanism: this can be subsidised by
   network interface for now.
2. Plumbing through libxl. It will be using rumprun to launch QEMU
   stubdom.
3. Integration with Raisin build system. This is the ulterior goal to
   get rid of all customised build scripts.

I haven't checked everything, but some of the rump kernel interfaces
are not yet declared stable. Relevant discussions should mainly take
place on rump kernel mailing list and CC xen-devel when necessary.

Here are the step-by-step guide for building a QEMU unikernel.

Note that all of my scripts have hardcoded paths (/local/scratch/...),
so anyone who is interested in using them would need to replace the
hardcoded paths with something sensible.

$XEN refers to Xen source code. I will always refer to upstream
repositories and only provide git branches unless there are local
changes needed.

Rumprun is a fast moving target so I will also state the working
version of things.

# Build rumprun toolchain (based on a5f58548)

$ git clone https://github.com/rumpkernel/rumprun.git rumprun.git
$ cd rumprun.git
$ export XEN_HEADERS=$XEN/xen/include/public
$ ./build-rr xen -- -j8

If nothing goes wrong, the toolchain will be available under app-tools
directory.

# Build QEMU dependency

Currently only a minimum set of libraries that QEMU depends on are
built. They are zlib, libffi, glib, libpixman, Xen libxenctrl and
libxenstore. More libraries might be needed as we move forward, but
overall the build process should be the same.

I have a git repository for build scripts.

$ git clone https://xenbits.xen.org/people/liuw/rump-build-scrpt-junk.git rump-build-script-junk
$ cd rump-build-script-junk
$ ./script 2>&1 | tee buildlog

The script will download tarballs from various websites and expects
Xen source tree resides at a fix path (see script).

If everything goes well, those libraries will be inside
/local/scratch/Rump-kernels/prefix. If something goes wrong, dig into
buildlog and figure out why. :-)

# Build QEMU

I have prepared a branch for my WIP patches for QEMU.

$ cd rump-build-junk-script
$ git clone https://xenbits.xen.org/people/liuw/qemu.git -b wip.rump qemu
$ ./script-qemu 2>&1 | tee buildlog

If everything goes well, qemu will be inside
/local/scratch/Rump-kernels/prefix-qemu/bin.

But that's not yet a working domain image.

# "Baking" QEMU into a unikernel

$ git clone https://xenbits.xen.org/people/liuw/qemu-rump-config.git qemu-rump-config
$ cd rump-build-junk-script
$ ./script-bake

Then the final binary will be in
/local/scratch/Rump-kernels/prefix-qemu/bin/qemu-system-i386-rump.

# Test the unikernel

Note that I omit path prefixes in following commands.

$ scp qemu-system-i386-rump testbox:~
$ scp app-tools/rumprun testbox:~

Then on testbox.

$ ./rumprun -S xen -i qemu-system-i386-rump --help

This rune should start the guest, attach to its console and show you
the help information from QEMU.

To add network interface, see ./rumprun -h.


Wei.

Re: Rumprun based upstream QEMU stubdom status update

[Eric Shelton]

[-- Attachment #1.1: Type: text/plain, Size: 11417 bytes --]

Wei and others,

It has been a long, long road to getting a rumprun-based stubdom
implemented.  The earliest discussion of pursuing this approach appears to
be a post by IanJ on August 21, 2013, with the first serious indication of
development work again posted by IanJ, on April 8, 2014 - over 19 months
ago:

http://lists.xenproject.org/archives/html/xen-devel/2014-04/msg01066.html

>From what is described below, it looks like there remains a long way to go,
as what appears to be running is only printing out the output from "qemu
--help" (pretty much what was described at XPDS15 more than three months
ago).

Once upon a time, one of the release managers was talking about making QEMU
upstream stubdom a blocker for 4.5:

http://lists.xen.org/archives/html/xen-devel/2014-04/msg01065.html

Obviously that did not happen.  Time and again, QEMU upstream stubdom has
gotten shelved in favor of other priorities, which testifies as to where it
falls on the Xen project's list of priorities.

I believe it is time to reassess the current development plan for QEMU
upstream stubdom.  From a project management perspective, the current plan
relies far too heavily on a very small handful of people that are simply
too busy with other things to bring it to a working, let alone releasable,
state.  Plus, the delay in realizing this has now reached a point where it
imposes frequent, tangible, and possibly significant drains on development
resources in order to cope with shortcomings of the current situation (QEMU
upstream in dom0, and separate qemu-traditional codebase for stubdom).
Plus, having QEMU upstream run in dom0 is likely Xen's single longest
running identified security issue - and certainly the longest-lived one
with an identified and actionable solution.  Consider how many of the
QEMU-related XSAs reported something along the lines of "systems using
qemu-dm stubdomain device models are NOT vulnerable."

Recently, following several XSAs demonstrating an interest in finding (and
likely exploiting) QEMU vulnerabilities, the Xen team has devoted resources
to running QEMU as non-root (now on patchset v10) - apparently trying to
replicate the bandaid used by KVM.  This effort would not have been
necessary if QEMU upstream stubdom had been implemented.  Additionally, the
Xen team bears the burdens of maintaining the qemu-traditional tree,
backporting fixes originating from various places into qemu-traditional,
and having libxl cope with the differences between QEMU upstream and
traditional and associated development headaches.  These represent
underappreciated and unnecessary drain on the Xen project's limited
resources imposed by not getting QEMU upstream stubdom implemented in a
timely manner.

Also, it looks like QEMU upstream would be the single most complex piece of
software to be implemented under rumprun, which puts the current plan
development way out on the bleeding edge.  It seems more like an experiment
with lots of unknowns than a development project with a defined and
predictable plan that simply needs man hours thrown at it.  Although
rumprun has technical "sex appeal," and it taps into the community's recent
interest in unikernels, rumprun still seems to be very immature at this
time.

On top of that, getting this project implemented - even at a basic level of
functionality - appears to require VERY specialized expertise held by VERY
few people.  Is there really anyone other than Wei, IanJ, and Antti that
has the skills needed to realize the current plan?  Further tightening the
resource crunch, those three are in very high demand for other projects:
Wei had (and has?) release management duties, IanJ is involved in pretty
much everything Xen, and Antti has broader pursuits to worry about.  The
fact that no one on xen-devel has commented on Wei's October update in over
a month indicates that he is essentially alone in his efforts.


When does anyone _realistically_ see a rumprun-based QEMU reaching
something approaching just a basic level of functionality as a stubdomain -
let's say being able to run Windows 7 with a working display and networking?


Unless the answer is along the lines of hitting at least tech preview
status in Xen 4.7, I suggest that rumprun is not the right approach at this
time.  I am fairly certain - based on my direct efforts in helping to move
it along - that a Linux-based stubdom is something that CAN be realized in
the 4.7 release timeframe.  Almost all of the problems you are trying to
solve at this time with rumprun are already resolved today under Linux.
Today, on a Linux-based QEMU upstream stubdom, you can SSH into a headless
Linux distribution with fully working networking - this is light years
beyond where rumprun is or will be any time soon.  Unsurprisingly, QEMU
runs just fine in a Linux environment.  The only thing that remains to
reach the "basic level of functionality" mentioned above is getting the
display and keyboard/mouse input working, which I suspect will not be that
big of an effort for someone familiar with QEMU upstream internals (and
would have to be done anyway for rumprun).  I also believe that what has
been described as the main problem for Linux-based stubdom - building a
Linux-based image - is not as big as it has been made out to be: raisin may
now reduce some of the previously perceived difficulties, and there is so
much Linux expertise in the Xen community that I find it incredible that it
is too hard to sort out how to build Linux on a Linux system that is
already capable of building Xen and QEMU upstream.

I am not suggesting that rumprun should be abandoned - just that in terms
of actually getting QEMU upstream stubdom released and finally kicking
qemu-traditional and other development resource drains to the curb, a
Linux-based stubdom represents a practical and effective solution that can
be used until rumprun is more mature and predictable.  Almost all of the
work (with the exception of building the Linux-based image) is the same
work that will have to be done to get the rumprun implementation working:
getting the display working, and getting QEMU upstream to provide its
current range of features from within a stubdom (as much of what is in
there today depends on running in dom0).  So, when the day arrives that a
rumprun-based stubdom is ready, little of what needs to be done for a
Linux-based stubdom will end up being wasted effort.  Plus, there might
just be some benefits from having QEMU running in a Linux PV domain
(firewalling provided in the stubdom and easier debug, as possible
examples) that could justify keeping it around even after rumprun is
practical.

Thank you for taking the time to read this.  I have been following, and
participating a bit in, this issue for more than 2.5 years, and I haven't
observed any progress in that time - if anything, things are beginning to
make backwards now that QEMU has become a target.  I believe it is time for
the Xen project to acknowledge that a rumprun-based stubdom is beyond the
level of resources and expertise that the project is actually willing or
able to commit to it (as Xen has - quite rationally - set it aside time and
again to pursue other matters).  In the face of finite and limited
development resources, it is sensible to set aside a "perfect" solution
aside, and instead pursue a path that has a genuine prospect of getting
released sometime soon.

All the best,
Eric

> Hi all
>
> I've been working on and off on upstream QEMU stubdom. Unfortunately in
> the last few months I didn't make much progress as I had expected. Some
> folks expressed interests during Xen developer summit on this project
> and I think it would be better I post what I have as soon as possible.
>
> In short, it is now possible to compile and run QEMU as a PV guest,
> but I haven't finished the plumbing of toolstack code, hence I don't
> know whether it functions as a device model.
>
> Note that there are quite a bit of hacks inside those scripts to work
> around problems in build systems of different piece of software.
>
> There is a small program to configure network topology inside the
> guest. It creates a bridge, then adds a bunch of tap interfaces and
> existing Xen vif interface(s) to the bridge.
>
> What are missing:
>
> 1. Inter-domain communication mechanism: this can be subsidised by
>    network interface for now.
> 2. Plumbing through libxl. It will be using rumprun to launch QEMU
>    stubdom.
> 3. Integration with Raisin build system. This is the ulterior goal to
>    get rid of all customised build scripts.
>
> I haven't checked everything, but some of the rump kernel interfaces
> are not yet declared stable. Relevant discussions should mainly take
> place on rump kernel mailing list and CC xen-devel when necessary.
>
> Here are the step-by-step guide for building a QEMU unikernel.
>
> Note that all of my scripts have hardcoded paths (/local/scratch/...),
> so anyone who is interested in using them would need to replace the
> hardcoded paths with something sensible.
>
> $XEN refers to Xen source code. I will always refer to upstream
> repositories and only provide git branches unless there are local
> changes needed.
>
> Rumprun is a fast moving target so I will also state the working
> version of things.
>
> # Build rumprun toolchain (based on a5f58548)
>
> $ git clone https://github.com/rumpkernel/rumprun.git rumprun.git
> $ cd rumprun.git
> $ export XEN_HEADERS=$XEN/xen/include/public
> $ ./build-rr xen -- -j8
>
> If nothing goes wrong, the toolchain will be available under app-tools
> directory.
>
> # Build QEMU dependency
>
> Currently only a minimum set of libraries that QEMU depends on are
> built. They are zlib, libffi, glib, libpixman, Xen libxenctrl and
> libxenstore. More libraries might be needed as we move forward, but
> overall the build process should be the same.
>
> I have a git repository for build scripts.
>
> $ git clone https://xenbits.xen.org/people/liuw/rump-build-scrpt-junk.git
rump-build-script-junk
> $ cd rump-build-script-junk
> $ ./script 2>&1 | tee buildlog
>
> The script will download tarballs from various websites and expects
> Xen source tree resides at a fix path (see script).
>
> If everything goes well, those libraries will be inside
> /local/scratch/Rump-kernels/prefix. If something goes wrong, dig into
> buildlog and figure out why. :-)
>
> # Build QEMU
>
> I have prepared a branch for my WIP patches for QEMU.
>
> $ cd rump-build-junk-script
> $ git clone https://xenbits.xen.org/people/liuw/qemu.git -b wip.rump qemu
> $ ./script-qemu 2>&1 | tee buildlog
>
> If everything goes well, qemu will be inside
> /local/scratch/Rump-kernels/prefix-qemu/bin.
>
> But that's not yet a working domain image.
>
> # "Baking" QEMU into a unikernel
>
> $ git clone https://xenbits.xen.org/people/liuw/qemu-rump-config.git qemu-
rump-config
> $ cd rump-build-junk-script
> $ ./script-bake
>
> Then the final binary will be in
> /local/scratch/Rump-kernels/prefix-qemu/bin/qemu-system-i386-rump.
>
> # Test the unikernel
>
> Note that I omit path prefixes in following commands.
>
> $ scp qemu-system-i386-rump testbox:~
> $ scp app-tools/rumprun testbox:~
>
> Then on testbox.
>
> $ ./rumprun -S xen -i qemu-system-i386-rump --help
>
> This rune should start the guest, attach to its console and show you
> the help information from QEMU.
>
> To add network interface, see ./rumprun -h.
>
>
> Wei.

[-- Attachment #1.2: Type: text/html, Size: 27129 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Rumprun based upstream QEMU stubdom status update

[Wei Liu]

Hi Eric

In fact we would like to have Linux-stubdom. Just that when it was
first posted back then we didn't know where to put it. Xen tree is not
an option. Now that Raisin is here it seems a good fit.

On Mon, Nov 23, 2015 at 02:52:47AM -0500, Eric Shelton wrote:
> Wei and others,
> 
> It has been a long, long road to getting a rumprun-based stubdom
> implemented.  The earliest discussion of pursuing this approach appears to
> be a post by IanJ on August 21, 2013, with the first serious indication of
> development work again posted by IanJ, on April 8, 2014 - over 19 months
> ago:
> 
> http://lists.xenproject.org/archives/html/xen-devel/2014-04/msg01066.html
> 
> From what is described below, it looks like there remains a long way to go,
> as what appears to be running is only printing out the output from "qemu
> --help" (pretty much what was described at XPDS15 more than three months
> ago).
> 
> Once upon a time, one of the release managers was talking about making QEMU
> upstream stubdom a blocker for 4.5:
> 
> http://lists.xen.org/archives/html/xen-devel/2014-04/msg01065.html
> 
> Obviously that did not happen.  Time and again, QEMU upstream stubdom has
> gotten shelved in favor of other priorities, which testifies as to where it
> falls on the Xen project's list of priorities.
> 
> I believe it is time to reassess the current development plan for QEMU
> upstream stubdom.  From a project management perspective, the current plan
> relies far too heavily on a very small handful of people that are simply
> too busy with other things to bring it to a working, let alone releasable,

Keep in mind that if it is the same set of people who are going to
work on Linux-based stubdom, the constraint is the same. Actually
there is more than just "making it work". I will elaborate on that
later.

> state.  Plus, the delay in realizing this has now reached a point where it
> imposes frequent, tangible, and possibly significant drains on development
> resources in order to cope with shortcomings of the current situation (QEMU
> upstream in dom0, and separate qemu-traditional codebase for stubdom).
> Plus, having QEMU upstream run in dom0 is likely Xen's single longest
> running identified security issue - and certainly the longest-lived one
> with an identified and actionable solution.  Consider how many of the
> QEMU-related XSAs reported something along the lines of "systems using
> qemu-dm stubdomain device models are NOT vulnerable."
> 
> Recently, following several XSAs demonstrating an interest in finding (and
> likely exploiting) QEMU vulnerabilities, the Xen team has devoted resources
> to running QEMU as non-root (now on patchset v10) - apparently trying to
> replicate the bandaid used by KVM.  This effort would not have been
> necessary if QEMU upstream stubdom had been implemented.  Additionally, the
> Xen team bears the burdens of maintaining the qemu-traditional tree,
> backporting fixes originating from various places into qemu-traditional,
> and having libxl cope with the differences between QEMU upstream and
> traditional and associated development headaches.  These represent
> underappreciated and unnecessary drain on the Xen project's limited
> resources imposed by not getting QEMU upstream stubdom implemented in a
> timely manner.
> 

While I agree current situation is not ideal, there is a flip side to
Linux-based or rump kernel based stubdom. Again, I will sum that up
later.

> Also, it looks like QEMU upstream would be the single most complex piece of
> software to be implemented under rumprun, which puts the current plan
> development way out on the bleeding edge.  It seems more like an experiment
> with lots of unknowns than a development project with a defined and
> predictable plan that simply needs man hours thrown at it.  Although
> rumprun has technical "sex appeal," and it taps into the community's recent
> interest in unikernels, rumprun still seems to be very immature at this
> time.
> 
> On top of that, getting this project implemented - even at a basic level of
> functionality - appears to require VERY specialized expertise held by VERY
> few people.  Is there really anyone other than Wei, IanJ, and Antti that
> has the skills needed to realize the current plan?  Further tightening the

Just to clarify some misunderstandings.

It's getting better at the moment. I think I managed to clear some
major obstacles (that is, writing drivers for rumprun, wrestling with
mini-os bugs or whatnot). It's in a better shape than before.

Another chunk of work is teaching QEMU to not initialised some
component or take a different path when initialising some
components. This is the same as Linux-based stubdom.

All that is left is plumbing toolstack and generating an image. Those
work items aren't really specific to rumprun based
stubdom. Linux-based stubdom is ahead in that regard, but there are
still patches waiting to be upstreamed and some other issues.

> resource crunch, those three are in very high demand for other projects:
> Wei had (and has?) release management duties, IanJ is involved in pretty
> much everything Xen, and Antti has broader pursuits to worry about.  The
> fact that no one on xen-devel has commented on Wei's October update in over
> a month indicates that he is essentially alone in his efforts.
> 
> 
> When does anyone _realistically_ see a rumprun-based QEMU reaching
> something approaching just a basic level of functionality as a stubdomain -
> let's say being able to run Windows 7 with a working display and networking?
>

There is no ETA at the moment TBH. I would only say as soon as
possible, when we find solutions to all external and internal issues
in this project. That includes but is not limited to doing the actual
work myself or waiting for external environment to become mature.

> 
> Unless the answer is along the lines of hitting at least tech preview
> status in Xen 4.7, I suggest that rumprun is not the right approach at this
> time.  I am fairly certain - based on my direct efforts in helping to move
> it along - that a Linux-based stubdom is something that CAN be realized in
> the 4.7 release timeframe.  Almost all of the problems you are trying to
> solve at this time with rumprun are already resolved today under Linux.
> Today, on a Linux-based QEMU upstream stubdom, you can SSH into a headless
> Linux distribution with fully working networking - this is light years
> beyond where rumprun is or will be any time soon.  Unsurprisingly, QEMU
> runs just fine in a Linux environment.  The only thing that remains to
> reach the "basic level of functionality" mentioned above is getting the
> display and keyboard/mouse input working, which I suspect will not be that
> big of an effort for someone familiar with QEMU upstream internals (and
> would have to be done anyway for rumprun).  I also believe that what has
> been described as the main problem for Linux-based stubdom - building a
> Linux-based image - is not as big as it has been made out to be: raisin may
> now reduce some of the previously perceived difficulties, and there is so
> much Linux expertise in the Xen community that I find it incredible that it
> is too hard to sort out how to build Linux on a Linux system that is
> already capable of building Xen and QEMU upstream.
> 

One lesson learned from QEMU-trad stubdom is that the build system
inevitably turns into a black hole that sucks in effort but yields no
clear return. That's why we would very much prefer doing things "the
right way", i.e. relying on well-established ways to deliver this
feature, and expanding the user base till it is large enough to draw
enough interest for this feature to be self-sustained.

What is really not clear at the moment is the path to Linux
distributions and BSDs. Xen project doesn't distribute binaries. Also
, to build an image requires specialised toolchain and build system
that distros are not interested in taking in. Building for *BSD is a
no-go at the moment for Linux-based stubdom and QEMU-trad stubdom.

That said, perfection is the enemy of good. I don't see a problem for
having Linux-based stubdom as a interim solution. As said, it was just
we didn't know where to put it.

> I am not suggesting that rumprun should be abandoned - just that in terms
> of actually getting QEMU upstream stubdom released and finally kicking
> qemu-traditional and other development resource drains to the curb, a
> Linux-based stubdom represents a practical and effective solution that can
> be used until rumprun is more mature and predictable.  Almost all of the
> work (with the exception of building the Linux-based image) is the same
> work that will have to be done to get the rumprun implementation working:
> getting the display working, and getting QEMU upstream to provide its
> current range of features from within a stubdom (as much of what is in
> there today depends on running in dom0).  So, when the day arrives that a
> rumprun-based stubdom is ready, little of what needs to be done for a
> Linux-based stubdom will end up being wasted effort.  Plus, there might
> just be some benefits from having QEMU running in a Linux PV domain
> (firewalling provided in the stubdom and easier debug, as possible
> examples) that could justify keeping it around even after rumprun is
> practical.
> 

I agree most of the things will be useful for both solutions.

>
> Thank you for taking the time to read this.  I have been following, and
> participating a bit in, this issue for more than 2.5 years, and I haven't
> observed any progress in that time - if anything, things are beginning to
> make backwards now that QEMU has become a target.  I believe it is time for
> the Xen project to acknowledge that a rumprun-based stubdom is beyond the
> level of resources and expertise that the project is actually willing or
> able to commit to it (as Xen has - quite rationally - set it aside time and
> again to pursue other matters).  In the face of finite and limited
> development resources, it is sensible to set aside a "perfect" solution
> aside, and instead pursue a path that has a genuine prospect of getting
> released sometime soon.
> 

Thanks for writing such long email. I see your point.

I discussed with Stefano and Anthony to reassess Linux-based stubdom,
with all the issues mentioned above, if we narrow down the scope of
the project -- ignore BSD support for now, integrate build with
Raisin, no distros integration -- it can actually be done within a
finite time frame.

Here are some issues with the existing patch series:

1. It's using Dracut build image, which is Redhat-centric.
2. Some patches are not yet complete and not suitable to upstream.

And these are the steps that need to be done:

1. Build with Raisin
  1.1 Build Linux kernel with a customised config.
  1.2 Build QEMU from a customised branch.
  1.3 Extract QEMU libraries dependency with generic tool.
  1.4 Generate a disk image with generic tool.
2. Toolstack plumbing and make work other components.
3. Teach OSStest to constantly test it.

Would you be up for working on Raisin integration?  That is, up to the
point of producing an image. We can discuss certain actions in
detailed and will help you with the issues you encounter along the
way. We appreciate help from the community as we're already quite
stretched with other projects.


Wei.

> All the best,
> Eric
>

Re: Rumprun based upstream QEMU stubdom status update

[Eric Shelton]

[-- Attachment #1.1: Type: text/plain, Size: 5999 bytes --]

Thanks for taking the time to consider my comments and respond.

On Tue, Nov 24, 2015 at 9:44 AM, Wei Liu <wei.liu2@citrix.com> wrote:

> Another chunk of work is teaching QEMU to not initialised some
> component or take a different path when initialising some
> components. This is the same as Linux-based stubdom.
>

When I was working on Linux-based stubdom earlier, it was the point where
serious QEMU-related work needed to be done  (in particular, working out
getting the display to work correctly via VNC or otherwise) that I reached
the limit of what I could accomplish on my own in a reasonable amount of
time.  There was simply too much for me to learn about QEMU internals and
how they are supposed to interact with Xen.  On top of that, getting QMP
and save/restore working across domains appears to require some serious
effort, and is for the most part outside of what I need QEMU stubdom for.


> One lesson learned from QEMU-trad stubdom is that the build system
> inevitably turns into a black hole that sucks in effort but yields no
> clear return. That's why we would very much prefer doing things "the
> right way", i.e. relying on well-established ways to deliver this
> feature, and expanding the user base till it is large enough to draw
> enough interest for this feature to be self-sustained.
>

That last bit is where I am interested in bringing this to - having the
basic functionality in place, so that others may be encouraged to drive to
completion the aspects that they have an interest or stake in.  Right now,
there is so much that needs to be done, that many would rather not touch
any of it.


> What is really not clear at the moment is the path to Linux
> distributions and BSDs. Xen project doesn't distribute binaries. Also
> , to build an image requires specialised toolchain and build system
> that distros are not interested in taking in. Building for *BSD is a
> no-go at the moment for Linux-based stubdom and QEMU-trad stubdom.
>

I agree that for the time being, *BSD would be off the table for
Linux-based stubdom - it would take a lot of bootstrapping to get an
environment up and running under *BSD to build the Linux-based stubdom on
*BSD.

It could be sensible to modify the no-binaries position if the notion of a
reproducible build could be implemented.  This could allow distribution of
a binary image for Linux-based stubdom (which would not be all that big)
for use by *BSD and others, yet demonstrate that the binary image does
correspond to the underlying source.  However, that is several step down
the road from just getting it to work on Linux.


> I discussed with Stefano and Anthony to reassess Linux-based stubdom,
> with all the issues mentioned above, if we narrow down the scope of
> the project -- ignore BSD support for now, integrate build with
> Raisin, no distros integration -- it can actually be done within a
> finite time frame.
>
> Here are some issues with the existing patch series:
>
> 1. It's using Dracut build image, which is Redhat-centric.
>

That issue was raised before, but I'm not sure how much it really is.
FWIW, I was using my updated version of Anthony's work on a Gentoo system,
and encountered no issues.  Is there a Linux distribution that we know
dracut does not work on?  It would be helpful to know.  Although I have not
really dug into dracut, it seemed to be effective for our purposes
(collecting the libraries that need to be included in the Linux image for
QEMU), and I'd rather not recreate dracut or part of dracut if we don't
need to.


> 2. Some patches are not yet complete and not suitable to upstream.
>

What kinds of things are being done by these patches?  You mentioned
component initialization; however, I did not encounter any problems
relating to that when getting QEMU upstream stubdom up to the point of
running SSH on a standard Linux distribution, etc.  Having to disable such
things makes sense as an issue under rumprun, given the likely difficulties
in getting certain libraries to run under rumprun, but I would expect it to
be less of an issue when QEMU gets run in its native Linux environment.


> And these are the steps that need to be done:
>
> 1. Build with Raisin
>   1.1 Build Linux kernel with a customised config.
>   1.2 Build QEMU from a customised branch.
>   1.3 Extract QEMU libraries dependency with generic tool.
>   1.4 Generate a disk image with generic tool.
> 2. Toolstack plumbing and make work other components.
> 3. Teach OSStest to constantly test it.
>
> Would you be up for working on Raisin integration?  That is, up to the
> point of producing an image. We can discuss certain actions in
> detailed and will help you with the issues you encounter along the
> way. We appreciate help from the community as we're already quite
> stretched with other projects.
>

I believe that is something I can work on, and likely won't be all that bad
given where things already are.  However, I know essentially nothing about
Raisin.  What is a good way to get up to speed?

One issue that was raised previously is where would the Raison/Xen build
process download the Linux kernel source from?  kernel.org?  A cloned
branch on xenbits?  I don't see any need to have a customized branch - the
stock Linux kernel should work just fine, and we just need to supply a
.config to get the appropriate features built into the kernel.

Is there anyone that can be tapped to address at least some of the initial
QEMU issues?  To get this up to a minimal "tech preview" status, we need to
get the display, keyboard input, and mouse input working - having Windows
working at a basic level under QEMU upstream stubdom seems like a good
target.  As I mentioned above, I don't have, and don't have the time to
acquire, the necessary QEMU skills to make that work.  If there is someone
who can look at it, a basic non-Raisin build approach can be provided that
should allow a developer to build and debug QEMU upstream stubdom.


> Wei.
>
> > All the best,
> > Eric
> >
>

[-- Attachment #1.2: Type: text/html, Size: 7529 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Rumprun based upstream QEMU stubdom status update

[Wei Liu]

On Tue, Nov 24, 2015 at 11:16:47PM -0500, Eric Shelton wrote:
[...]
> 
> > I discussed with Stefano and Anthony to reassess Linux-based stubdom,
> > with all the issues mentioned above, if we narrow down the scope of
> > the project -- ignore BSD support for now, integrate build with
> > Raisin, no distros integration -- it can actually be done within a
> > finite time frame.
> >
> > Here are some issues with the existing patch series:
> >
> > 1. It's using Dracut build image, which is Redhat-centric.
> >
> 
> That issue was raised before, but I'm not sure how much it really is.
> FWIW, I was using my updated version of Anthony's work on a Gentoo system,
> and encountered no issues.  Is there a Linux distribution that we know
> dracut does not work on?  It would be helpful to know.  Although I have not
> really dug into dracut, it seemed to be effective for our purposes
> (collecting the libraries that need to be included in the Linux image for
> QEMU), and I'd rather not recreate dracut or part of dracut if we don't
> need to.
> 

I can't recollect previous objections. If we know dracut is packaged in
most major distributions then I don't have objection to it.

Currently in Anthony's patch set, dracut is downloaded and built. That's
not ideal. To integrate with Raisin, the only viable way is to install
packaged Dracut -- building the tool to build image itself is not
acceptable to me.

> 
> > 2. Some patches are not yet complete and not suitable to upstream.
> >
> 
> What kinds of things are being done by these patches?  You mentioned
> component initialization; however, I did not encounter any problems
> relating to that when getting QEMU upstream stubdom up to the point of
> running SSH on a standard Linux distribution, etc.  Having to disable such
> things makes sense as an issue under rumprun, given the likely difficulties
> in getting certain libraries to run under rumprun, but I would expect it to
> be less of an issue when QEMU gets run in its native Linux environment.
> 

The current patch set disables some components due to Anthony didn't
have time to implement them. Those patches need to be reworked.

> 
> > And these are the steps that need to be done:
> >
> > 1. Build with Raisin
> >   1.1 Build Linux kernel with a customised config.
> >   1.2 Build QEMU from a customised branch.
> >   1.3 Extract QEMU libraries dependency with generic tool.
> >   1.4 Generate a disk image with generic tool.
> > 2. Toolstack plumbing and make work other components.
> > 3. Teach OSStest to constantly test it.
> >
> > Would you be up for working on Raisin integration?  That is, up to the
> > point of producing an image. We can discuss certain actions in
> > detailed and will help you with the issues you encounter along the
> > way. We appreciate help from the community as we're already quite
> > stretched with other projects.
> >
> 
> I believe that is something I can work on, and likely won't be all that bad
> given where things already are.  However, I know essentially nothing about
> Raisin.  What is a good way to get up to speed?
> 

There is a wiki page for Raisin.

http://wiki.xenproject.org/wiki/Raisin

I guess the most effective way is to just give it a try and post
questions to xen-devel.

> One issue that was raised previously is where would the Raison/Xen build
> process download the Linux kernel source from?  kernel.org?  A cloned
> branch on xenbits?  I don't see any need to have a customized branch - the
> stock Linux kernel should work just fine, and we just need to supply a
> .config to get the appropriate features built into the kernel.
> 

Right. These are the details.

I was thinking we pick one of our tested branch in xenbits. We also need
a minimal .config for the kernel. I think you can extract the one in
Anthony's patch series.

Another way is, as you said, to just use the stock kernel. Then we need
to make sure it is xen-capable.

Details can be discussed in another threads. I'm sure there are other
trivial things that we need to specify. Say, we also need to provide a
QEMU branch with customised patches on top before we upstream all of
them.

> Is there anyone that can be tapped to address at least some of the initial
> QEMU issues?  To get this up to a minimal "tech preview" status, we need to
> get the display, keyboard input, and mouse input working - having Windows
> working at a basic level under QEMU upstream stubdom seems like a good
> target.  As I mentioned above, I don't have, and don't have the time to
> acquire, the necessary QEMU skills to make that work.  If there is someone
> who can look at it, a basic non-Raisin build approach can be provided that
> should allow a developer to build and debug QEMU upstream stubdom.
> 

It's on our radar. As you said, getting those working would be
beneficial to both Linux-based stubdom and rump kernel stubdom.

Currently getting Raisin to generate image can help a developer to focus
on solving the real problems. I appreciate that a lot.

Wei.

> 
> > Wei.
> >
> > > All the best,
> > > Eric
> > >
> >

Re: Rumprun based upstream QEMU stubdom status update

[Anthony PERARD]

On Thu, Nov 26, 2015 at 11:45:13AM +0000, Wei Liu wrote:
> On Tue, Nov 24, 2015 at 11:16:47PM -0500, Eric Shelton wrote:
> [...]
> > 
> > > I discussed with Stefano and Anthony to reassess Linux-based stubdom,
> > > with all the issues mentioned above, if we narrow down the scope of
> > > the project -- ignore BSD support for now, integrate build with
> > > Raisin, no distros integration -- it can actually be done within a
> > > finite time frame.
> > >
> > > Here are some issues with the existing patch series:
> > >
> > > 1. It's using Dracut build image, which is Redhat-centric.
> > >
> > 
> > That issue was raised before, but I'm not sure how much it really is.
> > FWIW, I was using my updated version of Anthony's work on a Gentoo system,
> > and encountered no issues.  Is there a Linux distribution that we know
> > dracut does not work on?  It would be helpful to know.  Although I have not
> > really dug into dracut, it seemed to be effective for our purposes
> > (collecting the libraries that need to be included in the Linux image for
> > QEMU), and I'd rather not recreate dracut or part of dracut if we don't
> > need to.
> > 
> 
> I can't recollect previous objections. If we know dracut is packaged in
> most major distributions then I don't have objection to it.
> 
> Currently in Anthony's patch set, dracut is downloaded and built. That's
> not ideal. To integrate with Raisin, the only viable way is to install
> packaged Dracut -- building the tool to build image itself is not
> acceptable to me.

The only things that is use from dracut is dracut-install, I think. It
copies a binary and the libraries that the binary depends on into a
directory. I guess this could be replaced by a script if using tools
available in distros is an issue.

-- 
Anthony PERARD