qemu-upstream stubdom - status?

qemu-upstream stubdom - status?

[Eric Shelton]

[-- Attachment #1.1: Type: text/plain, Size: 1119 bytes --]

Where does this feature stand currently?  It pops up under a couple of
names and proposed implementations (e.g., "qemu-xen stubdomains",
"qemu-upstream stubdom, Linux", and "qemu-upstream stubdom, BSD libc"), so
it seems likely I missed something.

If I understand this email from Sept 2013 (
http://lists.xen.org/archives/html/xen-devel/2013-09/msg02881.html),
qemu-xen stubdomains might have made it into 4.4 if the schedule had been
slipped a little, suggesting some active effort.  On the other hand, the
same email designated the efforts attached to Anthony and Ian as
"prognosis: ?".  Anthony released a patchset/branch about a year ago, but
that got sidelined in favor of getting Xen on ARM.  However, I did not
identify any more recent patches.

It remains appealing to make use of the more recent developments in
qemu-upstream, but with the isolation provided by a stubdomain.  The many
improvements made in teasing driver domains out of dom0 strengthen this
appeal (for example, using OpenSolaris as a ZFS-based storage domain).

What are the most recent patches or repository for this feature?

Thanks,
Eric

[-- Attachment #1.2: Type: text/html, Size: 1370 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: qemu-upstream stubdom - status?

[George Dunlap]

On Mon, Apr 7, 2014 at 9:50 PM, Eric Shelton <eshelton@pobox.com> wrote:
> Where does this feature stand currently?  It pops up under a couple of names
> and proposed implementations (e.g., "qemu-xen stubdomains", "qemu-upstream
> stubdom, Linux", and "qemu-upstream stubdom, BSD libc"), so it seems likely
> I missed something.
>
> If I understand this email from Sept 2013
> (http://lists.xen.org/archives/html/xen-devel/2013-09/msg02881.html),
> qemu-xen stubdomains might have made it into 4.4 if the schedule had been
> slipped a little, suggesting some active effort.  On the other hand, the
> same email designated the efforts attached to Anthony and Ian as "prognosis:
> ?".  Anthony released a patchset/branch about a year ago, but that got
> sidelined in favor of getting Xen on ARM.  However, I did not identify any
> more recent patches.
>
> It remains appealing to make use of the more recent developments in
> qemu-upstream, but with the isolation provided by a stubdomain.  The many
> improvements made in teasing driver domains out of dom0 strengthen this
> appeal (for example, using OpenSolaris as a ZFS-based storage domain).
>
> What are the most recent patches or repository for this feature?

I think stubdomains for qemu-xen should be a blocker for 4.5.

The only thing that's missing for stubdoms, AFAIK, is an integrated
way to deploy the stubdom VM in which qemu-xen can run.
qemu-traditional uses minios with a version of newlib (IIRC) ported to
it.  The problem at the moment is that qemu-xen requires much more
from its libc than newlib provides.  There are several potential
solutions that have been investigated:

1. Use Linux with glibc.
2. Port a more fully-functional libc (e.g., a BSD libc) to minios
3. Use the "rump kernel" functionality
4. Use OSv.

#1 involves mainly trying to cut down the Linux kernel and the guest
FS as small as possible.  Anthony spent some time on #1 in the 4.3 and
4.4 development cycles, and I think got the image down to 32MB.
That's still fairly large, however; and I think he got stuck trying to
integrate an image builder into the Xen build system.

#2 involves dealing with the mismatch between the OS functionality
provided by minios and the functionality required by the libc:
threads, async IO, &c.  This entails either disabling / working around
the lack of functionality, or implementing the requisite
functionality.  Ian Jackson spent some time in the 4.3 development
cycle working on this, but ended up having to switch to something
else.

#3 on the surface looks promising: the "rump kernels" approach allows
you to lift bits of the BSD kernel into another context, like
user-space; or in our case, a PV "Cloud OS".  The advantage of this is
that it's using all the existing tested code that works together, but
just replacing certain bits of it: e.g., the lowest layers with PV,
and the systemcall boundary with a function call.  This has already
been used to make single-address-space PV BSD guests boot, IIRC; it
would just take someone familiar with both to sit down integrate it.
At the moment I don't think anyone is actively looking at this.

#4 would involve taking the existing OSv system and adding missing
functionality, if any.  I don't think anyone is looking at this
either.

If you're looking for a solution to use right now, I'm sure Anthony
can point you to wherever he left off.  If you're looking to maybe get
involved and contribute, I think the consensus was that #3 looked like
the most promising option -- I'm sure IanJ can give you some pointers
of where to look.

 -George

Re: qemu-upstream stubdom - status?

[Ian Jackson]

Eric Shelton writes ("qemu-upstream stubdom - status?"):
> Where does this feature stand currently?  It pops up under a couple
> of names and proposed implementations (e.g., "qemu-xen stubdomains",
> "qemu-upstream stubdom, Linux", and "qemu-upstream stubdom, BSD
> libc"), so it seems likely I missed something.

Hi.  I made a start on bsd libc + minios last release cycle but I
think this has now been overtaken by the rump kernels work by Antti
Kantee.  See https://github.com/rumpkernel/wiki/wiki etc.

> It remains appealing to make use of the more recent developments in
> qemu-upstream, but with the isolation provided by a stubdomain.  The
> many improvements made in teasing driver domains out of dom0
> strengthen this appeal (for example, using OpenSolaris as a
> ZFS-based storage domain).

Yes.

> What are the most recent patches or repository for this feature?

I've just started looking at the rump kernels with a view to
(eventually) trying to get qemu-upstream working but I don't have
anything I can show you yet.

Thanks,
Ian.

(CC Antti.)

Re: qemu-upstream stubdom - status?

[Anthony PERARD]

On Tue, Apr 08, 2014 at 11:49:43AM +0100, George Dunlap wrote:
> On Mon, Apr 7, 2014 at 9:50 PM, Eric Shelton <eshelton@pobox.com> wrote:
> > Where does this feature stand currently?  It pops up under a couple of names
> > and proposed implementations (e.g., "qemu-xen stubdomains", "qemu-upstream
> > stubdom, Linux", and "qemu-upstream stubdom, BSD libc"), so it seems likely
> > I missed something.
> >
> > If I understand this email from Sept 2013
> > (http://lists.xen.org/archives/html/xen-devel/2013-09/msg02881.html),
> > qemu-xen stubdomains might have made it into 4.4 if the schedule had been
> > slipped a little, suggesting some active effort.  On the other hand, the
> > same email designated the efforts attached to Anthony and Ian as "prognosis:
> > ?".  Anthony released a patchset/branch about a year ago, but that got
> > sidelined in favor of getting Xen on ARM.  However, I did not identify any
> > more recent patches.
> >
> > It remains appealing to make use of the more recent developments in
> > qemu-upstream, but with the isolation provided by a stubdomain.  The many
> > improvements made in teasing driver domains out of dom0 strengthen this
> > appeal (for example, using OpenSolaris as a ZFS-based storage domain).
> >
> > What are the most recent patches or repository for this feature?
> 
> I think stubdomains for qemu-xen should be a blocker for 4.5.
> 
> The only thing that's missing for stubdoms, AFAIK, is an integrated
> way to deploy the stubdom VM in which qemu-xen can run.
> qemu-traditional uses minios with a version of newlib (IIRC) ported to
> it.  The problem at the moment is that qemu-xen requires much more
> from its libc than newlib provides.  There are several potential
> solutions that have been investigated:
> 
> 1. Use Linux with glibc.
[...]

> #1 involves mainly trying to cut down the Linux kernel and the guest
> FS as small as possible.  Anthony spent some time on #1 in the 4.3 and
> 4.4 development cycles, and I think got the image down to 32MB.
> That's still fairly large, however; and I think he got stuck trying to
> integrate an image builder into the Xen build system.
[...]

> If you're looking for a solution to use right now, I'm sure Anthony
> can point you to wherever he left off.

I have this branch that is the most recent work I have:
git://xenbits.xen.org/people/aperard/xen-unstable.git
branch: stubdom

`make -C stubdom-linux` should make you an image.

Unfortunatly, there is no video output that can be seen, so a guest is
only accessible through the serial console or the network. And there are
probably many other issues.

Hope that help,

-- 
Anthony PERARD

Re: qemu-upstream stubdom - status?

[Antti Kantee]

On 08/04/14 10:49, Ian Jackson wrote:
> Eric Shelton writes ("qemu-upstream stubdom - status?"):
>> Where does this feature stand currently?  It pops up under a couple
>> of names and proposed implementations (e.g., "qemu-xen stubdomains",
>> "qemu-upstream stubdom, Linux", and "qemu-upstream stubdom, BSD
>> libc"), so it seems likely I missed something.
>
> Hi.  I made a start on bsd libc + minios last release cycle but I
> think this has now been overtaken by the rump kernels work by Antti
> Kantee.  See https://github.com/rumpkernel/wiki/wiki etc.
>
>> It remains appealing to make use of the more recent developments in
>> qemu-upstream, but with the isolation provided by a stubdomain.  The
>> many improvements made in teasing driver domains out of dom0
>> strengthen this appeal (for example, using OpenSolaris as a
>> ZFS-based storage domain).
>
> Yes.
>
>> What are the most recent patches or repository for this feature?
>
> I've just started looking at the rump kernels with a view to
> (eventually) trying to get qemu-upstream working but I don't have
> anything I can show you yet.

Last time we discussed this (ca. FOSDEM-time), I put up a page to list 
the issues with the hopes of a TODO-list making it easy to just tick 
items off quickly.  Wishful thinking.  I probably also forgot to send 
the URL to you.  Anyway, I updated the content now, here's the URL:

http://wiki.rumpkernel.org/Info:-Remaining-issues-for-%22OS-less%22-rump-kernels

I think Justin Cormack (also cc'd) has something yet unpublished for the 
build issue with some form of "compiler triplet".

Re: qemu-upstream stubdom - status?

[Ian Jackson]

Antti Kantee writes ("Re: qemu-upstream stubdom - status?"):
> On 08/04/14 10:49, Ian Jackson wrote:
> > I've just started looking at the rump kernels with a view to
> > (eventually) trying to get qemu-upstream working but I don't have
> > anything I can show you yet.
> 
> Last time we discussed this (ca. FOSDEM-time), I put up a page to list 
> the issues with the hopes of a TODO-list making it easy to just tick 
> items off quickly.  Wishful thinking.  I probably also forgot to send 
> the URL to you.  Anyway, I updated the content now, here's the URL:

Thanks.

The big one there is pthreads but I don't think we actually need that
for qemu-dm, provided we have aio, which it looks like we do ?

> I think Justin Cormack (also cc'd) has something yet unpublished for the 
> build issue with some form of "compiler triplet".

ATM I can't build the current rumpuser-xen tip, because I get
  ld: cannot find -lrumpkern_time
and indeed no librumpkern_time.a seems to have been built.

(I have edited rumpuser-xen/Config.mk to refer to my xen.git's
dist/install for XEN_HEADERS.)

Thanks,
Ian.

Re: qemu-upstream stubdom - status?

[Justin Cormack]

On Tue, Apr 8, 2014 at 6:03 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> ATM I can't build the current rumpuser-xen tip, because I get
>   ld: cannot find -lrumpkern_time
> and indeed no librumpkern_time.a seems to have been built.
>
> (I have edited rumpuser-xen/Config.mk to refer to my xen.git's
> dist/install for XEN_HEADERS.)

Sorry I broke that by accident - pull it again and should be fixed.

Justin

Re: qemu-upstream stubdom - status?

[Justin Cormack]

On Tue, Apr 8, 2014 at 6:03 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> The big one there is pthreads but I don't think we actually need that
> for qemu-dm, provided we have aio, which it looks like we do ?

The aio syscalls are not in the rump kernel at present. They could
easily be added but not sure that signal notification will work in
rump, although polling should. Unfortunately NetBSD does not have
kqueue aio support yet, which would be ideal.

Justin

Re: qemu-upstream stubdom - status?

[Ian Jackson]

Justin Cormack writes ("Re: qemu-upstream stubdom - status?"):
> On Tue, Apr 8, 2014 at 6:03 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> > The big one there is pthreads but I don't think we actually need that
> > for qemu-dm, provided we have aio, which it looks like we do ?
> 
> The aio syscalls are not in the rump kernel at present. They could
> easily be added but not sure that signal notification will work in
> rump, although polling should. Unfortunately NetBSD does not have
> kqueue aio support yet, which would be ideal.

Hmm.  I was going to say that qemu doesn't want signal notification.
But actually it is doing something with threads.  I think this can
probably be fixed in qemu.

Ian.

Re: qemu-upstream stubdom - status?

[Ian Jackson]

Justin Cormack writes ("Re: qemu-upstream stubdom - status?"):
> On Tue, Apr 8, 2014 at 6:03 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> > ATM I can't build the current rumpuser-xen tip, because I get
> >   ld: cannot find -lrumpkern_time
> > and indeed no librumpkern_time.a seems to have been built.
> >
> > (I have edited rumpuser-xen/Config.mk to refer to my xen.git's
> > dist/install for XEN_HEADERS.)
> 
> Sorry I broke that by accident - pull it again and should be fixed.

Ah, thanks, yes it is :-).

Thanks,
Ian.

Re: qemu-upstream stubdom - status?

[Antti Kantee]

On 08/04/14 17:47, Ian Jackson wrote:
> Justin Cormack writes ("Re: qemu-upstream stubdom - status?"):
>> On Tue, Apr 8, 2014 at 6:03 PM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
>>> The big one there is pthreads but I don't think we actually need that
>>> for qemu-dm, provided we have aio, which it looks like we do ?
>>
>> The aio syscalls are not in the rump kernel at present. They could
>> easily be added but not sure that signal notification will work in
>> rump, although polling should. Unfortunately NetBSD does not have
>> kqueue aio support yet, which would be ideal.
>
> Hmm.  I was going to say that qemu doesn't want signal notification.
> But actually it is doing something with threads.  I think this can
> probably be fixed in qemu.

Yea, it probably can be fixed in qemu, but I'd still like to have 
~generic(*) pthreads available for the rump kernel driver stack.  I 
started playing with pthreads support.  It shouldn't be too difficult in 
principle, but as usual, details are everything and saying that 
something isn't difficult tends to jinx it...  Meanwhile, don't let that 
stop you from creating a qemu-specific solution; insight from a 
"meet-in-the-middle" type of implementation attack never hurts.

I'll also include the aio syscall driver, for the sake of completeness, 
even if it's not strictly speaking required here.

If there are build problems in the future, check 
http://builds.rumpkernel.org.  We usually fix problems flagged there 
within an hour or two (does not apply to the NetBSD HEAD build, which 
also depends on the state of NetBSD HEAD itself).

*) in a cpu-bound thread without any I/O blocking points, the lack of a 
clock interrupt like scheduling facility is a bit of a bummer.  I'm 
assuming that's not an issue with qemu's use of threads.

Re: qemu-upstream stubdom - status?

[Eric Shelton]

On Tue, Apr 8, 2014 at 7:26 AM, Anthony PERARD
<anthony.perard@citrix.com> wrote:
> On Tue, Apr 08, 2014 at 11:49:43AM +0100, George Dunlap wrote:
>> On Mon, Apr 7, 2014 at 9:50 PM, Eric Shelton <eshelton@pobox.com> wrote:
>> > Where does this feature stand currently?  It pops up under a couple of names
>> > and proposed implementations (e.g., "qemu-xen stubdomains", "qemu-upstream
>> > stubdom, Linux", and "qemu-upstream stubdom, BSD libc"), so it seems likely
>> > I missed something.
>> >
>> > If I understand this email from Sept 2013
>> > (http://lists.xen.org/archives/html/xen-devel/2013-09/msg02881.html),
>> > qemu-xen stubdomains might have made it into 4.4 if the schedule had been
>> > slipped a little, suggesting some active effort.  On the other hand, the
>> > same email designated the efforts attached to Anthony and Ian as "prognosis:
>> > ?".  Anthony released a patchset/branch about a year ago, but that got
>> > sidelined in favor of getting Xen on ARM.  However, I did not identify any
>> > more recent patches.
>> >
>> > It remains appealing to make use of the more recent developments in
>> > qemu-upstream, but with the isolation provided by a stubdomain.  The many
>> > improvements made in teasing driver domains out of dom0 strengthen this
>> > appeal (for example, using OpenSolaris as a ZFS-based storage domain).
>> >
>> > What are the most recent patches or repository for this feature?
>>
>> I think stubdomains for qemu-xen should be a blocker for 4.5.
>>
>> The only thing that's missing for stubdoms, AFAIK, is an integrated
>> way to deploy the stubdom VM in which qemu-xen can run.
>> qemu-traditional uses minios with a version of newlib (IIRC) ported to
>> it.  The problem at the moment is that qemu-xen requires much more
>> from its libc than newlib provides.  There are several potential
>> solutions that have been investigated:
>>
>> 1. Use Linux with glibc.
> [...]
>
>> #1 involves mainly trying to cut down the Linux kernel and the guest
>> FS as small as possible.  Anthony spent some time on #1 in the 4.3 and
>> 4.4 development cycles, and I think got the image down to 32MB.
>> That's still fairly large, however; and I think he got stuck trying to
>> integrate an image builder into the Xen build system.
> [...]
>
>> If you're looking for a solution to use right now, I'm sure Anthony
>> can point you to wherever he left off.
>
> I have this branch that is the most recent work I have:
> git://xenbits.xen.org/people/aperard/xen-unstable.git
> branch: stubdom
>
> `make -C stubdom-linux` should make you an image.
>
> Unfortunatly, there is no video output that can be seen, so a guest is
> only accessible through the serial console or the network. And there are
> probably many other issues.
>
> Hope that help,
>
> --
> Anthony PERARD

With the impending rollout of another Xen release lacking qemu-xen
stubdom, I would like to campaign for a couple of things: (1) making
qemu-xen stubdom a blocker for Xen 4.6; and (2) using a Linux-based
stubdom, at least for the time being.

(1) Making qemu-xen a blocker for 4.6

This security issues presented by allowing qemu to run unrestricted
within dom0 have been appreciated for a long time (comments about
qemu-dm in XSA-109 illustrate this).  Just last month, a few
additional qemu escalation vulnerabilities were demonstrated.  Given
the size, complexity, and pace of development of qemu, we can
reasonably assume there will always be some escalation vulnerability.

Xen has generally taken known, and often far more obscure, security
issues very seriously.  I am puzzled that Xen continues to offer a
non-stubdom qemu.  The current qemu stubdom solution almost amounts to
a mitigation, rather than a solution, of a well-known issue.  The
options offered today are: (a) have access to all of the features and
bugfixes offered by upstream qemu (which have been significant thanks
to KVM's use of qemu, and are appreciable on and sometimes necessary
for some guest operating systems), at the cost of qemu running
unrestricted within dom0; or (b) have qemu contained in a separate
domain, but with what is essentially only a "good enough" six year old
version of qemu (ver 0.10.2).

Additionally, once qemu-xen stubdom is realized, we can move away from
and stop maintaining the forked qemu-traditional codebase.  Finally,
all of the Xen-specific code could be mainstreamed into upstream qemu.

If all that can managed is to release qemu-xen stub domains at the
level of a tech preview, it still represents a significant improvement
on both of the axes of functionality and security.

(2) Using Linux to implement qemu-xen stubdom

Efforts over the last 3 years at realizing qemu-xen stub domains seem
to illustrate the "perfect is the enemy of the good" phenomenon.  I
gather that rump kernel is the favored direction, and ultimately would
be more memory efficient than using Linux, but a lot of unresolved
technical issues lie down that path that are already solved by using
Linux, and resolving them has, and likely will continue to, stymie
getting something effective and reasonable out the door.

It is understandable that the Xen team has prioritized other items
over qemu-xen stubdom.  However, rather than allowing a lower priority
item to be left undone, that should inform us that an "ideal," but
more development intensive, path is not called for.  Instead, we
should just adopt the more easily implemented solution and move on.

Some reasons for picking up where Anthony left off with using Linux:
(a) Linux + qemu is a mature and well-tested codebase.  Although there
are some other environments under which qemu is built and run, far and
away the most common platform is Linux.
(b) Linux kernel + Xen is a mature and well-tested codebase.  Although
there was some disappointment that external patches had to be applied
to the Linux kernel, they were pretty minor.
(c) A lot things need to be done for rump kernel that already just
work with Linux.
(d) Developers are more familiar with Linux.  At one point along the
way, there was expressed interest in moving away from the obscure
mini-os.  With rump kernel, mini-os remains in the picture AND we are
adding NetBSD into the mix.  There are very few developers with the
skills needed to make rump kernel happen or contribute to it.
(e) The changes that need to be made to upstream qemu to run in a stub
domain are a fixed item - they are somewhat independent of the
underlying OS/execution environment.  If Xen starts with a Linux-based
stubdom, the transition can still be made to rump kernel.
(f) The memory overhead issue is overblown.  As time goes on, systems
have more memory and memory gets cheaper.  How hard do we work to
reduce the 32 MB used for Linux when we're allocating 2+ GB of memory
to an HVM domain anyway, and how much of a reduction would we realize
anyway?

When Anthony released his patches, there was some disappointment
expressed with the build process (which kernel version was being used
and reliance on busybox, for example).  None of the raised issues
strike me as significant problems - just decisions that have to be
made.  Anyways, I find it incredible to believe that a MiniOS + NetBSD
rump kernel + libc + whatever else solution is going to result in a
cleaner build process.

Given limited developer resources, sometimes tradeoffs have to be
made.  I propose that Linux-based stub domains are the right tradeoff
for incorporation into Xen 4.6.  Rump kernels may indeed be the
future, but they do not seem to be the immediate solution for qemu
stubdoms that we could use.

Thank you,
Eric