RFC: Cleaning up the Mini-OS namespace

RFC: Cleaning up the Mini-OS namespace

[Martin Lucina]

Hi,

In rumprun-xen [1] we use Mini-OS as a "base firmware" layer in our stack.
Currently we are using a slightly bastardized fork of the xen.git Mini-OS.

We would like to avoid this turning into a permanent fork. Following
previous discussion [2] on and openmirage-devel I would like to coordinate
upstreaming our changes if possible.

The biggest change which needs to be done is cleaning up the Mini-OS
namespace. This is necessary as we link Mini-OS with the application, rump
kernel and NetBSD libc to get a full application stack.

The changes as implemented in a semi-automated fashion for the Mini-OS used
by rumprun-xen can be viewed in the (since merged) pull request at [3]:

- All Mini-OS functions called by rumprun-xen are renamed to minios_* or
  _minios_* for strictly internal functions, except those in the
  blkfront_*, netfront_*, pcifront_* and xenbus_* driver namespaces.
- In the case of drivers, eg. init|shutdown_blkfront are renamed to
  blkfront_*.
- All global variables are either manually made local, or placed under
  the _minios_* namespace, with the exception of HYPERVISOR_shared_info,
  and those variables under driver namespaces kept above.
- All callers are updated to use the new names. Where it makes sense,
  macros such as alloc_page are also renamed into the minios_ namespace.

Questions:

- Is there a general interest in upstreaming this work?
- Upstream Mini-OS provides things we (rumpkernel.org) don't need (stub
  "libc", ...). If we are to move to upstream then we will need to do a
  more general cleanup of Mini-OS to make these pluggable. Again, is there
  upstream interest in this work?
- As there is no formal API for Mini-OS. My changes only address the
  "public" functionality used by rumprun-xen. Other users mileage will
  vary; who else should I coordinate with?
- I have not namespaced macros such as local_irq_save(). Should this be
  done? 
  
  Also, the driver namespaces have been preserved (since I was lazy), these
  should probably be renamed under the minios namespace; it's plausible
  some day someone will try to link an application with a function called
  blkfront_init().

All comments and review welcome.

Martin

[1] http://repo.rumpkernel.org/rumprun-xen
[2] http://thread.gmane.org/gmane.comp.rumpkernel.user/514
[3] https://github.com/rumpkernel/rumprun-xen/pull/10

Re: RFC: Cleaning up the Mini-OS namespace

[Andrew Cooper]

On 04/12/14 14:27, Martin Lucina wrote:
> Hi,
>
> In rumprun-xen [1] we use Mini-OS as a "base firmware" layer in our stack.
> Currently we are using a slightly bastardized fork of the xen.git Mini-OS.
>
> We would like to avoid this turning into a permanent fork. Following
> previous discussion [2] on and openmirage-devel I would like to coordinate
> upstreaming our changes if possible.
>
> The biggest change which needs to be done is cleaning up the Mini-OS
> namespace. This is necessary as we link Mini-OS with the application, rump
> kernel and NetBSD libc to get a full application stack.
>
> The changes as implemented in a semi-automated fashion for the Mini-OS used
> by rumprun-xen can be viewed in the (since merged) pull request at [3]:
>
> - All Mini-OS functions called by rumprun-xen are renamed to minios_* or
>   _minios_* for strictly internal functions, except those in the
>   blkfront_*, netfront_*, pcifront_* and xenbus_* driver namespaces.
> - In the case of drivers, eg. init|shutdown_blkfront are renamed to
>   blkfront_*.
> - All global variables are either manually made local, or placed under
>   the _minios_* namespace, with the exception of HYPERVISOR_shared_info,
>   and those variables under driver namespaces kept above.
> - All callers are updated to use the new names. Where it makes sense,
>   macros such as alloc_page are also renamed into the minios_ namespace.
>
> Questions:
>
> - Is there a general interest in upstreaming this work?
> - Upstream Mini-OS provides things we (rumpkernel.org) don't need (stub
>   "libc", ...). If we are to move to upstream then we will need to do a
>   more general cleanup of Mini-OS to make these pluggable. Again, is there
>   upstream interest in this work?
> - As there is no formal API for Mini-OS. My changes only address the
>   "public" functionality used by rumprun-xen. Other users mileage will
>   vary; who else should I coordinate with?
> - I have not namespaced macros such as local_irq_save(). Should this be
>   done? 
>   
>   Also, the driver namespaces have been preserved (since I was lazy), these
>   should probably be renamed under the minios namespace; it's plausible
>   some day someone will try to link an application with a function called
>   blkfront_init().
>
> All comments and review welcome.
>
> Martin

I think this is a very good idea, and I am completely in favour of it.

There are already-identified issues such as MiniOS leaking things like
ARRAY_SIZE() into linked namespaces, which I havn't yet had enough tuits
to fix.

I think splitting things like the stub libc away from the "MiniOS Xen
Framework" is also a good idea.  Ideally, the result of a "MiniOS Build"
would be a small set of .a's which can then be linked against some
normal C to make a minios guest.  (How feasible this is in reality
remains to be seen.)

>From a not-public-API point of view, all you have to worry about is that
the existing minios stuff in xen.git, including the stubdom stuff,
continues to work.  We have never made any guarantees to anyone using
minios out-of-tree.

~Andrew

Re: RFC: Cleaning up the Mini-OS namespace

[Antti Kantee]

On 04/12/14 14:40, Andrew Cooper wrote:
> There are already-identified issues such as MiniOS leaking things like
> ARRAY_SIZE() into linked namespaces, which I havn't yet had enough tuits
> to fix.
>
> I think splitting things like the stub libc away from the "MiniOS Xen
> Framework" is also a good idea.  Ideally, the result of a "MiniOS Build"
> would be a small set of .a's which can then be linked against some
> normal C to make a minios guest.  (How feasible this is in reality
> remains to be seen.)

I've become increasingly convinced that we (rump kernels) would like to 
use MiniOS as a small firmware library that just takes care of bootstrap 
and provides a high-level interface to the Xen hypervisor.

A componentized MiniOS is not critical from our perspective, as long as 
you can can compile things out.  We always want the minimal version, and 
can use build flags to produce it.  Notably, thanks to recent work by 
Martin, MiniOS is already compiled to a .o in the rumprun-xen 
repository, and then just linked into the final image.

What is critical for us, however, is reducing the amount of support 
routines needed by MiniOS.  Currently, the software stack in rumprun-xen 
is confusing because MiniOS partially uses libc which runs on top of the 
rump kernel which runs on top of MiniOS...  This confusion springs from 
MiniOS providing its own libc, and while it's ok for standalone MiniOS 
to use its own libc, we do not.  To make things as simple as possible, I 
don't want our [compiled] version of MiniOS to depend on anything from libc.

So, while I agree with everyone that merging is a good idea, the realist 
in me remains in doubt just like you do; is it feasible to both trim 
MiniOS to be minimal enough for our needs and keep it maximal enough for 
yours?  Or does that result in too many ifdef noodles?

>>From a not-public-API point of view, all you have to worry about is that
> the existing minios stuff in xen.git, including the stubdom stuff,
> continues to work.  We have never made any guarantees to anyone using
> minios out-of-tree.

I wonder if work is minimized if we attempt to merge before or after we 
(I?) take the carving knife for a second round in the rumprun-xen repo 
to minimize MiniOS to run only on top of itself.

   - antti

Re: RFC: Cleaning up the Mini-OS namespace

[Martin Lucina]

andrew.cooper3@citrix.com said:
> I think this is a very good idea, and I am completely in favour of it.
> 
> There are already-identified issues such as MiniOS leaking things like
> ARRAY_SIZE() into linked namespaces, which I havn't yet had enough tuits
> to fix.
> 
> I think splitting things like the stub libc away from the "MiniOS Xen
> Framework" is also a good idea.  Ideally, the result of a "MiniOS Build"
> would be a small set of .a's which can then be linked against some
> normal C to make a minios guest.  (How feasible this is in reality
> remains to be seen.)

The approach I used for rumprun-xen is to link all of MiniOS' object files
except the startfile into a final .o with "ld -r". This then allows me to
use "objcopy -w -GPREFIX..." to make all symbols in minios.o *except* those
starting with PREFIX local.

This has the advantage that I only had to rename symbols I really wanted to
keep global rather than going through all the MiniOS code adding "static"
in places where it was missing and sorting out the resulting
inter-dependencies.

> From a not-public-API point of view, all you have to worry about is that
> the existing minios stuff in xen.git, including the stubdom stuff,
> continues to work.  We have never made any guarantees to anyone using
> minios out-of-tree.

"Existing minios stuff" meaning the default build of extras/mini-os?

What's up with the -DHAVE_LIBC codepaths in mini-os? Who or what uses
these? Grepping around in stubdom/ doesn't come up with anything...

"Stubdom stuff" meaning the default build of stubdom/, plus the "make
c-stubdom" and "make caml-stubdom" examples documented in README?

Anything else? Sorry if this is obvious but I'm not that familiar with all
of xen.git.

Thanks,

Martin

Re: RFC: Cleaning up the Mini-OS namespace

[Martin Lucina]

pooka@iki.fi said:
> I wonder if work is minimized if we attempt to merge before or after
> we (I?) take the carving knife for a second round in the rumprun-xen
> repo to minimize MiniOS to run only on top of itself.

Before, I think. Minimizing our copy of Mini-OS duplicates what we would
need to do to the upstream copy.

I think the steps are roughly as follows:

  a) split the current rumprun-xen build out of the Mini-OS Makefile.
  b) replace our fork of Mini-OS with the vanilla upstream Mini-OS.
  c) re-apply my work and your work, while checking things keep working
  with upstream xen.git, until we get rumprun-xen working again.

c) will leave us with a set of patches to upstream.

Does this make sense? It's a fair amount of work but mostly retracing steps
we've already done.

It'd help if we had a full list of "what exactly needs to keep working
upstream", see my other reply to Andrew. Maybe also osstest building and
running Mini-OS related tests off our branch while we do the work? (Ian:
ping? Doable?)

Martin

Re: RFC: Cleaning up the Mini-OS namespace

[Antti Kantee]

On 05/12/14 18:31, Martin Lucina wrote:
> pooka@iki.fi said:
>> I wonder if work is minimized if we attempt to merge before or after
>> we (I?) take the carving knife for a second round in the rumprun-xen
>> repo to minimize MiniOS to run only on top of itself.
>
> Before, I think. Minimizing our copy of Mini-OS duplicates what we would
> need to do to the upstream copy.
>
> I think the steps are roughly as follows:
>
>    a) split the current rumprun-xen build out of the Mini-OS Makefile.
>    b) replace our fork of Mini-OS with the vanilla upstream Mini-OS.
>    c) re-apply my work and your work, while checking things keep working
>    with upstream xen.git, until we get rumprun-xen working again.
>
> c) will leave us with a set of patches to upstream.
>
> Does this make sense? It's a fair amount of work but mostly retracing steps
> we've already done.
>
> It'd help if we had a full list of "what exactly needs to keep working
> upstream", see my other reply to Andrew. Maybe also osstest building and
> running Mini-OS related tests off our branch while we do the work? (Ian:
> ping? Doable?)

It'd also help if we had a full list of "what exactly needs to be done 
to downstream".  That's why I'm not convinced by a list which ignores 
"d) perform the unknown steps to reach our goal" (which were painted in 
broad strokes in my previous mail).

Actually, it convinced me more of the opposite: wait before attempting 
full merge.  However, if someone's merge finger is twitching, a 
pseudo-merge with changes like the namespace protection and introducing 
a clean "libminios" split is nice.  But, again, is it more or less work 
doing that piecemeal or when all the puzzle pieces are known?

Re: RFC: Cleaning up the Mini-OS namespace

[Samuel Thibault]

Hello,

Martin Lucina, le Fri 05 Dec 2014 19:22:08 +0100, a écrit :
> What's up with the -DHAVE_LIBC codepaths in mini-os? Who or what uses
> these? Grepping around in stubdom/ doesn't come up with anything...

HAVE_LIBC gets defined by extra/mini-os/Config.mk when libc is y, and
libc is defined to $(stubdom), which is set to y from stubdom/Makefile

Basically everything in stubdom/ uses HAVE_LIBC.

I don't know where the qemu-stubdom part is currently stored, but it
uses it too.

Samuel

Re: RFC: Cleaning up the Mini-OS namespace

[Samuel Thibault]

Antti Kantee, le Thu 04 Dec 2014 22:52:05 +0000, a écrit :
> Currently, the software stack in rumprun-xen is confusing
> because MiniOS partially uses libc

Which part of libc? MiniOS itself is very independent of libc, it only
ships a couple of things. We can probably happily #ifdef them if needed.

Samuel

Re: RFC: Cleaning up the Mini-OS namespace

[Samuel Thibault]

Hello,

Martin Lucina, le Thu 04 Dec 2014 15:27:57 +0100, a écrit :
> - Is there a general interest in upstreaming this work?

I believe so.  That can only help people using minios.

> - All Mini-OS functions called by rumprun-xen are renamed to minios_* or
>   _minios_* for strictly internal functions, except those in the
>   blkfront_*, netfront_*, pcifront_* and xenbus_* driver namespaces.

That looks fine.

> - In the case of drivers, eg. init|shutdown_blkfront are renamed to
>   blkfront_*.

Right, good thing :)

> - All global variables are either manually made local, or placed under
>   the _minios_* namespace, with the exception of HYPERVISOR_shared_info,
>   and those variables under driver namespaces kept above.

That looks fine too.

> - All callers are updated to use the new names. Where it makes sense,
>   macros such as alloc_page are also renamed into the minios_ namespace.

The stubdom/ directory needs to be updated too, as well as the
qemu-stubdom directory (I unfortunately don't know where it is
maintained ATM).

> - Upstream Mini-OS provides things we (rumpkernel.org) don't need (stub
>   "libc", ...). If we are to move to upstream then we will need to do a
>   more general cleanup of Mini-OS to make these pluggable. Again, is there
>   upstream interest in this work?

Yes.  It should be feasible to make the tiny libc inside minios
pluggable, so you can plug things a better way for you. It's fine to
have #ifdefs in extra/mini-os/lib/

> - As there is no formal API for Mini-OS. My changes only address the
>   "public" functionality used by rumprun-xen. Other users mileage will
>   vary; who else should I coordinate with?

I guess anybody who will have seen this thread on this list :)

> - I have not namespaced macros such as local_irq_save(). Should this be
>   done? 

It should be fine not to do it.

>   Also, the driver namespaces have been preserved (since I was lazy), these
>   should probably be renamed under the minios namespace; it's plausible
>   some day someone will try to link an application with a function called
>   blkfront_init().

I'd say that while we are breaking the existing API, let's do it for
good and rename the driver namespace too.

Providing a header with backward-compatibility renaming macros for all
of this would however be useful, to be nicer to out-of-tree users.

Samuel

Re: RFC: Cleaning up the Mini-OS namespace

[Antti Kantee]

On 07/12/14 17:55, Samuel Thibault wrote:
> Antti Kantee, le Thu 04 Dec 2014 22:52:05 +0000, a écrit :
>> Currently, the software stack in rumprun-xen is confusing
>> because MiniOS partially uses libc
>
> Which part of libc? MiniOS itself is very independent of libc, it only
> ships a couple of things. We can probably happily #ifdef them if needed.

I said it unclearly.  I meant the use of

#include <lotofthings.h> (e.g. string.h, stdio.h, etc)

I'd rather see a well-defined interface there instead of ifdefs.

Re: RFC: Cleaning up the Mini-OS namespace

[Samuel Thibault]

Antti Kantee, le Sun 07 Dec 2014 18:03:33 +0000, a écrit :
> On 07/12/14 17:55, Samuel Thibault wrote:
> >Antti Kantee, le Thu 04 Dec 2014 22:52:05 +0000, a écrit :
> >>Currently, the software stack in rumprun-xen is confusing
> >>because MiniOS partially uses libc
> >
> >Which part of libc? MiniOS itself is very independent of libc, it only
> >ships a couple of things. We can probably happily #ifdef them if needed.
> 
> I said it unclearly.  I meant the use of
> 
> #include <lotofthings.h> (e.g. string.h, stdio.h, etc)

?

minios itself doesn't do this when it's not compiled with HAVE_LIBC.
Building with HAVE_LIBC is really not a requirement for using mini-os.
For rump projects, I would expect not using it, notably.

Samuel

Re: RFC: Cleaning up the Mini-OS namespace

[Antti Kantee]

On 07/12/14 18:09, Samuel Thibault wrote:
>> I said it unclearly.  I meant the use of
>>
>> #include <lotofthings.h> (e.g. string.h, stdio.h, etc)
>
> ?
>
> minios itself doesn't do this when it's not compiled with HAVE_LIBC.
> Building with HAVE_LIBC is really not a requirement for using mini-os.
> For rump projects, I would expect not using it, notably.

 From xen.git/extras/mini-os, irrespective of HAVE_LIBC:

netfront.c includes <errno.h>, kernel.c includes <fcntl.h>, pcifront.c 
includes <string.h> ...

Re: RFC: Cleaning up the Mini-OS namespace

[Samuel Thibault]

Antti Kantee, le Sun 07 Dec 2014 18:13:38 +0000, a écrit :
> On 07/12/14 18:09, Samuel Thibault wrote:
> >>I said it unclearly.  I meant the use of
> >>
> >>#include <lotofthings.h> (e.g. string.h, stdio.h, etc)
> >
> >?
> >
> >minios itself doesn't do this when it's not compiled with HAVE_LIBC.
> >Building with HAVE_LIBC is really not a requirement for using mini-os.
> >For rump projects, I would expect not using it, notably.
> 
> From xen.git/extras/mini-os, irrespective of HAVE_LIBC:
> 
> netfront.c includes <errno.h>,

That include should actually be #ifdef HAVE_LIBC

Samuel

Re: RFC: Cleaning up the Mini-OS namespace

[Ian Campbell]

On Sun, 2014-12-07 at 18:54 +0100, Samuel Thibault wrote:
> Hello,
> 
> Martin Lucina, le Fri 05 Dec 2014 19:22:08 +0100, a écrit :
> > What's up with the -DHAVE_LIBC codepaths in mini-os? Who or what uses
> > these? Grepping around in stubdom/ doesn't come up with anything...
> 
> HAVE_LIBC gets defined by extra/mini-os/Config.mk when libc is y, and
> libc is defined to $(stubdom), which is set to y from stubdom/Makefile
> 
> Basically everything in stubdom/ uses HAVE_LIBC.
> 
> I don't know where the qemu-stubdom part is currently stored, but it
> uses it too.

I think its in the regular qemu-xen-traditional git tree aka
git://xenbits.xen.org/qemu-xen-unstable.git.

There's isn't any stub qemu-xen-upstream yet.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel