Multiple priviliged domains

Multiple priviliged domains

[John L Griffin]

During my test machine's idle time I'm running the Linux Test Project on 
multiple unprivileged Xen domains (i.e., more domains than the number of 
real processors on the machine), for general stress testing & to see 
whether any problems crop up.

While running this workload I noticed occasional horrendously slow 
interactive performance in Domain-0 (in which I was not running the LTP). 
Although I haven't yet looked in depth at the source of the slowdown, my 
hypothesis is that Domain-0 blocks handling the high I/O load generated by 
the unprivileged domains, leading to slow keyboard responses.  This brings 
up several questions:

1. What is the model for allocating processor time to Domain-0?  Based on 
my read of the Xen docs to date, I would expect it to [at least be 
intended to] have an unbounded priority share of the total processing 
resources, with some attempt at allocating unprivileged-domain-specific 
processing (e.g., handling I/O or memory allocation requests) to the 
requesting unprivileged domain.  Along these lines, should there be a 
parameterizable configuration file for Domain-0?

2. Have there been discussions about allowing multiple simultaneous 
privileged domains, among which the physical resources are split?  Or 
perhaps "semiprivileged" domains -- for example, a domain that handles all 
the I/O requests to a particular storage device, or alternatively handles 
all the I/O requests for a particular class/subset of unprivileged 
domains?  I envision a desire for a master control partition (with 
priority resource allocation) that forms the root of a hierarchical domain 
structure, under which one or more I/O partitions execute.  (I recall 
reading about this sort of design in one of the older VMM papers, or 
perhaps a recent Denali paper?)

3. I don't seem to be able to create more than 8 VBDs. [I am using 
xen-2.0.1-src.tgz.]  While trying to start 5 domains, each of which had 
two "scsi" disks (/ and swap), I discovered that the 5th domain wouldn't 
start until I removed the swap disks from both the 4th and 5th domains' 
configuration files -- i.e., not exceeding 8 VBDs total.  A cursory search 
through the code didn't reveal any relevant #define's, and I didn't see 
anything about this in the Xen docs, so before I look further I thought 
I'd ask the list if this is a known limit.  (If so, I am surprised the VBD 
structures aren't allocated dynamically to prevent this?)  A search of the 
mailing list revealed a message from Ian on 2004-03-10 stating "Mark wrote 
a pretty good readme on VBDs and put it in the tree", but I couldn't find 
the readme; is it still part of the tree?

4. On a loosely related note, what regression tests are used by the Xen 
developers?  (I.e., what should be run before generating patches?)

-- 
Dr. John Linwood Griffin
Research Staff Member, Secure Systems Department
IBM T.J. Watson Research Center, Hawthorne, New York, USA
JLG at us.ibm.com, http://www.research.ibm.com/people/j/jlg/



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/

Re: Multiple priviliged domains

[Ian Pratt]

> 1. What is the model for allocating processor time to Domain-0?  Based on 
> my read of the Xen docs to date, I would expect it to [at least be 
> intended to] have an unbounded priority share of the total processing 
> resources, with some attempt at allocating unprivileged-domain-specific 
> processing (e.g., handling I/O or memory allocation requests) to the 
> requesting unprivileged domain.  Along these lines, should there be a 
> parameterizable configuration file for Domain-0?

At the current time, there's no attempt to 'recharge' CPU time
domain 0 spends doing I/O on behalf of other domains back to the
domains.  It would actually be pretty tricky to do this
accounting within Linux.  Fortunately, if you've got decent
DMA-capable hardware with interrupt coalescing then domain0
typically doesn't use too much CPU anyhow.  Obviously if you're
using loop files for VBDs or other higher-level processing the
situation might be different.
 
> 2. Have there been discussions about allowing multiple simultaneous 
> privileged domains, among which the physical resources are split?  Or 
> perhaps "semiprivileged" domains -- for example, a domain that handles all 
> the I/O requests to a particular storage device, or alternatively handles 
> all the I/O requests for a particular class/subset of unprivileged 
> domains?  I envision a desire for a master control partition (with 
> priority resource allocation) that forms the root of a hierarchical domain 
> structure, under which one or more I/O partitions execute.  (I recall 
> reading about this sort of design in one of the older VMM papers, or 
> perhaps a recent Denali paper?)

It's quite possible to have multiple privileged domains owning
different bits of hardware -- see the paper on I/O on the
website. This can even be used to enable the system to recover
from crashes due to buggy device drivers without a reboot.

The control tools haven't always made this easy to set up as its
not something that many people do.

> 3. I don't seem to be able to create more than 8 VBDs. [I am using 
> xen-2.0.1-src.tgz.]  While trying to start 5 domains, each of which had 
> two "scsi" disks (/ and swap), I discovered that the 5th domain wouldn't 
> start until I removed the swap disks from both the 4th and 5th domains' 
> configuration files -- i.e., not exceeding 8 VBDs total.  A cursory search 
> through the code didn't reveal any relevant #define's, and I didn't see 
> anything about this in the Xen docs, so before I look further I thought 
> I'd ask the list if this is a known limit.  (If so, I am surprised the VBD 
> structures aren't allocated dynamically to prevent this?)  A search of the 
> mailing list revealed a message from Ian on 2004-03-10 stating "Mark wrote 
> a pretty good readme on VBDs and put it in the tree", but I couldn't find 
> the readme; is it still part of the tree?

Are you using loop files? If so, Linux has a statically compiled
allocation, set to 8 by default. Fixing this and submitting the
patch to lkml would a good thing...

I don't believe the VBD code has any such limitations. (We tend
to use LVM rather than loop files).

> 4. On a loosely related note, what regression tests are used by the Xen 
> developers?  (I.e., what should be run before generating patches?)

ltp, ttcp and lmbench are useful. We run osdb/postgresql,
postmark and specweb/apache from time to time too. 

We have a nightly regression test procedure, but its rather old
now, being based on RH7.2. As such, it misses some of the
excitement caused by NPTL etc. It would be very useful to get an
automated regression test setup on something more modern, with as
many apps as possible.

Ian

> -- 
> Dr. John Linwood Griffin
> Research Staff Member, Secure Systems Department
> IBM T.J. Watson Research Center, Hawthorne, New York, USA
> JLG at us.ibm.com, http://www.research.ibm.com/people/j/jlg/
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now. 
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/

Re: Multiple priviliged domains

[Jacob Gorm Hansen]

Mark Williamson wrote:
l.
> 
> Obviously, dom0 will need control of some physical devices  inorder to have 
> something to boot from (unless  ofcourse you use a *really big* initrd :-) ).
> 

hi,

In my setup, you do not need a very large initrd to do this. 
Essentially, the only thing running in dom0 is a simple ICMP/ping 
server. You send a specially formatted and signed ping payload to dom0, 
and in response to that it fires up a new VM with a UIP TCP/IP stack 
inside.  You then connect to that TCP/IP stack, and self-migrate or 
(eventually, not implemented yet) bootstrap directly into the 
unprivileged guest. Dom0 does not need to have a TCP/IP stack, and at 
present the ping server main loop is only 50 lines of code.

With pre-NGIO version of Xen 1.3 that I am currently using, all you need 
in your dom0-initrd is the ping server and an ELF image from which to 
create UIP guest VMs. I am hoping that in the future the parts of xend 
that have to do with setting up network interfaces to domains will split 
out into a small C executable, as that would allow me to provide similar 
functionality for Xen 2.0 or 2.1.

Source and binaries are available from
http://www.diku.dk/~jacobg/self-migration/

Jacob

PS: I do apologize to the long-time subscribers who are fed with me 
spamming this list with ads for self-migration stuff every time a new 
user who could find this relevant shows up on the list. Perhaps I can 
convince the Xen webpage maintainers to link to my stuff as a related 
project?


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/

Re: Multiple priviliged domains

[Jan Kundrát]

Ian Pratt wrote:
>>3. I don't seem to be able to create more than 8 VBDs. [I am using 
>>xen-2.0.1-src.tgz.]  While trying to start 5 domains, each of which had 
>>two "scsi" disks (/ and swap), I discovered that the 5th domain wouldn't 
>>start until I removed the swap disks from both the 4th and 5th domains' 
>>configuration files -- i.e., not exceeding 8 VBDs total.  A cursory search 
>>through the code didn't reveal any relevant #define's, and I didn't see 
>>anything about this in the Xen docs, so before I look further I thought 
>>I'd ask the list if this is a known limit.  (If so, I am surprised the VBD 
>>structures aren't allocated dynamically to prevent this?)  A search of the 
>>mailing list revealed a message from Ian on 2004-03-10 stating "Mark wrote 
>>a pretty good readme on VBDs and put it in the tree", but I couldn't find 
>>the readme; is it still part of the tree?
> 
> 
> Are you using loop files? If so, Linux has a statically compiled
> allocation, set to 8 by default. Fixing this and submitting the
> patch to lkml would a good thing...
> 
> I don't believe the VBD code has any such limitations. (We tend
> to use LVM rather than loop files).

If you use files as backend for VBDs in domain0, you use loop device. If 
the CONFIG_BLK_DEV_LOOP in dom0 kernel is compiled as module, you can 
use max_loop parameter while doing modprobe/insmod. And if it's compiled 
directly into the kernel, use max_loop=n boot option.

jkt


-- 
cd /local/pub && more beer > /dev/mouth



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/

Re: Multiple priviliged domains

[Jacob Gorm Hansen]

Neugebauer, Rolf wrote:

> Yes, but in 1.3 device drivers are still in Xen. In 2.0 device drivers
> are run in a guest OS (which was what the original poster was looking
> for). In order to boot dom0 in this model without physical device
> drivers it would need a *really big* initrd (as Mark said) because it
> would need to create other VMs from it to provide dom0 with virtual
> devices. I'm not sure at all if xend supports this sort of setting.

Yep, it seems I misunderstood the original poster's intention. I guess 
that in an ideal world every device would provide a Xen VM driver in its 
firmware, similar to how Open Firmware provides Forth drivers, if I 
understand that part correctly. But then there is the issue of how to 
upgrade drivers etc.

Btw. I remember some discussion about wrapping NDIS drivers as Xen 
driver domains, did anyone every try doing that?

Jacob


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/

Re: Multiple priviliged domains

[Bin Ren]

On Sun, 19 Dec 2004 00:52:54 -0500, John L Griffin <jlg@us.ibm.com> wrote:

> 1. What is the model for allocating processor time to Domain-0?  Based on
> my read of the Xen docs to date, I would expect it to [at least be
> intended to] have an unbounded priority share of the total processing
> resources, with some attempt at allocating unprivileged-domain-specific
> processing (e.g., handling I/O or memory allocation requests) to the
> requesting unprivileged domain.  Along these lines, should there be a
> parameterizable configuration file for Domain-0?

Current schedulers in Xen don't treat Dom0 differently. In current
model, Dom0 plays two major roles: provides a control console for
user-interaction; provides I/O handling for other domains. These two
roles have different or even conflicting scheduling requirement: the
former is response time; the latter is I/O throughput and latency.
Ideally, we should put the two parts in two privileged domains and let
the scheduler treat them differently. This leads to your next question
about multiple privileged domains.

> 2. Have there been discussions about allowing multiple simultaneous
> privileged domains, among which the physical resources are split?  Or
> perhaps "semiprivileged" domains -- for example, a domain that handles all
> the I/O requests to a particular storage device, or alternatively handles
> all the I/O requests for a particular class/subset of unprivileged
> domains?  I envision a desire for a master control partition (with
> priority resource allocation) that forms the root of a hierarchical domain
> structure, under which one or more I/O partitions execute.  (I recall
> reading about this sort of design in one of the older VMM papers, or
> perhaps a recent Denali paper?)

I agree that multiple privleged domains make resource management both
conceptually and structurally clearer. So far, very vey few people
actually do that, for several reasons:
(1) It requires non-trivial changes to Xend, which people either don't
bother to hack or can't.
(2) With multiple privileged domains, the domain context switch
overhead can seriously decrease performance. It makes little sense on
current uni- or dual-processors.
(3) Managing multiple privilged domains is surely less easy than
managing one big Dom0.

Surprisingly enough, it's my phd topic at cambridge to do all above.
I've offloaded the entire tcp/ip stack to a privilged domain that's
shared by all others. My next step is to introduce the Makefile
changes, Xend changes to support multiple privilged domains (i.e. put
Xend, NIC device drivers, disk device drivers, tcp/ip stack in
different privileged domains). Then I'll investigate how to schedule
all the domains on multi-core chips. Mark Williamson has experiences
in putting NIC and disk device drivers into privliged domains. I'll
consult him when questions arise.

- Bin


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/

Re: Multiple priviliged domains

[Mark Williamson]

> 2. Have there been discussions about allowing multiple simultaneous
> privileged domains, among which the physical resources are split?  Or
> perhaps "semiprivileged" domains -- for example, a domain that handles all
> the I/O requests to a particular storage device, or alternatively handles
> all the I/O requests for a particular class/subset of unprivileged
> domains?

This has been done - it's possible to give domans direct control of aa 
particular device, whilst still limiting their ability to interact with other 
devices in the system.

It should also be possible to have unprivileged domains use different backend 
domains for different devices.  I think there's some support for this in 
Xend.

> I envision a desire for a master control partition (with 
> priority resource allocation) that forms the root of a hierarchical domain
> structure, under which one or more I/O partitions execute.  (I recall
> reading about this sort of design in one of the older VMM papers, or
> perhaps a recent Denali paper?)

You can do that sort of thing if you want, although it's not a particularly 
common setup so the control tools don't support it that well.

Obviously, dom0 will need control of some physical devices  inorder to have 
something to boot from (unless  ofcourse you use a *really big* initrd :-) ).

We're planning to maintain the flexibility to configure driver domains and 
possibly extend it some more - it's rather useful for various things we'll be 
looking at in the future.

HTH,
Mark


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/

RE: Multiple priviliged domains

[Neugebauer, Rolf]

> -----Original Message-----
> From: xen-devel-admin@lists.sourceforge.net [mailto:xen-devel-
> admin@lists.sourceforge.net] On Behalf Of Jacob Gorm Hansen
> Sent: 19 December 2004 11:52
> To: Mark Williamson
> Cc: xen-devel@lists.sourceforge.net; John L Griffin
> Subject: Re: [Xen-devel] Multiple priviliged domains
> 
> Mark Williamson wrote:
> l.
> >
> > Obviously, dom0 will need control of some physical devices  inorder
to
> have
> > something to boot from (unless  ofcourse you use a *really big*
initrd
> :-) ).
> >
> 
> hi,
> 
> In my setup, you do not need a very large initrd to do this.
> Essentially, the only thing running in dom0 is a simple ICMP/ping
> server. You send a specially formatted and signed ping payload to
dom0,
> and in response to that it fires up a new VM with a UIP TCP/IP stack
> inside.  You then connect to that TCP/IP stack, and self-migrate or
> (eventually, not implemented yet) bootstrap directly into the
> unprivileged guest. Dom0 does not need to have a TCP/IP stack, and at
> present the ping server main loop is only 50 lines of code.
> 
> With pre-NGIO version of Xen 1.3 that I am currently using, all you
need
> in your dom0-initrd is the ping server and an ELF image from which to
> create UIP guest VMs.

Yes, but in 1.3 device drivers are still in Xen. In 2.0 device drivers
are run in a guest OS (which was what the original poster was looking
for). In order to boot dom0 in this model without physical device
drivers it would need a *really big* initrd (as Mark said) because it
would need to create other VMs from it to provide dom0 with virtual
devices. I'm not sure at all if xend supports this sort of setting.

If you have a decent server class machine with multiple nics and scsi
controllers you could use some of these to get dom0 booting and then
fire off other VMs to provide isolated device services to other VMs.
That "should" be supported by xend...

rolf




> I am hoping that in the future the parts of xend
> that have to do with setting up network interfaces to domains will
split
> out into a small C executable, as that would allow me to provide
similar
> functionality for Xen 2.0 or 2.1.
> 
> Source and binaries are available from
> http://www.diku.dk/~jacobg/self-migration/
> 
> Jacob
> 
> PS: I do apologize to the long-time subscribers who are fed with me
> spamming this list with ads for self-migration stuff every time a new
> user who could find this relevant shows up on the list. Perhaps I can
> convince the Xen webpage maintainers to link to my stuff as a related
> project?
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real
users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/

Re: Multiple priviliged domains

[Ian Pratt]

> Yes, but in 1.3 device drivers are still in Xen. In 2.0 device drivers
> are run in a guest OS (which was what the original poster was looking
> for). In order to boot dom0 in this model without physical device
> drivers it would need a *really big* initrd (as Mark said) because it
> would need to create other VMs from it to provide dom0 with virtual
> devices. I'm not sure at all if xend supports this sort of setting.
> 
> If you have a decent server class machine with multiple nics and scsi
> controllers you could use some of these to get dom0 booting and then
> fire off other VMs to provide isolated device services to other VMs.
> That "should" be supported by xend...

One way we could do this would be to have Grub package up all the
driver domains as modules loaded ahead of dom0. Each driver
domain would have a very simple initrd containing one file that
enables it listen on the control interface and receive
configuration instructions.

Dom0 would then issue configuration instructions to the driver
domains and then bind its virtual drivers. This is another
argument for splitting this kind of functionality out of xend
into a set of standalone libraries with a daemon to give a
network facing interface to it.


Ian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/