Possibilities with Xen3 compared to IBM Power5 DLPAR

Possibilities with Xen3 compared to IBM Power5 DLPAR

[Julian Pawlowski]

Hey guys,

I'm just comparing the current development state of Xen with the established
technology from IBM on current Power5 series.
I could'nt find any information to these questions on the internet...

1. Does Xen support capacity entitlement for the CPUs, e.g.
   for processor pooling? (does Xen classify the CPU capacity
   into time slices? Would be the first step...)

2. If question 1 is "yes" or "will be in some day", is/will
   there be support for capped and uncapped domU's so that you can
   mix up things?

3. If uncapped use of a processor pool is possible, how about weighting
   different domU's? (e.g. dom0 should have uncapped mode with the
   highest weight)

4. Where can I find any further information about the dynamical
   change of active VCPUs and memory settings? E.g. What the domU
   guest system needs to have for supporting this.

5. Does Xen support SMT (simultanious multithreading) in the domU to
   also have logical processors in addition to virtual processors?
   Depends it on the physical processor so I would need a multi-core
   system which also supports SMT? But how would I control that I want to
   use SMT in some domU's and some not?

6. Can I really have a dedicated processor core assigned to dom0 or
   any domU and no other domU will use exactly that physical processing
   unit? (e.g. to make sure the L2 cache can be used more effective on
   multi-core CPUs)

I don't know which of these features have to be in hardware or can be
realized in software, so please don't blame me :-)


Maybe an interesting discussion this whole topic :-)

Looking forward to your answers!


Greetz
Julian

Re: Possibilities with Xen3 compared to IBM Power5 DLPAR

[Mark Williamson]

> I'm just comparing the current development state of Xen with the
> established technology from IBM on current Power5 series.
> I could'nt find any information to these questions on the internet...

Are we talking about Xen/PPC or Xen/x86 here?

> 1. Does Xen support capacity entitlement for the CPUs, e.g.
>    for processor pooling? (does Xen classify the CPU capacity
>    into time slices? Would be the first step...)
>
> 2. If question 1 is "yes" or "will be in some day", is/will
>    there be support for capped and uncapped domU's so that you can
>    mix up things?
>
> 3. If uncapped use of a processor pool is possible, how about weighting
>    different domU's? (e.g. dom0 should have uncapped mode with the
>    highest weight)

I think the credit scheduler supports at least some of this.

> 4. Where can I find any further information about the dynamical
>    change of active VCPUs and memory settings? E.g. What the domU
>    guest system needs to have for supporting this.

VCPU hotplug Just Works for XenLinux guests, I think.  Memory resizing also 
works for XenLinux but it's worth noting that if you shrink a domain too 
aggressively Linux gets confused about where its memory is going and things 
start to break!

> 5. Does Xen support SMT (simultanious multithreading) in the domU to
>    also have logical processors in addition to virtual processors?
>    Depends it on the physical processor so I would need a multi-core
>    system which also supports SMT? But how would I control that I want to
>    use SMT in some domU's and some not?

SMT is supported where the hardware has it (although it seems to be going out 
of fashion now on x86 boxes so maybe that's not such an issue at  the moment) 
DomU's VCPUs can be set to run on any thread context on the system.

I'm not sure to what extent guests are aware of the SMT so they can use it 
themselves; this would be particularly difficult for them to exploit given 
they could in theory be moved to other logical CPUs at any time.

Running dom0 in a dedicated thread context is useful though.

> 6. Can I really have a dedicated processor core assigned to dom0 or
>    any domU and no other domU will use exactly that physical processing
>    unit? (e.g. to make sure the L2 cache can be used more effective on
>    multi-core CPUs)

Yep.

Cheers,
Mark

> I don't know which of these features have to be in hardware or can be
> realized in software, so please don't blame me :-)
>
>
> Maybe an interesting discussion this whole topic :-)
>
> Looking forward to your answers!
>
>
> Greetz
> Julian
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: Possibilities with Xen3 compared to IBM Power5 DLPAR

[Julian Pawlowski]

Hey Mark,

Thanks for your answers!

On 15.03.2007 4:18 "Mark Williamson" <mark.williamson@cl.cam.ac.uk> wrote:

> Are we talking about Xen/PPC or Xen/x86 here?

Mainly I'm talking about the x86 platform - but the feature list should
apply to all platforms if possible :-)
Although the Power5 platform could offer more direct support in hardware...
I'm not sure if it where possible to use the hardware part of IBM's
hypervisor in xen.

> I think the credit scheduler supports at least some of this.

What exactly does that mean? :-)
Are you talking about the basic support so that one could step into these
features some day? I'm asking because I could not find any variables for the
config files to set an entitlement or so. At the moment I think the smallest
unit to assign processing power is per physical processor. But how can I
assign this more granular, e.g. only a half physical processor (0.5
entitlement) to 1 or x virtual processors? I'm also missing to have a
variable to weight important domU's with higher priority than some small and
unimportant domU's. At the moment each dom is equal to the others which
maybe is not what professionals would like to have...

> VCPU hotplug Just Works for XenLinux guests, I think.  Memory resizing also
> works for XenLinux but it's worth noting that if you shrink a domain too
> aggressively Linux gets confused about where its memory is going and things
> start to break!

Quite clear that the domU has to be xen-aware in order to re-read the memory
allocaion table outside the domU space. And I was not really thinking a
fully virtualized guest esp. like Windows could ever do so :-)

> SMT is supported where the hardware has it (although it seems to be going out
> of fashion now on x86 boxes so maybe that's not such an issue at  the moment)

Unfortunately yes, you only have this feature in some of Intels Extreme
versions
 
> DomU's VCPUs can be set to run on any thread context on the system.

Yes sure but that's not my point. As you describe a domU would use each
physical CPU it has access to but inside the domU each VCPU would look the
same, the guest cannot make any difference between logical CPUs and virtual
CPUs - it only has virtual CPUs then. But on the IBM platform the AIX guest
is still aware that there are two logical CPUs linked with one virtual CPU.
This is part of the hypervisor I think so the question is if Xen could also
virtualize logical CPUs. IBM has no SMT on the level of the hypervisor, here
they only have full physical CPUs. First in the logical partition you have
the ability to enable SMT in addition to the assigned VCPUs or to disable
it.

> I'm not sure to what extent guests are aware of the SMT so they can use it
> themselves; this would be particularly difficult for them to exploit given
> they could in theory be moved to other logical CPUs at any time.

The guest systems kernel does not really make a difference between logical
and virtual processors I think, it's only interest is to have two units to
send it's arithmetic problems to. But the whole guest operating system needs
to have the ability to enable or disable SMT. On AIX it is only one command
line but I'm sure it could be a point in the BIOS settings for Xen virtual
machines. Although I don't even know if I can edit the BIOS settings
directly when the vmachine starts up, never tried this :-) (with VMware you
can do this).

> Running dom0 in a dedicated thread context is useful though.

Absolutely. But I'm not sure if dom0_cpus=1 is everything I need because I
have no influence on Xen to only use one special core, in theory it can be
always another core in each time slice. Does Xen have the intelligence here
not to change the core? What about the guests, should I set cpus=1-7 in
their config files (with a total of 8 cores)? Would the hypervisor really
only use core 0 in that case?

Sorry for that annoying questions but I'd like to know it very exactly :-)


Cheers,
Julian

Re: Possibilities with Xen3 compared to IBM Power5 DLPAR

[Julian Pawlowski]

Just found some information about Xens scheduling procedures in the wiki:

http://wiki.xensource.com/xenwiki/Scheduling

Maybe this could be a part of what I'm looking for? Unfortunately the
informations there are not too detailed so is there another documentation so
that I can transfer my knowledge of the Power5 hypervisor with HMC to the
terms and definitions used in the Xen environment?


Cheers,
Julian

Re: Possibilities with Xen3 compared to IBM Power5 DLPAR

[Ewan Mellor]

On Thu, Mar 15, 2007 at 02:19:52PM +0100, Julian Pawlowski wrote:

> Just found some information about Xens scheduling procedures in the wiki:
> 
> http://wiki.xensource.com/xenwiki/Scheduling
> 
> Maybe this could be a part of what I'm looking for? Unfortunately the
> informations there are not too detailed so is there another documentation so
> that I can transfer my knowledge of the Power5 hypervisor with HMC to the
> terms and definitions used in the Xen environment?

That looks seriously out of date.  This is a lot more recent:

http://wiki.xensource.com/xenwiki/CreditScheduler

HTH,

Ewan.

Re: Possibilities with Xen3 compared to IBM Power5 DLPAR

[Julian Pawlowski]

Hello Ewan,

> http://wiki.xensource.com/xenwiki/CreditScheduler

thanks a lot, sounds great :-)

Is there a way to have those settings directly on the domU configuration
file to have these settings right after creating the VM without just a
second command line?


Best wishes,
Julian

Re: Possibilities with Xen3 compared to IBM Power5 DLPAR

[Ewan Mellor]

On Thu, Mar 15, 2007 at 07:27:17PM +0100, Julian Pawlowski wrote:

> Hello Ewan,
> 
> > http://wiki.xensource.com/xenwiki/CreditScheduler
> 
> thanks a lot, sounds great :-)
> 
> Is there a way to have those settings directly on the domU configuration
> file to have these settings right after creating the VM without just a
> second command line?

I think that cpu_weight and cpu_cap are the options you want.

Ewan.

Re: Possibilities with Xen3 compared to IBM Power5 DLPAR

[Julian Pawlowski]

On 15.03.2007 19:32 "Ewan Mellor" <ewan@xensource.com> wrote:

> I think that cpu_weight and cpu_cap are the options you want.

Yes, thank you.

I'm also looking for a (mostly) complete list of all possible options within
the configuration files (dom0 and domU) but unfortunately all examples on
the net seem to be more or less out of date or incomplete.
I couldn't find any informations on the Xen pages eather (they're not very
clearly arranged).
Do you have a clue?


Julian

Re: Possibilities with Xen3 compared to IBM Power5 DLPAR

[Ewan Mellor]

On Thu, Mar 15, 2007 at 07:41:09PM +0100, Julian Pawlowski wrote:

> On 15.03.2007 19:32 "Ewan Mellor" <ewan@xensource.com> wrote:
> 
> > I think that cpu_weight and cpu_cap are the options you want.
> 
> Yes, thank you.
> 
> I'm also looking for a (mostly) complete list of all possible options within
> the configuration files (dom0 and domU) but unfortunately all examples on
> the net seem to be more or less out of date or incomplete.
> I couldn't find any informations on the Xen pages eather (they're not very
> clearly arranged).
> Do you have a clue?

xm create --help_config is probably your best bet.  If there are things in
here that aren't in the user guide, we always appreciate patches to the
documentation.

Thanks,

Ewan.

Re: Possibilities with Xen3 compared to IBM Power5 DLPAR

[Julian Pawlowski]

> xm create --help_config is probably your best bet.  If there are things in
> here that aren't in the user guide, we always appreciate patches to the
> documentation.

Wahoo thats great, thnx :-)


Julian

Re: Possibilities with Xen3 compared to IBM Power5 DLPAR

[Mark Williamson]

> On 15.03.2007 4:18 "Mark Williamson" <mark.williamson@cl.cam.ac.uk> wrote:
> > Are we talking about Xen/PPC or Xen/x86 here?
>
> Mainly I'm talking about the x86 platform - but the feature list should
> apply to all platforms if possible :-)
> Although the Power5 platform could offer more direct support in hardware...
> I'm not sure if it where possible to use the hardware part of IBM's
> hypervisor in xen.

Well, the IBM guys are working on the PPC port but that's (as far as I know) 
currently directed towards PPC970 etc, rather than towards the higher-end 
POWER processors.  But who knows what we'll see!

> > I think the credit scheduler supports at least some of this.
>
> What exactly does that mean? :-)
> Are you talking about the basic support so that one could step into these
> features some day? I'm asking because I could not find any variables for
> the config files to set an entitlement or so. At the moment I think the
> smallest unit to assign processing power is per physical processor. But how
> can I assign this more granular, e.g. only a half physical processor (0.5
> entitlement) to 1 or x virtual processors? I'm also missing to have a
> variable to weight important domU's with higher priority than some small
> and unimportant domU's. At the moment each dom is equal to the others which
> maybe is not what professionals would like to have...

Hopefully the other information Ewan found for you has answered these 
questions.

My understanding of the credit scheduler is that it can indeed give out "n 
units worth" to a domain, added up across all its processors.  So for 
instance a 2 VCPU and a 1 VCPU domain with "1 unit's" entitlement will get 
the same total CPU bandwidth, even though one has two VCPUs being scheduled.

Usage can be capped so that a domain doesn't get more capacity than its paid 
for or slack time can (somehow) be distributed in the system.

That's how I *think* it works, but wasn't 100% which is why I glossed over 
it ;-)  Sounds roughly like what you wanted?

> > VCPU hotplug Just Works for XenLinux guests, I think.  Memory resizing
> > also works for XenLinux but it's worth noting that if you shrink a domain
> > too aggressively Linux gets confused about where its memory is going and
> > things start to break!
>
> Quite clear that the domU has to be xen-aware in order to re-read the
> memory allocaion table outside the domU space. And I was not really
> thinking a fully virtualized guest esp. like Windows could ever do so :-)

Well...  if the FV guest supported hotplug-capable hardware then maybe it 
could.  But in any case, you could in principle (it's not been done yet, and 
there were some problems to be solved) run the balloon driver in an HVM guest 
to enable it to change its memory footprint like a PV domain can.

I *suppose* one could do a similar thing with a driver that offlined CPUs in 
someway to support effective hotplugging.  This could probably also be done 
in an HVM guest with the addition of a few Xen-aware drivers.

> > DomU's VCPUs can be set to run on any thread context on the system.
>
> Yes sure but that's not my point. As you describe a domU would use each
> physical CPU it has access to but inside the domU each VCPU would look the
> same, the guest cannot make any difference between logical CPUs and virtual
> CPUs - it only has virtual CPUs then. But on the IBM platform the AIX guest
> is still aware that there are two logical CPUs linked with one virtual CPU.
> This is part of the hypervisor I think so the question is if Xen could also
> virtualize logical CPUs. IBM has no SMT on the level of the hypervisor,
> here they only have full physical CPUs. First in the logical partition you
> have the ability to enable SMT in addition to the assigned VCPUs or to
> disable it.

Ah.  Well that seems like a sensible way of working.  You could probably pin a 
domains VCPUs to a particular core if you wanted to achieve this sort of 
effect but I still don't know of a convenient way the domain can discover 
that they're siblings (and when they've changed relationships).  Maybe 
messing with CPUid can provide some hints.  There may be some topology stuff 
in there I don't know about, though, so I may be wrong.

Since Xen currently potentially schedules each VCPU on any logical thread in 
the system, there's probably not a very straightforward way for guests to 
make use of hyperthreading as the sibling relationships are potentially 
changing all the time.  In that case, maybe the IBM way is better, but if SMT 
is going to disappear on x86 machines then maybe it's not so important for 
now?  Just thinking out loud...

> > I'm not sure to what extent guests are aware of the SMT so they can use
> > it themselves; this would be particularly difficult for them to exploit
> > given they could in theory be moved to other logical CPUs at any time.
>
> The guest systems kernel does not really make a difference between logical
> and virtual processors I think, it's only interest is to have two units to
> send it's arithmetic problems to. But the whole guest operating system
> needs to have the ability to enable or disable SMT. On AIX it is only one
> command line but I'm sure it could be a point in the BIOS settings for Xen
> virtual machines. Although I don't even know if I can edit the BIOS
> settings directly when the vmachine starts up, never tried this :-) (with
> VMware you can do this).

I don't *think* you can do that with Bochs, but I don't know.

You couldn't disable hyperthreading in the guest as stands anyhow, but you 
*could* configure the guest such that hyperthreading is not used (by pinning 
CPUs) or you could switch it off for the whole host (in the host BIOS or noht 
on the Xen commandline should work).

> > Running dom0 in a dedicated thread context is useful though.
>
> Absolutely. But I'm not sure if dom0_cpus=1 is everything I need because I
> have no influence on Xen to only use one special core, in theory it can be
> always another core in each time slice. Does Xen have the intelligence here
> not to change the core? What about the guests, should I set cpus=1-7 in
> their config files (with a total of 8 cores)? Would the hypervisor really
> only use core 0 in that case?

There's a CPU-pinning syntax for specifying what host CPUs domains can / can't 
run on.  It's probably documented ... somewhere :-)

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!