RE: [Xen-users] Xen 3.0 Dom0 Smp enable?

RE: [Xen-users] Xen 3.0 Dom0 Smp enable?

[Ian Pratt]

[I've moved this to xen-devel from xen-users]

> > If you create a pincpu mask for the over domains/VCPUs that doesn't 
> > include the CPU in question, they won't use it. I guess a shorthand 
> > command for this would be useful.
> 
> Something like:
> 
> xm dedicate DOM CPU

It should probably be called cpu-dedicate or such like.

> > Ryan, if so, please can you dust off and resend. As I 
> recall, the key 
> > fix it needed was that it should be xend (not xm) that does the 
> > translation from the dot'ed form into CPU numbers.
> 
> Certainly it will be easier to do so in xend, but I'm more 
> concerned with the hypervisor being responsible responsible 
> for choosing the vcpu to cpu mapping.  I'd like to see dom 
> creation support the passing of a cpumap and have the 
> hypervisor cycle through the physical cpus marked therein.  
> This would remove the logic of mapping vcpus to cpus from the 
> hypervisor and let the dom creation tools build whatever 
> algorithm for distributing vcpus across cpus as it sees fit.

Sure - feel free to remove the logic from Xen. It's simply there because
until recently xend didn't know about number of hyperthreads, cores,
sockets etc.

The default should be to give Xen considerable freedom in how it
schedules VCPUs to CPUs: Stephan's load balancer is almost ready for
posting.

We should have some higher-level allocation control that can be tweaked
in xend e,g, "don't use hyperthreading", "allow only privileged domains
to use the 1st hyperthread on each CPU", "allow any logical CPU to be
used by any VCPU". [All additionally subject to cpu-dedicate
restrictions]

> If you are interested in this it means changing the hypercall 
> interface and so it should be done for 3.0, however I don't 
> want to push for significant changes so close to the testing 
> freeze as I want to help close features down rather than 
> create new ones.

Does it actually need a change to the hypercall interface? The domain
builder can just issue a pin for each VCPU. It might be cleaner to
remove the orignal CPU field, but that's such a small change we should
just do it anyway.

We could remove the current default allocation code from Xen.

Ian

Re: RE: [Xen-users] Xen 3.0 Dom0 Smp enable?

[Stephan Diestelhorst]

>>Certainly it will be easier to do so in xend, but I'm more

>>concerned with the hypervisor being responsible responsible 
>>for choosing the vcpu to cpu mapping.  I'd like to see dom 
>>creation support the passing of a cpumap and have the 
>>hypervisor cycle through the physical cpus marked therein.  
>>This would remove the logic of mapping vcpus to cpus from the 
>>hypervisor and let the dom creation tools build whatever 
>>algorithm for distributing vcpus across cpus as it sees fit.
>>    
>>
>
>Sure - feel free to remove the logic from Xen. It's simply there because
>until recently xend didn't know about number of hyperthreads, cores,
>sockets etc.
>
>  
>
I always thought that the so called logic in createdom is not very
clever and really doesn't belong there, although it has improved! It is
basically just something to get the domain going. Finer control is based
on xm pincpu. It would be more reasonable to give Xen a cpumask, already
when creating the domain. Rather than creating it and then setting the
cpumask immediatelly with xm pincpu...

>The default should be to give Xen considerable freedom in how it
>schedules VCPUs to CPUs: Stephan's load balancer is almost ready for
>posting.
>
>We should have some higher-level allocation control that can be tweaked
>in xend e,g, "don't use hyperthreading", "allow only privileged domains
>to use the 1st hyperthread on each CPU", "allow any logical CPU to be
>used by any VCPU". [All additionally subject to cpu-dedicate
>restrictions]
>  
>
Which could IMHO all expressed at a lower level (i.e. towards Xen) with
the cpumask.

>Does it actually need a change to the hypercall interface? The domain
>builder can just issue a pin for each VCPU. It might be cleaner to
>remove the orignal CPU field, but that's such a small change we should
>just do it anyway.
>  
>
I think the passing of the cpumask needs a little change in the code.

>We could remove the current default allocation code from Xen.
>  
>
Absolutely true.

Stephan

Re: RE: [Xen-users] Xen 3.0 Dom0 Smp enable?

[Ryan Harper]

* Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2005-08-08 12:05]:
> > > If you create a pincpu mask for the over domains/VCPUs that doesn't 
> > > include the CPU in question, they won't use it. I guess a shorthand 
> > > command for this would be useful.
> > 
> > Something like:
> > 
> > xm dedicate DOM CPU
> 
> It should probably be called cpu-dedicate or such like.

Sure.  Always open to names.

> 
> > > Ryan, if so, please can you dust off and resend. As I 
> > recall, the key 
> > > fix it needed was that it should be xend (not xm) that does the 
> > > translation from the dot'ed form into CPU numbers.
> > 
> > Certainly it will be easier to do so in xend, but I'm more 
> > concerned with the hypervisor being responsible responsible 
> > for choosing the vcpu to cpu mapping.  I'd like to see dom 
> > creation support the passing of a cpumap and have the 
> > hypervisor cycle through the physical cpus marked therein.  
> > This would remove the logic of mapping vcpus to cpus from the 
> > hypervisor and let the dom creation tools build whatever 
> > algorithm for distributing vcpus across cpus as it sees fit.
> 
> Sure - feel free to remove the logic from Xen. It's simply there because
> until recently xend didn't know about number of hyperthreads, cores,
> sockets etc.
> 
> The default should be to give Xen considerable freedom in how it
> schedules VCPUs to CPUs: Stephan's load balancer is almost ready for
> posting.
> 
> We should have some higher-level allocation control that can be tweaked
> in xend e,g, "don't use hyperthreading", "allow only privileged domains
> to use the 1st hyperthread on each CPU", "allow any logical CPU to be
> used by any VCPU". [All additionally subject to cpu-dedicate
> restrictions]
> 

Agreed. 

> > If you are interested in this it means changing the hypercall 
> > interface and so it should be done for 3.0, however I don't 
> > want to push for significant changes so close to the testing 
> > freeze as I want to help close features down rather than 
> > create new ones.
> 
> Does it actually need a change to the hypercall interface? The domain
> builder can just issue a pin for each VCPU. It might be cleaner to
> remove the orignal CPU field, but that's such a small change we should
> just do it anyway.

I suppose it is a toss up.  Should the DOM0_CREATEDOMAIN hypercall be
responsible for cpu allocation?  One could argue both ways.  Even now,
it is merely responsible for picking which physical cpu is used for a
dom's VCPU0. Then alloc_vcpu_struct() in schedule.c picks the
processor for additional vcpu as they get booted via do_boot_vcpu()

I think there is more work to do things cleanly.  We can make things
work with pinvcpu operations, but it feels hacky.  I think there is
quite a bit of work, and I'm still not convinced that this should be
done for 3.0 simply because of the size of the change.

1. create domain hypercall should take cpumap to indicate which physical
cpus any of a domain's vcpus can utilize
2. remove physical cpu selection from DOM0_CREATEDOMAIN in dom0_ops.c
3. remove physical cpu selection from alloc_vcpu_struct()
4. create new function, get_next_processor(struct domain *) which reads
the domain's cpumap and returns the next physical cpu the domain should
use.  Update DOM0_CREATEDOMAIN and alloc_vcpu_struct() functions to use
get_next_processor()
5. update tools/libs to pass in/require cpumap when creating domains
6. create high-level map abstractions (e.g. don't use hyperthreads)

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

RE: RE: [Xen-users] Xen 3.0 Dom0 Smp enable?

[Ian Pratt]

 
> I suppose it is a toss up.  Should the DOM0_CREATEDOMAIN hypercall be
> responsible for cpu allocation?  One could argue both ways.  Even now,
> it is merely responsible for picking which physical cpu is used for a
> dom's VCPU0. Then alloc_vcpu_struct() in schedule.c picks the
> processor for additional vcpu as they get booted via do_boot_vcpu()
> 
> I think there is more work to do things cleanly.  We can make things
> work with pinvcpu operations, but it feels hacky.  I think there is
> quite a bit of work, and I'm still not convinced that this should be
> done for 3.0 simply because of the size of the change.

None of the proposal touches anything at all complicated, so it's low
risk and I'd be inclined to make the change if I received a patch soon.
 
> 1. create domain hypercall should take cpumap to indicate 
> which physical
> cpus any of a domain's vcpus can utilize

The create hypercall effectively just allocates a domid. We set all
other parameters via separate ops these days.

We could move the per VCPU pin mask to the domain strcture and
initialise it with defaults when the domain is created. Alternatively we
could have a 'master' cpu map which is used to initialise the maps of
new vcpus. It could be set by the existing hypercall with a VCPU number
of -1.

> 2. remove physical cpu selection from DOM0_CREATEDOMAIN in dom0_ops.c
> 3. remove physical cpu selection from alloc_vcpu_struct()
> 4. create new function, get_next_processor(struct domain *) 
> which reads
> the domain's cpumap and returns the next physical cpu the 
> domain should
> use.  Update DOM0_CREATEDOMAIN and alloc_vcpu_struct() 
> functions to use
> get_next_processor()
> 5. update tools/libs to pass in/require cpumap when creating domains
> 6. create high-level map abstractions (e.g. don't use hyperthreads)


Ian