Re: RE: [Xen-users] Xen 3.0 Dom0 Smp enable?

[Ryan Harper <ryanh@us.ibm.com>]

* Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2005-08-08 12:05]:
> > > If you create a pincpu mask for the over domains/VCPUs that doesn't 
> > > include the CPU in question, they won't use it. I guess a shorthand 
> > > command for this would be useful.
> > 
> > Something like:
> > 
> > xm dedicate DOM CPU
> 
> It should probably be called cpu-dedicate or such like.

Sure.  Always open to names.

> 
> > > Ryan, if so, please can you dust off and resend. As I 
> > recall, the key 
> > > fix it needed was that it should be xend (not xm) that does the 
> > > translation from the dot'ed form into CPU numbers.
> > 
> > Certainly it will be easier to do so in xend, but I'm more 
> > concerned with the hypervisor being responsible responsible 
> > for choosing the vcpu to cpu mapping.  I'd like to see dom 
> > creation support the passing of a cpumap and have the 
> > hypervisor cycle through the physical cpus marked therein.  
> > This would remove the logic of mapping vcpus to cpus from the 
> > hypervisor and let the dom creation tools build whatever 
> > algorithm for distributing vcpus across cpus as it sees fit.
> 
> Sure - feel free to remove the logic from Xen. It's simply there because
> until recently xend didn't know about number of hyperthreads, cores,
> sockets etc.
> 
> The default should be to give Xen considerable freedom in how it
> schedules VCPUs to CPUs: Stephan's load balancer is almost ready for
> posting.
> 
> We should have some higher-level allocation control that can be tweaked
> in xend e,g, "don't use hyperthreading", "allow only privileged domains
> to use the 1st hyperthread on each CPU", "allow any logical CPU to be
> used by any VCPU". [All additionally subject to cpu-dedicate
> restrictions]
> 

Agreed. 

> > If you are interested in this it means changing the hypercall 
> > interface and so it should be done for 3.0, however I don't 
> > want to push for significant changes so close to the testing 
> > freeze as I want to help close features down rather than 
> > create new ones.
> 
> Does it actually need a change to the hypercall interface? The domain
> builder can just issue a pin for each VCPU. It might be cleaner to
> remove the orignal CPU field, but that's such a small change we should
> just do it anyway.

I suppose it is a toss up.  Should the DOM0_CREATEDOMAIN hypercall be
responsible for cpu allocation?  One could argue both ways.  Even now,
it is merely responsible for picking which physical cpu is used for a
dom's VCPU0. Then alloc_vcpu_struct() in schedule.c picks the
processor for additional vcpu as they get booted via do_boot_vcpu()

I think there is more work to do things cleanly.  We can make things
work with pinvcpu operations, but it feels hacky.  I think there is
quite a bit of work, and I'm still not convinced that this should be
done for 3.0 simply because of the size of the change.

1. create domain hypercall should take cpumap to indicate which physical
cpus any of a domain's vcpus can utilize
2. remove physical cpu selection from DOM0_CREATEDOMAIN in dom0_ops.c
3. remove physical cpu selection from alloc_vcpu_struct()
4. create new function, get_next_processor(struct domain *) which reads
the domain's cpumap and returns the next physical cpu the domain should
use.  Update DOM0_CREATEDOMAIN and alloc_vcpu_struct() functions to use
get_next_processor()
5. update tools/libs to pass in/require cpumap when creating domains
6. create high-level map abstractions (e.g. don't use hyperthreads)

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com