Re: Cache Allocation Technology(CAT) design for XEN

[Chao Peng <chao.p.peng@linux.intel.com>]

On Thu, Dec 18, 2014 at 05:40:28PM +0100, Tim Deegan wrote:
> Hi, 
> 
> Thanks for posting this - it's very useful.  I have a couple of
> questions about the interface design. 
Thanks Tim.
> 
> At 20:27 +0800 on 12 Dec (1418412477), Chao Peng wrote:
> > Design Overview
> > When enforcing cache allocation for VMs, the minimum granularity is
> > defined as the domain. All Virtual CPUs ("VCPUs") of a domain have the
> > same COS, and therefore, correspond to the same CBM. COS is used only in
> > hypervisor and is transparent to tool stack/user. System administrator
> > can specify the initial CBM for each domain or change it at runtime using 
> > tool stack. Hypervisor then choses a free COS to associate it with that
> > CBM or find a existed COS which has the same CBM.
> 
> What happens if there is no existing COS that matches, and all COSes
> are in use?  Does Xen return an error?  Or try to change COS->CMB
> mappings during context switches?

In the initial implementation, error is returned. It’s possible for
hypervisor to share COS for different CBMes and not to return error
here. But the problem is that COS shortage may still happen during
context switch. At that time we will have no idea for what to do. So I’d
prefer to return error directly here and leave the decision to user
space, e.g. if error is returned then it can clear CBM for some domain
and get free COS.
> 
> > - VCPU Schedule
> > When VCPU is scheduled on the physical CPU ("PCPU"), its COS value is
> > then written to MSR (IA32_PQR_ASSOC) of PCPU to notify hardware to use 
> > the new COS. The cache allocation is then enforced by hardware.
> > 
> > - Multi-Socket
> > In multi-socket environment, each VCPU may be scheduled on different
> > sockets. The hardware CAT ability(such as maximum supported COS and length
> > of CBM) maybe different among sockets. For such system, per-socket COS/CBM
> > configuration of a domain is specified. Hypervisor then use this per-socket
> > CBM information for VCPU schedule.
> 
> Is it OK to assume that in the common case all CPUs have the same CAT
> capabilities?  Then Xen can just report the smallest set of
> capabilities of any socket in the system, and the toolstack doesn't
> have to mess about with per-socket settings.
> 
> I guess you can add that syntactic sugar on the tools if you want and
> leave the more powerful hypecall interface in case it's useful. :)

Agreed, this is what I want to do. Basically the socketId is optional
for the caller. If more than one socket exists, omitting socketId means
the specified CBM applied to all sockets. While we still maintain
per-socket CBM in hypervisor internally and provide per-socket hypercall
interface in case it’s needed. In this way the interface should be user
friendly for most cases.

Chao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel