Re: [PATCH v4 12/12] docs: add xl-psr.markdown

[Chao Peng <chao.p.peng@linux.intel.com>]

On Thu, Apr 09, 2015 at 12:29:38PM +0100, Andrew Cooper wrote:
> On 09/04/15 10:18, Chao Peng wrote:
> > Add document to introduce basic concepts and terms in PSR family
> > techonologies and the xl/libxl interfaces.
> 
> Very nice!  A few minor comments...
> 
> >
> > Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>
> > ---
> >  docs/man/xl.pod.1         |   7 +++
> >  docs/misc/xl-psr.markdown | 111 ++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 118 insertions(+)
> >  create mode 100644 docs/misc/xl-psr.markdown
> >
> > +## Cache Monitoring Technology (CMT)
> > +
> > +Cache Monitoring Technology (CMT) is a new feature available on Intel Haswell
> > +and later server platforms that allows an OS or Hypervisor/VMM to determine
> > +the usage of cache(currently only L3 cache supported)
> 
> L3, or LLC?  This appears to be used ambiguously, but does have a
> material impact for system with L4 caches.

As there is no CMT+L4 working actually, I'd keep it as L3 right now.

> 
> >  by applications running
> > +on the platform. A Resource Monitoring ID (RMID) is the abstraction of the
> > +application(s) that will be monitored for its cache usage. The CMT hardware
> > +tracks cache utilization of memory accesses according to the RMID and reports
> > +monitored data via a counter register.
> > +
> > +Detailed information please refer to Intel SDM chapter 17.14.
> 
> Please put the chapter title as well, as the numbering does alter slowly
> over time.

Sure.

> 
> > +
> > +In Xen's implementation, each domain in the system can be assigned a RMID
> > +independently, while RMID=0 is reserved for monitoring domains that doesn't
> > +enable CMT service. RMID is opaque for xl/libxl and is only used in
> > +hypervisor.
> > +
> > +### xl interfaces
> > +
> > +A domain is assigned a RMID implicitly by attaching it to CMT service:
> > +
> > +xl psr-cmt-attach domid
> > +
> > +After that, cache usage for the domain can be showed by:
> > +
> > +xl psr-cmt-show cache_occupancy <domid>
> > +
> > +Once monitoring is not needed any more, the domain can be detached from the
> > +CMT service by:
> > +
> > +xl psr-cmt-detach domid
> > +
> > +The attaching may fail because of no free RMID available. In such case
> > +unused RMID(s) can be freed by detaching corresponding domains from CMT
> > +services. Maximum COS number in the system can also be obtained by:
> 
> You have not yet introduced COS as a term.  Perhaps this bit is better
> moving down to the CAT section?

Good catch. Thanks.

> 
> > +
> > +xl psr_cmt-show
> 
> "psr-cmt-show"
> 
> I am not sure how wise it is to dump information like max rmid/max cos
> into cmt-show.
> 
> Is it perhaps worth having an `xl psr-hwinfo` (or equivalent) which will
> dump the hardware capabilities, per-socket limits etc, as a consise way
> to obtain all relevant information?

Sounds reasonable. I will do it.
> 
> > +
> > +## Memory Bandwidth Monitoring (MBM)
> > +
> > +Memory Bandwidth Monitoring(MBM) is a new hardware feature available on Intel
> > +Broadwell and later server platforms which builds on the CMT infrastructure to
> > +allow monitoring of system memory bandwidth. It introduces two new monitoring
> > +event type to monitor system total/local memory bandwidth. The same RMID can
> > +be used to monitor both cache usage and memory bandwidth at the same time.
> > +
> > +Detailed information please refer to Intel SDM chapter 17.14.
> > +
> > +In Xen's implementation, MBM shares the same set of underlying monitoring
> > +service with CMT and can be used to monitor memory bandwidth on domain basis.
> > +
> > +The xl/libxl interface is the same with that of CMT. The difference is the
> > +monitor type is corresponding memory monitoring type(local_mem_bandwidth/
> > +total_mem_bandwidth) but not cache_occupancy.
> > +
> > +## Cache Allocation Technology (CAT)
> > +
> > +Cache Allocation Technology (CAT) is a new feature available on Intel
> > +Broadwell and later server platforms that allows an OS or Hypervisor/VMM to
> > +partition cache allocation(i.e. L3 cache) based on application priority or
> > +Class of Service(COS). Each COS is configured using capacity bitmasks (CBM)
> > +which represent cache capacity and indicate the degree of overlap and
> > +isolation between classes. System cache resource is divided into numbers of
> > +minimum portions which is then made up into subset for cache partition. Each
> > +portion corresponds to a bit in CBM and the set bit represents the
> > +corresponding cache portion is available.
> > +
> > +Detailed information please refer to Intel SDM chapter 17.15.
> > +
> > +In Xen's implementation, CBM can be set/get with libxl/xl interfaces but COS
> 
> Strictly speaking that should be "set/got" in english, but "configured"
> would be a better alternative.

Exactly, thanks.

Chao