Re: [PATCH v4 12/12] docs: add xl-psr.markdown

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 09/04/15 10:18, Chao Peng wrote:
> Add document to introduce basic concepts and terms in PSR family
> techonologies and the xl/libxl interfaces.

Very nice!  A few minor comments...

>
> Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>
> ---
>  docs/man/xl.pod.1         |   7 +++
>  docs/misc/xl-psr.markdown | 111 ++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 118 insertions(+)
>  create mode 100644 docs/misc/xl-psr.markdown
>
> diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
> index dfab921..b71d6e6 100644
> --- a/docs/man/xl.pod.1
> +++ b/docs/man/xl.pod.1
> @@ -1472,6 +1472,9 @@ occupancy monitoring share the same set of underlying monitoring service. Once
>  a domain is attached to the monitoring service, monitoring data can be showed
>  for any of these monitoring types.
>  
> +See L<http://xenbits.xen.org/docs/unstable/misc/xl-psr.html> for more
> +informations.
> +
>  =over 4
>  
>  =item B<psr-cmt-attach> [I<domain-id>]
> @@ -1501,6 +1504,9 @@ applications. In Xen implementation, CAT is used to control cache allocation
>  on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
>  (CBM) for the domain.
>  
> +See L<http://xenbits.xen.org/docs/unstable/misc/xl-psr.html> for more
> +informations.
> +
>  =over 4
>  
>  =item B<psr-cat-cbm-set> [I<OPTIONS>] [I<domain-id>] [I<cbm>]
> @@ -1546,6 +1552,7 @@ And the following documents on the xen.org website:
>  L<http://xenbits.xen.org/docs/unstable/misc/xl-network-configuration.html>
>  L<http://xenbits.xen.org/docs/unstable/misc/xl-disk-configuration.txt>
>  L<http://xenbits.xen.org/docs/unstable/misc/xsm-flask.txt>
> +L<http://xenbits.xen.org/docs/unstable/misc/xl-psr.html>
>  
>  For systems that don't automatically bring CPU online:
>  
> diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown
> new file mode 100644
> index 0000000..44f6f8c
> --- /dev/null
> +++ b/docs/misc/xl-psr.markdown
> @@ -0,0 +1,111 @@
> +# Intel Platform Shared Resource Monitoring/Control in xl/libxl
> +
> +This document introduces Intel Platform Shared Resource Monitoring/Control
> +technologies, their basic concepts and the xl/libxl interfaces.
> +
> +## Cache Monitoring Technology (CMT)
> +
> +Cache Monitoring Technology (CMT) is a new feature available on Intel Haswell
> +and later server platforms that allows an OS or Hypervisor/VMM to determine
> +the usage of cache(currently only L3 cache supported)

L3, or LLC?  This appears to be used ambiguously, but does have a
material impact for system with L4 caches.

>  by applications running
> +on the platform. A Resource Monitoring ID (RMID) is the abstraction of the
> +application(s) that will be monitored for its cache usage. The CMT hardware
> +tracks cache utilization of memory accesses according to the RMID and reports
> +monitored data via a counter register.
> +
> +Detailed information please refer to Intel SDM chapter 17.14.

Please put the chapter title as well, as the numbering does alter slowly
over time.

> +
> +In Xen's implementation, each domain in the system can be assigned a RMID
> +independently, while RMID=0 is reserved for monitoring domains that doesn't
> +enable CMT service. RMID is opaque for xl/libxl and is only used in
> +hypervisor.
> +
> +### xl interfaces
> +
> +A domain is assigned a RMID implicitly by attaching it to CMT service:
> +
> +xl psr-cmt-attach domid
> +
> +After that, cache usage for the domain can be showed by:
> +
> +xl psr-cmt-show cache_occupancy <domid>
> +
> +Once monitoring is not needed any more, the domain can be detached from the
> +CMT service by:
> +
> +xl psr-cmt-detach domid
> +
> +The attaching may fail because of no free RMID available. In such case
> +unused RMID(s) can be freed by detaching corresponding domains from CMT
> +services. Maximum COS number in the system can also be obtained by:

You have not yet introduced COS as a term.  Perhaps this bit is better
moving down to the CAT section?

> +
> +xl psr_cmt-show

"psr-cmt-show"

I am not sure how wise it is to dump information like max rmid/max cos
into cmt-show.

Is it perhaps worth having an `xl psr-hwinfo` (or equivalent) which will
dump the hardware capabilities, per-socket limits etc, as a consise way
to obtain all relevant information?

> +
> +## Memory Bandwidth Monitoring (MBM)
> +
> +Memory Bandwidth Monitoring(MBM) is a new hardware feature available on Intel
> +Broadwell and later server platforms which builds on the CMT infrastructure to
> +allow monitoring of system memory bandwidth. It introduces two new monitoring
> +event type to monitor system total/local memory bandwidth. The same RMID can
> +be used to monitor both cache usage and memory bandwidth at the same time.
> +
> +Detailed information please refer to Intel SDM chapter 17.14.
> +
> +In Xen's implementation, MBM shares the same set of underlying monitoring
> +service with CMT and can be used to monitor memory bandwidth on domain basis.
> +
> +The xl/libxl interface is the same with that of CMT. The difference is the
> +monitor type is corresponding memory monitoring type(local_mem_bandwidth/
> +total_mem_bandwidth) but not cache_occupancy.
> +
> +## Cache Allocation Technology (CAT)
> +
> +Cache Allocation Technology (CAT) is a new feature available on Intel
> +Broadwell and later server platforms that allows an OS or Hypervisor/VMM to
> +partition cache allocation(i.e. L3 cache) based on application priority or
> +Class of Service(COS). Each COS is configured using capacity bitmasks (CBM)
> +which represent cache capacity and indicate the degree of overlap and
> +isolation between classes. System cache resource is divided into numbers of
> +minimum portions which is then made up into subset for cache partition. Each
> +portion corresponds to a bit in CBM and the set bit represents the
> +corresponding cache portion is available.
> +
> +Detailed information please refer to Intel SDM chapter 17.15.
> +
> +In Xen's implementation, CBM can be set/get with libxl/xl interfaces but COS

Strictly speaking that should be "set/got" in english, but "configured"
would be a better alternative.

~Andrew

> +is maintained in hypervisor only. The cache partition granularity is per
> +domain, each domain has COS=0 assigned by default, the corresponding CBM is
> +all-ones, which means all the cache resource can be used by default.
> +
> +### xl interfaces
> +
> +The simplest way to change a domain's CBM from its default is running:
> +
> +xl psr-cat-cbm-set  [OPTIONS] <domid> <cbm>
> +
> +where cbm is a decimal/hexadecimal number to represent the corresponding cache
> +subset can be used.
> +
> +A cbm is valid only when:
> +
> + * Set bits only exist in the range of [0, cbm_len), where cbm_len can be
> +   obtained with 'xl psr-cat-show'.
> + * All the set bits is contiguous.
> + * Is not the same with the current cbm of the domain.
> +
> +In multi-sockets system, the same cbm will be set to each socket by default.
> +Per socket cbm can be specified with '--socket SOCKET' option.
> +
> +The cbm may be not set successfully because of no enough COS available. In such
> +case unused COS(es) may be freed by setting CBM of all related domains to its
> +default value(all-ones).
> +
> +System CAT information(such as maximum COS and CBM length) and per domain CBM
> +settings can be showed by:
> +
> +xl psr-cat-show
> +
> +## Reference
> +
> +[1] Intel SDM
> +(http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html).