Re: [PATCH v4 12/12] docs: add xl-psr.markdown

[Ian Campbell <ian.campbell@citrix.com>]

On Thu, 2015-04-09 at 17:18 +0800, Chao Peng wrote:
> Add document to introduce basic concepts and terms in PSR family
> techonologies and the xl/libxl interfaces.

"technologies"
> 
> Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>
> ---
>  docs/man/xl.pod.1         |   7 +++
>  docs/misc/xl-psr.markdown | 111 ++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 118 insertions(+)
>  create mode 100644 docs/misc/xl-psr.markdown
> 
> diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
> index dfab921..b71d6e6 100644
> --- a/docs/man/xl.pod.1
> +++ b/docs/man/xl.pod.1
> @@ -1472,6 +1472,9 @@ occupancy monitoring share the same set of underlying monitoring service. Once
>  a domain is attached to the monitoring service, monitoring data can be showed
>  for any of these monitoring types.
>  
> +See L<http://xenbits.xen.org/docs/unstable/misc/xl-psr.html> for more
> +informations.
> +
>  =over 4
>  
>  =item B<psr-cmt-attach> [I<domain-id>]
> @@ -1501,6 +1504,9 @@ applications. In Xen implementation, CAT is used to control cache allocation
>  on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
>  (CBM) for the domain.
>  
> +See L<http://xenbits.xen.org/docs/unstable/misc/xl-psr.html> for more
> +informations.

Aha, this is what I was missing in the previous patch, thanks.

I think it would still be useful to either briefly explain what the
bitmap contains here or to explicitly refer people to this new document
for that bit of information.

information should be singular not plural both here and above.


> +
>  =over 4
>  
>  =item B<psr-cat-cbm-set> [I<OPTIONS>] [I<domain-id>] [I<cbm>]
> @@ -1546,6 +1552,7 @@ And the following documents on the xen.org website:
>  L<http://xenbits.xen.org/docs/unstable/misc/xl-network-configuration.html>
>  L<http://xenbits.xen.org/docs/unstable/misc/xl-disk-configuration.txt>
>  L<http://xenbits.xen.org/docs/unstable/misc/xsm-flask.txt>
> +L<http://xenbits.xen.org/docs/unstable/misc/xl-psr.html>
>  
>  For systems that don't automatically bring CPU online:
>  
> diff --git a/docs/misc/xl-psr.markdown b/docs/misc/xl-psr.markdown
> new file mode 100644
> index 0000000..44f6f8c
> --- /dev/null
> +++ b/docs/misc/xl-psr.markdown
> @@ -0,0 +1,111 @@
> +# Intel Platform Shared Resource Monitoring/Control in xl/libxl
> +
> +This document introduces Intel Platform Shared Resource Monitoring/Control
> +technologies, their basic concepts and the xl/libxl interfaces.

AFAICT this document only covers xl and not libxl, which I think is fine
(this doc is for users, not developers of other toolstacks), but I think
you should omit mention of libxl.

Whether or not other users of libxl (i.e. the libvirt developers) would
appreciate a dev focused doc covering the libxl interface I'm not sure.
Given the libxl.h prototypes and this doc I suppose it will be pretty
straight forwards.

BTW, do you know if someone is planning to work on libvirt integration
for this stuff?

> +
> +## Cache Monitoring Technology (CMT)
> +
> +Cache Monitoring Technology (CMT) is a new feature available on Intel Haswell
> +and later server platforms that allows an OS or Hypervisor/VMM to determine
> +the usage of cache(currently only L3 cache supported) by applications running
> +on the platform. A Resource Monitoring ID (RMID) is the abstraction of the
> +application(s) that will be monitored for its cache usage. The CMT hardware
> +tracks cache utilization of memory accesses according to the RMID and reports
> +monitored data via a counter register.
> +
> +Detailed information please refer to Intel SDM chapter 17.14.
> +
> +In Xen's implementation, each domain in the system can be assigned a RMID
> +independently, while RMID=0 is reserved for monitoring domains that doesn't
> +enable CMT service. RMID is opaque for xl/libxl and is only used in
> +hypervisor.
> +
> +### xl interfaces
> +
> +A domain is assigned a RMID implicitly by attaching it to CMT service:
> +
> +xl psr-cmt-attach domid

<domid> to match the syntax used in the next example?

If you wrap the command line examples in backticks (like `xl
do-a-thing`) 

> +
> +After that, cache usage for the domain can be showed by:
> +
> +xl psr-cmt-show cache_occupancy <domid>

(Aside: "cache-occupancy" would be more in keeping with the interfaces,
oh well, you can fix if you feel like it, or not bother if you like).

> +Once monitoring is not needed any more, the domain can be detached from the
> +CMT service by:
> +
> +xl psr-cmt-detach domid

<domid> again?

> +
> +The attaching may fail because of no free RMID available. In such case

I think "An attach may fail..." here.

> +unused RMID(s) can be freed by detaching corresponding domains from CMT
> +services. Maximum COS number in the system can also be obtained by:

I think COS hasn't been defined at this point.

> +
> +xl psr_cmt-show

psr-cmt-show? (Hopefully, if the interface is actually inconsistent we
should fix it)

> +
> +## Memory Bandwidth Monitoring (MBM)
> +
> +Memory Bandwidth Monitoring(MBM) is a new hardware feature available on Intel
> +Broadwell and later server platforms which builds on the CMT infrastructure to
> +allow monitoring of system memory bandwidth. It introduces two new monitoring
> +event type to monitor system total/local memory bandwidth. The same RMID can
> +be used to monitor both cache usage and memory bandwidth at the same time.
> +
> +Detailed information please refer to Intel SDM chapter 17.14.
> +
> +In Xen's implementation, MBM shares the same set of underlying monitoring
> +service with CMT and can be used to monitor memory bandwidth on domain basis.

"...on a per domain basis"

> +
> +The xl/libxl interface is the same with that of CMT. The difference is the
> +monitor type is corresponding memory monitoring type(local_mem_bandwidth/

                                                       ^ missing space

> +total_mem_bandwidth) but not cache_occupancy.

I think add: e.g. After a `xl psr-attach`:

`xl psr-cmt-show local_mem_bandwidth <domid>`
`xl psr-cmt-show total_mem_bandwidth <domid>`

To make it clear that the paragraph refers to the argument to
psr-cmt-show and that psr-attach is needed.


> +
> +## Cache Allocation Technology (CAT)
> +
> +Cache Allocation Technology (CAT) is a new feature available on Intel
> +Broadwell and later server platforms that allows an OS or Hypervisor/VMM to
> +partition cache allocation(i.e. L3 cache) based on application priority or

                             ^ missing space

> +Class of Service(COS).

                   ^ missing space

Here is the definition of COS I was looking for.

I'm not sure if the previous mention of "xl psr_cmt-show" getting the
maximum COS is relevant in the context of CMT where it is now, if not
then maybe move that here?

If it is then perhaps some reordering of the sections would allow COS to
be defined first?

>  Each COS is configured using capacity bitmasks (CBM)
> +which represent cache capacity and indicate the degree of overlap and
> +isolation between classes. System cache resource is divided into numbers of
> +minimum portions which is then made up into subset for cache partition. Each
> +portion corresponds to a bit in CBM and the set bit represents the
> +corresponding cache portion is available.
> +
> +Detailed information please refer to Intel SDM chapter 17.15.

Perhaps a few simple examples, would make the basics clearer without
having to hit the SDM for the full gory detail e.g.

        For example, assuming a system with 8 portions and 3 domains:
        
        A CBM of 0xff for every domain means each domain can access the
        whole cache. This is the default.
        
        Giving one domain a CBM of 0x0F and the other two domain's 0xF0
        means that the first domain gets exclusive access to half of the
        cache (half of the portions) and the other two will share the
        other half.
        
        Giving one domain a CBM of 0x0F, one 0x30 and the last 0xc0
        would give the first domain exclusive access to half the cache,
        and the other two exclusive access to one quarter each.
        
Then have the reference the SDM for more detailed stuff.

> +
> +In Xen's implementation, CBM can be set/get with libxl/xl interfaces but COS
> +is maintained in hypervisor only. The cache partition granularity is per
> +domain, each domain has COS=0 assigned by default, the corresponding CBM is
> +all-ones, which means all the cache resource can be used by default.
> +
> +### xl interfaces
> +
> +The simplest way to change a domain's CBM from its default is running:
> +
> +xl psr-cat-cbm-set  [OPTIONS] <domid> <cbm>
> +
> +where cbm is a decimal/hexadecimal number to represent the corresponding cache
> +subset can be used.
> +
> +A cbm is valid only when:
> +
> + * Set bits only exist in the range of [0, cbm_len), where cbm_len can be
> +   obtained with 'xl psr-cat-show'.
> + * All the set bits is contiguous.

"are contiguous".

> + * Is not the same with the current cbm of the domain.

Can't we just implement this as a NOP and avoid this restriction?

If not then the text should "Is not the same as the current..."

> +In multi-sockets system, the same cbm will be set to each socket by default.

"In a multi-socket system, the same cbm will be set on each socket...".

> +Per socket cbm can be specified with '--socket SOCKET' option.

"specified with the `--socket SOCKET`.

> +
> +The cbm may be not set successfully because of no enough COS available.

Setting the CBM may not be successful if insufficient COS are available.

>  In such
> +case unused COS(es) may be freed by setting CBM of all related domains to its
> +default value(all-ones).
> +
> +System CAT information(such as maximum COS and CBM length) and per domain CBM

                         ^space

> +settings can be showed by:

"shown"

> +
> +xl psr-cat-show
> +
> +## Reference
> +
> +[1] Intel SDM
> +(http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html).