Re: [PATCH v2 0/8] enable Cache QoS Monitoring (CQM) feature

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 21/11/13 07:20, dongxiao.xu@intel.com wrote:
> From: Dongxiao Xu <dongxiao.xu@intel.com>

You will need to CC the relevant maintainers.  This would be Jan and
Keir for the Hypervisor changes, and IanC, IanJ and Stefano for the
tools changes.

I would also suggest that you group the tools changes together at the
end of the series rather than having them interspersed.

~Andrew

>
> Changes from v1:
>  - Address comments from Andrew Cooper, including:
>    * Change function names, e.g., alloc_cqm_rmid(), system_supports_cqm(), etc.
>    * Change some structure element order to save packing cost.
>    * Correct some function's return value.
>    * Some programming styles change.
>    * ...
>
> Future generations of Intel Xeon processor may offer monitoring capability in
> each logical processor to measure specific quality-of-service metric,
> for example, the Cache QoS Monitoring to get L3 cache occupancy.
> Detailed information please refer to Intel SDM chapter 17.14.
>
> Cache QoS Monitoring provides a layer of abstraction between applications and
> logical processors through the use of Resource Monitoring IDs (RMIDs).
> In Xen design, each guest in the system can be assigned an RMID independently,
> while RMID=0 is reserved for monitoring domains that doesn't enable CQM service.
> When any of the domain's vcpu is scheduled on a logical processor, the domain's
> RMID will be activated by programming the value into one specific MSR, and when
> the vcpu is scheduled out, a RMID=0 will be programmed into that MSR.
> The Cache QoS Hardware tracks cache utilization of memory accesses according to
> the RMIDs and reports monitored data via a counter register. With this solution,
> we can get the knowledge how much L3 cache is used by a certain guest.
>
> To attach CQM service to a certain guest, two approaches are provided:
> 1) Create the guest with "pqos_cqm=1" set in configuration file.
> 2) Use "xl pqos-attach cqm domid" for a running guest.
>
> To detached CQM service from a guest, users can:
> 1) Use "xl pqos-detach cqm domid" for a running guest.
> 2) Also destroying a guest will detach the CQM service.
>
> To get the L3 cache usage, users can use the command of:
> $ xl pqos-list cqm (domid)
>
> The below data is just an example showing how the CQM related data is exposed to
> end user.
>
> [root@localhost]# xl pqos-list cqm
> RMID count    56        RMID available    53
> Name               ID  SocketID        L3C_Usage       SocketID        L3C_Usage
> Domain-0            0         0         20127744              1         25231360
> ExampleHVMDomain    1         0          3211264              1         10551296
>
> Dongxiao Xu (8):
>   x86: detect and initialize Cache QoS Monitoring feature
>   x86: handle CQM resource when creating/destroying guests
>   tools: handle CQM resource when creating/destroying guests
>   x86: dynamically attach/detach CQM service for a guest
>   tools: dynamically attach/detach CQM service for a guest
>   x86: get per domain CQM information
>   tools: get per domain CQM information
>   x86: enable CQM monitoring for each domain RMID
>
>  tools/libxc/xc_domain.c          |   51 +++++++++
>  tools/libxc/xenctrl.h            |   14 +++
>  tools/libxl/Makefile             |    3 +-
>  tools/libxl/libxl.h              |    8 ++
>  tools/libxl/libxl_create.c       |    3 +
>  tools/libxl/libxl_pqos.c         |  102 +++++++++++++++++
>  tools/libxl/libxl_types.idl      |    1 +
>  tools/libxl/xl.h                 |    3 +
>  tools/libxl/xl_cmdimpl.c         |  129 ++++++++++++++++++++++
>  tools/libxl/xl_cmdtable.c        |   15 +++
>  xen/arch/x86/Makefile            |    1 +
>  xen/arch/x86/cpu/intel.c         |    6 +
>  xen/arch/x86/domain.c            |   14 +++
>  xen/arch/x86/domctl.c            |   40 +++++++
>  xen/arch/x86/pqos.c              |  224 ++++++++++++++++++++++++++++++++++++++
>  xen/arch/x86/setup.c             |    3 +
>  xen/arch/x86/sysctl.c            |   64 +++++++++++
>  xen/common/domctl.c              |    5 +-
>  xen/include/asm-x86/cpufeature.h |    1 +
>  xen/include/asm-x86/domain.h     |    2 +
>  xen/include/asm-x86/msr-index.h  |    5 +
>  xen/include/asm-x86/pqos.h       |   56 ++++++++++
>  xen/include/public/domctl.h      |   31 ++++++
>  xen/include/public/sysctl.h      |   15 +++
>  xen/include/xen/sched.h          |    3 +
>  25 files changed, 797 insertions(+), 2 deletions(-)
>  create mode 100644 tools/libxl/libxl_pqos.c
>  create mode 100644 xen/arch/x86/pqos.c
>  create mode 100644 xen/include/asm-x86/pqos.h
>