[PATCH v22] enable Cache Monitoring Technology (CMT) feature(tools)

[PATCH v22] enable Cache Monitoring Technology (CMT) feature(tools)

[Chao Peng]

Changes from v21:
 - Fix build error on ARM platform reported by Ian:
   * ifdef CMT code with LIBXL_HAVE_PSR_CMT in libxl/xl.

Changes from v20:
 - Address comments from Wei:
   * remove randomization in 'pick_socket_cpu' and move it to libxl_psr.c
 - drop #1-#9 of this patch serial for review as they have been acked.
 - rebase.

Changes from v19:
 - Address xen comments from Jan and tools comments from Wei.

Changes from v18:
 - Address comments from Andrew and Jan, including:
   * Fix return value/cpu check/other misc errors in resource_op;
   * Fix cmd option/rmid_max/rmid_mask error for psr initialization;
   * Make get_l3_cache_size sysctl to get per-socket value;
   * MSR names update;

Changes from v17:
 - Address comments from Andrew and Jan, including:
   * Add per-entry return value for resource_op;
   * Refine code for resource_op;
    
Changes from v16:
 - Address comments from Konrad, Andrew and Jan, including:
   * Correct copyin/copyout for resource_op.
   * Improve documentation/coding style
   * Other minor fix.

Changes from v15:
 - Keywords change: Intel changed the names for PQOS/CQM in latest SDM.
   Adjust the code accordingly:
     PQOS(Platform QOS) => PSR(Platform Shared Resource)
     CQM(Cache QoS Monitoring) => CMT(Cache Monitoring Technology)
 - Make resource operation more clean:
   * do_platform_op is the minimum unit for non-preemptible operation, it
     accepts Small, non-preempt operations as well as single operation.
   * Other preemptible batch operations are performed with multicall mechanism.
 - Add padding field in xenpf_resource_data structure and check for that.

Changes from v14:
 - Address comments from Jan and Andrew, including:
   * Add non-preemption ability to multicall;
   * Build the resource batch operation on top of multicall;
   * Simplify pqos option.

Changes from v13:
 - Address comments from Jan and Andrew, including:
   * Support mixed resource types in one invocation;
   * Remove some unused fields(rmid_min/rmid_inuse);
   * Other minor changes and code clean up;

Changes from v12:
 - Address comments from Jan, like QoS feature setting when booting,
   avoid unbound memory allocation in Xen, put resource access
   hypercall in platform_hypercall.c to avoid creating new files, 
   specifically enumerate L3 cache size for CQM (instead of
   x86_cache_size), get random socket CPU in user space tools, and 
   also some coding styles. However the continue_hypercall_on_cpu()
   suggestion is not adopted in this version due to the potential
   issue in our usage case.
 - Add a white list to limit the capability for resource access from
   tool side.
 - Address comments from Ian on the xl/libxl/libxc side.

Changes from v11:
 - Turn off pqos and pqos_monitor in Xen command line by default.
 - Modify the original specific MSR access hypercall into a generic
   resource access hypercall. This hypercall could be used to access
   MSR, Port I/O, etc. Use platform_op to replace sysctl so that both
   dom0 kernel and userspace could use this hypercall.
 - Address various comments from Jan, Ian, Konrad, and Daniel.

Changes from v10:
 - Re-design and re-implement the whole logic. In this version,
   hypervisor provides basic mechanisms (like access MSRs) while all
   policies are put in user space.
   patch 1-3 provide a generic MSR hypercall for toolstack to access
   patch 4-9 implement the cache QoS monitoring feature

Changes from v9:
 - Revise the readonly mapping mechanism to share data between Xen and
   userspace. We create L3C buffer for each socket, share both buffer
   address MFNs and buffer MFNs to userspace.
 - Split the pqos.c into pqos.c and cqm.c for better code structure.
 - Show the total L3 cache size when issueing xl pqos-list cqm command.
 - Abstract a libxl_getcqminfo() function to fetch cqm data from Xen.
 - Several coding style fixes.

Changes from v8:
 - Address comments from Ian Campbell, including:
   * Modify the return handling for xc_sysctl();
   * Add man page items for platform QoS related commands.
   * Fix typo in commit message.

Changes from v7:
 - Address comments from Andrew Cooper, including:
   * Check CQM capability before allocating cpumask memory.
   * Move one function declaration into the correct patch.

Changes from v6:
 - Address comments from Jan Beulich, including:
   * Remove the unnecessary CPUID feature check.
   * Remove the unnecessary socket_cpu_map.
   * Spin_lock related changes, avoid spin_lock_irqsave().
   * Use readonly mapping to pass cqm data between Xen/Userspace,
     to avoid data copying.
   * Optimize RDMSR/WRMSR logic to avoid unnecessary calls.
   * Misc fixes including __read_mostly prefix, return value, etc.

Changes from v5:
 - Address comments from Dario Faggioli, including:
   * Define a new libxl_cqminfo structure to avoid reference of xc
     structure in libxl functions.
   * Use LOGE() instead of the LIBXL__LOG() functions.

Changes from v4:
 - When comparing xl cqm parameter, use strcmp instead of strncmp,
   otherwise, "xl pqos-attach cqmabcd domid" will be considered as
   a valid command line.
 - Address comments from Andrew Cooper, including:
   * Adjust the pqos parameter parsing function.
   * Modify the pqos related documentation.
   * Add a check for opt_cqm_max_rmid in initialization code.
   * Do not IPI CPU that is in same socket with current CPU.
 - Address comments from Dario Faggioli, including:
   * Fix an typo in export symbols.
   * Return correct libxl error code for qos related functions.
   * Abstract the error printing logic into a function.
 - Address comment from Daniel De Graaf, including:
   * Add return value in for pqos related check.
 - Address comments from Konrad Rzeszutek Wilk, including:
   * Modify the GPLv2 related file header, remove the address.

Changes from v3:
 - Use structure to better organize CQM related global variables.
 - Address comments from Andrew Cooper, including:
   * Remove the domain creation flag for CQM RMID allocation.
   * Adjust the boot parameter format, use custom_param().
   * Add documentation for the new added boot parameter.
   * Change QoS type flag to be uint64_t.
   * Initialize the per socket cpu bitmap in system boot time.
   * Remove get_cqm_avail() function.
   * Misc of format changes.
 - Address comment from Daniel De Graaf, including:
   * Use avc_current_has_perm() for XEN2__PQOS_OP that belongs to SECCLASS_XEN2.

Changes from v2:
 - Address comments from Andrew Cooper, including:
   * Merging tools stack changes into one patch.
   * Reduce the IPI number to one per socket.
   * Change structures for CQM data exchange between tools and Xen.
   * Misc of format/variable/function name changes.
 - Address comments from Konrad Rzeszutek Wilk, including:
   * Simplify the error printing logic.
   * Add xsm check for the new added hypercalls.

Changes from v1:
 - Address comments from Andrew Cooper, including:
   * Change function names, e.g., alloc_cqm_rmid(), system_supports_cqm(), etc.
   * Change some structure element order to save packing cost.
   * Correct some function's return value.
   * Some programming styles change.
   * ...

The Intel Xeon processor E5 v3 family introduced resource monitoring capability
in each logical processor to measure specific platform shared resource metrics,
for example, L3 cache occupancy. Detailed information please refer to Intel SDM
chapter 17.14.

Cache Monitoring Technology provides a layer of abstraction between applications
and logical processors through the use of Resource Monitoring IDs (RMIDs).
In Xen design, each guest in the system can be assigned an RMID independently,
while RMID=0 is reserved for monitoring domains that doesn't enable CMT service.
When any of the domain's vcpu is scheduled on a logical processor, the domain's
RMID will be activated by programming the value into one specific MSR, and when
the vcpu is scheduled out, a RMID=0 will be programmed into that MSR.
The CMT Hardware tracks cache utilization of memory accesses according to the
RMIDs and reports monitored data via a counter register. With this solution,
we can get the knowledge how much L3 cache is used by a certain guest.

To attach CMT service to a certain guest:
xl psr-cmt-attach domid

To detached CMT service from a guest:
xl psr-cmt-detach domid

To get the L3 cache usage:
$ xl psr-cmt-show cache_occupancy <domid>

The below data is just an example showing how the CMT related data is exposed to
end user.

[root@localhost]# xl psr-cmt-show cache_occupancy
Total RMID: 31
Name                                        ID        Socket 0        Socket 1
Total L3 Cache Size                                   20480 KB        20480 KB
Domain-0                                     0        14240 KB        14976 KB
ExampleHVMDomain                             1         4200 KB         2352 KB

Chao Peng (1):
  tools: CMDs and APIs for Cache Monitoring Technology

 docs/man/xl.pod.1             |   25 +++++
 tools/libxc/Makefile          |    1 +
 tools/libxc/include/xenctrl.h |   19 ++++
 tools/libxc/xc_msr_x86.h      |   36 +++++++
 tools/libxc/xc_psr.c          |  215 ++++++++++++++++++++++++++++++++++++++++
 tools/libxl/Makefile          |    2 +-
 tools/libxl/libxl.h           |   21 ++++
 tools/libxl/libxl_psr.c       |  217 +++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_types.idl   |    4 +
 tools/libxl/xl.h              |    5 +
 tools/libxl/xl_cmdimpl.c      |  160 ++++++++++++++++++++++++++++++
 tools/libxl/xl_cmdtable.c     |   19 ++++
 12 files changed, 723 insertions(+), 1 deletion(-)
 create mode 100644 tools/libxc/xc_msr_x86.h
 create mode 100644 tools/libxc/xc_psr.c
 create mode 100644 tools/libxl/libxl_psr.c

-- 
1.7.9.5

[PATCH v22] tools: CMDs and APIs for Cache Monitoring Technology

[Chao Peng]

Introduced some new xl commands to enable/disable Cache Monitoring
Technology(CMT) feature.

The following two commands is to attach/detach the CMT feature
to/from a certain domain.

$ xl psr-cmt-attach domid
$ xl psr-cmt-detach domid

This command is to display the CMT information, such as L3 cache
occupancy.

$ xl psr-cmt-show cache_occupancy <domid>

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>
---
 docs/man/xl.pod.1             |   25 +++++
 tools/libxc/Makefile          |    1 +
 tools/libxc/include/xenctrl.h |   19 ++++
 tools/libxc/xc_msr_x86.h      |   36 +++++++
 tools/libxc/xc_psr.c          |  215 ++++++++++++++++++++++++++++++++++++++++
 tools/libxl/Makefile          |    2 +-
 tools/libxl/libxl.h           |   21 ++++
 tools/libxl/libxl_psr.c       |  217 +++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_types.idl   |    4 +
 tools/libxl/xl.h              |    5 +
 tools/libxl/xl_cmdimpl.c      |  160 ++++++++++++++++++++++++++++++
 tools/libxl/xl_cmdtable.c     |   19 ++++
 12 files changed, 723 insertions(+), 1 deletion(-)
 create mode 100644 tools/libxc/xc_msr_x86.h
 create mode 100644 tools/libxc/xc_psr.c
 create mode 100644 tools/libxl/libxl_psr.c

diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 362e92f..925cd7d 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -1440,6 +1440,31 @@ Load FLASK policy from the given policy file. The initial policy is provided to
 the hypervisor as a multiboot module; this command allows runtime updates to the
 policy. Loading new security policy will reset runtime changes to device labels.
 
+=head1 Cache Monitoring Technology
+
+Intel Haswell and later server platforms offer monitoring capability in each
+logical processor to measure specific platform shared resource metric, for
+example, L3 cache occupancy. In Xen implementation, the monitoring granularity
+is domain level. To monitor a specific domain, just attach the domain id with
+the monitoring service. When the domain doesn't need to be monitored any more,
+detach the domain id from the monitoring service.
+
+=over 4
+
+=item B<psr-cmt-attach> [I<domain-id>]
+
+attach: Attach the platform shared resource monitoring service to a domain.
+
+=item B<psr-cmt-detach> [I<domain-id>]
+
+detach: Detach the platform shared resource monitoring service from a domain.
+
+=item B<psr-cmt-show> [I<psr-monitor-type>] [I<domain-id>]
+
+Show monitoring data for a certain domain or all domains. Current supported
+monitor types are:
+ - "cache-occupancy": showing the L3 cache occupancy.
+
 =back
 
 =head1 TO BE DOCUMENTED
diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
index 0851273..bd2ca6c 100644
--- a/tools/libxc/Makefile
+++ b/tools/libxc/Makefile
@@ -36,6 +36,7 @@ CTRL_SRCS-y       += xc_kexec.c
 CTRL_SRCS-y       += xtl_core.c
 CTRL_SRCS-y       += xtl_logger_stdio.c
 CTRL_SRCS-y       += xc_resource.c
+CTRL_SRCS-$(CONFIG_X86) += xc_psr.c
 CTRL_SRCS-$(CONFIG_X86) += xc_pagetab.c
 CTRL_SRCS-$(CONFIG_Linux) += xc_linux.c xc_linux_osdep.c
 CTRL_SRCS-$(CONFIG_FreeBSD) += xc_freebsd.c xc_freebsd_osdep.c
diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h
index b73b157..564e187 100644
--- a/tools/libxc/include/xenctrl.h
+++ b/tools/libxc/include/xenctrl.h
@@ -2679,6 +2679,25 @@ struct xc_resource_op {
 typedef struct xc_resource_op xc_resource_op_t;
 int xc_resource_op(xc_interface *xch, uint32_t nr_ops, xc_resource_op_t *ops);
 
+#if defined(__i386__) || defined(__x86_64__)
+enum xc_psr_cmt_type {
+    XC_PSR_CMT_L3_OCCUPANCY,
+};
+typedef enum xc_psr_cmt_type xc_psr_cmt_type;
+int xc_psr_cmt_attach(xc_interface *xch, uint32_t domid);
+int xc_psr_cmt_detach(xc_interface *xch, uint32_t domid);
+int xc_psr_cmt_get_domain_rmid(xc_interface *xch, uint32_t domid,
+    uint32_t *rmid);
+int xc_psr_cmt_get_total_rmid(xc_interface *xch, uint32_t *total_rmid);
+int xc_psr_cmt_get_l3_upscaling_factor(xc_interface *xch,
+    uint32_t *upscaling_factor);
+int xc_psr_cmt_get_l3_cache_size(xc_interface *xch, uint32_t cpu,
+    uint32_t *l3_cache_size);
+int xc_psr_cmt_get_data(xc_interface *xch, uint32_t rmid,
+    uint32_t cpu, uint32_t psr_cmt_type, uint64_t *monitor_data);
+int xc_psr_cmt_enabled(xc_interface *xch);
+#endif
+
 #endif /* XENCTRL_H */
 
 /*
diff --git a/tools/libxc/xc_msr_x86.h b/tools/libxc/xc_msr_x86.h
new file mode 100644
index 0000000..7c3e1a3
--- /dev/null
+++ b/tools/libxc/xc_msr_x86.h
@@ -0,0 +1,36 @@
+/*
+ * xc_msr_x86.h
+ *
+ * MSR definition macros
+ *
+ * Copyright (C) 2014      Intel Corporation
+ * Author Dongxiao Xu <dongxiao.xu@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#ifndef XC_MSR_X86_H
+#define XC_MSR_X86_H
+
+#define MSR_IA32_CMT_EVTSEL     0x00000c8d
+#define MSR_IA32_CMT_CTR        0x00000c8e
+
+#endif
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxc/xc_psr.c b/tools/libxc/xc_psr.c
new file mode 100644
index 0000000..872e6dc
--- /dev/null
+++ b/tools/libxc/xc_psr.c
@@ -0,0 +1,215 @@
+/*
+ * xc_psr.c
+ *
+ * platform shared resource related API functions.
+ *
+ * Copyright (C) 2014      Intel Corporation
+ * Author Dongxiao Xu <dongxiao.xu@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "xc_private.h"
+#include "xc_msr_x86.h"
+
+#define IA32_CMT_CTR_ERROR_MASK         (0x3ull << 62)
+
+#define EVTID_L3_OCCUPANCY             0x1
+
+int xc_psr_cmt_attach(xc_interface *xch, uint32_t domid)
+{
+    DECLARE_DOMCTL;
+
+    domctl.cmd = XEN_DOMCTL_psr_cmt_op;
+    domctl.domain = (domid_t)domid;
+    domctl.u.psr_cmt_op.cmd = XEN_DOMCTL_PSR_CMT_OP_ATTACH;
+
+    return do_domctl(xch, &domctl);
+}
+
+int xc_psr_cmt_detach(xc_interface *xch, uint32_t domid)
+{
+    DECLARE_DOMCTL;
+
+    domctl.cmd = XEN_DOMCTL_psr_cmt_op;
+    domctl.domain = (domid_t)domid;
+    domctl.u.psr_cmt_op.cmd = XEN_DOMCTL_PSR_CMT_OP_DETACH;
+
+    return do_domctl(xch, &domctl);
+}
+
+int xc_psr_cmt_get_domain_rmid(xc_interface *xch, uint32_t domid,
+                                    uint32_t *rmid)
+{
+    int rc;
+    DECLARE_DOMCTL;
+
+    domctl.cmd = XEN_DOMCTL_psr_cmt_op;
+    domctl.domain = (domid_t)domid;
+    domctl.u.psr_cmt_op.cmd = XEN_DOMCTL_PSR_CMT_OP_QUERY_RMID;
+
+    rc = do_domctl(xch, &domctl);
+
+    if ( !rc )
+        *rmid = domctl.u.psr_cmt_op.data;
+
+    return rc;
+}
+
+int xc_psr_cmt_get_total_rmid(xc_interface *xch, uint32_t *total_rmid)
+{
+    static int val = 0;
+    int rc;
+    DECLARE_SYSCTL;
+
+    if ( val )
+    {
+        *total_rmid = val;
+        return 0;
+    }
+
+    sysctl.cmd = XEN_SYSCTL_psr_cmt_op;
+    sysctl.u.psr_cmt_op.cmd = XEN_SYSCTL_PSR_CMT_get_total_rmid;
+    sysctl.u.psr_cmt_op.flags = 0;
+
+    rc = xc_sysctl(xch, &sysctl);
+    if ( !rc )
+        val = *total_rmid = sysctl.u.psr_cmt_op.u.data;
+
+    return rc;
+}
+
+int xc_psr_cmt_get_l3_upscaling_factor(xc_interface *xch,
+                                            uint32_t *upscaling_factor)
+{
+    static int val = 0;
+    int rc;
+    DECLARE_SYSCTL;
+
+    if ( val )
+    {
+        *upscaling_factor = val;
+        return 0;
+    }
+
+    sysctl.cmd = XEN_SYSCTL_psr_cmt_op;
+    sysctl.u.psr_cmt_op.cmd =
+        XEN_SYSCTL_PSR_CMT_get_l3_upscaling_factor;
+    sysctl.u.psr_cmt_op.flags = 0;
+
+    rc = xc_sysctl(xch, &sysctl);
+    if ( !rc )
+        val = *upscaling_factor = sysctl.u.psr_cmt_op.u.data;
+
+    return rc;
+}
+
+int xc_psr_cmt_get_l3_cache_size(xc_interface *xch, uint32_t cpu,
+                                      uint32_t *l3_cache_size)
+{
+    static int val = 0;
+    int rc;
+    DECLARE_SYSCTL;
+
+    if ( val )
+    {
+        *l3_cache_size = val;
+        return 0;
+    }
+
+    sysctl.cmd = XEN_SYSCTL_psr_cmt_op;
+    sysctl.u.psr_cmt_op.cmd =
+        XEN_SYSCTL_PSR_CMT_get_l3_cache_size;
+    sysctl.u.psr_cmt_op.flags = 0;
+    sysctl.u.psr_cmt_op.u.l3_cache.cpu = cpu;
+
+    rc = xc_sysctl(xch, &sysctl);
+    if ( !rc )
+        val = *l3_cache_size= sysctl.u.psr_cmt_op.u.data;
+
+    return rc;
+}
+
+int xc_psr_cmt_get_data(xc_interface *xch, uint32_t rmid,
+    uint32_t cpu, xc_psr_cmt_type type, uint64_t *monitor_data)
+{
+    xc_resource_op_t op;
+    xc_resource_entry_t entries[2];
+    uint32_t evtid;
+    int rc;
+
+    switch ( type )
+    {
+    case XC_PSR_CMT_L3_OCCUPANCY:
+        evtid = EVTID_L3_OCCUPANCY;
+        break;
+    default:
+        return -1;
+    }
+
+    entries[0].u.cmd = XEN_RESOURCE_OP_MSR_WRITE;
+    entries[0].idx = MSR_IA32_CMT_EVTSEL;
+    entries[0].val = (uint64_t)rmid << 32 | evtid;
+    entries[0].rsvd = 0;
+
+    entries[1].u.cmd = XEN_RESOURCE_OP_MSR_READ;
+    entries[1].idx = MSR_IA32_CMT_CTR;
+    entries[1].val = 0;
+    entries[1].rsvd = 0;
+
+    op.cpu = cpu;
+    op.nr_entries = 2;
+    op.entries = entries;
+
+    rc = xc_resource_op(xch, 1, &op);
+    if ( rc < 0 )
+        return rc;
+
+    if ( op.result !=2 || entries[1].val & IA32_CMT_CTR_ERROR_MASK )
+        return -1;
+
+    *monitor_data = entries[1].val;
+
+    return 0;
+}
+
+int xc_psr_cmt_enabled(xc_interface *xch)
+{
+    static int val = -1;
+    int rc;
+    DECLARE_SYSCTL;
+
+    if ( val >= 0 )
+        return val;
+
+    sysctl.cmd = XEN_SYSCTL_psr_cmt_op;
+    sysctl.u.psr_cmt_op.cmd = XEN_SYSCTL_PSR_CMT_enabled;
+    sysctl.u.psr_cmt_op.flags = 0;
+
+    rc = do_sysctl(xch, &sysctl);
+    if ( !rc )
+    {
+        val = sysctl.u.psr_cmt_op.u.data;
+        return val;
+    }
+
+    return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 0bf666f..df08c8a 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -58,7 +58,7 @@ endif
 
 LIBXL_OBJS-y += libxl_remus_device.o libxl_remus_disk_drbd.o
 
-LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o
+LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o
 
 ifeq ($(CONFIG_NetBSD),y)
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 2700cc1..3568e1e 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -659,6 +659,15 @@ typedef uint8_t libxl_mac[6];
 #define LIBXL_MAC_BYTES(mac) mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]
 void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, libxl_mac *src);
 
+#if defined(__i386__) || defined(__x86_64__)
+/*
+ * LIBXL_HAVE_PSR_CMT
+ *
+ * If this is defined, the Cache Monitoring Technology feature is supported.
+ */
+#define LIBXL_HAVE_PSR_CMT 1
+#endif
+
 typedef char **libxl_string_list;
 void libxl_string_list_dispose(libxl_string_list *sl);
 int libxl_string_list_length(const libxl_string_list *sl);
@@ -1400,6 +1409,18 @@ bool libxl_ms_vm_genid_is_zero(const libxl_ms_vm_genid *id);
 void libxl_ms_vm_genid_copy(libxl_ctx *ctx, libxl_ms_vm_genid *dst,
                             libxl_ms_vm_genid *src);
 
+#ifdef LIBXL_HAVE_PSR_CMT
+int libxl_psr_cmt_attach(libxl_ctx *ctx, uint32_t domid);
+int libxl_psr_cmt_detach(libxl_ctx *ctx, uint32_t domid);
+int libxl_psr_cmt_domain_attached(libxl_ctx *ctx, uint32_t domid);
+int libxl_psr_cmt_enabled(libxl_ctx *ctx);
+int libxl_psr_cmt_get_total_rmid(libxl_ctx *ctx, uint32_t *total_rmid);
+int libxl_psr_cmt_get_l3_cache_size(libxl_ctx *ctx, uint32_t socketid,
+    uint32_t *l3_cache_size);
+int libxl_psr_cmt_get_cache_occupancy(libxl_ctx *ctx, uint32_t domid,
+    uint32_t socketid, uint32_t *l3_cache_occupancy);
+#endif
+
 /* misc */
 
 /* Each of these sets or clears the flag according to whether the
diff --git a/tools/libxl/libxl_psr.c b/tools/libxl/libxl_psr.c
new file mode 100644
index 0000000..0437465
--- /dev/null
+++ b/tools/libxl/libxl_psr.c
@@ -0,0 +1,217 @@
+/*
+ * Copyright (C) 2014      Intel Corporation
+ * Author Dongxiao Xu <dongxiao.xu@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+#include "libxl_internal.h"
+
+
+#define IA32_QM_CTR_ERROR_MASK         (0x3ul << 62)
+
+static void libxl__psr_cmt_log_err_msg(libxl__gc *gc, int err)
+{
+    char *msg;
+
+    switch (err) {
+    case ENOSYS:
+        msg = "unsupported operation";
+        break;
+    case ENODEV:
+        msg = "CMT is not supported in this system";
+        break;
+    case EEXIST:
+        msg = "CMT is already attached to this domain";
+        break;
+    case ENOENT:
+        msg = "CMT is not attached to this domain";
+        break;
+    case EUSERS:
+        msg = "no free RMID available";
+        break;
+    case ESRCH:
+        msg = "invalid domain ID";
+        break;
+    case EFAULT:
+        msg = "failed to exchange data with Xen";
+        break;
+    default:
+        msg = "unknown error";
+        break;
+    }
+
+    LOGE(ERROR, "%s", msg);
+}
+
+static int libxl__pick_socket_cpu(libxl__gc *gc, uint32_t socketid)
+{
+    int i, nr_cpus;
+    libxl_cputopology *topology;
+    int cpu = ERROR_FAIL;
+
+    topology = libxl_get_cpu_topology(CTX, &nr_cpus);
+    if (!topology)
+        return ERROR_FAIL;
+
+    for (i = 0; i < nr_cpus; i++)
+        if (topology[i].socket == socketid) {
+            cpu = i;
+            break;
+        }
+
+    libxl_cputopology_list_free(topology, nr_cpus);
+    return cpu;
+}
+
+int libxl_psr_cmt_attach(libxl_ctx *ctx, uint32_t domid)
+{
+    GC_INIT(ctx);
+    int rc;
+
+    rc = xc_psr_cmt_attach(ctx->xch, domid);
+    if (rc < 0) {
+        libxl__psr_cmt_log_err_msg(gc, errno);
+        rc = ERROR_FAIL;
+    }
+
+    GC_FREE;
+    return rc;
+}
+
+int libxl_psr_cmt_detach(libxl_ctx *ctx, uint32_t domid)
+{
+    GC_INIT(ctx);
+    int rc;
+
+    rc = xc_psr_cmt_detach(ctx->xch, domid);
+    if (rc < 0) {
+        libxl__psr_cmt_log_err_msg(gc, errno);
+        rc = ERROR_FAIL;
+    }
+
+    GC_FREE;
+    return rc;
+}
+
+int libxl_psr_cmt_domain_attached(libxl_ctx *ctx, uint32_t domid)
+{
+    int rc;
+    uint32_t rmid;
+
+    rc = xc_psr_cmt_get_domain_rmid(ctx->xch, domid, &rmid);
+    if (rc < 0)
+        return 0;
+
+    return !!rmid;
+}
+
+int libxl_psr_cmt_enabled(libxl_ctx *ctx)
+{
+    return xc_psr_cmt_enabled(ctx->xch);
+}
+
+int libxl_psr_cmt_get_total_rmid(libxl_ctx *ctx, uint32_t *total_rmid)
+{
+    GC_INIT(ctx);
+    int rc;
+
+    rc = xc_psr_cmt_get_total_rmid(ctx->xch, total_rmid);
+    if (rc < 0) {
+        libxl__psr_cmt_log_err_msg(gc, errno);
+        rc = ERROR_FAIL;
+    }
+
+    GC_FREE;
+    return rc;
+}
+
+int libxl_psr_cmt_get_l3_cache_size(libxl_ctx *ctx, uint32_t socketid,
+                                         uint32_t *l3_cache_size)
+{
+    GC_INIT(ctx);
+
+    int rc;
+    int cpu = libxl__pick_socket_cpu(gc, socketid);
+
+    if (cpu < 0) {
+        LOGE(ERROR, "failed to get socket cpu");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    rc = xc_psr_cmt_get_l3_cache_size(ctx->xch, cpu, l3_cache_size);
+    if (rc < 0) {
+        libxl__psr_cmt_log_err_msg(gc, errno);
+        rc = ERROR_FAIL;
+    }
+
+out:
+    GC_FREE;
+    return rc;
+}
+
+int libxl_psr_cmt_get_cache_occupancy(libxl_ctx *ctx, uint32_t domid,
+    uint32_t socketid, uint32_t *l3_cache_occupancy)
+{
+    GC_INIT(ctx);
+
+    unsigned int rmid;
+    uint32_t upscaling_factor;
+    uint64_t monitor_data;
+    int cpu, rc;
+    xc_psr_cmt_type type;
+
+    rc = xc_psr_cmt_get_domain_rmid(ctx->xch, domid, &rmid);
+    if (rc < 0 || rmid == 0) {
+        LOGE(ERROR, "fail to get the domain rmid, "
+            "or domain is not attached with platform QoS monitoring service");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    cpu = libxl__pick_socket_cpu(gc, socketid);
+    if (cpu < 0) {
+        LOGE(ERROR, "failed to get socket cpu");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    type = XC_PSR_CMT_L3_OCCUPANCY;
+    rc = xc_psr_cmt_get_data(ctx->xch, rmid, cpu, type, &monitor_data);
+    if (rc < 0) {
+        LOGE(ERROR, "failed to get monitoring data");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    rc = xc_psr_cmt_get_l3_upscaling_factor(ctx->xch, &upscaling_factor);
+    if (rc < 0) {
+        LOGE(ERROR, "failed to get L3 upscaling factor");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    *l3_cache_occupancy = upscaling_factor * monitor_data / 1024;
+    rc = 0;
+out:
+    GC_FREE;
+    return rc;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index bbb03e2..df572c7 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -652,3 +652,7 @@ libxl_event = Struct("event",[
                                  ])),
            ("domain_create_console_available", None),
            ]))])
+
+libxl_psr_cmt_type = Enumeration("psr_cmt_type", [
+    (1, "CACHE_OCCUPANCY"),
+    ])
diff --git a/tools/libxl/xl.h b/tools/libxl/xl.h
index 6c7aa8e..378245d 100644
--- a/tools/libxl/xl.h
+++ b/tools/libxl/xl.h
@@ -111,6 +111,11 @@ int main_loadpolicy(int argc, char **argv);
 int main_remus(int argc, char **argv);
 #endif
 int main_devd(int argc, char **argv);
+#ifdef LIBXL_HAVE_PSR_CMT
+int main_psr_cmt_attach(int argc, char **argv);
+int main_psr_cmt_detach(int argc, char **argv);
+int main_psr_cmt_show(int argc, char **argv);
+#endif
 
 void help(const char *command);
 
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index c734f79..988ee28 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -7626,6 +7626,166 @@ out:
     return ret;
 }
 
+#ifdef LIBXL_HAVE_PSR_CMT
+static void psr_cmt_print_domain_cache_occupancy(libxl_dominfo *dominfo,
+                                                    uint32_t nr_sockets)
+{
+    char *domain_name;
+    uint32_t socketid;
+    uint32_t l3_cache_occupancy;
+
+    if (!libxl_psr_cmt_domain_attached(ctx, dominfo->domid))
+        return;
+
+    domain_name = libxl_domid_to_name(ctx, dominfo->domid);
+    printf("%-40s %5d", domain_name, dominfo->domid);
+    free(domain_name);
+
+    for (socketid = 0; socketid < nr_sockets; socketid++) {
+        if ( !libxl_psr_cmt_get_cache_occupancy(ctx, dominfo->domid,
+                 socketid, &l3_cache_occupancy) )
+            printf("%13u KB", l3_cache_occupancy);
+    }
+
+    printf("\n");
+}
+
+static int psr_cmt_show_cache_occupancy(uint32_t domid)
+{
+    uint32_t i, socketid, nr_sockets, total_rmid;
+    uint32_t l3_cache_size;
+    libxl_physinfo info;
+    int rc, nr_domains;
+
+    if (!libxl_psr_cmt_enabled(ctx)) {
+        fprintf(stderr, "CMT is disabled in the system\n");
+        return -1;
+    }
+
+    libxl_physinfo_init(&info);
+    rc = libxl_get_physinfo(ctx, &info);
+    if (rc < 0) {
+        fprintf(stderr, "Failed getting physinfo, rc: %d\n", rc);
+        libxl_physinfo_dispose(&info);
+        return -1;
+    }
+    nr_sockets = info.nr_cpus / info.threads_per_core / info.cores_per_socket;
+    libxl_physinfo_dispose(&info);
+
+    rc = libxl_psr_cmt_get_total_rmid(ctx, &total_rmid);
+    if (rc < 0) {
+        fprintf(stderr, "Failed to get max RMID value\n");
+        return -1;
+    }
+
+    printf("Total RMID: %d\n", total_rmid);
+
+    /* Header */
+    printf("%-40s %5s", "Name", "ID");
+    for (socketid = 0; socketid < nr_sockets; socketid++)
+        printf("%14s %d", "Socket", socketid);
+    printf("\n");
+
+    /* Total L3 cache size */
+    printf("%-46s", "Total L3 Cache Size");
+    for (socketid = 0; socketid < nr_sockets; socketid++) {
+        rc = libxl_psr_cmt_get_l3_cache_size(ctx, socketid, &l3_cache_size);
+        if (rc < 0) {
+            fprintf(stderr, "Failed to get system l3 cache size for socket:%d\n",
+                            socketid);
+            return -1;
+        }
+        printf("%13u KB", l3_cache_size);
+    }
+    printf("\n");
+
+    /* Each domain */
+    if (domid != INVALID_DOMID) {
+        libxl_dominfo dominfo;
+        if (!libxl_domain_info(ctx, &dominfo, domid)) {
+            fprintf(stderr, "Failed to get domain info for %d\n", domid);
+            return -1;
+        }
+        psr_cmt_print_domain_cache_occupancy(&dominfo, nr_sockets);
+    }
+    else
+    {
+        libxl_dominfo *list;
+        if (!(list = libxl_list_domain(ctx, &nr_domains))) {
+            fprintf(stderr, "Failed to get domain info for domain list.\n");
+            return -1;
+        }
+        for (i = 0; i < nr_domains; i++)
+            psr_cmt_print_domain_cache_occupancy(list + i, nr_sockets);
+        libxl_dominfo_list_free(list, nr_domains);
+    }
+    return 0;
+}
+
+int main_psr_cmt_attach(int argc, char **argv)
+{
+    uint32_t domid;
+    int opt, ret = 0;
+
+    SWITCH_FOREACH_OPT(opt, "", NULL, "psr-cmt-attach", 1) {
+        /* No options */
+    }
+
+    domid = find_domain(argv[optind]);
+    ret = libxl_psr_cmt_attach(ctx, domid);
+
+    return ret;
+}
+
+int main_psr_cmt_detach(int argc, char **argv)
+{
+    uint32_t domid;
+    int opt, ret = 0;
+
+    SWITCH_FOREACH_OPT(opt, "", NULL, "psr-cmt-detach", 1) {
+        /* No options */
+    }
+
+    domid = find_domain(argv[optind]);
+    ret = libxl_psr_cmt_detach(ctx, domid);
+
+    return ret;
+}
+
+int main_psr_cmt_show(int argc, char **argv)
+{
+    int opt, ret = 0;
+    uint32_t domid;
+    libxl_psr_cmt_type type;
+
+    SWITCH_FOREACH_OPT(opt, "", NULL, "psr-cmt-show", 1) {
+        /* No options */
+    }
+
+    libxl_psr_cmt_type_from_string(argv[optind], &type);
+
+    if (optind + 1 >= argc)
+        domid = INVALID_DOMID;
+    else if (optind + 1 == argc - 1)
+        domid = find_domain(argv[optind + 1]);
+    else {
+        help("psr-cmt-show");
+        return 2;
+    }
+
+    switch (type) {
+    case LIBXL_PSR_CMT_TYPE_CACHE_OCCUPANCY:
+        ret = psr_cmt_show_cache_occupancy(domid);
+        break;
+    default:
+        help("psr-cmt-show");
+        return 2;
+    }
+
+    return ret;
+}
+#endif
+
 /*
  * Local variables:
  * mode: C
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index f93ee4f..8ed8f45 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -517,6 +517,25 @@ struct cmd_spec cmd_table[] = {
       "[options]",
       "-F                      Run in the foreground",
     },
+#ifdef LIBXL_HAVE_PSR_CMT
+    { "psr-cmt-attach",
+      &main_psr_cmt_attach, 0, 1,
+      "Attach Cache Monitoring Technology service to a domain",
+      "<Domain>",
+    },
+    { "psr-cmt-detach",
+      &main_psr_cmt_detach, 0, 1,
+      "Detach Cache Monitoring Technology service from a domain",
+      "<Domain>",
+    },
+    { "psr-cmt-show",
+      &main_psr_cmt_show, 0, 1,
+      "Show Cache Monitoring Technology information",
+      "<PSR-CMT-Type> <Domain>",
+      "Available monitor types:\n"
+      "\"cache_occupancy\":         Show L3 cache occupancy\n",
+    },
+#endif
 };
 
 int cmdtable_len = sizeof(cmd_table)/sizeof(struct cmd_spec);
-- 
1.7.9.5

Re: [PATCH v22] tools: CMDs and APIs for Cache Monitoring Technology

[Wei Liu]

On Thu, Oct 09, 2014 at 12:02:04PM +0800, Chao Peng wrote:
> Introduced some new xl commands to enable/disable Cache Monitoring
> Technology(CMT) feature.
> 
> The following two commands is to attach/detach the CMT feature
> to/from a certain domain.
> 
> $ xl psr-cmt-attach domid
> $ xl psr-cmt-detach domid
> 
> This command is to display the CMT information, such as L3 cache
> occupancy.
> 
> $ xl psr-cmt-show cache_occupancy <domid>
> 
> Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>

Acked-by: Wei Liu <wei.liu2@citrix.com>

In your later contribution please don't forget to carry over any tags
you got during review.

Wei.

Re: [PATCH v22] tools: CMDs and APIs for Cache Monitoring Technology

[Chao Peng]

On Thu, Oct 09, 2014 at 10:09:08AM +0100, Wei Liu wrote:
> On Thu, Oct 09, 2014 at 12:02:04PM +0800, Chao Peng wrote:
> > Introduced some new xl commands to enable/disable Cache Monitoring
> > Technology(CMT) feature.
> > 
> > The following two commands is to attach/detach the CMT feature
> > to/from a certain domain.
> > 
> > $ xl psr-cmt-attach domid
> > $ xl psr-cmt-detach domid
> > 
> > This command is to display the CMT information, such as L3 cache
> > occupancy.
> > 
> > $ xl psr-cmt-show cache_occupancy <domid>
> > 
> > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> > Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>
> 
> Acked-by: Wei Liu <wei.liu2@citrix.com>
> 
> In your later contribution please don't forget to carry over any tags
> you got during review.
> 
> Wei.

I apologize. I know this but still missed it this time.

Thanks for raising again.

Chao

Re: [PATCH v22] tools: CMDs and APIs for Cache Monitoring Technology

[Wei Liu]

On Thu, Oct 09, 2014 at 05:24:20PM +0800, Chao Peng wrote:
> On Thu, Oct 09, 2014 at 10:09:08AM +0100, Wei Liu wrote:
> > On Thu, Oct 09, 2014 at 12:02:04PM +0800, Chao Peng wrote:
> > > Introduced some new xl commands to enable/disable Cache Monitoring
> > > Technology(CMT) feature.
> > > 
> > > The following two commands is to attach/detach the CMT feature
> > > to/from a certain domain.
> > > 
> > > $ xl psr-cmt-attach domid
> > > $ xl psr-cmt-detach domid
> > > 
> > > This command is to display the CMT information, such as L3 cache
> > > occupancy.
> > > 
> > > $ xl psr-cmt-show cache_occupancy <domid>
> > > 
> > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> > > Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>
> > 
> > Acked-by: Wei Liu <wei.liu2@citrix.com>
> > 
> > In your later contribution please don't forget to carry over any tags
> > you got during review.
> > 
> > Wei.
> 
> I apologize. I know this but still missed it this time.
> 

No need to apologise. It's just a gentle reminder. :-)

> Thanks for raising again.
> 
> Chao

Re: [PATCH v22] tools: CMDs and APIs for Cache Monitoring Technology

[Chao Peng]

On Thu, Oct 09, 2014 at 10:09:08AM +0100, Wei Liu wrote:
> On Thu, Oct 09, 2014 at 12:02:04PM +0800, Chao Peng wrote:
> > Introduced some new xl commands to enable/disable Cache Monitoring
> > Technology(CMT) feature.
> > 
> > The following two commands is to attach/detach the CMT feature
> > to/from a certain domain.
> > 
> > $ xl psr-cmt-attach domid
> > $ xl psr-cmt-detach domid
> > 
> > This command is to display the CMT information, such as L3 cache
> > occupancy.
> > 
> > $ xl psr-cmt-show cache_occupancy <domid>
> > 
> > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> > Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>
> 
> Acked-by: Wei Liu <wei.liu2@citrix.com>
> 
> In your later contribution please don't forget to carry over any tags
> you got during review.
> 
> Wei.

So konrad, is it the time to allow this patch in?
You know we are eager to meet the deadline for this :)

Thanks,
Chao

Re: [PATCH v22] tools: CMDs and APIs for Cache Monitoring Technology

[Ian Campbell]

On Fri, 2014-10-10 at 14:29 +0800, Chao Peng wrote:
> On Thu, Oct 09, 2014 at 10:09:08AM +0100, Wei Liu wrote:
> > On Thu, Oct 09, 2014 at 12:02:04PM +0800, Chao Peng wrote:
> > > Introduced some new xl commands to enable/disable Cache Monitoring
> > > Technology(CMT) feature.
> > > 
> > > The following two commands is to attach/detach the CMT feature
> > > to/from a certain domain.
> > > 
> > > $ xl psr-cmt-attach domid
> > > $ xl psr-cmt-detach domid
> > > 
> > > This command is to display the CMT information, such as L3 cache
> > > occupancy.
> > > 
> > > $ xl psr-cmt-show cache_occupancy <domid>
> > > 
> > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> > > Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>
> > 
> > Acked-by: Wei Liu <wei.liu2@citrix.com>
> > 
> > In your later contribution please don't forget to carry over any tags
> > you got during review.
> > 
> > Wei.
> 
> So konrad, is it the time to allow this patch in?

Has Konrad not already acked this series? Given that the hypervisor side
went in earlier in the week I was going to assume so (i.e. I even tried
to apply v21) and just go ahead.

Ian.

Re: [PATCH v22] tools: CMDs and APIs for Cache Monitoring Technology

[Jan Beulich]

>>> On 10.10.14 at 10:26, <Ian.Campbell@citrix.com> wrote:
> On Fri, 2014-10-10 at 14:29 +0800, Chao Peng wrote:
>> On Thu, Oct 09, 2014 at 10:09:08AM +0100, Wei Liu wrote:
>> > On Thu, Oct 09, 2014 at 12:02:04PM +0800, Chao Peng wrote:
>> > > Introduced some new xl commands to enable/disable Cache Monitoring
>> > > Technology(CMT) feature.
>> > > 
>> > > The following two commands is to attach/detach the CMT feature
>> > > to/from a certain domain.
>> > > 
>> > > $ xl psr-cmt-attach domid
>> > > $ xl psr-cmt-detach domid
>> > > 
>> > > This command is to display the CMT information, such as L3 cache
>> > > occupancy.
>> > > 
>> > > $ xl psr-cmt-show cache_occupancy <domid>
>> > > 
>> > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
>> > > Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>
>> > 
>> > Acked-by: Wei Liu <wei.liu2@citrix.com>
>> > 
>> > In your later contribution please don't forget to carry over any tags
>> > you got during review.
>> > 
>> > Wei.
>> 
>> So konrad, is it the time to allow this patch in?
> 
> Has Konrad not already acked this series? Given that the hypervisor side
> went in earlier in the week I was going to assume so (i.e. I even tried
> to apply v21) and just go ahead.

Iirc he explicitly excluded this patch from his ack, since he wants
maintainer acks to be in place before giving his.

Jan

Re: [PATCH v22] tools: CMDs and APIs for Cache Monitoring Technology

[Ian Campbell]

On Fri, 2014-10-10 at 09:31 +0100, Jan Beulich wrote:
> >>> On 10.10.14 at 10:26, <Ian.Campbell@citrix.com> wrote:
> > On Fri, 2014-10-10 at 14:29 +0800, Chao Peng wrote:
> >> On Thu, Oct 09, 2014 at 10:09:08AM +0100, Wei Liu wrote:
> >> > On Thu, Oct 09, 2014 at 12:02:04PM +0800, Chao Peng wrote:
> >> > > Introduced some new xl commands to enable/disable Cache Monitoring
> >> > > Technology(CMT) feature.
> >> > > 
> >> > > The following two commands is to attach/detach the CMT feature
> >> > > to/from a certain domain.
> >> > > 
> >> > > $ xl psr-cmt-attach domid
> >> > > $ xl psr-cmt-detach domid
> >> > > 
> >> > > This command is to display the CMT information, such as L3 cache
> >> > > occupancy.
> >> > > 
> >> > > $ xl psr-cmt-show cache_occupancy <domid>
> >> > > 
> >> > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> >> > > Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>
> >> > 
> >> > Acked-by: Wei Liu <wei.liu2@citrix.com>
> >> > 
> >> > In your later contribution please don't forget to carry over any tags
> >> > you got during review.
> >> > 
> >> > Wei.
> >> 
> >> So konrad, is it the time to allow this patch in?
> > 
> > Has Konrad not already acked this series? Given that the hypervisor side
> > went in earlier in the week I was going to assume so (i.e. I even tried
> > to apply v21) and just go ahead.
> 
> Iirc he explicitly excluded this patch from his ack, since he wants
> maintainer acks to be in place before giving his.

Ah, ok. I'll wait then


Ian.

Re: [PATCH v22] tools: CMDs and APIs for Cache Monitoring Technology

[Wei Liu]

On Fri, Oct 10, 2014 at 09:37:24AM +0100, Ian Campbell wrote:
> On Fri, 2014-10-10 at 09:31 +0100, Jan Beulich wrote:
> > >>> On 10.10.14 at 10:26, <Ian.Campbell@citrix.com> wrote:
> > > On Fri, 2014-10-10 at 14:29 +0800, Chao Peng wrote:
> > >> On Thu, Oct 09, 2014 at 10:09:08AM +0100, Wei Liu wrote:
> > >> > On Thu, Oct 09, 2014 at 12:02:04PM +0800, Chao Peng wrote:
> > >> > > Introduced some new xl commands to enable/disable Cache Monitoring
> > >> > > Technology(CMT) feature.
> > >> > > 
> > >> > > The following two commands is to attach/detach the CMT feature
> > >> > > to/from a certain domain.
> > >> > > 
> > >> > > $ xl psr-cmt-attach domid
> > >> > > $ xl psr-cmt-detach domid
> > >> > > 
> > >> > > This command is to display the CMT information, such as L3 cache
> > >> > > occupancy.
> > >> > > 
> > >> > > $ xl psr-cmt-show cache_occupancy <domid>
> > >> > > 
> > >> > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> > >> > > Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>
> > >> > 
> > >> > Acked-by: Wei Liu <wei.liu2@citrix.com>
> > >> > 
> > >> > In your later contribution please don't forget to carry over any tags
> > >> > you got during review.
> > >> > 
> > >> > Wei.
> > >> 
> > >> So konrad, is it the time to allow this patch in?
> > > 
> > > Has Konrad not already acked this series? Given that the hypervisor side
> > > went in earlier in the week I was going to assume so (i.e. I even tried
> > > to apply v21) and just go ahead.
> > 
> > Iirc he explicitly excluded this patch from his ack, since he wants
> > maintainer acks to be in place before giving his.
> 
> Ah, ok. I'll wait then
> 

See <20141008124917.GD13391@laptop.dumpdata.com>

I think he implicitly acked this patch.

Wei.

> 
> Ian.

Re: [PATCH v22] tools: CMDs and APIs for Cache Monitoring Technology

[Konrad Rzeszutek Wilk]

On Fri, Oct 10, 2014 at 09:56:17AM +0100, Wei Liu wrote:
> On Fri, Oct 10, 2014 at 09:37:24AM +0100, Ian Campbell wrote:
> > On Fri, 2014-10-10 at 09:31 +0100, Jan Beulich wrote:
> > > >>> On 10.10.14 at 10:26, <Ian.Campbell@citrix.com> wrote:
> > > > On Fri, 2014-10-10 at 14:29 +0800, Chao Peng wrote:
> > > >> On Thu, Oct 09, 2014 at 10:09:08AM +0100, Wei Liu wrote:
> > > >> > On Thu, Oct 09, 2014 at 12:02:04PM +0800, Chao Peng wrote:
> > > >> > > Introduced some new xl commands to enable/disable Cache Monitoring
> > > >> > > Technology(CMT) feature.
> > > >> > > 
> > > >> > > The following two commands is to attach/detach the CMT feature
> > > >> > > to/from a certain domain.
> > > >> > > 
> > > >> > > $ xl psr-cmt-attach domid
> > > >> > > $ xl psr-cmt-detach domid
> > > >> > > 
> > > >> > > This command is to display the CMT information, such as L3 cache
> > > >> > > occupancy.
> > > >> > > 
> > > >> > > $ xl psr-cmt-show cache_occupancy <domid>
> > > >> > > 
> > > >> > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> > > >> > > Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com>
> > > >> > 
> > > >> > Acked-by: Wei Liu <wei.liu2@citrix.com>
> > > >> > 
> > > >> > In your later contribution please don't forget to carry over any tags
> > > >> > you got during review.
> > > >> > 
> > > >> > Wei.
> > > >> 
> > > >> So konrad, is it the time to allow this patch in?
> > > > 
> > > > Has Konrad not already acked this series? Given that the hypervisor side
> > > > went in earlier in the week I was going to assume so (i.e. I even tried
> > > > to apply v21) and just go ahead.
> > > 
> > > Iirc he explicitly excluded this patch from his ack, since he wants
> > > maintainer acks to be in place before giving his.
> > 
> > Ah, ok. I'll wait then
> > 
> 
> See <20141008124917.GD13391@laptop.dumpdata.com>
> 
> I think he implicitly acked this patch.

I was waiting for Wei or Ian's Ack before giving it an Ack.

And since that has been given (see above):

Release-Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>

Thank you!
> 
> Wei.
> 
> > 
> > Ian.

Re: [PATCH v22] tools: CMDs and APIs for Cache Monitoring Technology

[Chao Peng]

> I was waiting for Wei or Ian's Ack before giving it an Ack.
> 
> And since that has been given (see above):
> 
> Release-Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> 
> Thank you!

Thank you guys. It's all of you making this happen.

Actually I thought it was almost done when I took over this patch serial
from Dongxiao. But you guys still made several 'good catches' as well as
abundant other comments afterwards. I'm impressed with your strictness
on code but you guys also never hesitated to give me suggestion and
guidance.

As a newbie to XEN, I leant much from the discussion with you guys.

Special thanks to Jan and Andrew, you reviewed almost every new versions
for this patch serial. It took lot of your time I know.  I just feel sorry
for this and I think this is what we can improve in the future.

Regards,
Chao

Re: [PATCH v22] tools: CMDs and APIs for Cache Monitoring Technology

[Ian Campbell]

On Mon, 2014-10-13 at 17:15 +0800, Chao Peng wrote:
> > I was waiting for Wei or Ian's Ack before giving it an Ack.
> > 
> > And since that has been given (see above):
> > 
> > Release-Acked-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > 
> > Thank you!
> 
> Thank you guys. It's all of you making this happen.

Applied, thanks!

Ian.