[RFC 01/16] docs: create Memory Bandwidth Allocation (MBA) feature document.

[Yi Sun <yi.y.sun@linux.intel.com>]

This patch creates MBA feature document in doc/features/. It
describes details for MBA.

Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com>
---
 docs/features/intel_psr_mba.pandoc | 226 +++++++++++++++++++++++++++++++++++++
 1 file changed, 226 insertions(+)
 create mode 100644 docs/features/intel_psr_mba.pandoc

diff --git a/docs/features/intel_psr_mba.pandoc b/docs/features/intel_psr_mba.pandoc
new file mode 100644
index 0000000..8c04f01
--- /dev/null
+++ b/docs/features/intel_psr_mba.pandoc
@@ -0,0 +1,226 @@
+% Intel Memory Bandwidth Allocation (MBA) Feature
+% Revision 1.0
+
+\clearpage
+
+# Basics
+
+---------------- ----------------------------------------------------
+         Status: **Tech Preview**
+
+Architecture(s): Intel x86
+
+   Component(s): Hypervisor, toolstack
+
+       Hardware: MBA is supported on Skylake Server and beyond
+---------------- ----------------------------------------------------
+
+# Overview
+
+The Memory Bandwidth Allocation (MBA) feature provides indirect and approximate
+control over memory bandwidth available per-core. This feature provides OS/VMMs
+the ability to slow misbehaving apps/VMs or create advanced closed-loop control
+system via exposing control over a credit-based throttling mechanism.
+
+## Terminology
+
+* CAT         Cache Allocation Technology
+* COS/CLOS    Class of Service
+* MSRs        Machine Specific Registers
+* PSR         Intel Platform Shared Resource
+* VMM         Virtual Machine Monitor
+* THRTL       Throttle value or delay value
+
+# User details
+
+* Feature Enabling:
+
+  Add "psr=mba" to boot line parameter to enable MBA feature.
+
+* xl interfaces:
+
+  1. `psr-mba-show [domain-id]`:
+
+     Show system/domain MBA information.
+
+  2. `psr-mba-set [OPTIONS] domain-id throttling`:
+
+     Set memory bandwidth throttling for domain.
+
+     Options:
+     '-s': Specify the socket to process, otherwise all sockets are processed.
+
+     Throttling value set in register implies memory bandwidth blocked, i.e.
+     higher throttling value results in lower bandwidth. The max throttling
+     value can be got through CPUID.
+
+     The response of the throttling value could be linear mode or non-linear
+     mode.
+
+     Linear mode: the input precision is defined as 100-(MBA_MAX). For instance,
+     if the MBA_MAX value is 90, the input precision is 10%. Values not an even
+     multiple of the precision (e.g., 12%) will be rounded down (e.g., to 10%
+     delay applied) by HW automatically.
+
+     Non-linear mode: input delay values are powers-of-two from zero to the
+     MBA_MAX value from CPUID. In this case any values not a power of two will
+     be rounded down the next nearest power of two by HW automatically.
+
+# Technical details
+
+MBA is a member of Intel PSR features, it would share some base PSR
+infrastructure in Xen.
+
+## Hardware perspective
+
+MBA provides an architectural consistent method to map cores’ to a Class
+of Service (COS). This infrastructure will be shared with the previously
+introduced CAT technologies.
+
+Furthermore, MBA also defines a new range MSRs to support specifying a
+delay value (Thrtl) per COS, with details below.
+
++----------------------------+----------------+
+| MSR (per socket)           |    Address     |
++----------------------------+----------------+
+| IA32_L2_QOS_Ext_BW_Thrtl_0 |     0xD50      |
++----------------------------+----------------+
+| ...                        |  ...           |
++----------------------------+----------------+
+| IA32_L2_QOS_Ext_BW_Thrtl_n | 0xD50+n (n<64) |
++----------------------------+----------------+
+
+When context switch happens, the COS of VCPU is written to per-thread
+MSR `IA32_PQR_ASSOC`, and then hardware enforces bandwidth allocation
+according to the throttling value corresponding to the COS.
+
+## The relationship between MBA and CAT/CDP
+
+Generally speaking, MBA is completely independent of CAT/CDP, and any
+combination may be applied at any time, e.g. enabling MBA with CAT 
+disabled.
+
+But it needs to be noticed that MBA shares COS infrastructure with CAT,
+although MBA is enumerated by different CPUID leaf from CAT (which
+indicates that the max COS of MBA may be different from CAT).
+
+## Design Overview
+
+* Core COS/Thrtl association
+
+  When enforcing Memory Bandwidth Allocation, all cores of domains have
+  the same default COS (COS0) which correspond to the same Thrtl (0).
+  The default COS is used only in hypervisor and is transparent to tool
+  stack and user.
+
+  System administrator can change PSR allocation policy at runtime by
+  tool stack. Since MBA shares COS with CAT/CDP, a COS corresponds to a
+  2-tuple, like [CBM, Thrtl] with only-CAT enalbed, when CDP is enable,
+  the COS corresponds to a 3-tuple, like [Code_CBM, Data_CBM, Thrtl]. If
+  neither CAT nor CDP is enabled, things would be easier, one COS
+  corresponds to one Thrtl.
+
+* VCPU schedule
+
+  This part reuses CAT COS infrastructure.
+
+* Multi-sockets
+
+  Different sockets may have different MBA ability (like max COS)
+  although it is consistent on the same socket. So the capability
+  of per-socket MBA is specified.
+
+## Implementation Description
+
+* Hypervisor interfaces:
+
+  1. Boot line param: "psr=mba" to enable the feature.
+
+  2. SYSCTL:
+          - XEN_SYSCTL_PSR_MBA_get_info: Get system MBA information.
+
+  3. DOMCTL:
+          - XEN_DOMCTL_PSR_MBA_OP_GET_THRTL: Get Throttling for a domain.
+          - XEN_DOMCTL_PSR_MBA_OP_SET_THRTL: Set Throttling for a domain.
+
+* xl interfaces:
+
+  1. psr-mba-show [domain-id]
+          Show system/runtime MBA information.
+          => XEN_SYSCTL_PSR_MBA_get_info/XEN_DOMCTL_PSR_MBA_OP_GET_THRTL
+
+  2. psr-mba-set [OPTIONS] domain-id throttling
+          Set bandwidth throttling for a domain.
+          => XEN_DOMCTL_PSR_MBA_OP_SET_THRTL
+
+* Key data structure:
+
+  1. Feature HW info
+
+     ```
+     struct psr_mba_hw_info {
+         unsigned int thrtl_max;
+         unsigned int cos_max;
+         unsigned int linear;
+     };
+
+     - Member `thrtl_max`
+
+       `thrtl_max` is the max throttling value to be set.
+
+     - Member `cos_max`
+
+       `cos_max` is one of the hardware info of CAT.
+
+     - Member `linear`
+
+       `thrtl_max` means the response of delay value is linear or not.
+
+     As mentioned above, MBA is a member of Intel PSR features, it would
+     share some base PSR infrastructure in Xen. So, for other data structure
+     details, please refer 'intel_psr_l2_cat.pandoc'.
+
+# Limitations
+
+MBA can only work on HW which enables it (check by CPUID).
+
+# Testing
+
+We can execute these commands to verify MBA on different HWs supporting them.
+
+For example:
+    root@:~$ xl psr-hwinfo --mba
+    Memory Bandwidth Allocation (MBA):
+    Socket ID       : 0
+    Linear Mode     : Enabled
+    Maximum COS     : 7
+    Maximum Throttling Value: 90
+    Default Throttling Value: 0
+
+    root@:~$ xl psr-mba-set 1 0xa
+
+    root@:~$ xl psr-mba-show 1
+    Socket ID       : 0
+    Default THRTL   : 0
+       ID                     NAME             CBM
+        1                 ubuntu14             0xa
+
+# Areas for improvement
+
+N/A
+
+# Known issues
+
+N/A
+
+# References
+
+"INTEL® RESOURCE DIRECTOR TECHNOLOGY (INTEL® RDT) ALLOCATION FEATURES" [Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html)
+
+# History
+
+------------------------------------------------------------------------
+Date       Revision Version  Notes
+---------- -------- -------- -------------------------------------------
+2017-01-10 1.0      Xen 4.9  Design document written
+---------- -------- -------- -------------------------------------------
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel