Re: [PATCH v2 01/15] docs: create Memory Bandwidth Allocation (MBA) feature document

[Yi Sun <yi.y.sun@linux.intel.com>]

Thanks a lot for the review comments!

On 17-08-29 12:46:49, Roger Pau Monn� wrote:
> On Thu, Aug 24, 2017 at 09:14:35AM +0800, Yi Sun wrote:
> > +# Overview
> > +
> > +The Memory Bandwidth Allocation (MBA) feature provides indirect and approximate
> > +control over memory bandwidth available per-core. This feature provides OS/
> > +hypervisor the ability to slow misbehaving apps/domains or create advanced
> > +closed-loop control system via exposing control over a credit-based throttling
> > +mechanism.
> 
> I don't really understand what "advanced closed-loop control system
> via exposing..." means. From my understand it is clear/simpler to
> write it as:
> 
> "... the ability to slow misbehaving apps/domains by using a
> credit-based throttling mechanism".
> 
Thanks, 'closed-loop' looks redundant, will remove it.

> > +
> > +# User details
> > +
> > +* Feature Enabling:
> > +
> > +  Add "psr=mba" to boot line parameter to enable MBA feature.
> > +
> > +* xl interfaces:
> > +
> > +  1. `psr-mba-show [domain-id]`:
> > +
> > +     Show memory bandwidth throttling for domain. For linear mode, it shows the
> > +     decimal value. For non-linear mode, it shows hexadecimal value.
> 
> You should first explain what are the linear and non-linear modes.
> 
Ok, will move below linear/non-linear modes explanation to here.

> > +
> > +  2. `psr-mba-set [OPTIONS] <domain-id> <throttling>`:
> > +
> > +     Set memory bandwidth throttling for domain.
> > +
> > +     Options:
> > +     '-s': Specify the socket to process, otherwise all sockets are processed.
> > +
> > +     Throttling value set in register implies memory bandwidth blocked, i.e.
> > +     higher throttling value results in lower bandwidth. The max throttling
> > +     value can be got through CPUID.
> 
> This is also hard to understand IMHO. I would rather write it as:
> 
> "The throttling value describes the amount of blocked bandwidth".

Thanks! This is not accurate. In fact, throttling value means the approximate
amount of delaying the traffic between core and memory.

> Although I have to admit I don't really understand this interface,
> wouldn't it be easier to specify the memory bandwidth allowed
> per-domain, rather the amount of bandwidth removed?
> 
> Using your approach the user has to first get the total bandwidth, and
> then subtract the removed bandwidth in order to know the remaining
> bandwidth for a domain.
> 
The HW only provides throttling set method to control the bandwidth. So, I
think it is straightforward to set throttling in tools layer. The 'psr-mba-set'
is designed as a simple command to set what HW needs.

Also, mentioned by SDM, "The throttling values exposed by MBA are approximate,
and are calibrated to specific traffic patterns.". So, it is hard to provide
exact bandwidth control in 'psr-mba-set'.

> Also, IMHO you should provide a command to print the max throttling,

The 'psr-hwinfo' can show the max throttling. Because it is part of MBA HW info.

> remember that from Xen's PoV Dom0 is just another domain, and the
> CPUID values reported to Dom0 don't need to be the same as found on
> bare metal.
> 
But the CPUID values got through 'psr' commands should be ones found on bare
metal, right? Because these commands directly get the values from hypervisor
through domctl/sysctl.

> > +
> > +     The response of the throttling value could be linear mode or non-linear
> > +     mode.
> > +
> > +     Linear mode: the input precision is defined as 100-(MBA_MAX). For instance,
> 
> What's MBA_MAX? I don't see any reference/description of it above.
> 
Sorry, will explain it.

> > +     if the MBA_MAX value is 90, the input precision is 10%. Values not an even
> > +     multiple of the precision (e.g., 12%) will be rounded down (e.g., to 10%
> > +     delay applied) by HW automatically.
> > +
> > +     Non-linear mode: input delay values are powers-of-two from zero to the
> > +     MBA_MAX value from CPUID. In this case any values not a power of two will
> > +     be rounded down the next nearest power of two by HW automatically.
> > +
> > +# Technical details
> > +
> > +MBA is a member of Intel PSR features, it shares the base PSR infrastructure
> > +in Xen.
> > +
> > +## Hardware perspective
> > +
> > +  MBA defines a range of MSRs to support specifying a delay value (Thrtl) per
> > +  COS, with details below.
> > +
> > +  ```
> > +   +----------------------------+----------------+
> > +   | MSR (per socket)           |    Address     |
> > +   +----------------------------+----------------+
> > +   | IA32_L2_QOS_Ext_BW_Thrtl_0 |     0xD50      |
> > +   +----------------------------+----------------+
> > +   | ...                        |  ...           |
> > +   +----------------------------+----------------+
> > +   | IA32_L2_QOS_Ext_BW_Thrtl_n | 0xD50+n (n<64) |
> > +   +----------------------------+----------------+
> > +  ```
> 
> Are you sure you want to hardcode this n<64? Isn't there a chance this
> is going to be bumped in newer hardware?
> 
This is just a HW limitation declared in SDM. In fact, there is no such hard
code limitation. Hypervisor side checks the 'cos_max' got through CPUID.

> > +
> > +  When context switch happens, the COS ID of domain is written to per-thread MSR
> > +  `IA32_PQR_ASSOC`, and then hardware enforces bandwidth allocation according
> > +  to the throttling value stored in the Thrtl MSR register.
> > +
> > +## The relationship between MBA and CAT/CDP
> > +
> > +  Generally speaking, MBA is completely independent of CAT/CDP, and any
> > +  combination may be applied at any time, e.g. enabling MBA with CAT
> > +  disabled.
> > +
> > +  But it needs to be noticed that MBA shares COS infrastructure with CAT,
> > +  although MBA is enumerated by different CPUID leaf from CAT (which
> > +  indicates that the max COS of MBA may be different from CAT). In some
> > +  cases, a domain is permitted to have a COS that is beyond one (or more)
> > +  of PSR features but within the others. For instance, let's assume the max
> > +  COS of MBA is 8 but the max COS of L3 CAT is 16, when a domain is assigned
> > +  9 as COS, the L3 CAT CBM associated to COS 9 would be enforced, but for MBA,
> > +  the HW works as default value is set since COS 9 is beyond the max COS (8)
> > +  of MBA.
> > +
> > +## Design Overview
> > +
> > +* Core COS/Thrtl association
> > +
> > +  When enforcing Memory Bandwidth Allocation, all cores of domains have
> > +  the same default Thrtl MSR (COS0) which stores the same Thrtl (0). The
> > +  default Thrtl MSR is used only in hypervisor and is transparent to tool stack
> > +  and user.
> > +
> > +  System administrator can change PSR allocation policy at runtime by
>                         ^s
> > +  tool stack. Since MBA shares COS ID with CAT/CDP, a COS ID corresponds to a
>      ^ using the tool ...

Will modify this and below words.

> > +  2-tuple, like [CBM, Thrtl] with only-CAT enalbed, when CDP is enabled,
>                                               ^ enabled
> > +  the COS ID corresponds to a 3-tuple, like [Code_CBM, Data_CBM, Thrtl]. If
> > +  neither CAT nor CDP is enabled, things would be easier, one COS ID corresponds
>                                             ^ are easier, since one...
> > +  to one Thrtl.
> > +
> > +* VCPU schedule
> > +
> > +  This part reuses CAT COS infrastructure.
> > +
> > +* Multi-sockets
> > +
> > +  Different sockets may have different MBA ability (like max COS)
> > +  although it is consistent on the same socket. So the capability
> > +  of per-socket MBA is specified.
> > +
> > +  This part reuses CAT COS infrastructure.
> > +
> > +## Implementation Description
> > +
> > +* Hypervisor interfaces:
> > +
> > +  1. Boot line param: "psr=mba" to enable the feature.
> > +
> > +  2. SYSCTL:
> > +          - XEN_SYSCTL_PSR_MBA_get_info: Get system MBA information.
> > +
> > +  3. DOMCTL:
> > +          - XEN_DOMCTL_PSR_MBA_OP_GET_THRTL: Get throttling for a domain.
> > +          - XEN_DOMCTL_PSR_MBA_OP_SET_THRTL: Set throttling for a domain.
> > +
> > +* xl interfaces:
> > +
> > +  1. psr-mba-show [domain-id]
> > +          Show system/domain runtime MBA throttling value. For linear mode,
> > +          it shows the decimal value. For non-linear mode, it shows hexadecimal
> > +          value.
> > +          => XEN_SYSCTL_PSR_MBA_get_info/XEN_DOMCTL_PSR_MBA_OP_GET_THRTL
> > +
> > +  2. psr-mba-set [OPTIONS] <domain-id> <throttling>
> > +          Set bandwidth throttling for a domain.
> > +          => XEN_DOMCTL_PSR_MBA_OP_SET_THRTL
> > +
> > +  3. psr-hwinfo
> > +          Show PSR HW information, including L3 CAT/CDP/L2 CAT/MBA.
> > +          => XEN_SYSCTL_PSR_MBA_get_info
> > +
> > +* Key data structure:
> > +
> > +  1. Feature HW info
> > +
> > +     ```
> > +     struct {
> > +         unsigned int thrtl_max;
> > +         unsigned int linear;
> > +     } mba_info;
> 
> Is this a domctl structure? a libxl one?
> 
Nope, this is a hypervisor side data structure used in 'psr.c'.

> > +
> > +     - Member `thrtl_max`
> > +
> > +       `thrtl_max` is the max throttling value to be set.
> > +
> > +     - Member `linear`
> > +
> > +       `linear` means the response of delay value is linear or not.
> > +
> > +     As mentioned above, MBA is a member of Intel PSR features, it would
> > +     share the base PSR infrastructure in Xen. For example, the 'cos_max'
> > +     is a common HW property for all features. So, for other data structure
> > +     details, please refer 'intel_psr_cat_cdp.pandoc'.
> > +
> > +# Limitations
> > +
> > +MBA can only work on HW which enables it (check by CPUID).
> > +
> > +# Testing
> > +
> > +We can execute these commands to verify MBA on different HWs supporting them.
> > +
> > +For example:
> > +    root@:~$ xl psr-hwinfo --mba
> > +    Memory Bandwidth Allocation (MBA):
> > +    Socket ID       : 0
> > +    Linear Mode     : Enabled
> > +    Maximum COS     : 7
> > +    Maximum Throttling Value: 90
> > +    Default Throttling Value: 0
> > +
> > +    root@:~$ xl psr-mba-set 1 0xa
> 
> Could you elaborate a little bit on why '0xa' is used here? IMHO The
> example should provide some context.
> 
Sure, I will explain the meaning of '0xa' or '10' here.

> Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel