From: Tony Luck <tony.luck@intel.com>
To: Fenghua Yu <fenghua.yu@intel.com>,
Reinette Chatre <reinette.chatre@intel.com>,
Peter Newman <peternewman@google.com>,
Jonathan Corbet <corbet@lwn.net>,
Shuah Khan <skhan@linuxfoundation.org>,
x86@kernel.org
Cc: Shaopeng Tan <tan.shaopeng@fujitsu.com>,
James Morse <james.morse@arm.com>,
Jamie Iles <quic_jiles@quicinc.com>,
Babu Moger <babu.moger@amd.com>,
Randy Dunlap <rdunlap@infradead.org>,
Drew Fustini <dfustini@baylibre.com>,
linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org,
patches@lists.linux.dev, Tony Luck <tony.luck@intel.com>
Subject: [PATCH v15 8/8] x86/resctrl: Update documentation with Sub-NUMA cluster changes
Date: Wed, 28 Feb 2024 11:37:03 -0800 [thread overview]
Message-ID: <20240228193717.8170-10-tony.luck@intel.com> (raw)
In-Reply-To: <20240228112935.8087-tony.luck@intel.com>
With Sub-NUMA Cluster mode enabled the scope of monitoring resources is
per-NODE instead of per-L3 cache. Suffixes of directories with "L3" in
their name refer to Sub-NUMA nodes instead of L3 cache ids.
Users should be aware that SNC mode also affects the amount of L3 cache
available for allocation within each SNC node.
Signed-off-by: Tony Luck <tony.luck@intel.com>
---
Documentation/arch/x86/resctrl.rst | 25 +++++++++++++++++++++----
1 file changed, 21 insertions(+), 4 deletions(-)
diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
index a6279df64a9d..15f1cff6ee76 100644
--- a/Documentation/arch/x86/resctrl.rst
+++ b/Documentation/arch/x86/resctrl.rst
@@ -366,10 +366,10 @@ When control is enabled all CTRL_MON groups will also contain:
When monitoring is enabled all MON groups will also contain:
"mon_data":
- This contains a set of files organized by L3 domain and by
- RDT event. E.g. on a system with two L3 domains there will
- be subdirectories "mon_L3_00" and "mon_L3_01". Each of these
- directories have one file per event (e.g. "llc_occupancy",
+ This contains a set of files organized by L3 domain or by NUMA
+ node (depending on whether Sub-NUMA Cluster (SNC) mode is disabled
+ or enabled respectively) and by RDT event. Each of these
+ directories has one file per event (e.g. "llc_occupancy",
"mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
files provide a read out of the current value of the event for
all tasks in the group. In CTRL_MON groups these files provide
@@ -478,6 +478,23 @@ if non-contiguous 1s value is supported. On a system with a 20-bit mask
each bit represents 5% of the capacity of the cache. You could partition
the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.
+Notes on Sub-NUMA Cluster mode
+==============================
+When SNC mode is enabled, Linux may load balance tasks between Sub-NUMA
+nodes much more readily than between regular NUMA nodes since the CPUs
+on Sub-NUMA nodes share the same L3 cache and the system may report
+the NUMA distance between Sub-NUMA nodes with a lower value than used
+for regular NUMA nodes. Users who do not bind tasks to the CPUs of a
+specific Sub-NUMA node must read the "llc_occupancy", "mbm_total_bytes",
+and "mbm_local_bytes" for all Sub-NUMA nodes where the tasks may execute
+to get the full view of traffic for which the tasks were the source.
+
+The cache allocation feature still provides the same number of
+bits in a mask to control allocation into the L3 cache, but each
+of those ways has its capacity reduced because the cache is divided
+between the SNC nodes. The values reported in the resctrl
+"size" files are adjusted accordingly.
+
Memory bandwidth Allocation and monitoring
==========================================
--
2.43.0
next prev parent reply other threads:[~2024-02-28 19:37 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-28 19:36 Cover-cover letter for two resctrl patch sets Tony Luck
2024-02-28 19:36 ` [PATCH v4 0/2] x86/resctrl: Pass domain to target CPU Tony Luck
2024-02-28 19:36 ` [PATCH v4 1/2] " Tony Luck
2024-03-04 23:07 ` Reinette Chatre
2024-03-05 0:17 ` Luck, Tony
2024-03-05 1:31 ` Reinette Chatre
2024-03-05 16:37 ` Luck, Tony
2024-03-05 16:48 ` Reinette Chatre
2024-03-08 18:26 ` James Morse
2024-03-08 18:50 ` Luck, Tony
2024-02-28 19:36 ` [PATCH v4 2/2] x86/resctrl: Simplify call convention for MSR update functions Tony Luck
2024-03-04 23:07 ` Reinette Chatre
2024-02-28 19:36 ` [PATCH v15 0/8] Add support for Sub-NUMA cluster (SNC) systems Tony Luck
2024-02-28 19:36 ` [PATCH v15 1/8] x86/resctrl: Prepare for new domain scope Tony Luck
2024-02-28 19:36 ` [PATCH v15 2/8] x86/resctrl: Prepare to split rdt_domain structure Tony Luck
2024-02-28 19:36 ` [PATCH v15 3/8] x86/resctrl: Prepare for different scope for control/monitor operations Tony Luck
2024-02-28 19:36 ` [PATCH v15 4/8] x86/resctrl: Split the rdt_domain and rdt_hw_domain structures Tony Luck
2024-02-28 19:37 ` [PATCH v15 5/8] x86/resctrl: Add node-scope to the options for feature scope Tony Luck
2024-02-28 19:37 ` [PATCH v15 6/8] x86/resctrl: Introduce snc_nodes_per_l3_cache Tony Luck
2024-02-28 19:37 ` [PATCH v15 7/8] x86/resctrl: Sub NUMA Cluster detection and enable Tony Luck
2024-02-28 19:37 ` Tony Luck [this message]
2024-03-04 23:07 ` Cover-cover letter for two resctrl patch sets Reinette Chatre
2024-03-05 22:02 ` Konstantin Ryabitsev
2024-03-05 22:27 ` Luck, Tony
2024-03-06 3:11 ` Konstantin Ryabitsev
2024-03-05 7:28 ` Maciej Wieczor-Retman
2024-03-05 17:51 ` Luck, Tony
2024-03-06 7:22 ` Maciej Wieczor-Retman
2024-03-06 7:28 ` Maciej Wieczor-Retman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240228193717.8170-10-tony.luck@intel.com \
--to=tony.luck@intel.com \
--cc=babu.moger@amd.com \
--cc=corbet@lwn.net \
--cc=dfustini@baylibre.com \
--cc=fenghua.yu@intel.com \
--cc=james.morse@arm.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=patches@lists.linux.dev \
--cc=peternewman@google.com \
--cc=quic_jiles@quicinc.com \
--cc=rdunlap@infradead.org \
--cc=reinette.chatre@intel.com \
--cc=skhan@linuxfoundation.org \
--cc=tan.shaopeng@fujitsu.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox