public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: Tony Luck <tony.luck@intel.com>
To: Fenghua Yu <fenghua.yu@intel.com>,
	Reinette Chatre <reinette.chatre@intel.com>,
	Peter Newman <peternewman@google.com>,
	Jonathan Corbet <corbet@lwn.net>,
	Shuah Khan <skhan@linuxfoundation.org>,
	x86@kernel.org
Cc: Shaopeng Tan <tan.shaopeng@fujitsu.com>,
	James Morse <james.morse@arm.com>,
	Jamie Iles <quic_jiles@quicinc.com>,
	Babu Moger <babu.moger@amd.com>,
	Randy Dunlap <rdunlap@infradead.org>,
	Drew Fustini <dfustini@baylibre.com>,
	linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org,
	patches@lists.linux.dev, Tony Luck <tony.luck@intel.com>
Subject: [PATCH v15 8/8] x86/resctrl: Update documentation with Sub-NUMA cluster changes
Date: Wed, 28 Feb 2024 11:37:03 -0800	[thread overview]
Message-ID: <20240228193717.8170-10-tony.luck@intel.com> (raw)
In-Reply-To: <20240228112935.8087-tony.luck@intel.com>

With Sub-NUMA Cluster mode enabled the scope of monitoring resources is
per-NODE instead of per-L3 cache. Suffixes of directories with "L3" in
their name refer to Sub-NUMA nodes instead of L3 cache ids.

Users should be aware that SNC mode also affects the amount of L3 cache
available for allocation within each SNC node.

Signed-off-by: Tony Luck <tony.luck@intel.com>
---
 Documentation/arch/x86/resctrl.rst | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/Documentation/arch/x86/resctrl.rst b/Documentation/arch/x86/resctrl.rst
index a6279df64a9d..15f1cff6ee76 100644
--- a/Documentation/arch/x86/resctrl.rst
+++ b/Documentation/arch/x86/resctrl.rst
@@ -366,10 +366,10 @@ When control is enabled all CTRL_MON groups will also contain:
 When monitoring is enabled all MON groups will also contain:
 
 "mon_data":
-	This contains a set of files organized by L3 domain and by
-	RDT event. E.g. on a system with two L3 domains there will
-	be subdirectories "mon_L3_00" and "mon_L3_01".	Each of these
-	directories have one file per event (e.g. "llc_occupancy",
+	This contains a set of files organized by L3 domain or by NUMA
+	node (depending on whether Sub-NUMA Cluster (SNC) mode is disabled
+	or enabled respectively) and by RDT event.  Each of these
+	directories has one file per event (e.g. "llc_occupancy",
 	"mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
 	files provide a read out of the current value of the event for
 	all tasks in the group. In CTRL_MON groups these files provide
@@ -478,6 +478,23 @@ if non-contiguous 1s value is supported. On a system with a 20-bit mask
 each bit represents 5% of the capacity of the cache. You could partition
 the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.
 
+Notes on Sub-NUMA Cluster mode
+==============================
+When SNC mode is enabled, Linux may load balance tasks between Sub-NUMA
+nodes much more readily than between regular NUMA nodes since the CPUs
+on Sub-NUMA nodes share the same L3 cache and the system may report
+the NUMA distance between Sub-NUMA nodes with a lower value than used
+for regular NUMA nodes.  Users who do not bind tasks to the CPUs of a
+specific Sub-NUMA node must read the "llc_occupancy", "mbm_total_bytes",
+and "mbm_local_bytes" for all Sub-NUMA nodes where the tasks may execute
+to get the full view of traffic for which the tasks were the source.
+
+The cache allocation feature still provides the same number of
+bits in a mask to control allocation into the L3 cache, but each
+of those ways has its capacity reduced because the cache is divided
+between the SNC nodes. The values reported in the resctrl
+"size" files are adjusted accordingly.
+
 Memory bandwidth Allocation and monitoring
 ==========================================
 
-- 
2.43.0


  parent reply	other threads:[~2024-02-28 19:37 UTC|newest]

Thread overview: 29+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-02-28 19:36 Cover-cover letter for two resctrl patch sets Tony Luck
2024-02-28 19:36 ` [PATCH v4 0/2] x86/resctrl: Pass domain to target CPU Tony Luck
2024-02-28 19:36   ` [PATCH v4 1/2] " Tony Luck
2024-03-04 23:07     ` Reinette Chatre
2024-03-05  0:17       ` Luck, Tony
2024-03-05  1:31         ` Reinette Chatre
2024-03-05 16:37           ` Luck, Tony
2024-03-05 16:48             ` Reinette Chatre
2024-03-08 18:26     ` James Morse
2024-03-08 18:50       ` Luck, Tony
2024-02-28 19:36   ` [PATCH v4 2/2] x86/resctrl: Simplify call convention for MSR update functions Tony Luck
2024-03-04 23:07     ` Reinette Chatre
2024-02-28 19:36 ` [PATCH v15 0/8] Add support for Sub-NUMA cluster (SNC) systems Tony Luck
2024-02-28 19:36   ` [PATCH v15 1/8] x86/resctrl: Prepare for new domain scope Tony Luck
2024-02-28 19:36   ` [PATCH v15 2/8] x86/resctrl: Prepare to split rdt_domain structure Tony Luck
2024-02-28 19:36   ` [PATCH v15 3/8] x86/resctrl: Prepare for different scope for control/monitor operations Tony Luck
2024-02-28 19:36   ` [PATCH v15 4/8] x86/resctrl: Split the rdt_domain and rdt_hw_domain structures Tony Luck
2024-02-28 19:37   ` [PATCH v15 5/8] x86/resctrl: Add node-scope to the options for feature scope Tony Luck
2024-02-28 19:37   ` [PATCH v15 6/8] x86/resctrl: Introduce snc_nodes_per_l3_cache Tony Luck
2024-02-28 19:37   ` [PATCH v15 7/8] x86/resctrl: Sub NUMA Cluster detection and enable Tony Luck
2024-02-28 19:37   ` Tony Luck [this message]
2024-03-04 23:07 ` Cover-cover letter for two resctrl patch sets Reinette Chatre
2024-03-05 22:02   ` Konstantin Ryabitsev
2024-03-05 22:27     ` Luck, Tony
2024-03-06  3:11       ` Konstantin Ryabitsev
2024-03-05  7:28 ` Maciej Wieczor-Retman
2024-03-05 17:51   ` Luck, Tony
2024-03-06  7:22     ` Maciej Wieczor-Retman
2024-03-06  7:28 ` Maciej Wieczor-Retman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240228193717.8170-10-tony.luck@intel.com \
    --to=tony.luck@intel.com \
    --cc=babu.moger@amd.com \
    --cc=corbet@lwn.net \
    --cc=dfustini@baylibre.com \
    --cc=fenghua.yu@intel.com \
    --cc=james.morse@arm.com \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=patches@lists.linux.dev \
    --cc=peternewman@google.com \
    --cc=quic_jiles@quicinc.com \
    --cc=rdunlap@infradead.org \
    --cc=reinette.chatre@intel.com \
    --cc=skhan@linuxfoundation.org \
    --cc=tan.shaopeng@fujitsu.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox