[PATCH 0/5] selftests/resctrl: Run the MBA test on arm64 MPAM via the NVIDIA SCF PMU

[PATCH 0/5] selftests/resctrl: Run the MBA test on arm64 MPAM via the NVIDIA SCF PMU

[Richard Cheng]

The resctrl MBA selftest is currently Intel-only: the test is
vendor-gated to GenuineIntel and its reference bandwidth measurement
is hard-wired to the Intel iMC uncore PMU. MPAM on arm64 exposes the
same MB resource through resctrl, and NVIDIA Grace/Vera systems
provide an uncore PMU (the SoC coherency fabric, SCF) that can serve
as the same kind of independent reference that the iMC provides on
x86.

Patches 1-2 decouple the test machinery from the iMC with no
functional change and no NVIDIA code: a mechanical rename, then a
small backend abstraction (detect()/setup_counters()) with the iMC as
the only backend. Patch 3 teaches the framework the MPAM memory
bandwidth monitoring layout (NUMA-node-keyed MB domains, MB_MON's
mbm_total_bytes, ABMC counter assignment). Patch 4 switches the MBA
test from vendor gating to feature detection. Patch 5 adds the NVIDIA
SCF PMU as a reference bandwidth backend, making the MBA test
runnable on arm64.

The series builds after each patch, and mid-series behavior is sane:
on arm64 the MBA test skips cleanly until patch 5, after which it
runs and passes. x86 behavior is preserved throughout - the iMC stays
the default backend and the mbm_local_bytes comparison path is
unchanged.

This series applies on top of the
26.04_linux-nvidia.glue.others.cpu_less.hardlimit branch of

  https://github.com/fyu1/NV-Kernels.fenghuay.baseos

The base-commit recorded below is the current tip of that branch.

Validated on a two-socket VR-NVL72 (Vera) system: the SCF-measured
bandwidth and resctrl mbm_total_bytes agree within 3-4% across all
MBA schemata levels (10%..100%), against the test's 8% threshold.

Richard Cheng (5):
  selftests/resctrl: Make memory bandwidth measurement vendor-neutral
  selftests/resctrl: Abstract reference bandwidth counters behind a
    backend
  selftests/resctrl: Support the MPAM memory bandwidth monitoring layout
  selftests/resctrl: Gate the MBA test on features, not CPU vendor
  selftests/resctrl: Add NVIDIA SCF reference bandwidth backend

 tools/testing/selftests/resctrl/mba_test.c    |  38 +-
 tools/testing/selftests/resctrl/mbm_test.c    |  23 +-
 tools/testing/selftests/resctrl/resctrl.h     |   6 +-
 tools/testing/selftests/resctrl/resctrl_val.c | 530 ++++++++++++++----
 tools/testing/selftests/resctrl/resctrlfs.c   |  45 +-
 5 files changed, 519 insertions(+), 123 deletions(-)


base-commit: 475f355d0408698d9f632ba3b9a5dfb1f91e928b
-- 
2.43.0

[PATCH 1/5] selftests/resctrl: Make memory bandwidth measurement vendor-neutral

[Richard Cheng]

The MBA and MBM tests validate resctrl's bandwidth accounting against
an independent reference: the Intel iMC uncore PMU read through perf.
Most of the machinery involved (counter bookkeeping, the one second
measurement window, multiplexing correction, result logging and the
comparison with the resctrl value) is not iMC-specific, yet it is all
named after the iMC.

Rename the shared machinery and the result labels to vendor-neutral
names in preparation for supporting reference memory bandwidth PMUs
on other architectures:

  - struct imc_counter_config         -> struct mem_bw_counter
  - imc_counters_config[], imcs       -> bw_counters[], nr_bw_counters
  - MAX_IMCS                          -> MAX_BW_COUNTERS
  - initialize_read_mem_bw_imc()      -> initialize_mem_bw_ref()
  - perf_open/close_imc_read_mem_bw() -> perf_open/close_mem_bw_counters()
  - do_imc_read_mem_bw_test()         -> do_mem_bw_test()
  - get_read_mem_bw_imc()             -> get_mem_bw_ref()
  - "Mem_BW_iMC" result file label    -> "Mem_BW_ref"
  - bw_imc / avg_bw_imc               -> bw_ref / avg_bw_ref

check_results() parses result lines by field position, not by label,
so the label change has no effect on parsing.

While touching it, reflow the result-file fprintf() so the previously
overlong line (101 columns since before this change) fits within the
100 column limit, keeping the format string unsplit.

Functions whose logic is genuinely iMC-specific (num_of_imcs(),
read_from_imc_dir()) keep their names.

No functional change intended.

Signed-off-by: Richard Cheng <icheng@nvidia.com>
---
 tools/testing/selftests/resctrl/mba_test.c    |  26 +--
 tools/testing/selftests/resctrl/mbm_test.c    |  22 +--
 tools/testing/selftests/resctrl/resctrl.h     |   4 +-
 tools/testing/selftests/resctrl/resctrl_val.c | 178 +++++++++---------
 4 files changed, 114 insertions(+), 116 deletions(-)

diff --git a/tools/testing/selftests/resctrl/mba_test.c b/tools/testing/selftests/resctrl/mba_test.c
index c7e9adc0368f..468ba8accc18 100644
--- a/tools/testing/selftests/resctrl/mba_test.c
+++ b/tools/testing/selftests/resctrl/mba_test.c
@@ -21,7 +21,7 @@ static int mba_init(const struct resctrl_val_param *param, int domain_id)
 {
 	int ret;
 
-	ret = initialize_read_mem_bw_imc();
+	ret = initialize_mem_bw_ref();
 	if (ret)
 		return ret;
 
@@ -71,7 +71,7 @@ static int mba_measure(const struct user_params *uparams,
 	return measure_read_mem_bw(uparams, param, bm_pid);
 }
 
-static bool show_mba_info(unsigned long *bw_imc, unsigned long *bw_resc)
+static bool show_mba_info(unsigned long *bw_ref, unsigned long *bw_resc)
 {
 	unsigned int allocation;
 	bool ret = false;
@@ -81,27 +81,27 @@ static bool show_mba_info(unsigned long *bw_imc, unsigned long *bw_resc)
 	/* Memory bandwidth from 100% down to 10% */
 	for (allocation = 0; allocation < ALLOCATION_MAX / ALLOCATION_STEP;
 	     allocation++) {
-		unsigned long sum_bw_imc = 0, sum_bw_resc = 0;
-		long avg_bw_imc, avg_bw_resc;
+		unsigned long sum_bw_ref = 0, sum_bw_resc = 0;
+		long avg_bw_ref, avg_bw_resc;
 		int avg_diff_per;
 		float avg_diff;
 
 		for (runs = NUM_OF_RUNS * allocation;
 		     runs < NUM_OF_RUNS * allocation + NUM_OF_RUNS ; runs++) {
-			sum_bw_imc += bw_imc[runs];
+			sum_bw_ref += bw_ref[runs];
 			sum_bw_resc += bw_resc[runs];
 		}
 
-		avg_bw_imc = sum_bw_imc / NUM_OF_RUNS;
+		avg_bw_ref = sum_bw_ref / NUM_OF_RUNS;
 		avg_bw_resc = sum_bw_resc / NUM_OF_RUNS;
-		if (avg_bw_imc < THROTTLE_THRESHOLD || avg_bw_resc < THROTTLE_THRESHOLD) {
+		if (avg_bw_ref < THROTTLE_THRESHOLD || avg_bw_resc < THROTTLE_THRESHOLD) {
 			ksft_print_msg("Bandwidth below threshold (%d MiB). Dropping results from MBA schemata %u.\n",
 				       THROTTLE_THRESHOLD,
 				       ALLOCATION_MIN + ALLOCATION_STEP * allocation);
 			continue;
 		}
 
-		avg_diff = (float)labs(avg_bw_resc - avg_bw_imc) / avg_bw_imc;
+		avg_diff = (float)labs(avg_bw_resc - avg_bw_ref) / avg_bw_ref;
 		avg_diff_per = (int)(avg_diff * 100);
 
 		ksft_print_msg("%s Check MBA diff within %d%% for schemata %u\n",
@@ -111,7 +111,7 @@ static bool show_mba_info(unsigned long *bw_imc, unsigned long *bw_resc)
 			       ALLOCATION_MIN + ALLOCATION_STEP * allocation);
 
 		ksft_print_msg("avg_diff_per: %d%%\n", avg_diff_per);
-		ksft_print_msg("avg_bw_imc: %lu\n", avg_bw_imc);
+		ksft_print_msg("avg_bw_ref: %lu\n", avg_bw_ref);
 		ksft_print_msg("avg_bw_resc: %lu\n", avg_bw_resc);
 		if (avg_diff_per > MAX_DIFF_PERCENT)
 			ret = true;
@@ -128,7 +128,7 @@ static bool show_mba_info(unsigned long *bw_imc, unsigned long *bw_resc)
 static int check_results(void)
 {
 	unsigned long bw_resc[NUM_OF_RUNS * ALLOCATION_MAX / ALLOCATION_STEP];
-	unsigned long bw_imc[NUM_OF_RUNS * ALLOCATION_MAX / ALLOCATION_STEP];
+	unsigned long bw_ref[NUM_OF_RUNS * ALLOCATION_MAX / ALLOCATION_STEP];
 	char *token_array[8], output[] = RESULT_FILE_NAME, temp[512];
 	int runs;
 	FILE *fp;
@@ -150,8 +150,8 @@ static int check_results(void)
 			token = strtok(NULL, ":\t");
 		}
 
-		/* Field 3 is perf imc value */
-		bw_imc[runs] = strtoul(token_array[3], NULL, 0);
+		/* Field 3 is the reference PMU value */
+		bw_ref[runs] = strtoul(token_array[3], NULL, 0);
 		/* Field 5 is resctrl value */
 		bw_resc[runs] = strtoul(token_array[5], NULL, 0);
 		runs++;
@@ -159,7 +159,7 @@ static int check_results(void)
 
 	fclose(fp);
 
-	return show_mba_info(bw_imc, bw_resc);
+	return show_mba_info(bw_ref, bw_resc);
 }
 
 static void mba_test_cleanup(void)
diff --git a/tools/testing/selftests/resctrl/mbm_test.c b/tools/testing/selftests/resctrl/mbm_test.c
index 84d8bc250539..001a87608835 100644
--- a/tools/testing/selftests/resctrl/mbm_test.c
+++ b/tools/testing/selftests/resctrl/mbm_test.c
@@ -15,21 +15,21 @@
 #define NUM_OF_RUNS		5
 
 static int
-show_bw_info(unsigned long *bw_imc, unsigned long *bw_resc, size_t span)
+show_bw_info(unsigned long *bw_ref, unsigned long *bw_resc, size_t span)
 {
-	unsigned long sum_bw_imc = 0, sum_bw_resc = 0;
-	long avg_bw_imc = 0, avg_bw_resc = 0;
+	unsigned long sum_bw_ref = 0, sum_bw_resc = 0;
+	long avg_bw_ref = 0, avg_bw_resc = 0;
 	int runs, ret, avg_diff_per;
 	float avg_diff = 0;
 
 	for (runs = 0; runs < NUM_OF_RUNS; runs++) {
-		sum_bw_imc += bw_imc[runs];
+		sum_bw_ref += bw_ref[runs];
 		sum_bw_resc += bw_resc[runs];
 	}
 
-	avg_bw_imc = sum_bw_imc / NUM_OF_RUNS;
+	avg_bw_ref = sum_bw_ref / NUM_OF_RUNS;
 	avg_bw_resc = sum_bw_resc / NUM_OF_RUNS;
-	avg_diff = (float)labs(avg_bw_resc - avg_bw_imc) / avg_bw_imc;
+	avg_diff = (float)labs(avg_bw_resc - avg_bw_ref) / avg_bw_ref;
 	avg_diff_per = (int)(avg_diff * 100);
 
 	ret = avg_diff_per > MAX_DIFF_PERCENT;
@@ -38,7 +38,7 @@ show_bw_info(unsigned long *bw_imc, unsigned long *bw_resc, size_t span)
 	ksft_print_msg("avg_diff_per: %d%%\n", avg_diff_per);
 	if (span)
 		ksft_print_msg("Span (MB): %zu\n", span / MB);
-	ksft_print_msg("avg_bw_imc: %lu\n", avg_bw_imc);
+	ksft_print_msg("avg_bw_ref: %lu\n", avg_bw_ref);
 	ksft_print_msg("avg_bw_resc: %lu\n", avg_bw_resc);
 
 	return ret;
@@ -46,7 +46,7 @@ show_bw_info(unsigned long *bw_imc, unsigned long *bw_resc, size_t span)
 
 static int check_results(size_t span)
 {
-	unsigned long bw_imc[NUM_OF_RUNS], bw_resc[NUM_OF_RUNS];
+	unsigned long bw_ref[NUM_OF_RUNS], bw_resc[NUM_OF_RUNS];
 	char temp[1024], *token_array[8];
 	char output[] = RESULT_FILE_NAME;
 	int runs, ret;
@@ -72,11 +72,11 @@ static int check_results(size_t span)
 		}
 
 		bw_resc[runs] = strtoul(token_array[5], NULL, 0);
-		bw_imc[runs] = strtoul(token_array[3], NULL, 0);
+		bw_ref[runs] = strtoul(token_array[3], NULL, 0);
 		runs++;
 	}
 
-	ret = show_bw_info(bw_imc, bw_resc, span);
+	ret = show_bw_info(bw_ref, bw_resc, span);
 
 	fclose(fp);
 
@@ -87,7 +87,7 @@ static int mbm_init(const struct resctrl_val_param *param, int domain_id)
 {
 	int ret;
 
-	ret = initialize_read_mem_bw_imc();
+	ret = initialize_mem_bw_ref();
 	if (ret)
 		return ret;
 
diff --git a/tools/testing/selftests/resctrl/resctrl.h b/tools/testing/selftests/resctrl/resctrl.h
index afe635b6e48d..0de72327a085 100644
--- a/tools/testing/selftests/resctrl/resctrl.h
+++ b/tools/testing/selftests/resctrl/resctrl.h
@@ -49,7 +49,7 @@
 
 /*
  * Memory bandwidth (in MiB) below which the bandwidth comparisons
- * between iMC and resctrl are considered unreliable. For example RAS
+ * between the reference PMU and resctrl are considered unreliable. For example RAS
  * features or memory performance features that generate memory traffic
  * may drive accesses that are counted differently by performance counters
  * and MBM respectively, for instance generating "overhead" traffic which
@@ -187,7 +187,7 @@ unsigned char *alloc_buffer(size_t buf_size, bool memflush);
 void mem_flush(unsigned char *buf, size_t buf_size);
 void fill_cache_read(unsigned char *buf, size_t buf_size, bool once);
 ssize_t get_fill_buf_size(int cpu_no, const char *cache_type);
-int initialize_read_mem_bw_imc(void);
+int initialize_mem_bw_ref(void);
 int measure_read_mem_bw(const struct user_params *uparams,
 			struct resctrl_val_param *param, pid_t bm_pid);
 void initialize_mem_bw_resctrl(const struct resctrl_val_param *param,
diff --git a/tools/testing/selftests/resctrl/resctrl_val.c b/tools/testing/selftests/resctrl/resctrl_val.c
index 7c08e936572d..ee44fbae2cbf 100644
--- a/tools/testing/selftests/resctrl/resctrl_val.c
+++ b/tools/testing/selftests/resctrl/resctrl_val.c
@@ -14,7 +14,7 @@
 #define READ_FILE_NAME		"events/cas_count_read"
 #define DYN_PMU_PATH		"/sys/bus/event_source/devices"
 #define SCALE			0.00006103515625
-#define MAX_IMCS		20
+#define MAX_BW_COUNTERS		20
 #define MAX_TOKENS		5
 
 #define CON_MBM_LOCAL_BYTES_PATH		\
@@ -27,7 +27,7 @@ struct membw_read_format {
 	__u64 id;            /* if PERF_FORMAT_ID */
 };
 
-struct imc_counter_config {
+struct mem_bw_counter {
 	__u32 type;
 	__u64 event;
 	__u64 umask;
@@ -37,36 +37,36 @@ struct imc_counter_config {
 };
 
 static char mbm_total_path[1024];
-static int imcs;
-static struct imc_counter_config imc_counters_config[MAX_IMCS];
+static int nr_bw_counters;
+static struct mem_bw_counter bw_counters[MAX_BW_COUNTERS];
 static const struct resctrl_test *current_test;
 
 static void read_mem_bw_initialize_perf_event_attr(int i)
 {
-	memset(&imc_counters_config[i].pe, 0,
+	memset(&bw_counters[i].pe, 0,
 	       sizeof(struct perf_event_attr));
-	imc_counters_config[i].pe.type = imc_counters_config[i].type;
-	imc_counters_config[i].pe.size = sizeof(struct perf_event_attr);
-	imc_counters_config[i].pe.disabled = 1;
-	imc_counters_config[i].pe.inherit = 1;
-	imc_counters_config[i].pe.exclude_guest = 0;
-	imc_counters_config[i].pe.config =
-		imc_counters_config[i].umask << 8 |
-		imc_counters_config[i].event;
-	imc_counters_config[i].pe.sample_type = PERF_SAMPLE_IDENTIFIER;
-	imc_counters_config[i].pe.read_format =
+	bw_counters[i].pe.type = bw_counters[i].type;
+	bw_counters[i].pe.size = sizeof(struct perf_event_attr);
+	bw_counters[i].pe.disabled = 1;
+	bw_counters[i].pe.inherit = 1;
+	bw_counters[i].pe.exclude_guest = 0;
+	bw_counters[i].pe.config =
+		bw_counters[i].umask << 8 |
+		bw_counters[i].event;
+	bw_counters[i].pe.sample_type = PERF_SAMPLE_IDENTIFIER;
+	bw_counters[i].pe.read_format =
 		PERF_FORMAT_TOTAL_TIME_ENABLED | PERF_FORMAT_TOTAL_TIME_RUNNING;
 }
 
 static void read_mem_bw_ioctl_perf_event_ioc_reset_enable(int i)
 {
-	ioctl(imc_counters_config[i].fd, PERF_EVENT_IOC_RESET, 0);
-	ioctl(imc_counters_config[i].fd, PERF_EVENT_IOC_ENABLE, 0);
+	ioctl(bw_counters[i].fd, PERF_EVENT_IOC_RESET, 0);
+	ioctl(bw_counters[i].fd, PERF_EVENT_IOC_ENABLE, 0);
 }
 
 static void read_mem_bw_ioctl_perf_event_ioc_disable(int i)
 {
-	ioctl(imc_counters_config[i].fd, PERF_EVENT_IOC_DISABLE, 0);
+	ioctl(bw_counters[i].fd, PERF_EVENT_IOC_DISABLE, 0);
 }
 
 /*
@@ -88,21 +88,21 @@ static void get_read_event_and_umask(char *cas_count_cfg, int count)
 		if (!token[i])
 			break;
 		if (strcmp(token[i], "event") == 0)
-			imc_counters_config[count].event = strtol(token[i + 1], NULL, 16);
+			bw_counters[count].event = strtol(token[i + 1], NULL, 16);
 		if (strcmp(token[i], "umask") == 0)
-			imc_counters_config[count].umask = strtol(token[i + 1], NULL, 16);
+			bw_counters[count].umask = strtol(token[i + 1], NULL, 16);
 	}
 }
 
 static int open_perf_read_event(int i, int cpu_no)
 {
-	imc_counters_config[i].fd =
-		perf_event_open(&imc_counters_config[i].pe, -1, cpu_no, -1,
+	bw_counters[i].fd =
+		perf_event_open(&bw_counters[i].pe, -1, cpu_no, -1,
 				PERF_FLAG_FD_CLOEXEC);
 
-	if (imc_counters_config[i].fd == -1) {
+	if (bw_counters[i].fd == -1) {
 		fprintf(stderr, "Error opening leader %llx\n",
-			imc_counters_config[i].pe.config);
+			bw_counters[i].pe.config);
 
 		return -1;
 	}
@@ -124,7 +124,7 @@ static int read_from_imc_dir(char *imc_dir, int count)
 
 		return -1;
 	}
-	if (fscanf(fp, "%u", &imc_counters_config[count].type) <= 0) {
+	if (fscanf(fp, "%u", &bw_counters[count].type) <= 0) {
 		ksft_perror("Could not get iMC type");
 		fclose(fp);
 
@@ -221,46 +221,46 @@ static int num_of_imcs(void)
 	return count;
 }
 
-int initialize_read_mem_bw_imc(void)
+int initialize_mem_bw_ref(void)
 {
-	int imc;
+	int i;
 
-	imcs = num_of_imcs();
-	if (imcs <= 0)
-		return imcs;
+	nr_bw_counters = num_of_imcs();
+	if (nr_bw_counters <= 0)
+		return nr_bw_counters;
 
 	/* Initialize perf_event_attr structures for all iMC's */
-	for (imc = 0; imc < imcs; imc++)
-		read_mem_bw_initialize_perf_event_attr(imc);
+	for (i = 0; i < nr_bw_counters; i++)
+		read_mem_bw_initialize_perf_event_attr(i);
 
 	return 0;
 }
 
-static void perf_close_imc_read_mem_bw(void)
+static void perf_close_mem_bw_counters(void)
 {
-	int mc;
+	int i;
 
-	for (mc = 0; mc < imcs; mc++) {
-		if (imc_counters_config[mc].fd != -1)
-			close(imc_counters_config[mc].fd);
+	for (i = 0; i < nr_bw_counters; i++) {
+		if (bw_counters[i].fd != -1)
+			close(bw_counters[i].fd);
 	}
 }
 
 /*
- * perf_open_imc_read_mem_bw - Open perf fds for IMCs
+ * perf_open_mem_bw_counters - Open perf fds for the reference-bandwidth counters
  * @cpu_no: CPU number that the benchmark PID is bound to
  *
  * Return: = 0 on success. < 0 on failure.
  */
-static int perf_open_imc_read_mem_bw(int cpu_no)
+static int perf_open_mem_bw_counters(int cpu_no)
 {
-	int imc, ret;
+	int i, ret;
 
-	for (imc = 0; imc < imcs; imc++)
-		imc_counters_config[imc].fd = -1;
+	for (i = 0; i < nr_bw_counters; i++)
+		bw_counters[i].fd = -1;
 
-	for (imc = 0; imc < imcs; imc++) {
-		ret = open_perf_read_event(imc, cpu_no);
+	for (i = 0; i < nr_bw_counters; i++) {
+		ret = open_perf_read_event(i, cpu_no);
 		if (ret)
 			goto close_fds;
 	}
@@ -268,56 +268,54 @@ static int perf_open_imc_read_mem_bw(int cpu_no)
 	return 0;
 
 close_fds:
-	perf_close_imc_read_mem_bw();
+	perf_close_mem_bw_counters();
 	return -1;
 }
 
 /*
- * do_imc_read_mem_bw_test - Perform memory bandwidth test
+ * do_mem_bw_test - Perform memory bandwidth test
  *
  * Runs memory bandwidth test over one second period. Also, handles starting
- * and stopping of the IMC perf counters around the test.
+ * and stopping of the reference perf counters around the test.
  */
-static void do_imc_read_mem_bw_test(void)
+static void do_mem_bw_test(void)
 {
-	int imc;
+	int i;
 
-	for (imc = 0; imc < imcs; imc++)
-		read_mem_bw_ioctl_perf_event_ioc_reset_enable(imc);
+	for (i = 0; i < nr_bw_counters; i++)
+		read_mem_bw_ioctl_perf_event_ioc_reset_enable(i);
 
 	sleep(1);
 
 	/* Stop counters after a second to get results. */
-	for (imc = 0; imc < imcs; imc++)
-		read_mem_bw_ioctl_perf_event_ioc_disable(imc);
+	for (i = 0; i < nr_bw_counters; i++)
+		read_mem_bw_ioctl_perf_event_ioc_disable(i);
 }
 
 /*
- * get_read_mem_bw_imc - Memory read bandwidth as reported by iMC counters
+ * get_mem_bw_ref - Memory bandwidth as reported by the reference PMU counters
  *
- * Memory read bandwidth utilized by a process on a socket can be calculated
- * using iMC counters' read events. Perf events are used to read these
- * counters.
+ * Sum all configured reference counters, scaled to MiB: read CAS counts on
+ * the x86 iMC.
  *
  * Return: = 0 on success. < 0 on failure.
  */
-static int get_read_mem_bw_imc(float *bw_imc)
+static int get_mem_bw_ref(float *bw_ref)
 {
-	float reads = 0, of_mul_read = 1;
-	int imc;
+	float bw = 0, of_mul = 1;
+	int i;
 
 	/*
-	 * Log read event values from all iMC counters into
-	 * struct imc_counter_config.
+	 * Log event values from all reference counters into
+	 * struct mem_bw_counter.
 	 * Take overflow into consideration before calculating total bandwidth.
 	 */
-	for (imc = 0; imc < imcs; imc++) {
-		struct imc_counter_config *r =
-			&imc_counters_config[imc];
+	for (i = 0; i < nr_bw_counters; i++) {
+		struct mem_bw_counter *r = &bw_counters[i];
 
 		if (read(r->fd, &r->return_value,
 			 sizeof(struct membw_read_format)) == -1) {
-			ksft_perror("Couldn't get read bandwidth through iMC");
+			ksft_perror("Couldn't read reference bandwidth counter");
 			return -1;
 		}
 
@@ -325,13 +323,13 @@ static int get_read_mem_bw_imc(float *bw_imc)
 		__u64 r_time_running = r->return_value.time_running;
 
 		if (r_time_enabled != r_time_running)
-			of_mul_read = (float)r_time_enabled /
+			of_mul = (float)r_time_enabled /
 					(float)r_time_running;
 
-		reads += r->return_value.value * of_mul_read * SCALE;
+		bw += r->return_value.value * of_mul * SCALE;
 	}
 
-	*bw_imc = reads;
+	*bw_ref = bw;
 	return 0;
 }
 
@@ -435,19 +433,19 @@ void signal_handler_unregister(void)
  * print_results_bw:	the memory bandwidth results are stored in a file
  * @filename:		file that stores the results
  * @bm_pid:		child pid that runs benchmark
- * @bw_imc:		perf imc counter value
+ * @bw_ref:		reference PMU counter value
  * @bw_resc:		memory bandwidth value
  *
  * Return:		0 on success, < 0 on error.
  */
-static int print_results_bw(char *filename, pid_t bm_pid, float bw_imc,
+static int print_results_bw(char *filename, pid_t bm_pid, float bw_ref,
 			    unsigned long bw_resc)
 {
-	unsigned long diff = fabs(bw_imc - bw_resc);
+	unsigned long diff = fabs(bw_ref - bw_resc);
 	FILE *fp;
 
 	if (strcmp(filename, "stdio") == 0 || strcmp(filename, "stderr") == 0) {
-		printf("Pid: %d \t Mem_BW_iMC: %f \t ", (int)bm_pid, bw_imc);
+		printf("Pid: %d \t Mem_BW_ref: %f \t ", (int)bm_pid, bw_ref);
 		printf("Mem_BW_resc: %lu \t Difference: %lu\n", bw_resc, diff);
 	} else {
 		fp = fopen(filename, "a");
@@ -456,8 +454,9 @@ static int print_results_bw(char *filename, pid_t bm_pid, float bw_imc,
 
 			return -1;
 		}
-		if (fprintf(fp, "Pid: %d \t Mem_BW_iMC: %f \t Mem_BW_resc: %lu \t Difference: %lu\n",
-			    (int)bm_pid, bw_imc, bw_resc, diff) <= 0) {
+		if (fprintf(fp,
+			    "Pid: %d \t Mem_BW_ref: %f \t Mem_BW_resc: %lu \t Difference: %lu\n",
+			    (int)bm_pid, bw_ref, bw_resc, diff) <= 0) {
 			ksft_print_msg("Could not log results\n");
 			fclose(fp);
 
@@ -475,10 +474,9 @@ static int print_results_bw(char *filename, pid_t bm_pid, float bw_imc,
  * @param:		Parameters passed to resctrl_val()
  * @bm_pid:		PID that runs the benchmark
  *
- * Measure memory bandwidth from resctrl and from another source which is
- * perf imc value or could be something else if perf imc event is not
- * available. Compare the two values to validate resctrl value. It takes
- * 1 sec to measure the data.
+ * Measure memory bandwidth from resctrl and from the independent reference
+ * PMU. Compare the two values to validate resctrl value. It takes 1 sec to
+ * measure the data.
  * resctrl does not distinguish between read and write operations so
  * its data includes all memory operations.
  */
@@ -487,42 +485,42 @@ int measure_read_mem_bw(const struct user_params *uparams,
 {
 	unsigned long bw_resc, bw_resc_start, bw_resc_end;
 	FILE *mem_bw_fp;
-	float bw_imc;
+	float bw_ref;
 	int ret;
 
 	mem_bw_fp = open_mem_bw_resctrl(mbm_total_path);
 	if (!mem_bw_fp)
 		return -1;
 
-	ret = perf_open_imc_read_mem_bw(uparams->cpu);
+	ret = perf_open_mem_bw_counters(uparams->cpu);
 	if (ret < 0)
 		goto close_fp;
 
 	ret = get_mem_bw_resctrl(mem_bw_fp, &bw_resc_start);
 	if (ret < 0)
-		goto close_imc;
+		goto close_counters;
 
 	rewind(mem_bw_fp);
 
-	do_imc_read_mem_bw_test();
+	do_mem_bw_test();
 
 	ret = get_mem_bw_resctrl(mem_bw_fp, &bw_resc_end);
 	if (ret < 0)
-		goto close_imc;
+		goto close_counters;
 
-	ret = get_read_mem_bw_imc(&bw_imc);
+	ret = get_mem_bw_ref(&bw_ref);
 	if (ret < 0)
-		goto close_imc;
+		goto close_counters;
 
-	perf_close_imc_read_mem_bw();
+	perf_close_mem_bw_counters();
 	fclose(mem_bw_fp);
 
 	bw_resc = (bw_resc_end - bw_resc_start) / MB;
 
-	return print_results_bw(param->filename, bm_pid, bw_imc, bw_resc);
+	return print_results_bw(param->filename, bm_pid, bw_ref, bw_resc);
 
-close_imc:
-	perf_close_imc_read_mem_bw();
+close_counters:
+	perf_close_mem_bw_counters();
 close_fp:
 	fclose(mem_bw_fp);
 	return ret;
-- 
2.43.0

[PATCH 2/5] selftests/resctrl: Abstract reference bandwidth counters behind a backend

[Richard Cheng]

The reference bandwidth measurement is hard-wired to the Intel iMC:
discovery, event configuration and perf open all assume the iMC PMU
sysfs layout. Other architectures expose equivalent uncore PMUs that
could serve as the same kind of independent reference.

Introduce struct mem_bw_backend with two operations, detect() (is this
backend's PMU present?) and setup_counters() (populate bw_counters[]
for measuring the benchmark CPU), plus a backend table currently
containing only the iMC. Shared code selects a backend at init time
and no longer contains any PMU-specific knowledge.

Counter configuration moves from test init time to perf open time via
setup_counters(), because a backend may need the benchmark CPU to pick
per-socket PMU instances. Each counter now carries the CPU to open it
on. The added per-measurement discovery cost is a handful of sysfs
reads, negligible against the one second measurement window.

Add mem_bw_ref_available() so feature checks can verify that a
reference PMU exists: the MBA test is now skipped instead of failing
at init on machines without iMC counters (e.g. virtual machines).
Add mem_bw_ref_cleanup() to reset the backend selection between tests.

Signed-off-by: Richard Cheng <icheng@nvidia.com>
---
 tools/testing/selftests/resctrl/mba_test.c    |   4 +-
 tools/testing/selftests/resctrl/mbm_test.c    |   1 +
 tools/testing/selftests/resctrl/resctrl.h     |   2 +
 tools/testing/selftests/resctrl/resctrl_val.c | 145 ++++++++++++++++--
 4 files changed, 141 insertions(+), 11 deletions(-)

diff --git a/tools/testing/selftests/resctrl/mba_test.c b/tools/testing/selftests/resctrl/mba_test.c
index 468ba8accc18..8b891d3e6650 100644
--- a/tools/testing/selftests/resctrl/mba_test.c
+++ b/tools/testing/selftests/resctrl/mba_test.c
@@ -164,6 +164,7 @@ static int check_results(void)
 
 static void mba_test_cleanup(void)
 {
+	mem_bw_ref_cleanup();
 	remove(RESULT_FILE_NAME);
 }
 
@@ -210,7 +211,8 @@ static int mba_run_test(const struct resctrl_test *test, const struct user_param
 static bool mba_feature_check(const struct resctrl_test *test)
 {
 	return test_resource_feature_check(test) &&
-	       resctrl_mon_feature_exists("L3_MON", "mbm_local_bytes");
+	       resctrl_mon_feature_exists("L3_MON", "mbm_local_bytes") &&
+	       mem_bw_ref_available();
 }
 
 struct resctrl_test mba_test = {
diff --git a/tools/testing/selftests/resctrl/mbm_test.c b/tools/testing/selftests/resctrl/mbm_test.c
index 001a87608835..873acf6aa648 100644
--- a/tools/testing/selftests/resctrl/mbm_test.c
+++ b/tools/testing/selftests/resctrl/mbm_test.c
@@ -123,6 +123,7 @@ static int mbm_measure(const struct user_params *uparams,
 
 static void mbm_test_cleanup(void)
 {
+	mem_bw_ref_cleanup();
 	remove(RESULT_FILE_NAME);
 }
 
diff --git a/tools/testing/selftests/resctrl/resctrl.h b/tools/testing/selftests/resctrl/resctrl.h
index 0de72327a085..2bd19c22a344 100644
--- a/tools/testing/selftests/resctrl/resctrl.h
+++ b/tools/testing/selftests/resctrl/resctrl.h
@@ -188,6 +188,8 @@ void mem_flush(unsigned char *buf, size_t buf_size);
 void fill_cache_read(unsigned char *buf, size_t buf_size, bool once);
 ssize_t get_fill_buf_size(int cpu_no, const char *cache_type);
 int initialize_mem_bw_ref(void);
+bool mem_bw_ref_available(void);
+void mem_bw_ref_cleanup(void);
 int measure_read_mem_bw(const struct user_params *uparams,
 			struct resctrl_val_param *param, pid_t bm_pid);
 void initialize_mem_bw_resctrl(const struct resctrl_val_param *param,
diff --git a/tools/testing/selftests/resctrl/resctrl_val.c b/tools/testing/selftests/resctrl/resctrl_val.c
index ee44fbae2cbf..a2b919bd4839 100644
--- a/tools/testing/selftests/resctrl/resctrl_val.c
+++ b/tools/testing/selftests/resctrl/resctrl_val.c
@@ -34,12 +34,29 @@ struct mem_bw_counter {
 	struct perf_event_attr pe;
 	struct membw_read_format return_value;
 	int fd;
+	int cpu;
+};
+
+/*
+ * mem_bw_backend - Source of the independent reference bandwidth that
+ *		    validates resctrl MBM
+ * @name:		Human-readable PMU name
+ * @detect:		True if this backend's PMU is present on the system
+ * @setup_counters:	Populate bw_counters[] / nr_bw_counters with the perf
+ *			events measuring memory bandwidth generated from
+ *			@bench_cpu. Returns 0 on success, < 0 on failure.
+ */
+struct mem_bw_backend {
+	const char *name;
+	bool (*detect)(void);
+	int (*setup_counters)(int bench_cpu);
 };
 
 static char mbm_total_path[1024];
 static int nr_bw_counters;
 static struct mem_bw_counter bw_counters[MAX_BW_COUNTERS];
 static const struct resctrl_test *current_test;
+static const struct mem_bw_backend *mem_bw_backend;
 
 static void read_mem_bw_initialize_perf_event_attr(int i)
 {
@@ -94,10 +111,11 @@ static void get_read_event_and_umask(char *cas_count_cfg, int count)
 	}
 }
 
-static int open_perf_read_event(int i, int cpu_no)
+static int open_perf_read_event(int i)
 {
 	bw_counters[i].fd =
-		perf_event_open(&bw_counters[i].pe, -1, cpu_no, -1,
+		perf_event_open(&bw_counters[i].pe, -1,
+				bw_counters[i].cpu, -1,
 				PERF_FLAG_FD_CLOEXEC);
 
 	if (bw_counters[i].fd == -1) {
@@ -221,21 +239,108 @@ static int num_of_imcs(void)
 	return count;
 }
 
-int initialize_mem_bw_ref(void)
+/* True if any x86 uncore iMC PMU ("uncore_imc_<n>") is present. */
+static bool intel_imc_present(void)
 {
-	int i;
+	struct dirent *ep;
+	bool found = false;
+	char *temp;
+	DIR *dp;
 
-	nr_bw_counters = num_of_imcs();
-	if (nr_bw_counters <= 0)
-		return nr_bw_counters;
+	dp = opendir(DYN_PMU_PATH);
+	if (!dp)
+		return false;
+	while ((ep = readdir(dp))) {
+		temp = strstr(ep->d_name, UNCORE_IMC);
+		if (!temp)
+			continue;
+		temp += sizeof(UNCORE_IMC);
+		if (temp[0] >= '0' && temp[0] <= '9') {
+			found = true;
+			break;
+		}
+	}
+	closedir(dp);
+	return found;
+}
 
-	/* Initialize perf_event_attr structures for all iMC's */
-	for (i = 0; i < nr_bw_counters; i++)
+/*
+ * imc_setup_counters - Configure iMC CAS-count-read counters
+ * @bench_cpu: CPU the benchmark is bound to
+ *
+ * Discovers all iMC PMUs and configures one read-bandwidth counter per iMC,
+ * opened on the benchmark CPU (each iMC counts socket-wide regardless).
+ *
+ * Return: 0 on success, < 0 on failure.
+ */
+static int imc_setup_counters(int bench_cpu)
+{
+	int i, count;
+
+	count = num_of_imcs();
+	if (count <= 0)
+		return -1;
+
+	nr_bw_counters = count;
+	for (i = 0; i < nr_bw_counters; i++) {
 		read_mem_bw_initialize_perf_event_attr(i);
+		bw_counters[i].cpu = bench_cpu;
+	}
 
 	return 0;
 }
 
+static const struct mem_bw_backend mem_bw_backends[] = {
+	{
+		.name = "Intel iMC",
+		.detect = intel_imc_present,
+		.setup_counters = imc_setup_counters,
+	},
+};
+
+/*
+ * mem_bw_ref_available - Is an independent reference-bandwidth PMU present?
+ *
+ * Used by the MBA/MBM feature checks so the test is SKIPped (not failed) on
+ * hardware lacking a usable memory-bandwidth PMU.
+ */
+bool mem_bw_ref_available(void)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(mem_bw_backends); i++) {
+		if (mem_bw_backends[i].detect())
+			return true;
+	}
+
+	return false;
+}
+
+/*
+ * initialize_mem_bw_ref - Select the reference-bandwidth backend for this system
+ *
+ * Counter discovery happens at perf-open time through the backend's
+ * setup_counters() since it can depend on the benchmark CPU.
+ *
+ * Return: 0 on success, < 0 if no reference PMU is available.
+ */
+int initialize_mem_bw_ref(void)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(mem_bw_backends); i++) {
+		if (mem_bw_backends[i].detect()) {
+			mem_bw_backend = &mem_bw_backends[i];
+			ksft_print_msg("Using %s as memory bandwidth reference\n",
+				       mem_bw_backend->name);
+			return 0;
+		}
+	}
+
+	ksft_print_msg("No memory bandwidth reference PMU found\n");
+	return -1;
+}
+
 static void perf_close_mem_bw_counters(void)
 {
 	int i;
@@ -250,17 +355,27 @@ static void perf_close_mem_bw_counters(void)
  * perf_open_mem_bw_counters - Open perf fds for the reference-bandwidth counters
  * @cpu_no: CPU number that the benchmark PID is bound to
  *
+ * Counters are (re)configured by the selected backend first since their
+ * discovery can depend on the benchmark CPU (e.g. per-socket PMU instances).
+ *
  * Return: = 0 on success. < 0 on failure.
  */
 static int perf_open_mem_bw_counters(int cpu_no)
 {
 	int i, ret;
 
+	if (!mem_bw_backend)
+		return -1;
+
+	ret = mem_bw_backend->setup_counters(cpu_no);
+	if (ret < 0)
+		return ret;
+
 	for (i = 0; i < nr_bw_counters; i++)
 		bw_counters[i].fd = -1;
 
 	for (i = 0; i < nr_bw_counters; i++) {
-		ret = open_perf_read_event(i, cpu_no);
+		ret = open_perf_read_event(i);
 		if (ret)
 			goto close_fds;
 	}
@@ -345,6 +460,16 @@ void initialize_mem_bw_resctrl(const struct resctrl_val_param *param,
 		param->ctrlgrp, domain_id);
 }
 
+/*
+ * mem_bw_ref_cleanup - Reset the selected reference-bandwidth backend
+ *
+ * Safe to call when no backend was selected.
+ */
+void mem_bw_ref_cleanup(void)
+{
+	mem_bw_backend = NULL;
+}
+
 /*
  * Open file to read MBM local bytes from resctrl FS
  */
-- 
2.43.0

[PATCH 3/5] selftests/resctrl: Support the MPAM memory bandwidth monitoring layout

[Richard Cheng]

Arm MPAM systems running resctrl expose memory bandwidth differently
from x86:

 - MB control/monitor domains are keyed by NUMA node id, which need
   not match the L3 cache id (on a Grace/Vera system L3 ids are {1,2}
   while MB domains are {0,1,2,10,...}).
 - Monitoring lives in a separate MB_MON resource that provides only
   mbm_total_bytes; there is no local/total split and no L3_MON
   bandwidth event.
 - In "mbm_event" (ABMC) assignment mode a hardware counter must be
   explicitly assigned to a group+domain+event before the bandwidth
   file returns a value (it reads "Unassigned" otherwise), and
   removing a group does not release its counter, so it must be
   unassigned explicitly or it leaks.

Teach the test framework this layout:

 - get_domain_id() resolves MB domains via the CPU's NUMA node when
   the MB_MON resource exists. x86 (no MB_MON) is unaffected.
 - initialize_mem_bw_resctrl() falls back to
   mon_MB_<domain>/mbm_total_bytes when mbm_local_bytes is absent,
   and assigns an ABMC counter when the group's mbm_MB_assignments
   file is present. Note the assignment parser requires a trailing
   newline; without it the write fails with EINVAL.
 - mem_bw_ref_cleanup() unassigns the counter again.
 - The MBA feature check accepts MB_MON's mbm_total_bytes as the
   monitoring source.

Comparing a reference PMU total (read+write) with mbm_total_bytes is
a like-for-like comparison; the iMC path keeps comparing read
bandwidth with mbm_local_bytes as before.

No effect yet on any vendor: the MBA test is still gated to Intel.

Signed-off-by: Richard Cheng <icheng@nvidia.com>
---
 tools/testing/selftests/resctrl/mba_test.c    |  3 +-
 tools/testing/selftests/resctrl/resctrl_val.c | 69 +++++++++++++++++--
 tools/testing/selftests/resctrl/resctrlfs.c   | 45 +++++++++++-
 3 files changed, 111 insertions(+), 6 deletions(-)

diff --git a/tools/testing/selftests/resctrl/mba_test.c b/tools/testing/selftests/resctrl/mba_test.c
index 8b891d3e6650..6c85107c32ee 100644
--- a/tools/testing/selftests/resctrl/mba_test.c
+++ b/tools/testing/selftests/resctrl/mba_test.c
@@ -211,7 +211,8 @@ static int mba_run_test(const struct resctrl_test *test, const struct user_param
 static bool mba_feature_check(const struct resctrl_test *test)
 {
 	return test_resource_feature_check(test) &&
-	       resctrl_mon_feature_exists("L3_MON", "mbm_local_bytes") &&
+	       (resctrl_mon_feature_exists("L3_MON", "mbm_local_bytes") ||
+		resctrl_mon_feature_exists("MB_MON", "mbm_total_bytes")) &&
 	       mem_bw_ref_available();
 }
 
diff --git a/tools/testing/selftests/resctrl/resctrl_val.c b/tools/testing/selftests/resctrl/resctrl_val.c
index a2b919bd4839..0c3079d31340 100644
--- a/tools/testing/selftests/resctrl/resctrl_val.c
+++ b/tools/testing/selftests/resctrl/resctrl_val.c
@@ -20,6 +20,21 @@
 #define CON_MBM_LOCAL_BYTES_PATH		\
 	"%s/%s/mon_data/mon_L3_%02d/mbm_local_bytes"
 
+/*
+ * MPAM exposes only total (read+write) bandwidth, under MB_MON, and per the
+ * NUMA-node-keyed MB domain rather than the L3 cache id.
+ */
+#define CON_MBM_TOTAL_BYTES_PATH		\
+	"%s/%s/mon_data/mon_MB_%02d/mbm_total_bytes"
+
+/*
+ * In MPAM "mbm_event" (ABMC) counter-assignment mode a hardware counter must
+ * be explicitly assigned to a group+domain+event before the bandwidth file
+ * reads a value; otherwise it returns "Unassigned".
+ */
+#define MBM_MB_ASSIGN_PATH			\
+	"%s/%s/mbm_MB_assignments"
+
 struct membw_read_format {
 	__u64 value;         /* The value of the event */
 	__u64 time_enabled;  /* if PERF_FORMAT_TOTAL_TIME_ENABLED */
@@ -58,6 +73,10 @@ static struct mem_bw_counter bw_counters[MAX_BW_COUNTERS];
 static const struct resctrl_test *current_test;
 static const struct mem_bw_backend *mem_bw_backend;
 
+/* MPAM ABMC counter assignment state, see initialize_mem_bw_resctrl(). */
+static char mbm_assign_path[1024];
+static int mbm_assign_domain = -1;
+
 static void read_mem_bw_initialize_perf_event_attr(int i)
 {
 	memset(&bw_counters[i].pe, 0,
@@ -452,21 +471,63 @@ static int get_mem_bw_ref(float *bw_ref)
  * initialize_mem_bw_resctrl:	Appropriately populate "mbm_total_path"
  * @param:	Parameters passed to resctrl_val()
  * @domain_id:	Domain ID (cache ID; for MB, L3 cache ID)
+ *
+ * x86 resctrl reports per-RMID local bandwidth under L3_MON. MPAM exposes only
+ * total (read+write) bandwidth under MB_MON, indexed by the NUMA-node-keyed MB
+ * domain, and in "mbm_event" (ABMC) mode a hardware counter must be assigned to
+ * the group+domain before the file reads a value.
  */
 void initialize_mem_bw_resctrl(const struct resctrl_val_param *param,
 			       int domain_id)
 {
-	sprintf(mbm_total_path, CON_MBM_LOCAL_BYTES_PATH, RESCTRL_PATH,
-		param->ctrlgrp, domain_id);
+	const char *grp = param->ctrlgrp ? param->ctrlgrp : "";
+
+	if (resctrl_mon_feature_exists("L3_MON", "mbm_local_bytes")) {
+		sprintf(mbm_total_path, CON_MBM_LOCAL_BYTES_PATH, RESCTRL_PATH,
+			grp, domain_id);
+		return;
+	}
+
+	/* MPAM: total bandwidth under MB_MON. */
+	sprintf(mbm_total_path, CON_MBM_TOTAL_BYTES_PATH, RESCTRL_PATH,
+		grp, domain_id);
+
+	/* Assign an ABMC counter for this group+domain if in mbm_event mode. */
+	snprintf(mbm_assign_path, sizeof(mbm_assign_path), MBM_MB_ASSIGN_PATH,
+		 RESCTRL_PATH, grp);
+	if (access(mbm_assign_path, W_OK) == 0) {
+		FILE *fp = fopen(mbm_assign_path, "w");
+
+		if (fp) {
+			/* The assignment parser requires a trailing newline. */
+			fprintf(fp, "mbm_total_bytes:%d=e\n", domain_id);
+			fclose(fp);
+			mbm_assign_domain = domain_id;
+		}
+	} else {
+		/* Not in counter-assignment mode (or no MB_MON); nothing to do. */
+		mbm_assign_path[0] = '\0';
+	}
 }
 
 /*
- * mem_bw_ref_cleanup - Reset the selected reference-bandwidth backend
+ * mem_bw_ref_cleanup - Release an assigned ABMC counter and reset backend state
  *
- * Safe to call when no backend was selected.
+ * Group removal does not reclaim assigned MPAM counters, so the test must
+ * unassign explicitly. Safe to call when nothing was assigned.
  */
 void mem_bw_ref_cleanup(void)
 {
+	if (mbm_assign_domain >= 0 && mbm_assign_path[0]) {
+		FILE *fp = fopen(mbm_assign_path, "w");
+
+		if (fp) {
+			fprintf(fp, "mbm_total_bytes:%d=_\n", mbm_assign_domain);
+			fclose(fp);
+		}
+	}
+	mbm_assign_domain = -1;
+	mbm_assign_path[0] = '\0';
 	mem_bw_backend = NULL;
 }
 
diff --git a/tools/testing/selftests/resctrl/resctrlfs.c b/tools/testing/selftests/resctrl/resctrlfs.c
index b9c1bfb6cc02..893f6748c68c 100644
--- a/tools/testing/selftests/resctrl/resctrlfs.c
+++ b/tools/testing/selftests/resctrl/resctrlfs.c
@@ -121,11 +121,40 @@ static int get_resource_cache_level(const char *resource)
 	return get_cache_level(resource);
 }
 
+/*
+ * cpu_to_numa_node - NUMA node a logical CPU belongs to
+ * @cpu_no:	CPU number
+ *
+ * Reads the "node<N>" symlink in the CPU's sysfs directory.
+ *
+ * Return: NUMA node id (>= 0) on success, < 0 on failure.
+ */
+static int cpu_to_numa_node(int cpu_no)
+{
+	char cpu_dir[1024];
+	struct dirent *ep;
+	int node = -1;
+	DIR *dp;
+
+	snprintf(cpu_dir, sizeof(cpu_dir), "%s%d", PHYS_ID_PATH, cpu_no);
+	dp = opendir(cpu_dir);
+	if (!dp)
+		return -1;
+	while ((ep = readdir(dp))) {
+		if (!strncmp(ep->d_name, "node", 4) && isdigit(ep->d_name[4])) {
+			node = atoi(ep->d_name + 4);
+			break;
+		}
+	}
+	closedir(dp);
+	return node;
+}
+
 /*
  * get_domain_id - Get resctrl domain ID for a specified CPU
  * @resource:	resource name
  * @cpu_no:	CPU number
- * @domain_id:	domain ID (cache ID; for MB, L3 cache ID)
+ * @domain_id:	domain ID (cache ID; for MB, L3 cache ID, or NUMA node on MPAM)
  *
  * Return: >= 0 on success, < 0 on failure.
  */
@@ -135,6 +164,20 @@ int get_domain_id(const char *resource, int cpu_no, int *domain_id)
 	int cache_num;
 	FILE *fp;
 
+	/*
+	 * On MPAM the MB monitoring/allocation domains are keyed by NUMA node
+	 * (surfaced via a separate MB_MON group) and need not match the L3 cache
+	 * id used on x86, so resolve the MB domain from the CPU's NUMA node.
+	 */
+	if (!strcmp(resource, "MB") && resctrl_resource_exists("MB_MON")) {
+		int node = cpu_to_numa_node(cpu_no);
+
+		if (node >= 0) {
+			*domain_id = node;
+			return 0;
+		}
+	}
+
 	cache_num = get_resource_cache_level(resource);
 	if (cache_num < 0)
 		return cache_num;
-- 
2.43.0

[PATCH 4/5] selftests/resctrl: Gate the MBA test on features, not CPU vendor

[Richard Cheng]

The MBA test is skipped on every CPU that does not report
"GenuineIntel", although everything the test needs is discoverable at
runtime: an MB resource with a bandwidth monitor in resctrl, and a
reference PMU backend to validate against. The vendor gate also can
never match on architectures whose /proc/cpuinfo has no vendor_id
line at all, such as arm64.

Drop .vendor_specific from the MBA test and rely on the feature
check. On systems where any required piece is missing - including
architectures for which no reference bandwidth backend exists yet -
the test reports SKIP exactly as it did under the vendor gate.

Signed-off-by: Richard Cheng <icheng@nvidia.com>
---
 tools/testing/selftests/resctrl/mba_test.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/resctrl/mba_test.c b/tools/testing/selftests/resctrl/mba_test.c
index 6c85107c32ee..22a887a1b4f8 100644
--- a/tools/testing/selftests/resctrl/mba_test.c
+++ b/tools/testing/selftests/resctrl/mba_test.c
@@ -208,6 +208,12 @@ static int mba_run_test(const struct resctrl_test *test, const struct user_param
 	return ret;
 }
 
+/*
+ * The MBA test runs wherever resctrl exposes an MB resource, a memory-bandwidth
+ * monitor (x86 local bytes under L3_MON or MPAM total bytes under MB_MON), and
+ * an independent reference-bandwidth PMU exists to
+ * validate against. It is not gated on CPU vendor.
+ */
 static bool mba_feature_check(const struct resctrl_test *test)
 {
 	return test_resource_feature_check(test) &&
@@ -219,7 +225,6 @@ static bool mba_feature_check(const struct resctrl_test *test)
 struct resctrl_test mba_test = {
 	.name = "MBA",
 	.resource = "MB",
-	.vendor_specific = ARCH_INTEL,
 	.feature_check = mba_feature_check,
 	.run_test = mba_run_test,
 	.cleanup = mba_test_cleanup,
-- 
2.43.0

[PATCH 5/5] selftests/resctrl: Add NVIDIA SCF reference bandwidth backend

[Richard Cheng]

NVIDIA Grace/Vera systems expose the SoC coherency fabric (SCF)
uncore PMU, which counts per-socket DRAM traffic and can serve as the
independent reference for the MBA test, in the same role the iMC
plays on x86. Together with the MPAM MB/MB_MON support this makes the
MBA test runnable on arm64.

Implementation notes, all verified on a two-socket Vera system:

 - The arm_cspmu driver registers these PMUs under the generic
   nvidia_uncore_pmu_<n> name, so instances are identified by the
   PMIIDR value in the sysfs "identifier" file (product id 0x2cf =
   SCF) rather than by device name. This also keeps working should a
   future kernel register the PMU under a dedicated name.
 - The driver registers no named events on current silicon, and the
   documented byte counters (cmem_rd_data, gmem_*) read zero under
   load, so the backend programs raw events: 0xF1 (LLC refill, i.e.
   reads) and 0xF3 (LLC writeback, i.e. writes). Each event counts
   64-byte cache lines, the same scale as iMC CAS counts, and
   refill+writeback matches the read+write semantics of MPAM's
   mbm_total_bytes.
 - The SCF PMU is per-socket; the backend selects the instance(s)
   whose cpumask belongs to the benchmark CPU's package.
 - The events are opened system-wide (pid == -1), where inherit must
   be 0 or perf_event_open() fails on this PMU.

Validated against the MPAM MB resource: SCF and mbm_total_bytes agree
within 3-4% across all MBA schemata levels (10%..100%), against the
8% test threshold.

Signed-off-by: Richard Cheng <icheng@nvidia.com>
---
 tools/testing/selftests/resctrl/mba_test.c    |   2 +-
 tools/testing/selftests/resctrl/resctrl_val.c | 164 +++++++++++++++++-
 2 files changed, 162 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/resctrl/mba_test.c b/tools/testing/selftests/resctrl/mba_test.c
index 22a887a1b4f8..40352df5303c 100644
--- a/tools/testing/selftests/resctrl/mba_test.c
+++ b/tools/testing/selftests/resctrl/mba_test.c
@@ -211,7 +211,7 @@ static int mba_run_test(const struct resctrl_test *test, const struct user_param
 /*
  * The MBA test runs wherever resctrl exposes an MB resource, a memory-bandwidth
  * monitor (x86 local bytes under L3_MON or MPAM total bytes under MB_MON), and
- * an independent reference-bandwidth PMU exists to
+ * an independent reference-bandwidth PMU (Intel iMC or NVIDIA SCF) exists to
  * validate against. It is not gated on CPU vendor.
  */
 static bool mba_feature_check(const struct resctrl_test *test)
diff --git a/tools/testing/selftests/resctrl/resctrl_val.c b/tools/testing/selftests/resctrl/resctrl_val.c
index 0c3079d31340..e7b6eb255e3b 100644
--- a/tools/testing/selftests/resctrl/resctrl_val.c
+++ b/tools/testing/selftests/resctrl/resctrl_val.c
@@ -17,6 +17,23 @@
 #define MAX_BW_COUNTERS		20
 #define MAX_TOKENS		5
 
+/*
+ * NVIDIA SoC uncore/SCF (System Coherency Fabric) PMU: the ARM64 analog of the
+ * Intel iMC reference counters used to cross-check resctrl MBM. On Grace/Vera
+ * the fabric PMU registers generically as "nvidia_uncore_pmu_<n>"; the SCF
+ * instances (one per socket) are identified by the PMIIDR product-id field
+ * (0x2cf), independent of the device name. The driver does not register named
+ * events on this silicon, so the read (LLC refill) and write (LLC writeback)
+ * bandwidth events are encoded raw. Each event counts 64-byte cacheline
+ * transfers, i.e. the same SCALE as Intel CAS counts. Summing read+write
+ * matches resctrl's MBM total bytes (which includes reads and writes).
+ */
+#define UNCORE_NV_SCF		"nvidia_uncore_pmu"
+#define NV_SCF_PRODID		0x2cf
+#define NV_SCF_EVENT_READ	0xF1	/* scf_cache_refill */
+#define NV_SCF_EVENT_WRITE	0xF3	/* scf_cache_wb */
+#define CPU_PKG_ID_PATH		"/sys/devices/system/cpu/cpu%d/topology/physical_package_id"
+
 #define CON_MBM_LOCAL_BYTES_PATH		\
 	"%s/%s/mon_data/mon_L3_%02d/mbm_local_bytes"
 
@@ -258,6 +275,55 @@ static int num_of_imcs(void)
 	return count;
 }
 
+/* Read an unsigned value from a PMU device sysfs attribute. */
+static int read_pmu_value(const char *pmu, const char *attr, int base,
+			  unsigned long *val)
+{
+	char path[600], buf[64];
+	FILE *fp;
+
+	snprintf(path, sizeof(path), "%s/%s/%s", DYN_PMU_PATH, pmu, attr);
+	fp = fopen(path, "r");
+	if (!fp)
+		return -1;
+	if (!fgets(buf, sizeof(buf), fp)) {
+		fclose(fp);
+		return -1;
+	}
+	fclose(fp);
+	*val = strtoul(buf, NULL, base);
+	return 0;
+}
+
+/* physical_package_id (socket) of a logical CPU, < 0 on failure. */
+static int cpu_package_id(int cpu)
+{
+	char path[128];
+	int pkg = -1;
+	FILE *fp;
+
+	snprintf(path, sizeof(path), CPU_PKG_ID_PATH, cpu);
+	fp = fopen(path, "r");
+	if (!fp)
+		return -1;
+	if (fscanf(fp, "%d", &pkg) != 1)
+		pkg = -1;
+	fclose(fp);
+	return pkg;
+}
+
+/* True if @pmu is an NVIDIA SCF instance (PMIIDR product id == NV_SCF_PRODID). */
+static bool is_nvidia_scf_pmu(const char *pmu)
+{
+	unsigned long id;
+
+	if (strncmp(pmu, UNCORE_NV_SCF, sizeof(UNCORE_NV_SCF) - 1))
+		return false;
+	if (read_pmu_value(pmu, "identifier", 16, &id))
+		return false;
+	return ((id >> 20) & 0xfff) == NV_SCF_PRODID;
+}
+
 /* True if any x86 uncore iMC PMU ("uncore_imc_<n>") is present. */
 static bool intel_imc_present(void)
 {
@@ -283,6 +349,93 @@ static bool intel_imc_present(void)
 	return found;
 }
 
+/* True if any NVIDIA SCF PMU instance is present. */
+static bool nvidia_scf_present(void)
+{
+	struct dirent *ep;
+	bool found = false;
+	DIR *dp;
+
+	dp = opendir(DYN_PMU_PATH);
+	if (!dp)
+		return false;
+	while ((ep = readdir(dp))) {
+		if (is_nvidia_scf_pmu(ep->d_name)) {
+			found = true;
+			break;
+		}
+	}
+	closedir(dp);
+	return found;
+}
+
+/*
+ * nvidia_scf_setup_counters - Configure SCF read+write bandwidth counters
+ * @bench_cpu: CPU the benchmark is bound to
+ *
+ * The SCF PMU is per-socket; pick the instance(s) on the benchmark's socket and
+ * configure two raw counters each (LLC refill = reads, LLC writeback = writes).
+ * Per-socket so traffic from the throttled MB domain is attributed correctly.
+ *
+ * Return: 0 on success, < 0 on failure.
+ */
+static int nvidia_scf_setup_counters(int bench_cpu)
+{
+	static const __u64 events[] = { NV_SCF_EVENT_READ, NV_SCF_EVENT_WRITE };
+	int bench_pkg = cpu_package_id(bench_cpu);
+	struct dirent *ep;
+	int n = 0;
+	DIR *dp;
+
+	if (bench_pkg < 0) {
+		ksft_print_msg("Could not determine socket of CPU %d\n", bench_cpu);
+		return -1;
+	}
+
+	dp = opendir(DYN_PMU_PATH);
+	if (!dp) {
+		ksft_perror("Unable to open PMU directory");
+		return -1;
+	}
+
+	while ((ep = readdir(dp))) {
+		unsigned long type, pmu_cpu;
+		size_t e;
+
+		if (!is_nvidia_scf_pmu(ep->d_name))
+			continue;
+		/* cpumask holds the representative CPU of this PMU instance. */
+		if (read_pmu_value(ep->d_name, "cpumask", 10, &pmu_cpu))
+			continue;
+		if (cpu_package_id((int)pmu_cpu) != bench_pkg)
+			continue;
+		if (read_pmu_value(ep->d_name, "type", 10, &type))
+			continue;
+
+		for (e = 0; e < ARRAY_SIZE(events) && n < MAX_BW_COUNTERS; e++) {
+			memset(&bw_counters[n], 0,
+			       sizeof(bw_counters[n]));
+			bw_counters[n].type = (__u32)type;
+			bw_counters[n].event = events[e];
+			bw_counters[n].umask = 0;
+			bw_counters[n].cpu = (int)pmu_cpu;
+			read_mem_bw_initialize_perf_event_attr(n);
+			/* System-wide (pid == -1) CPU event: inherit is invalid. */
+			bw_counters[n].pe.inherit = 0;
+			n++;
+		}
+	}
+	closedir(dp);
+
+	if (n == 0) {
+		ksft_print_msg("No NVIDIA SCF PMU found for socket %d\n", bench_pkg);
+		return -1;
+	}
+
+	nr_bw_counters = n;
+	return 0;
+}
+
 /*
  * imc_setup_counters - Configure iMC CAS-count-read counters
  * @bench_cpu: CPU the benchmark is bound to
@@ -315,6 +468,11 @@ static const struct mem_bw_backend mem_bw_backends[] = {
 		.detect = intel_imc_present,
 		.setup_counters = imc_setup_counters,
 	},
+	{
+		.name = "NVIDIA SCF",
+		.detect = nvidia_scf_present,
+		.setup_counters = nvidia_scf_setup_counters,
+	},
 };
 
 /*
@@ -430,7 +588,7 @@ static void do_mem_bw_test(void)
  * get_mem_bw_ref - Memory bandwidth as reported by the reference PMU counters
  *
  * Sum all configured reference counters, scaled to MiB: read CAS counts on
- * the x86 iMC.
+ * the x86 iMC, LLC refill + writeback counts on the NVIDIA SCF.
  *
  * Return: = 0 on success. < 0 on failure.
  */
@@ -661,8 +819,8 @@ static int print_results_bw(char *filename, pid_t bm_pid, float bw_ref,
  * @bm_pid:		PID that runs the benchmark
  *
  * Measure memory bandwidth from resctrl and from the independent reference
- * PMU. Compare the two values to validate resctrl value. It takes 1 sec to
- * measure the data.
+ * PMU (iMC on x86, SCF on NVIDIA ARM64). Compare the two values to validate
+ * resctrl value. It takes 1 sec to measure the data.
  * resctrl does not distinguish between read and write operations so
  * its data includes all memory operations.
  */
-- 
2.43.0