[PATCH v6 0/2] selftests/resctrl: Check the effect of the CBM in CMT test

[PATCH v6 0/2] selftests/resctrl: Check the effect of the CBM in CMT test

[Richard Cheng]

Previous version [1] added cl_flush() and sb() for aarch64.
As discussed there this does not work on arm64, and Ben and
Reinette suggested to enhance the CMT test with a forgiving
check instead.
This series does that without needing a cache flush.

Patch 1 fixes the CMT test failing when all cache bits are tested.

Patch 2 measures a second time with all CBM bits set while the same
benchmark keeps running, and checks that LLC occupancy increased by
more than 15%.

Tested on one MPAM system (16 bit CBM, 172 MB L3).
- Default (5 bits) and -n 2, 4, 8: occupancy increases by 75% to 130%
  with the wider CBM, test passes.
- -n 9 and -n 16: second measurement is skipped, test passes. -n 16
  fails without patch 1 (write of "L3:1=0" fails).
- With the second schemata write changed to not widen the CBM the new
  check fails as it should. Widening 8 bits to 9 (+11%) fails, to 10
  (+23%) passes.
- -n 1 fails the existing check with and without this series,
  occupancy is 35% below the expected value on this system.
- The CMT test takes about 5 seconds longer.

Changes since v5:
- Drop "Implement cl_flus() and sb() for aarch64", replaced by
  patch 2.
- Add patch 1.
- "Recognise aarch64 as a vendor for L3_NONCONT_CAT" is sent
  separately, it does not depend on this series.

[1] https://lore.kernel.org/r/20261012074729.40042-1-icheng@nvidia.com


Richard Cheng (2):
  selftests/resctrl: Fix CMT test failure when all cache bits are tested
  selftests/resctrl: Check that CBM limits LLC occupancy in CMT test

 tools/testing/selftests/resctrl/cmt_test.c | 85 +++++++++++++++++++---
 1 file changed, 73 insertions(+), 12 deletions(-)


base-commit: 30af56a227e27b892d34bc44c23298b5274913e3
-- 
2.43.0

[PATCH v6 1/2] selftests/resctrl: Fix CMT test failure when all cache bits are tested

[Richard Cheng]

The CMT test accepts any number of bits from 1 to the length of the full
cache bit mask via "-n". Since cmt_init() gives the default resource
group the inverse of the tested mask to keep other tasks out of the
tested cache portion.

When all bits are tested the inverse mask is empty. resctrl rejects an
empty mask with "Mask out of range" on systems that require at least one
bit to be set, cmt_init() fails and the test fails before taking any
measurement.

Only write the schemata of the default resource group when there're bits
left for it. When all bits are tested the default resource group keeps
its allocation.

Fixes: 7a214b4ec1c9 ("selftests/resctrl: Improve accuracy of cache occupancy test")
Signed-off-by: Richard Cheng <icheng@nvidia.com>
---
 tools/testing/selftests/resctrl/cmt_test.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/resctrl/cmt_test.c b/tools/testing/selftests/resctrl/cmt_test.c
index ccb6fe881a94..5d0bb27b7817 100644
--- a/tools/testing/selftests/resctrl/cmt_test.c
+++ b/tools/testing/selftests/resctrl/cmt_test.c
@@ -42,10 +42,13 @@ static int cmt_init(const struct resctrl_test *test,
 	if (ret)
 		return ret;
 
-	snprintf(schemata, sizeof(schemata), "%lx", ~param->mask & full_mask);
-	ret = write_schemata("", schemata, uparams->cpu, test->resource);
-	if (ret)
-		return ret;
+	/* No bits are left for the default group when all bits are tested. */
+	if (~param->mask & full_mask) {
+		snprintf(schemata, sizeof(schemata), "%lx", ~param->mask & full_mask);
+		ret = write_schemata("", schemata, uparams->cpu, test->resource);
+		if (ret)
+			return ret;
+	}
 
 	snprintf(schemata, sizeof(schemata), "%lx", param->mask);
 	ret = write_schemata(param->ctrlgrp, schemata, uparams->cpu, test->resource);
-- 
2.43.0

[PATCH v6 2/2] selftests/resctrl: Check that CBM limits LLC occupancy in CMT test

[Richard Cheng]

The CMT test measures LLC occupancy with one CBM and never checks that
the CBM limits it.

Measure again with all CBM bits set while the benchmark keeps running.
This needs no cache flush, what the group has in the cache is covered by
the wider CBM too.

The buffer is twi the tested portion, so occupancy should about double.
Only require an increase above MAX_DIFF_PERCENT.

Skip this for an user-provided benchmark or a buffer larger than the
cache.

Signed-off-by: Richard Cheng <icheng@nvidia.com>
---
 tools/testing/selftests/resctrl/cmt_test.c | 74 +++++++++++++++++++---
 1 file changed, 66 insertions(+), 8 deletions(-)

diff --git a/tools/testing/selftests/resctrl/cmt_test.c b/tools/testing/selftests/resctrl/cmt_test.c
index 5d0bb27b7817..5962aeb0cd8b 100644
--- a/tools/testing/selftests/resctrl/cmt_test.c
+++ b/tools/testing/selftests/resctrl/cmt_test.c
@@ -19,6 +19,12 @@
 #define CON_MON_LCC_OCCUP_PATH		\
 	"%s/%s/mon_data/mon_L3_%02d/llc_occupancy"
 
+/*
+ * CBM that the control group being tested is widened to after the first
+ * NUM_OF_RUNS measurements. Zero when measuring with a single CBM only.
+ */
+static unsigned long wide_mask;
+
 /*
  * Initialize capacity bitmasks (CBMs) of:
  * - control group being tested per test parameters,
@@ -62,10 +68,26 @@ static int cmt_setup(const struct resctrl_test *test,
 		     const struct user_params *uparams,
 		     struct resctrl_val_param *p)
 {
-	/* Run NUM_OF_RUNS times */
-	if (p->num_of_runs >= NUM_OF_RUNS)
+	char schemata[64];
+	int ret;
+
+	/* Run NUM_OF_RUNS times with each CBM */
+	if (p->num_of_runs >= (wide_mask ? 2 * NUM_OF_RUNS : NUM_OF_RUNS))
 		return END_OF_TESTS;
 
+	/*
+	 * Widen the CBM while the benchmark keeps running. All that the
+	 * control group has in the cache was allocated using the narrower
+	 * CBM and thus is covered by the wider CBM also, no flush is needed
+	 * between the two sets of measurements.
+	 */
+	if (wide_mask && p->num_of_runs == NUM_OF_RUNS) {
+		snprintf(schemata, sizeof(schemata), "%lx", wide_mask);
+		ret = write_schemata(p->ctrlgrp, schemata, uparams->cpu, test->resource);
+		if (ret)
+			return ret;
+	}
+
 	p->num_of_runs++;
 
 	return 0;
@@ -107,10 +129,12 @@ static int show_results_info(unsigned long sum_llc_val, int no_of_bits,
 
 static int check_results(struct resctrl_val_param *param, size_t span, int no_of_bits)
 {
+	unsigned long sum_llc_occu_resc = 0, sum_llc_occu_wide = 0;
 	char *token_array[8], temp[512];
-	unsigned long sum_llc_occu_resc = 0;
-	int runs = 0;
+	int runs = 0, runs_wide = 0;
+	unsigned long llc_occu, avg, avg_wide;
 	FILE *fp;
+	int ret, ret_wide;
 
 	ksft_print_msg("Checking for pass/fail\n");
 	fp = fopen(param->filename, "r");
@@ -130,13 +154,36 @@ static int check_results(struct resctrl_val_param *param, size_t span, int no_of
 		}
 
 		/* Field 3 is llc occ resc value */
-		sum_llc_occu_resc += strtoul(token_array[3], NULL, 0);
-		runs++;
+		llc_occu = strtoul(token_array[3], NULL, 0);
+		if (runs < NUM_OF_RUNS) {
+			sum_llc_occu_resc += llc_occu;
+			runs++;
+		} else {
+			sum_llc_occu_wide += llc_occu;
+			runs_wide++;
+		}
 	}
 	fclose(fp);
 
-	return show_results_info(sum_llc_occu_resc, no_of_bits, span,
-				 MAX_DIFF, MAX_DIFF_PERCENT, runs, true);
+	ret = show_results_info(sum_llc_occu_resc, no_of_bits, span,
+				MAX_DIFF, MAX_DIFF_PERCENT, runs, true);
+	if (!runs_wide)
+		return ret;
+
+	/*
+	 * The buffer is twice the size of the cache portion being tested and
+	 * LLC occupancy is expected to about double with the wider CBM. Only
+	 * check that it increased by more than the tolerated difference.
+	 */
+	avg = sum_llc_occu_resc / runs;
+	avg_wide = sum_llc_occu_wide / runs_wide;
+	ret_wide = avg_wide <= avg + avg * MAX_DIFF_PERCENT / 100;
+	ksft_print_msg("%s Check cache occupancy increased with wider CBM\n",
+		       ret_wide ? "Fail:" : "Pass:");
+	ksft_print_msg("Number of bits: %u\n", count_bits(wide_mask));
+	ksft_print_msg("Average LLC val: %lu\n", avg_wide);
+
+	return ret || ret_wide;
 }
 
 static void cmt_test_cleanup(void)
@@ -193,6 +240,17 @@ static int cmt_run_test(const struct resctrl_test *test, const struct user_param
 		param.fill_buf = &fill_buf;
 	}
 
+	/*
+	 * The buffer is twice the size of the cache portion being tested. If
+	 * the entire buffer fits in the cache then also measure with the full
+	 * CBM to check that the tested CBM is what limits the LLC occupancy.
+	 */
+	wide_mask = 0;
+	if (param.fill_buf && 2 * n <= count_of_bits)
+		wide_mask = long_mask;
+	else
+		ksft_print_msg("Not measuring with wider CBM\n");
+
 	remove(RESULT_FILE_NAME);
 
 	ret = resctrl_val(test, uparams, &param);
-- 
2.43.0