Re: [PATCH v2 1/9] selftests/resctrl: Improve accuracy of cache occupancy test

Reinette Chatre
2026-03-06T19:24:53Z

Hi Ilpo,

On 3/6/26 1:47 AM, Ilpo Järvinen wrote:
> On Tue, 3 Mar 2026, Reinette Chatre wrote:
>
>> Dave Martin reported inconsistent CMT test failures. In one experiment
>> the first run of the CMT test failed because of too large (24%) difference
>> between measured and achievable cache occupancy while the second run passed
>> with an acceptable 4% difference.
>>
>> The CMT test is susceptible to interference from the rest of the system.
>> This can be demonstrated with a utility like stress-ng by running the CMT
>> test while introducing cache misses using:
>>
>> stress-ng --matrix-3d 0 --matrix-3d-zyx
>>
>> Below shows an example of the CMT test failing because of a significant
>> difference between measured and achievable cache occupancy when run with
>> interference:
>> # Starting CMT test ...
>> # Mounting resctrl to "/sys/fs/resctrl"
>> # Cache size :56623104
>> # Writing benchmark parameters to resctrl FS
>> # Benchmark PID: 3275
>> # Checking for pass/fail
>> # Fail: Check cache miss rate within 15%
>> # Percent diff=97
>> # Number of bits: 5
>> # Average LLC val: 501350
>> # Cache span (bytes): 23592960
>> not ok 1 CMT: test
>>
>> The CMT test creates a new control group that is also capable of monitoring
>> and assigns the workload to it. The workload allocates a buffer that by
>> default fills a portion of the L3 and keeps reading from the buffer,
>> measuring the L3 occupancy at intervals. The test passes if the workload's
>> L3 occupancy is within 15% of the buffer size.
>>
>> By not adjusting any capacity bitmasks the workload shares the cache with
>> the rest of the system. Any other task that may be running could evict
>> the workload's data from the cache causing it to have low cache occupancy.
>>
>> Reduce interference from the rest of the system by ensuring that the
>> workload's control group uses the capacity bitmask found in the user
>> parameters for L3 and that the rest of the system can only allocate into
>> the inverse of the workload's L3 cache portion. Other tasks can thus no
>> longer evict the workload's data from L3.
>>
>> Take the L2 cache into account to further improve test accuracy.
>> By default the buffer size is the same as the L3 portion that the workload
>> can allocate into. This buffer size does not take into account that some
>> of the workload's data may land in L2/L1. Address this in two ways:
>> - Reduce the amount of L2 cache the workload can allocate into to the
>
> "into to the" sounds wrong.

How about:
"Reduce the workload's L2 cache allocation to the minimum on systems that
support L2 cache allocation."

>
>> minimum on systems that support L2 cache allocation.
>> - Increase the buffer size to accommodate data that may be allocated into
>> the L2 cache. Use a buffer size double the L3 portion to keep using the
>> L3 portion size as goal for L3 occupancy while taking into account that
>> some of the data may be in L2.
>
> To me the this control over L2 looks logically pretty separate from the
> inverse of L3 portion control so it would seem like logically belong to a
> separate change.

Sure. Both adjustments are needed to get results described below so I'll move
all descriptions surrounding the before/after performance numbers to the cover
letter. Looks like partial fixes cannot use the Closes tag so instead of Closes
I'll use the Link tag in both patches to point to the original report.

>
>> With the above adjustments the CMT test is more consistent. Repeating the
>> CMT test while generating interference with stress-ng on a sample
>> system after applying the fixes show significant improvement in test
>> accuracy:
>>
>> # Starting CMT test ...
>> # Mounting resctrl to "/sys/fs/resctrl"
>> # Cache size :56623104
>> # Writing benchmark parameters to resctrl FS
>> # Write schema "L3:0=fe0" to resctrl FS
>> # Write schema "L3:0=1f" to resctrl FS
>> # Benchmark PID: 3223
>> # Checking for pass/fail
>> # Pass: Check cache miss rate within 15%
>> # Percent diff=3
>> # Number of bits: 5
>> # Average LLC val: 22811443
>> # Cache span (bytes): 23592960
>> ok 1 CMT: test
>>
>> Reported-by: Dave Martin <Dave.Martin@arm.com>
>> Closes: https://lore.kernel.org/lkml/aO+7MeSMV29VdbQs@e133380.arm.com/
>> Signed-off-by: Reinette Chatre <reinette.chatre@intel.com>
>> ---
>> Changes since v1:
>> - Fix typo in changelog: "data my be in L2" -> "data may be in L2".
>> ---
>> tools/testing/selftests/resctrl/cmt_test.c | 35 ++++++++++++++++---
>> tools/testing/selftests/resctrl/mba_test.c | 4 ++-
>> tools/testing/selftests/resctrl/mbm_test.c | 4 ++-
>> tools/testing/selftests/resctrl/resctrl.h | 4 ++-
>> tools/testing/selftests/resctrl/resctrl_val.c | 2 +-
>> 5 files changed, 41 insertions(+), 8 deletions(-)
>>
>> diff --git a/tools/testing/selftests/resctrl/cmt_test.c b/tools/testing/selftests/resctrl/cmt_test.c
>> index d09e693dc739..44e9938dfafd 100644
>> --- a/tools/testing/selftests/resctrl/cmt_test.c
>> +++ b/tools/testing/selftests/resctrl/cmt_test.c
>> @@ -19,12 +19,39 @@
>> #define CON_MON_LCC_OCCUP_PATH \
>> "%s/%s/mon_data/mon_L3_%02d/llc_occupancy"
>>
>> -static int cmt_init(const struct resctrl_val_param *param, int domain_id)
>> +/*
>> + * Initialize capacity bitmasks (CBMs) for control group being tested,
>> + * default resource group to prevent its tasks from interfering with test,
>> + * and L2 resource of control group to minimize allocations into L2 if
>> + * possible to better predict L3 occupancy.
>> + */
>> +static int cmt_init(const struct resctrl_test *test,
>> + const struct user_params *uparams,
>> + const struct resctrl_val_param *param, int domain_id)
>> {
>> + unsigned long long_mask;
>> + char schemata[64];
>> + int ret;
>> +
>> sprintf(llc_occup_path, CON_MON_LCC_OCCUP_PATH, RESCTRL_PATH,
>> param->ctrlgrp, domain_id);
>>
>> - return 0;
>> + ret = get_full_cbm(test->resource, &long_mask);
>> + if (ret)
>> + return ret;
>> +
>> + snprintf(schemata, sizeof(schemata), "%lx", ~param->mask & long_mask);
>
> I don't know why this is variable is called "long_mask", the type seems
> pretty unrelated to its use. Perhaps change it to e.g. full_mask?

It is called long_mask to be consistent with the same value used elsewhere
in this file. This is not required. I can rename to full_mask.

>
> Seems otherwise fine AFAICT.
>

Thank you very much for taking a look.

Reinette