Re: [PATCH v3] sched/topology: Enable topology_span_sane check only for debug builds

[Naman Jain <namjain@linux.microsoft.com>]

On 2/11/2025 11:22 AM, Shrikanth Hegde wrote:
> 
> 
> On 2/5/25 15:18, K Prateek Nayak wrote:
>> Hello all,
>>
>> On 2/3/2025 5:17 PM, Naman Jain wrote:
>>> [..snip..]
>>>
>>> Adding a link to the other patch which is under review.
>>> https://lore.kernel.org/lkml/20241031200431.182443-1-steve.wahl@hpe.com/
>>> Above patch tries to optimize the topology sanity check, whereas this
>>> patch makes it optional. We believe both patches can coexist, as even
>>> with optimization, there will still be some performance overhead for
>>> this check.
>>
>> I would like to discuss this parallelly here. Going back to the original
>> problem highlighted in [1], the topology_span_sane() came to be as a
>> result of how drivers/base/arch_topology.c computed the
>> cpu_coregroup_mask().
>>
>> [1] https://lore.kernel.org/all/1577088979-8545-1-git-send-email- 
>> prime.zeng@hisilicon.com/
>>
>> Originally described problematic topology is as follows:
>>
>>      **************************
>>      NUMA:               0-2,  3-7
>>      core_siblings:   0-3,  4-7
>>      **************************
>>
>> with the problematic bit in the handling being:
>>
>>      const struct cpumask *cpu_coregroup_mask(int cpu)
>>      {
>>              const cpumask_t *core_mask = 
>> cpumask_of_node(cpu_to_node(cpu));
>>
>>              ...
>>
>>              if (last_level_cache_is_valid(cpu)) {
>>                      /* If the llc_sibling is subset of node return 
>> llc_sibling */
>>                      if 
>> (cpumask_subset(&cpu_topology[cpu].llc_sibling, core_mask))
>>                              core_mask = &cpu_topology[cpu].llc_sibling;
>>
>>                      /* else the core_mask remains cpumask_of_node() */
>>              }
>>
>>              ...
>>
>>              return core_mask;
>>      }
>>
>> For CPU3, the llc_sibling 0-3 is not a subset of node mask 3-7, and the
>> fallback is to use 3-7. For CPUs 4-7, the llc_sibling 4-7 is a subset of
>> the node mask 3-7 and the coremask is returned a 4-7.
>>
>> In case of x86 (and perhaps other arch too) the arch/x86 bits ensure
>> that this inconsistency never happens for !NUMA domains using the
>> topology IDs. If a set of IDs match between two CPUs, the CPUs are set
>> in each other's per-CPU topology mask (see link_mask() usage and
>> match_*() functions in arch/x86/kernel/smpboot.c)
>>
>> If the set of IDs match with one CPU, it should match with all other
>> CPUs set in the cpumask for a given topology level. If it doesn't match
>> with one, it will not match with any other CPUs in the cpumask either.
>> The cpumasks of two CPUs can either be equal or disjoint at any given
>> level. Steve's optimization reverses this to check if the the cpumask
>> of set of CPUs match.
>>
>> Have there been any reports on an x86 system / VM where
>> topology_span_sane() was tripped? Looking at the implementation it
>> does not seem possible (at least to my eyes) with one exception of
>> AMD Fam 0x15 processors which set "cu_id" and match_smt() will look at
>> cu_id if the core_id doesn't match between 2 CPUs. It may so happen
>> that core IDs may match with one set of CPUs and cu_id may match with
>> another set of CPUs if the information from CPUID is faulty.
>>
>> What I'm getting to is that the arch specific topology parsing code
>> can set a "SDTL_ARCH_VERIFIED" flag indicating that the arch specific
>> bits have verified that the cpumasks are either equal or disjoint and
>> since sched_debug() is "false" by default, topology_span_sane() can
>> bail out if:
>>
>>      if (!sched_debug() && (tl->flags & SDTL_ARCH_VERIFIED))
>>          return;
>>
> 
> it would simpler to use sched_debug(). no?
> 
> Since it can be enabled at runtime by "echo Y > verbose", if one one 
> needs to enable even after boot. Wouldn't that suffice to run 
> topology_span_sane by doing a hotplug?
> 

I agree with your point. We are keeping it the same. Thanks.


Regards,
Naman

>> In case arch specific parsing was wrong, "sched_verbose" can always
>> be used to double check the topology and for the arch that require
>> this sanity check, Steve's optimized version of
>> topology_span_sane() can be run to be sure of the sanity.
>>
>> All this justification is in case folks want to keep
>> topology_span_sane() around but if no one cares, Naman and Saurabh's
>> approach works as intended.
>>