Re: [PATCH v5 00/20] x86/resctrl : Support AMD Assignable Bandwidth Monitoring Counters (ABMC)

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Peter,

On 8/15/24 4:06 PM, Peter Newman wrote:
> On Wed, Aug 14, 2024 at 10:37 AM Reinette Chatre
> <reinette.chatre@intel.com> wrote:
>>
>> Hi Peter,
>>
>> On 8/2/24 3:50 PM, Peter Newman wrote:
>>> On Fri, Aug 2, 2024 at 1:55 PM Reinette Chatre
>>> <reinette.chatre@intel.com> wrote:
>>>> On 8/2/24 11:49 AM, Peter Newman wrote:
>>>>> On Fri, Aug 2, 2024 at 9:14 AM Reinette Chatre
>>>>>> I am of course not familiar with details of the software implementation
>>>>>> - could there be benefits to using it even if hardware counters are
>>>>>> supported?
>>>>>
>>>>> I can't see any situation where the user would want to choose software
>>>>> over hardware counters. The number of groups which can be monitored by
>>>>> software assignable counters will always be less than with hardware,
>>>>> due to the need for consuming one RMID (and the counters automatically
>>>>> allocated to it by the AMD hardware) for all unassigned groups.
>>>>
>>>> Thank you for clarifying. This seems specific to this software implementation,
>>>> and I missed that there was a shift from soft-RMIDs to soft-ABMC. If I remember
>>>> correctly this depends on undocumented hardware specific knowledge.
>>>
>>> For the benefit of anyone else who needs to monitor bandwidth on a
>>> large number of monitoring groups on pre-ABMC AMD implementations,
>>> hopefully a future AMD publication will clarify, at least on some
>>> existing, pre-ABMC models, exactly when the QM_CTR.U bit is set.
>>>
>>>
>>>>>
>>>>> The behavior as I've implemented today is:
>>>>>
>>>>> # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_events
>>>>> 0
>>>>>
>>>>> # cat /sys/fs/resctrl/info/L3_MON/mbm_control
>>>>> test//0=_;1=_;
>>>>> //0=_;1=_;
>>>>>
>>>>> # echo "test//1+l" > /sys/fs/resctrl/info/L3_MON/mbm_control
>>>>> # cat /sys/fs/resctrl/info/L3_MON/mbm_control
>>>>> test//0=_;1=tl;
>>>>> //0=_;1=_;
>>>>>
>>>>> # echo "test//1-t" > /sys/fs/resctrl/info/L3_MON/mbm_control
>>>>> # cat /sys/fs/resctrl/info/L3_MON/mbm_control
>>>>> test//0=_;1=_;
>>>>> //0=_;1=_;
>>>>>
>>>>>
>>>>
>>>> This highlights how there cannot be a generic/consistent interface between hardware
>>>> and software implementation. If resctrl implements something like above without any
>>>> other hints to user space then it will push complexity to user space since user space
>>>> would not know if setting one flag results in setting more than that flag, which may
>>>> force a user space implementation to always follow a write with a read that
>>>> needs to confirm what actually resulted from the write. Similarly, that removing a
>>>> flag impacts other flags needs to be clear without user space needing to "try and
>>>> see what happens".
>>>
>>> I'll return to this topic in the context of MPAM below...
>>>
>>>> It is not clear to me how to interpret the above example when it comes to the
>>>> RMID management though. If the RMID assignment is per group then I expected all
>>>> the domains of a group to have the same flag(s)?
>>>
>>> The group RMIDs are never programmed into any MSRs and the RMID space
>>> is independent in each domain, so it is still possible to do
>>> per-domain assignment. (and like with soft RMIDs, this enables us to
>>> create unlimited groups, but we've never been limited by the size of
>>> the RMID space)
>>>
>>> However, in our use cases, jobs are not confined to any domain, so
>>> bandwidth measurements must be done simultaneously in all domains, so
>>> we have no current use for per-domain assignment. But if any Google
>>> users did begin to see value in confining jobs to domains, this could
>>> change.
>>>
>>>>
>>>>>>
>>>>>>> However, If we don't expect to see these semantics in any other
>>>>>>> implementation, these semantics could be implicit in the definition of
>>>>>>> a SW assignable counter.
>>>>>>
>>>>>> It is not clear to me how implementation differences between hardware
>>>>>> and software assignment can be hidden from user space. It is possible
>>>>>> to let user space enable individual events and then silently upgrade it
>>>>>> to all events. I see two options here, either "mbm_control" needs to
>>>>>> explicitly show this "silent upgrade" so that user space knows which
>>>>>> events are actually enabled, or "mbm_control" only shows flags/events enabled
>>>>>> from user space perspective. In the former scenario, this needs more
>>>>>> user space support since a generic user space cannot be confident which
>>>>>> flags are set after writing to "mbm_control". In the latter scenario,
>>>>>> meaning of "num_mbm_cntrs" becomes unclear since user space is expected
>>>>>> to rely on it to know which events can be enabled and if some are
>>>>>> actually "silently enabled" when user space still thinks it needs to be
>>>>>> enabled the number of available counters becomes vague.
>>>>>>
>>>>>> It is not clear to me how to present hardware and software assignable
>>>>>> counters with a single consistent interface. Actually, what if the
>>>>>> "mbm_mode" is what distinguishes how counters are assigned instead of how
>>>>>> it is backed (hw vs sw)? What if, instead of "mbm_cntr_assignable" and
>>>>>> "mbm_cntr_sw_assignable" MBM modes the terms "mbm_cntr_event_assignable"
>>>>>> and "mbm_cntr_group_assignable" is used? Could that replace a
>>>>>> potential "mbm_assign_events" while also supporting user space in
>>>>>> interactions with "mbm_control"?
>>>>>
>>>>> If I understand this correctly, is this a preference that the info
>>>>> node be named differently if its value will have different units,
>>>>> rather than a second node to indicate what the value of num_mbm_cntrs
>>>>> actually means? This sounds reasonable to me.
>>>>
>>>> Indeed. As you highlighted, user space may not need to know if
>>>> counters are backed by hardware or software, but user space needs to
>>>> know what to expect from (how to interact with) interface.
>>>>
>>>>> I think it's also important to note that in MPAM, the MBWU (memory
>>>>> bandwidth usage) monitors don't have a concept of local versus total
>>>>> bandwidth, so event assignment would likely not apply there either.
>>>>> What the counted bandwidth actually represents is more implicit in the
>>>>> monitor's position in the memory system in the particular
>>>>> implementation. On a theoretical multi-socket system, resctrl would
>>>>> require knowledge about the system's architecture to stitch together
>>>>> the counts from different types of monitors to produce a local and
>>>>> total value. I don't know if we'd program this SoC-specific knowledge
>>>>> into the kernel to produce a unified MBM resource like we're
>>>>> accustomed to now or if we'd present multiple MBM resources, each only
>>>>> providing an mbm_total_bytes event. In this case, the counters would
>>>>> have to be assigned separately in each MBM resource, especially if the
>>>>> different MBM resources support a different number of counters.
>>>>>
>>>>
>>>> "total" and "local" bandwidth is already in grey area after the
>>>> introduction of mbm_total_bytes_config/mbm_local_bytes_config where
>>>> user space could set values reported to not be constrained by the
>>>> "total" and "local" terms. We keep sticking with it though, even in
>>>> this implementation that uses the "t" and "l" flags, knowing that
>>>> what is actually monitored when "l" is set is just what the user
>>>> configured via mbm_local_bytes_config, which theoretically
>>>> can be "total" bandwidth.
>>>
>>> If it makes sense to support a separate, group-assignment interface at
>>> least for MPAM, this would be a better fit for soft-ABMC, even if it
>>> does have to stay downstream.
>>
>> (apologies for the delay)
>>
>> Could we please take a step back and confirm/agree what is meant with "group-
>> assignment"? In a previous message [1] I latched onto the statement
>> "the implementation is assigning RMIDs to groups, assignment results in all
>> events being counted.". In this I understood "groups" to be resctrl groups
>> and I understood this to mean that when a (soft-ABMC) counter is assigned
>> it applies to the entire resctrl group (all domains, all events). The
>> subsequent example in [2] was thus unexpected to me when the interface
>> was used to assign a (soft-ABMC) counter to the group but not all domains
>> were impacted.
>>
>> Considering this, could you please elaborate what is meant with
>> "group assignment"?
> 
> By "group assignment", I just mean assigning counters to individual
> MBM events is not possible, or that assignment results in counters
> being assigned to all MBM events for a group in a domain.

Thank you for clarifying. I still think it is possible to use an entry
in "mbm_mode" to indicate to user space what to expect from the mbm_control
interface but I withdraw my original naming suggestions since it would create
confusion about what is meant by "group".

> 
> I only omitted per-domain assignment in soft-ABMC before because
> Google doesn't have a use-case for it. I started the prototype before
> Babu's proposed interface required domain-scoped assignments[1]. Now
> that some sort of domain selector is required, I'm reconsidering.

Could you please elaborate what you mean with the required "domain selector"?
The latest ABMC version (v6) added support for assigning all domains using '*'.

Reinette