Re: [PATCH v13 00/27] x86/resctrl : Support AMD Assignable Bandwidth Monitoring Counters (ABMC)

[Peter Newman <peternewman@google.com>]

Hi Babu/Reinette,

On Wed, May 21, 2025 at 1:44 AM Reinette Chatre
<reinette.chatre@intel.com> wrote:
>
> Hi Babu,
>
> On 5/20/25 4:25 PM, Moger, Babu wrote:
> > Hi Reinette,
> >
> > On 5/20/2025 1:23 PM, Reinette Chatre wrote:
> >> Hi Babu,
> >>
> >> On 5/20/25 10:51 AM, Moger, Babu wrote:
> >>> Hi Reinette,
> >>>
> >>> On 5/20/25 11:06, Reinette Chatre wrote:
> >>>> Hi Babu,
> >>>>
> >>>> On 5/20/25 8:28 AM, Moger, Babu wrote:
> >>>>> On 5/19/25 10:59, Peter Newman wrote:
> >>>>>> On Fri, May 16, 2025 at 12:52 AM Babu Moger <babu.moger@amd.com> wrote:
> >>>>
> >>>> ...
> >>>>
> >>>>>>> /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs: Reports the number of monitoring
> >>>>>>> counters available for assignment.
> >>>>>>
> >>>>>> Earlier I discussed with Reinette[1] what num_mbm_cntrs should
> >>>>>> represent in a "soft-ABMC" implementation where assignment is
> >>>>>> implemented by assigning an RMID, which would result in all events
> >>>>>> being assigned at once.
> >>>>>>
> >>>>>> My main concern is how many "counters" you can assign by assigning
> >>>>>> RMIDs. I recall Reinette proposed reporting the number of groups which
> >>>>>> can be assigned separately from counters which can be assigned.
> >>>>>
> >>>>> More context may be needed here. Currently, num_mbm_cntrs indicates the
> >>>>> number of counters available per domain, which is 32.
> >>>>>
> >>>>> At the moment, we can assign 2 counters to each group, meaning each RMID
> >>>>> can be associated with 2 hardware counters. In theory, it's possible to
> >>>>> assign all 32 hardware counters to a group—allowing one RMID to be linked
> >>>>> with up to 32 counters. However, we currently lack the interface to
> >>>>> support that level of assignment.
> >>>>>
> >>>>> For now, the plan is to support basic assignment and expand functionality
> >>>>> later once we have the necessary data structure and requirements.
> >>>>
> >>>> Looks like some requirements did not make it into this implementation.
> >>>> Do you recall the discussion that resulted in you writing [2]? Looks like
> >>>> there is a question to Peter in there on how to determine how many "counters"
> >>>> are available in soft-ABMC. I interpreted [3] at that time to mean that this
> >>>> information would be available in a future AMD publication.
> >>>
> >>> We already have a method to determine the number of counters in soft-ABMC
> >>> mode, which Peter has addressed [4].
> >>>
> >>> [4]
> >>> https://lore.kernel.org/lkml/20250203132642.2746754-1-peternewman@google.com/
> >>>
> >>> This appears to be more of a workaround, and I doubt it will be included
> >>> in any official AMD documentation. Additionally, the long-term direction
> >>> is moving towards ABMC.
> >>>
> >>> I don’t believe this workaround needs to be part of the current series. It
> >>> can be added later when soft-ABMC is implemented.
> >>
> >> Agreed. What about the plans described in [2]? (Thanks to Peter for
> >> catching this!).
> >>
> >> It is important to keep track of requirements while working on a feature to
> >> ensure that the implementation supports the planned use cases. Re-reading that
> >> thread it is not clear to me how soft-ABMC's per-group assignment would look.
> >> Could you please share how you see it progress from this implementation?
> >> This includes the single event vs. multiple event assignment. I would like to
> >> highlight that this is not a request for this to be supported in this implementation
> >> but there needs to be a plan for how this can be supported on top of interfaces
> >> established by this work.
> >>
> >
> > Here’s my current understanding of soft-ABMC. Peter may have a more in-depth perspective on this.
> >
> > Soft-ABMC:
> > a. num_mbm_cntrs: This is a software-defined limit based on the number of active RMIDs that can be supported. The value can be obtained using the code referenced in [4].

I would call it a hardware-defined limit that can be probed by software.

The main question is whether this file returns the exact number of
RMIDs hardware can track or double that number (mbm_total_bytes +
mbm_local_bytes) so that the value is always measured in events.

There's also the mongroup-RMID overcommit use case I described
above[1]. On Intel we can safely assume that there are counters to
back all RMIDs, so num_mbm_cntrs would be calculated directly from
num_rmids.

I realized this use case is more difficult to implement on MPAM,
because a PARTID is effectively a CLOSID+RMID, so deferring assigning
a unique PARTID to a group also results in it being in a different
allocation group. It will work if the unmonitored groups could find a
way to share PARTIDs, but this has consequences on allocation - but
hopefully no worse than sharing CLOSIDs on x86.

There's a lot of interest in monitoring ID overcommit in Google, so I
think it's worth it for me to investigate the additional structural
changes needed in resctrl (i.e., breaking the FS-level association
between mongroups and HW monitoring IDs). Such a framework could be a
better fit for soft-ABMC. For example, if overcommit is allowed, we
would just report the number of simultaneous RMIDs we were able to
probe as num_rmids. I would want the same shared assignment scheduler
to be able to work with RMIDs and counters, though.

Thanks,
-Peter

[1] https://lore.kernel.org/lkml/CALPaoChSzzU5mzMZsdT6CeyEn0WD1qdT9fKCoNW_ty4tojtrkw@mail.gmail.com/