Re: [RFD] resctrl: reassigning a running container's CTRL_MON group

[Reinette Chatre <reinette.chatre@intel.com>]

Hi James,

On 10/19/2022 6:57 AM, James Morse wrote:
> Hi Peter,
> 
> On 17/10/2022 11:15, Peter Newman wrote:
>> On Wed, Oct 12, 2022 at 6:55 PM James Morse <james.morse@arm.com> wrote:
>>> You originally asked:
>>> | Any concerns about the CLOSID-reusing behavior?
>>>
>>> I don't think this will work well with MPAM ... I expect it will mess up the bandwidth
>>> counters.
>>>
>>> MPAM's equivalent to RMID is PMG. While on x86 CLOSID and RMID are independent numbers,
>>> this isn't true for PARTID (MPAM's version of CLOSID) and PMG. The PMG bits effectively
>>> extended the PARTID with bits that aren't used to look up the configuration.
>>>
>>> x86's monitors match only on RMID, and there are 'enough' RMID... MPAMs monitors are more
>>> complicated. I've seen details of a system that only has 1 bit of PMG space.
>>>
>>> While MPAM's bandwidth monitors can match just the PMG, there aren't expected to be enough
>>> unique PMG for every control/monitor group to have a unique value. Instead, MPAM's
>>> monitors are expected to be used with both the PARTID and PMG.
>>>
>>> ('bandwidth monitors' is relevant here, MPAM's 'cache storage utilisation' monitors can't
>>> match on just PMG at all - they have to be told the PARTID too)
>>>
>>>
>>> If you're re-using CLOSID like this, I think you'll end up with noisy measurements on MPAM
>>> systems as the caches hold PARTID/PMG values from before the re-use pattern changed, and
>>> the monitors have to match on both.
> 
>> Yes, that sounds like it would be an issue.
>>
>> Following your refactoring changes, hopefully the MPAM driver could
>> offer alternative methods for managing PARTIDs and PMGs depending on the
>> available hardware resources.
> 
> Mmmm, I don't think anything other than one-partid per control group and one-pmg per
> monitor group makes much sense.
> 
> 
>> If there are a lot more PARTIDs than PMGs, then it would fit well with a
>> user who never creates child MON groups. In case the number of MON
>> groups gets ahead of the number of CTRL_MON groups and you've run out of
>> PMGs, perhaps you would just try to allocate another PARTID and program
>> the same partitioning configuration before giving up.
> 
> User-space can choose to do this.
> If the kernel tries to be clever and do this behind user-space's back, it needs to
> allocate two monitors for this secretly-two-control-groups, and always sum the counters
> before reporting them to user-space.

If I understand this scenario correctly, the kernel is already doing this.
As implemented in mon_event_count() the monitor data of a CTRL_MON group is
the sum of the parent CTRL_MON group and all its child MON groups.

> If monitors are a contended resource, then you may be unable to monitor the
> secretly-two-control-groups group once the kernel has done this.

I am not viewing this as "secretly-two-control-groups" - there would still be
only one parent CTRL_MON group that dictates all the allocations. MON groups already
have a CLOSID (PARTID) property but at this time it is always identical to the parent
CTRL_MON group. The difference introduced is that some of the child MON groups
may have a different CLOSID (PARTID) from the parent.

> 
> I don't think the kernel should try to be too clever here.
> 

That is a fair concern but it may be worth exploring as it seems to address
a few ABI concerns and user space seems to be eyeing using a future "num_closid"
info as a check of "RDT/PQoS" vs "MPAM".

Reinette