From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D543643B6EB for ; Mon, 24 Aug 2026 14:40:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582410; cv=none; b=TdBJfpqo2EKblLxSekgjWbUhg9ACKsUADX7dSy33/tXPr88b99v5RmXom1w9RKhAxYo6kd38Je3QatQb/WX6PcSxMOkU8RhHYpC9rBzQ6vGVWEMXCTOAVxY9e/0+H5ocfHpyzB8eoN9D3WWtkGmck9oJsXth3NJ53In6UJE7MRk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787582410; c=relaxed/simple; bh=CJs7/+6q+c2W64mnjNGjBdb6+bSf98+Jn0J6O8nZBHo=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=FdMQKYFtlptRp5GWTv23h9WXVvqr8WVEW04wk1FpkkRtzj+yZ5XQlf9Tqwb926gRgfSse3DAOKEHZoo0ur0Kdh9vhYvqX5xR9ZTZu1pWNom1qsqm70vNk2RbEgoiqPPElqp+A/EgQEQ4UwK5ME8TJ97o63LaZAPpzfrsauIylik= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=WiEALrsQ; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="WiEALrsQ" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 283B61756; Mon, 24 Aug 2026 07:40:03 -0700 (PDT) Received: from [10.2.212.8] (e134344.arm.com [10.2.212.8]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1FBBF3F85F; Mon, 24 Aug 2026 07:40:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1787582407; bh=CJs7/+6q+c2W64mnjNGjBdb6+bSf98+Jn0J6O8nZBHo=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=WiEALrsQFV4ijPphQUoWDeizvWqUqmGV0k9Me+VifZlFHcY/4Asgog51jolCVlF2g xNrswHij1YuoWC8fKwl8ovrj6t9ixM5XMEqQK/4idkcFjCvtQsizJkXv6TXcABMQEq +TgAhgEpj5ASHgkqxvtneT+ed1iVejsuUJI4BHJM= Message-ID: <35bc4039-e7a1-4349-bcb4-54be988ab4bd@arm.com> Date: Mon, 24 Aug 2026 15:40:03 +0100 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC v2] arm,x86,fs/resctrl: Generic schema description Proof of Concept To: Reinette Chatre , "Moger, Babu" , "Luck, Tony" Cc: James Morse , Dave Martin , Babu Moger , Drew Fustini , Fenghua Yu , Chen Yu , Borislav Petkov , Thomas Gleixner , Dave Hansen , Peter Newman , "x86@kernel.org" , "linux-kernel@vger.kernel.org" References: <7a8b6cc8-d194-4af2-9fb9-2438d2dcae01@intel.com> <9d0239e6-318e-4a59-937d-a64b839e02e9@intel.com> <81f465d7-d12f-4414-9824-85d3cba5a36b@amd.com> Content-Language: en-US From: Ben Horgan In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hi Reinette, Babu, On 07/08/2026 23:53, Reinette Chatre wrote: > Hi Babu and Ben, > > On 8/7/26 11:33 AM, Moger, Babu wrote: >> Hi Reinette/Ben, >> >> On 8/7/2026 10:35 AM, Reinette Chatre wrote: >>> Hi Babu, >>> >>> On 8/7/26 7:14 AM, Moger, Babu wrote: >>>> On 8/6/2026 12:11 PM, Reinette Chatre wrote: >>>>> On 8/6/26 10:04 AM, Luck, Tony wrote: >>>>>> On Wed, Aug 05, 2026 at 04:57:04PM -0700, Reinette Chatre wrote: >>>>>>> On 8/5/26 9:59 AM, Ben Horgan wrote: >>>>>>>> Just given a go at running this on a model with MPAM and I can mount resctrl with this >>>>>>>> small patch to initialise the emulated_by lists so that list_empty() behaves. >>>>>>>> >>>>>>>> diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c >>>>>>>> index 21dcbf5764cc..638095649151 100644 >>>>>>>> --- a/drivers/resctrl/mpam_resctrl.c >>>>>>>> +++ b/drivers/resctrl/mpam_resctrl.c >>>>>>>> @@ -1018,6 +1018,7 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) >>>>>>>>           case RDT_RESOURCE_L3: >>>>>>>>                   mpam_ctrl->r_ctrl.type = RESCTRL_CTRL_BITMAP; >>>>>>>>                   mpam_ctrl->r_ctrl.name = RESCTRL_CTRL_NAME_DEF; >>>>>>>> +               INIT_LIST_HEAD(&mpam_ctrl->r_ctrl.emulated_by); >>>>>>>>                   INIT_LIST_HEAD_RCU(&mpam_ctrl->r_ctrl.domains); >>>>>>>>                   __set_bit(RESCTRL_BITMAP_FLAG_SPARSE, mpam_ctrl->r_ctrl.bitmap.flags); >>>>>>>>                   mpam_ctrl->r_ctrl.bitmap.cbm_len = class->props.cpbm_wd; >>>>>>>> @@ -1048,6 +1049,7 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) >>>>>>>>                   r->ctrl_scope = RESCTRL_L3_CACHE; >>>>>>>>                   mpam_ctrl->r_ctrl.type = RESCTRL_CTRL_SCALAR; >>>>>>>>                   mpam_ctrl->r_ctrl.name = RESCTRL_CTRL_NAME_DEF; >>>>>>>> +               INIT_LIST_HEAD(&mpam_ctrl->r_ctrl.emulated_by); >>>>>>>>                   INIT_LIST_HEAD_RCU(&mpam_ctrl->r_ctrl.domains); >>>>>>>> >>>>>>>>                   r->bw_throttle_mode = THREAD_THROTTLE_UNDEFINED; >>>>>>> >>>>>>> Thank you for this. Added this and it is now available in branch resctrl/controls_rfc_v2.1 >>>>>> >>>>>> Is same needed for x86? I don't see any initialization of the >>>>>> "r_ctrl.emulated_by" lists in similar initialization functions. >>>>> x86 "emulated_by" list initialization should be in both branches. >>>>> >>>>> Branch resctrl/controls_rfc_v2.1 combined all "emulated_by" list initialization >>>>> (for x86 and MPAM) into commit: >>>>> 3407e523c988 ("fs/resctrl: Introduce emulated controls and control mode") >>>>> >>>>> In the original resctrl/controls_rfc_v2 the x86 "emulated_by" list initialization can >>>>> be found in commit fbed64f80515 ("x86/resctrl: SAMPLE: Emulated controls") >>>>> >>>> >>>> Looking at the commit: >>>> >>>> commit 3407e523c988 ("fs/resctrl: Introduce emulated controls and control mode") >>>> >>>> Based on the patch description, emulated controls are intended to be >>>> used only when there is a difference between the native and legacy >>>> controls. If no such difference exists, both modes should operate >>>> identically. >>> >>> The original AMD MBA enabling did not follow the original percentage based MBA >>> control so resctrl essentially has *two* "legacy" MB controls today: one for AMD >>> and one for Intel, MPAM, and RISC-V (planned afaik). That cannot be changed now. >>> AMD would continue to expose the MB control that is not the percentage based "legacy" >>> control but actually AMD's native control. In short, yes, on AMD's "MB" control >>> the "legacy" and "native" control modes should operate identically. >>> At least now users could use the files in info/MB/schemata/MB/* to learn the >>> properties of the control. >>>> For MBA(AMD) and GMBA, there does not appear to be any difference >>>> between the two modes. Is that understanding correct? >>> It is not clear to me how resctrl should support AMD's GLBE. Note I am intentionally >>> not using GMBA since that already makes an assumption on how resctrl will support this. >>> Do you perhaps have an answer for Ben's question in >>> https://lore.kernel.org/lkml/1444b515-752e-4167-87f5-30ace189e05c@arm.com/ ? >> >> Yes. We are already at RFC v2. Let me respond here. Please see my response below. >> >>> >>> GLBE claims to enable users to allocate memory bandwidth at node scope but the memory >>> is managed at L3. This results in scenarios where, for example, a NUMA node can be >>> online and used but resctrl cannot expose it for bandwidth allocation when all the CPUs >>> at that NUMA node scope are offline. GLBE is thus not actually allocating memory bandwidth >>> at the NUMA node. >> >> Yea. That is correct. >> >> There are a couple of key differences when compared to the pure NUMA scope. >> >> 1. In some cases, a NUMA node is treated as the entire system. >> >>   >> https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/ >> >> 2. When a user updates the settings on a GLBE for a specific node, we need to update the MSRs within that node at the L3 scope: >> >> https://lore.kernel.org/lkml/a2a06bd290e68f902be9e7cc3ad35f0a2211b950.1776980182.git.babu.moger@amd.com/ >> >> >> Considering these differences, I think we should probably treat GLBE >> separate from MB_NODE scope. What do you think? > > I think so too, but to Ben's point we need to be clear on terminology here. There are two usages > of "scope" to consider: > 1) The scope of the *resource* being allocated. > Here "scope" applies to the resource. For cache there is L2 and L3 that indicates the > scope of the cache resource. There is also now two different scope to consider for > memory bandwidth allocation, "L3 scope" for memory bandwidth at egress from L3 and "NUMA/node" > scope for memory bandwidth at ingress to NUMA node. > > 2) The scope of the *control* used to allocate the resource. > So far the scope of the resource has been assumed that to be the same as the scope of the control. > That is, a resource at particular scope is allocated at that same scope. GLBE taught us this is not > always the case - memory bandwidth at L3 scope can be allocated at NUMA(with some caveat as you point out) > scope. > > We have gone back and forth on this. Currently upstream resctrl has control scope as a property of > the resource with expectation that the resource and its controls have the same scope. As a change > to this RFC v1 of this PoC had "scope" as a property of the control to support a control to have > different scope as resource. After deciding to treat MB_NODE as a new resource to support the CPU-less > nodes, RFC v2 changed this back to have scope a property of the resource. > > First, it looks to me as though the RFC v1 separating control scope from resource scope needs to > return. > > Second, how to name these resources/controls needs to be decided. This is what prompted my question > about future considerations to you because my original proposal in > https://lore.kernel.org/lkml/f5b6cec4-03d8-4a11-884d-d4579dab6b22@intel.com/ > was able to convey the resource and allocation scope via info hierarchy but it used the same > name in the schemata file that Ben highlighted could be problematic. > > I believe there is agreement that the resource should include the resource scope in its name. This > is currently done for L2 and L3, and planned to be done for MB_NODE (the "MB" resource does not have > "L3" in its name, this cannot be changed now, but "MB" resource is implicitly "L3" scope). The "NODE" > in MB_NODE is thus memory bandwidth allocation at NUMA node scope - "NODE" in MB_NODE is the *resource* > scope. > > Considering this I do not think that GLBE should use MB_NODE as you also state above. Since it does > not allocate memory bandwidth *resource* at node scope. > > We also discussed before (https://lore.kernel.org/lkml/c78169bc-e2d6-4583-96ec-09fa6dd6653a@intel.com/ ) > of having the control's scope part of the control's name. > > We could have a rule of thumb to include both resource scope and control scope in the > control name but only one instance is displayed if they are the same. > > For example, what do you think of something like below as an alternate proposal of what I mentioned in > https://lore.kernel.org/lkml/f5b6cec4-03d8-4a11-884d-d4579dab6b22@intel.com/: > > GLBE (booted with NPS < 4): > info/ > └── MB/ /* memory bandwidth allocation implicitly/legacy at L3 scope, think of this as "MB_L3" */ > └── schemata/ > ├── MB/ /* control scope = L3 */ > └── MB_L3NODE/ /* control scope = node ("L3" resource scope + "node" control scope) */ > > GLBE (booted with NPS = 4): > info/ > └── MB/ /* memory bandwidth allocation implicitly/legacy at L3 scope, think of this as "MB_L3" */ > └── schemata/ > ├── MB/ /* control scope = L3 */ > └── MB_L3SYSTEM/ /* control scope = system ("L3" resource scope + "system" control scope*/ I think this separation of control scope and resource scope works. My understanding is that, as the scope of the resource is L3, we can infer that the lifecycle of the resctrl domain is linked to the L3 cache and so the cpu hotplug lifecycle. The new part is that the size of the domain is determined by the control scope, a single L3 (L3 control scope as per current resctrl) or a collection of L3 instances as identified by a NUMA node id (NODE) or the whole system (SYSTEM). > > MPAM MSC: > info/ > ├── MB/ /* memory bandwidth allocation at L3 scope */ > │   └── schemata/ > │   └── MB/ /* control scope = L3 */ > └── MB_NODE/ /* memory bandwidth allocation at node scope */ > └── schemata/ > └── MB_NODE/ /* control scope = node */ Yes to the general structure. You may have just omitted this for clarity but I don't think we should introduce a control just called MB_NODE, it should be specify the type too, so one of MB_NODE_MIN, MB_NODE_MAX, MB_NODE_PROP etc. Thanks, Ben > > The GLBE controls make it clear that the *same* resource is allocated using two > different controls that have different scope. This creates an implicit dependency > between the two that is not quite captured but having two controls for the same > resource would already give user space some insight that their control values > need to be considered with care. > > I'd appreciate your, Ben's, and anybody else's thoughts on this. > >>> Should resctrl prepare for some future where an AMD system may support memory bandwidth >>> allocation at the NUMA node while also supporting GLBE (memory bandwidth allocation at L3)? >> >> No, I don't believe that's the case. At least, I haven't come across any information suggesting that. > Apologies, this was an unreasonable question from my side. We cannot predict the future. > We try to prepare for it with what we know today but new features always seem to come with unique > capabilities. > > Reinette >