From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9A13137F337 for ; Tue, 25 Aug 2026 15:20:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671235; cv=none; b=Nbtni7XFx/RmCDGpWmwaBm6soy5h+6gEhU1xsQ0nK+9nH7vs4BJRTjt/xnvpQOBUPZGl9BBG6PNTf6M5x7A9izLQcTBcViRSjFeuT3DRYLrpjYfO+YTqSy6pgXGerC4M03JqhtKj1t/cDx+Ab88bq0wDO+nSI1BEzEiJXy8KgPE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787671235; c=relaxed/simple; bh=rpo5GEAqTlZVcdReOhHEewMr9Cy8UANUvVBX8ipTKEs=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=TVWfhjovgHkOwiKuspkGXw7PL+1rFDrZknUy18qfnTJ9ssbHxnmQ8sOHaJAv2vVab5GkhcIflllDCi7S6hWT4vHuDYKFtL9w6n6/JOk89PNsgS6XInQTjiwq3adDknkaEhajwZifoUo4VPtP0uBgu8kEIYChI354xZEJKdY0mCc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=ZYasYhIi; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="ZYasYhIi" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9A80E1BA8; Tue, 25 Aug 2026 08:20:22 -0700 (PDT) Received: from [10.2.212.8] (e134344.arm.com [10.2.212.8]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5FBEA3F85F; Tue, 25 Aug 2026 08:20:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1787671226; bh=rpo5GEAqTlZVcdReOhHEewMr9Cy8UANUvVBX8ipTKEs=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=ZYasYhIieG+am11UyR86ra+4jxIA6S9k3hI3uf9GEuM+GzaLTZOL+n3KZZsIdEkgw L6LLRuj81KVdmgFgRYOdpTE0l8Pr4I7L2tZ4njHYuw8KavTe9I7+uKffuLtb5/WG1b 2E3fZA4yd3P8jXiCZoh5Vqn4f9WTxOuXxL2XpnS8= Message-ID: <7de132bf-4d84-4e44-b334-a5e4083d968b@arm.com> Date: Tue, 25 Aug 2026 16:20:22 +0100 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC v2] arm,x86,fs/resctrl: Generic schema description Proof of Concept To: Reinette Chatre , "Moger, Babu" , "Luck, Tony" Cc: James Morse , Dave Martin , Babu Moger , Drew Fustini , Fenghua Yu , Chen Yu , Borislav Petkov , Thomas Gleixner , Dave Hansen , Peter Newman , "x86@kernel.org" , "linux-kernel@vger.kernel.org" References: <7a8b6cc8-d194-4af2-9fb9-2438d2dcae01@intel.com> <9d0239e6-318e-4a59-937d-a64b839e02e9@intel.com> <81f465d7-d12f-4414-9824-85d3cba5a36b@amd.com> <35bc4039-e7a1-4349-bcb4-54be988ab4bd@arm.com> Content-Language: en-US From: Ben Horgan In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hi Reinette, On 24/08/2026 19:07, Reinette Chatre wrote: > Hi Ben, > > On 8/24/26 7:40 AM, Ben Horgan wrote: >> On 07/08/2026 23:53, Reinette Chatre wrote: >>> On 8/7/26 11:33 AM, Moger, Babu wrote: >>>> On 8/7/2026 10:35 AM, Reinette Chatre wrote: >>>>> On 8/7/26 7:14 AM, Moger, Babu wrote: >>>>>> On 8/6/2026 12:11 PM, Reinette Chatre wrote: >>>>>>> On 8/6/26 10:04 AM, Luck, Tony wrote: >>>>>>>> On Wed, Aug 05, 2026 at 04:57:04PM -0700, Reinette Chatre wrote: >>>>>>>>> On 8/5/26 9:59 AM, Ben Horgan wrote: >>>>>>>>>> Just given a go at running this on a model with MPAM and I can mount resctrl with this >>>>>>>>>> small patch to initialise the emulated_by lists so that list_empty() behaves. >>>>>>>>>> >>>>>>>>>> diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c >>>>>>>>>> index 21dcbf5764cc..638095649151 100644 >>>>>>>>>> --- a/drivers/resctrl/mpam_resctrl.c >>>>>>>>>> +++ b/drivers/resctrl/mpam_resctrl.c >>>>>>>>>> @@ -1018,6 +1018,7 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) >>>>>>>>>>           case RDT_RESOURCE_L3: >>>>>>>>>>                   mpam_ctrl->r_ctrl.type = RESCTRL_CTRL_BITMAP; >>>>>>>>>>                   mpam_ctrl->r_ctrl.name = RESCTRL_CTRL_NAME_DEF; >>>>>>>>>> +               INIT_LIST_HEAD(&mpam_ctrl->r_ctrl.emulated_by); >>>>>>>>>>                   INIT_LIST_HEAD_RCU(&mpam_ctrl->r_ctrl.domains); >>>>>>>>>>                   __set_bit(RESCTRL_BITMAP_FLAG_SPARSE, mpam_ctrl->r_ctrl.bitmap.flags); >>>>>>>>>>                   mpam_ctrl->r_ctrl.bitmap.cbm_len = class->props.cpbm_wd; >>>>>>>>>> @@ -1048,6 +1049,7 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res) >>>>>>>>>>                   r->ctrl_scope = RESCTRL_L3_CACHE; >>>>>>>>>>                   mpam_ctrl->r_ctrl.type = RESCTRL_CTRL_SCALAR; >>>>>>>>>>                   mpam_ctrl->r_ctrl.name = RESCTRL_CTRL_NAME_DEF; >>>>>>>>>> +               INIT_LIST_HEAD(&mpam_ctrl->r_ctrl.emulated_by); >>>>>>>>>>                   INIT_LIST_HEAD_RCU(&mpam_ctrl->r_ctrl.domains); >>>>>>>>>> >>>>>>>>>>                   r->bw_throttle_mode = THREAD_THROTTLE_UNDEFINED; >>>>>>>>> >>>>>>>>> Thank you for this. Added this and it is now available in branch resctrl/controls_rfc_v2.1 >>>>>>>> >>>>>>>> Is same needed for x86? I don't see any initialization of the >>>>>>>> "r_ctrl.emulated_by" lists in similar initialization functions. >>>>>>> x86 "emulated_by" list initialization should be in both branches. >>>>>>> >>>>>>> Branch resctrl/controls_rfc_v2.1 combined all "emulated_by" list initialization >>>>>>> (for x86 and MPAM) into commit: >>>>>>> 3407e523c988 ("fs/resctrl: Introduce emulated controls and control mode") >>>>>>> >>>>>>> In the original resctrl/controls_rfc_v2 the x86 "emulated_by" list initialization can >>>>>>> be found in commit fbed64f80515 ("x86/resctrl: SAMPLE: Emulated controls") >>>>>>> >>>>>> >>>>>> Looking at the commit: >>>>>> >>>>>> commit 3407e523c988 ("fs/resctrl: Introduce emulated controls and control mode") >>>>>> >>>>>> Based on the patch description, emulated controls are intended to be >>>>>> used only when there is a difference between the native and legacy >>>>>> controls. If no such difference exists, both modes should operate >>>>>> identically. >>>>> >>>>> The original AMD MBA enabling did not follow the original percentage based MBA >>>>> control so resctrl essentially has *two* "legacy" MB controls today: one for AMD >>>>> and one for Intel, MPAM, and RISC-V (planned afaik). That cannot be changed now. >>>>> AMD would continue to expose the MB control that is not the percentage based "legacy" >>>>> control but actually AMD's native control. In short, yes, on AMD's "MB" control >>>>> the "legacy" and "native" control modes should operate identically. >>>>> At least now users could use the files in info/MB/schemata/MB/* to learn the >>>>> properties of the control. >>>>>> For MBA(AMD) and GMBA, there does not appear to be any difference >>>>>> between the two modes. Is that understanding correct? >>>>> It is not clear to me how resctrl should support AMD's GLBE. Note I am intentionally >>>>> not using GMBA since that already makes an assumption on how resctrl will support this. >>>>> Do you perhaps have an answer for Ben's question in >>>>> https://lore.kernel.org/lkml/1444b515-752e-4167-87f5-30ace189e05c@arm.com/ ? >>>> >>>> Yes. We are already at RFC v2. Let me respond here. Please see my response below. >>>> >>>>> >>>>> GLBE claims to enable users to allocate memory bandwidth at node scope but the memory >>>>> is managed at L3. This results in scenarios where, for example, a NUMA node can be >>>>> online and used but resctrl cannot expose it for bandwidth allocation when all the CPUs >>>>> at that NUMA node scope are offline. GLBE is thus not actually allocating memory bandwidth >>>>> at the NUMA node. >>>> >>>> Yea. That is correct. >>>> >>>> There are a couple of key differences when compared to the pure NUMA scope. >>>> >>>> 1. In some cases, a NUMA node is treated as the entire system. >>>> >>>>   >>>> https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/ >>>> >>>> 2. When a user updates the settings on a GLBE for a specific node, we need to update the MSRs within that node at the L3 scope: >>>> >>>> https://lore.kernel.org/lkml/a2a06bd290e68f902be9e7cc3ad35f0a2211b950.1776980182.git.babu.moger@amd.com/ >>>> >>>> >>>> Considering these differences, I think we should probably treat GLBE >>>> separate from MB_NODE scope. What do you think? >>> >>> I think so too, but to Ben's point we need to be clear on terminology here. There are two usages >>> of "scope" to consider: >>> 1) The scope of the *resource* being allocated. >>> Here "scope" applies to the resource. For cache there is L2 and L3 that indicates the >>> scope of the cache resource. There is also now two different scope to consider for >>> memory bandwidth allocation, "L3 scope" for memory bandwidth at egress from L3 and "NUMA/node" >>> scope for memory bandwidth at ingress to NUMA node. >>> >>> 2) The scope of the *control* used to allocate the resource. >>> So far the scope of the resource has been assumed that to be the same as the scope of the control. >>> That is, a resource at particular scope is allocated at that same scope. GLBE taught us this is not >>> always the case - memory bandwidth at L3 scope can be allocated at NUMA(with some caveat as you point out) >>> scope. >>> >>> We have gone back and forth on this. Currently upstream resctrl has control scope as a property of >>> the resource with expectation that the resource and its controls have the same scope. As a change >>> to this RFC v1 of this PoC had "scope" as a property of the control to support a control to have >>> different scope as resource. After deciding to treat MB_NODE as a new resource to support the CPU-less >>> nodes, RFC v2 changed this back to have scope a property of the resource. >>> >>> First, it looks to me as though the RFC v1 separating control scope from resource scope needs to >>> return. >>> >>> Second, how to name these resources/controls needs to be decided. This is what prompted my question >>> about future considerations to you because my original proposal in >>> https://lore.kernel.org/lkml/f5b6cec4-03d8-4a11-884d-d4579dab6b22@intel.com/ >>> was able to convey the resource and allocation scope via info hierarchy but it used the same >>> name in the schemata file that Ben highlighted could be problematic. >>> >>> I believe there is agreement that the resource should include the resource scope in its name. This >>> is currently done for L2 and L3, and planned to be done for MB_NODE (the "MB" resource does not have >>> "L3" in its name, this cannot be changed now, but "MB" resource is implicitly "L3" scope). The "NODE" >>> in MB_NODE is thus memory bandwidth allocation at NUMA node scope - "NODE" in MB_NODE is the *resource* >>> scope. >>> >>> Considering this I do not think that GLBE should use MB_NODE as you also state above. Since it does >>> not allocate memory bandwidth *resource* at node scope. >>> >>> We also discussed before (https://lore.kernel.org/lkml/c78169bc-e2d6-4583-96ec-09fa6dd6653a@intel.com/ ) >>> of having the control's scope part of the control's name. >>> >>> We could have a rule of thumb to include both resource scope and control scope in the >>> control name but only one instance is displayed if they are the same. >>> >>> For example, what do you think of something like below as an alternate proposal of what I mentioned in >>> https://lore.kernel.org/lkml/f5b6cec4-03d8-4a11-884d-d4579dab6b22@intel.com/: >>> >>> GLBE (booted with NPS < 4): >>> info/ >>> └── MB/ /* memory bandwidth allocation implicitly/legacy at L3 scope, think of this as "MB_L3" */ >>> └── schemata/ >>> ├── MB/ /* control scope = L3 */ >>> └── MB_L3NODE/ /* control scope = node ("L3" resource scope + "node" control scope) */ >>> >>> GLBE (booted with NPS = 4): >>> info/ >>> └── MB/ /* memory bandwidth allocation implicitly/legacy at L3 scope, think of this as "MB_L3" */ >>> └── schemata/ >>> ├── MB/ /* control scope = L3 */ >>> └── MB_L3SYSTEM/ /* control scope = system ("L3" resource scope + "system" control scope*/ >> >> I think this separation of control scope and resource scope works. My understanding is that, as the >> scope of the resource is L3, we can infer that the lifecycle of the resctrl domain is linked to the >> L3 cache and so the cpu hotplug lifecycle. The new part is that the size of the domain is determined >> by the control scope, a single L3 (L3 control scope as per current resctrl) or a collection of L3 >> instances as identified by a NUMA node id (NODE) or the whole system (SYSTEM). > > Thank you very much. I agree. An additional way I use to simplify this is that the new "control scope" > is how resctrl communicates to user space what the domain ID used in, for example, schemata file > represents. > > One caveat is that the above layout is not something that the filesystem can enforce so for this > interface to remain consistent will depend on the support from all architectures. > >>> >>> MPAM MSC: >>> info/ >>> ├── MB/ /* memory bandwidth allocation at L3 scope */ >>> │   └── schemata/ >>> │   └── MB/ /* control scope = L3 */ >>> └── MB_NODE/ /* memory bandwidth allocation at node scope */ >>> └── schemata/ >>> └── MB_NODE/ /* control scope = node */ >> >> Yes to the general structure. You may have just omitted this for clarity but I don't think we should >> introduce a control just called MB_NODE, it should be specify the type too, so one of MB_NODE_MIN, >> MB_NODE_MAX, MB_NODE_PROP etc. > > I agree. One uncertainty here is how the MB_NODE resource's control mode should be presented to > user space. I think it may be most intuitive to present "native" as the one and only control mode > option? Yes, "native" being the only option here makes sense to me. Thanks, Ben > In the current implementation the "native" and "legacy" control mode of a resource without > any emulated controls will result in the same controls enabled. > > Reinette > >