From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9294A3590AE for ; Thu, 6 Aug 2026 09:24:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786008288; cv=none; b=GpKrBXWHHQkqgLV9hJ+xIp0ingKoe2xH4e9ysn2u9U5r6VrSGW3x3E6fBIB94/ACAFJrwBEfshGPDzZDDtO8D21rABSHbf9x++2axYcexyAr0aTvPxvsf6qMojvYe0Hp3awRP3tQZUgV6ujxRrkWiJNLr0EvF23zsxE8U9umGFA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786008288; c=relaxed/simple; bh=L5i7Ftb/5iFtwN15+aqudVSkRAIkZsI2TYbXYC2uHWI=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=Kcj7QgEhSAJZvFYFd/UsB4NQ7P8KPIp1G89AayHa0L9I86n2LZvxu7ZcqgqvUneGUyXjb7jNQqAljuqxqYQ/d/5M1S7/a+rq1GvjtdTyrcm6SODam6osPbLzZW5Vwi7+W9doeXErl8g3Q0K+Q7gV9VC/EJ6d/sHOP42tpva7KD0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=XMVwO/LV; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="XMVwO/LV" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1A16F153B; Thu, 6 Aug 2026 02:24:36 -0700 (PDT) Received: from [10.2.212.8] (e134344.arm.com [10.2.212.8]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6E3F93F66F; Thu, 6 Aug 2026 02:24:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1786008280; bh=L5i7Ftb/5iFtwN15+aqudVSkRAIkZsI2TYbXYC2uHWI=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=XMVwO/LVrjubhpsuugu/yvYl2BFA/jzwaPBbocmF5oXF5HqEtDfDzgPOBGYXJOSqe lJ6Jq7EmSgZeRbnmPyEUradUUT0VW4oOi6cNrOwzqcmPsAks8gKQGy7U9nMxpOECqs YPukkzL4GGz1rarumCdP/QD6QaoKny4MlUZPUoIo= Message-ID: <1444b515-752e-4167-87f5-30ace189e05c@arm.com> Date: Thu, 6 Aug 2026 10:24:37 +0100 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept To: Reinette Chatre , Babu Moger , Fenghua Yu , Tony Luck , James Morse , Dave Martin , Drew Fustini , Chen Yu Cc: Borislav Petkov , Thomas Gleixner , Dave Hansen , Peter Newman , "x86@kernel.org" , "linux-kernel@vger.kernel.org" References: <0fc6df54-26c7-43fa-948a-528cd94937f1@arm.com> <9049378c-699a-4155-b1e4-737a1d7265d5@intel.com> <57740b97-80ee-4632-bca3-dc43cd7776c2@arm.com> <44f26cd4-be79-476e-b002-7ccfb7705179@intel.com> <749bd904-523d-4e9d-8493-0e8cfd79949e@arm.com> <9db33feb-cf04-420c-a99a-e31e4b8e4954@arm.com> <8fd6caed-820f-457a-a1ef-a0a006fa52aa@intel.com> <4ef15dde-2fbb-4763-93b6-4333b02d6859@arm.com> <7b751c28-2f04-42b7-b957-af6447e7f824@intel.com> <34b95afb-8b60-4680-9ad1-90c5b24e8fb7@arm.com> <08f016bc-2ba6-439e-bb3e-20061166402c@intel.com> <21e614b8-50fa-49e4-87c3-e6bdb4e83ab1@amd.com> <653c9a0c-c665-4016-90b2-3f55e06050b3@amd.com> <618ba724-d734-43d3-b90e-11ba7f31c2b6@arm.com> Content-Language: en-US From: Ben Horgan In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hi Reinette, On 05/08/2026 07:06, Reinette Chatre wrote: > Hi Ben and Babu, > > On 8/4/26 8:11 AM, Ben Horgan wrote: >> On 8/4/26 15:09, Babu Moger wrote: >>> On 8/4/26 04:11, Ben Horgan wrote: >>>> On 7/22/26 18:02, Babu Moger wrote: >>>>> On 7/22/26 05:47, Ben Horgan wrote: >>>>>> On 7/21/26 21:02, Babu Moger wrote: >>>>>>> On 7/21/26 12:30, Reinette Chatre wrote: >>>>>>>> On 7/21/26 6:23 AM, Ben Horgan wrote: >>>>>>>>> On 7/20/26 23:54, Reinette Chatre wrote: >>>>>>>>>> On 7/20/26 6:30 AM, Ben Horgan wrote: >>>>>>>> ...> >>>>>>>>> The former, info/ contains a directory for each allocation scope of each resource. >>>>> >>>>>>>>>> >>>>>>>>>> The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory >>>>>>>>>> bandwidth allocation >>>>>>>>>> in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains >>>>>>>>>> controlled at L3 scope), >>>>>>>>>> except for one configuration where it is "SYSTEM"(?) scope. >>>>>>>>>> Ref.: https://lore.kernel.org/ >>>>>>>>>> lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@amd.com/ >>>>>>>>> >>>>>>>>> Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems >>>>>>>>> to be >>>>>>>>> accidental that it maps to the NUMA node but really the scope is just a grouping of L3 >>>>>>>>> instances. >>>>>>>>> For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync >>>>>>>>> with >>>>>>>>> the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether >>>>>>>>> any of >>>>>>>>> the CPUs and so L3 instances in the group are online. Am I correct here? >>>>>>>>> >>>>>>>>> Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the >>>>>>>>> traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 >>>>>>>>> group >>>>>>>>> then a task running on a CPU affine to that L3 group won't be throttled unless that particular >>>>>>>>> domain is throttled but with NUMA node domains it may be throttled if it has traffic going to >>>>>>>>> that >>>>>>>>> domain. >>>>>>>> >>>>>>>> I'll defer to Babu for accurate answers about this hardware capability. >>>>>>>> >>>>>>> >>>>>>> To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may >>>>>>> appear >>>>>>> as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire >>>>>>> system. In such cases, there is only one schemata entry controlling the whole system. >>>>>>> >>>>>>> Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still >>>>>>> performed at the L3 level, but that implementation detail can be hidden from users and does not >>>>>>> need >>>>>>> to be exposed through the interface. >>>>>> >>>>>> We seem to have two things that can both, somewhat reasonably, be called NODE scope in the resctrl >>>>>> user interface but the behaviour required for an MPAM system and an AMD system appears different >>>>>> from the point of view of lifecycle of the resctrl domain. >>>>>> >>>>>> For MPAM NUMA scope the MSC instance (MPAM hardware interface) is at the memory controller and so >>>>>> goes on and offline based on whether the NUMA node is offline or online. For AMD NUMA scope it >>>>>> looks >>>>>> to me that the lifecycle of the resctrl domains would be tied to the CPUs associated with the NUMA >>>>>> node. To me it does seem odd that a control with a domain associated with an offline NUMA node can >>>>>> continue to throttle (cross-NUMA) traffic. >>>>>> >>>>>> Is there any GLBE Control Domain ID or similar that is exposed to the user, e.g. is sysfs, or is >>>>>> this just implicitly the NUMA id? >>>>>> >>>>> Yes, the GLBE Control Domain ID is exposed to the user. It is essentially equivalent to the NUMA ID. >>>> >>>> What's the on/off lifecycle of these nodes? Does it follow the NUMA lifecycle as managed by the NUMA >>>> node notifiers documented in Documentation/core-api/memory-hotplug.rst or is it just linked to the >>>> cpu hotplug as is done currently for the resctrl cache based domains. >>>> >>> It will follow the CPU hotplug lifecycle, similar to how it is currently handled for the resctrl >>> cache-based domains. >> >> Ok. MPAM MSC are associated with the memory controllers and so MPAM controls that have NUMA scope >> will follow the NUMA memory hotplug lifecycle. This points to them being different resources. Do you >> have any thoughts on how we would handle this difference? > If I understand correctly, on an AMD GLBE system, even though it is "NUMA node scoped" it does not > support memory bandwidth allocation for a NUMA node that is online but all its CPUs are offline. This is > because GLBE is essentially L3 MBA that allows to set limits across multiple L3 domains. I think this > is a good match for a control associated with the legacy MB resource that just has a different scope > of allocation but how to do so without creating confusion with a "real" NUMA bandwidth allocation is > not clear to me since it may end up looking like: > > GLBE (make clear allocation is at L3 scope but domains are node scoped): > info/ > └── MB/ > └── schemata/ > ├── MB/ /* scope of domain ID = L3 */ > └── MB_NODE/ /* scope of domain ID = node */ > > MPAM MSC (make clear allocation is at node scope: > info/ > ├── MB/ > │   └── schemata/ > │   └── MB/ /* scope of domain ID = L3 */ > └── MB_NODE/ > └── schemata/ > └── MB_NODE/ /* scope of domain ID = node */ These info/ trees do express the difference between the two, GLBE and MPAM MSC at the memory. These both result in the same schemata file though. MB:=... MB_NODE:... This gives a potential conflict if GLBE is paired with memory allocation controls at the memory. I don't know if this is likely though. > > Any suggestions? > > I do not see how these can be considered different resources though. To me it looks like different > controls operating at different scope for the same ("memory bandwidth") resource? This is likely just a terminology issue. My thoughts are that, in a similar way to the L2 and L3 both being cache resources but a different level of the hierarchy, you can think of memory bandwidth at egress from L3 as a different resource from memory bandwidth at ingress to the memory controllers. Sure, they are interdependent but on an MPAM system you could have MSC at both locations. On a system with one L3 and one NUMA node and no other caches in between they are the same resource as they are just opposite ends of the same link but that's not the case when you can have cross NUMA traffic. So in your diagram above info/MB and info/MB_NODE are different "resources" and info/MB/MB_NODE is node based scoping of the info/MB "resource". Do you have a good way of referring to these resource concepts so that we can be more precise in our language? Thanks, Ben > > Reinette