From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from SN4PR2101CU001.outbound.protection.outlook.com (mail-southcentralusazon11012006.outbound.protection.outlook.com [40.93.195.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C18726738C for ; Fri, 10 Jul 2026 21:00:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.195.6 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783717210; cv=fail; b=r7zkRgPE++g2or/FGcPjBd5cIlOe09H9CjuDLmNd9UPbSBdsULHhPhjsTHBM1GURTw88L6U8ecer6ny8FdtyqjsO3nH7Zsj7TXz14TRk4n1GpZhJJzGV4L51E/2c5pWD7EyNczpr9MRPBkbM9AS79dZ9cGyWQgSzSKN8KIwQfwI= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783717210; c=relaxed/simple; bh=E9yl/e6Hrsj24WPK2fWuqQ/B7/AG2uOEjxLU2+C2Wlw=; h=Message-ID:Date:Subject:To:Cc:References:From:In-Reply-To: Content-Type:MIME-Version; b=oL73Qgkntm7XBjSqHYDs5MwYkieWVwja1gEyIrxQWRQEjZA4mZk8bvJI9pLoI58jz69NEDgx/3DP9sVidlyn+pGuX26+paCh8W+JpM9IPiErK/L5hIqyDwCbOarEmh/gMbbih0pvp6vsnhJiYwX3Mbye4Ib43AebAzPoqYy0V40= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=GrINkUzX; arc=fail smtp.client-ip=40.93.195.6 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="GrINkUzX" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=LE9D7bRVr+cxrthB/fx01nOw/5fAUkmtx9LO4k2KNmWGoF3lJ+VKX3xiLSTTbJHceUvqxYdEt6mozT0Xsblr1sWdRGfjdQSzHW14NMPaJrS0rZTRQEQJ9orLbtd3X1HPQ+h3mFOIEjOllOI2IYsz/Li1yWuyTT/T04/LQshpoUL3ME+Pa74Hk9SvwxnLiU5tVIz9DBRxVgO1xFbOcs4KcGaOBxaaW1jRGS+htM6nKJVHBF+zCsFU0wFyCgkNKZ+DyKB77e/3hr5wYPWD+PSRuoGq+xnnx86g0bAEg99mZvYhUr94TN07U+3OICn8buc/LT/yL5YB8ubeQW8kzFbFwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TN6euYGoo4XuK9l93LTtcrfOzJSgrsbrBr3YX8evr0w=; b=xqCh9+wa34z5A8CTRriuaSaeMxtWWMBUY4uUyb/7DjlZkkdZArHs8dGBAAonIEOEO//7L0jUvj/lvAQa623mTcjymuBD+DxVuCQ8ivanOGdlT4ol8e36EaZfQxoQ6GX+6nwCNvlQ0TlYB7udp2Yb4cHo5nb/VtrIz2paNtYN0NNpmVgAr2y2wpRRnCGB8ig2x7eqVHTlLvbN1XH04mMcmapIIpblm4wRCZFMOY5kTmO/AczexIqbhLgyJI+P8oeS3FOntyE93wy+ZaTZMvTvqyhkWamDjLTbC3OWLyyIOAxTpOXPw5IX/IESqtb2FJ/voPxT2Nr0CdY1woCCHmCYLA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TN6euYGoo4XuK9l93LTtcrfOzJSgrsbrBr3YX8evr0w=; b=GrINkUzXws9DQtKF/mb37sF+dAGnuNrAxOcj61BQqifgYcZ5h6v9Qq/mW0CaEPgII9zhzBnZvJOgIzbegvLml5pCAaHbbn8TlQfjW5R8AwfDF1/3MiqUIkTlH3cFoJeKPlRvNZ5/Ss8faZoGvAEWzjNFL+UF3OYjmUoD5rX66PT4mMeH+269cxsXIfwIRFnoZkQdyUYH5dCs6KYIuj36QyWyuL02VGG70USPrbggkTGtbJ6vncUvBq30gV13Ge4X4Sztd35VyjxjmMAGLUYTA+Wfy2abjUzpmNqL+AZaWQ1nFRQ3QgpS/qEkSqNYxkqoOo3ZduXj9OjLmpRAD8Flqg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM4PR12MB5230.namprd12.prod.outlook.com (2603:10b6:5:399::11) by PH7PR12MB7844.namprd12.prod.outlook.com (2603:10b6:510:27b::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.181.16; Fri, 10 Jul 2026 20:59:58 +0000 Received: from DM4PR12MB5230.namprd12.prod.outlook.com ([fe80::6e87:1bde:1853:3b73]) by DM4PR12MB5230.namprd12.prod.outlook.com ([fe80::6e87:1bde:1853:3b73%5]) with mapi id 15.21.0181.017; Fri, 10 Jul 2026 20:59:58 +0000 Message-ID: <8f9f78dd-e3f5-4b35-bc72-0eb5dafdcedf@nvidia.com> Date: Fri, 10 Jul 2026 13:59:56 -0700 User-Agent: Mozilla Thunderbird Subject: Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept To: Reinette Chatre , Tony Luck , Ben Horgan , James Morse , Dave Martin , Babu Moger , Drew Fustini , Chen Yu Cc: Borislav Petkov , Thomas Gleixner , Dave Hansen , Peter Newman , "x86@kernel.org" , "linux-kernel@vger.kernel.org" References: <5ee87762-1898-4b62-94da-85b3e9917ecc@intel.com> <62701203-c4a3-4ec2-a9af-602e1fc15863@nvidia.com> Content-Language: en-US From: Fenghua Yu In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: BY5PR04CA0026.namprd04.prod.outlook.com (2603:10b6:a03:1d0::36) To DM4PR12MB5230.namprd12.prod.outlook.com (2603:10b6:5:399::11) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR12MB5230:EE_|PH7PR12MB7844:EE_ X-MS-Office365-Filtering-Correlation-Id: 49e7691c-5a93-42af-02a0-08dedec63343 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016|7416014|23010399003|4143699003|13003099007|22082099003|18002099003|3023799007|56012099006|6133799003|5023799004|11063799006; X-Microsoft-Antispam-Message-Info: BcmHNL3KMY2gH4FAkWsWZTEg5FjvWVlUOBf84ig59UH2iFdhFpzIimBdVsMz7rnCZ0O9Bvcffy0NhiZ2LRHMAKy4rKXA5Sj2sGKFBaKN4yHXJ7ohXmPNaGD1/XKN9Py2oY2HzUpKlIJen2vNdmDRfHjakC6SAVIlyDRnkxJUyZeMlG4XODrOHHXH086rA3q4QkqBoG36AnxuohMNQgT5RrZFcdqG51PZ9WpYG6vehcoCS2gND79gGHld23Yi7yAJSwwTQL0Wjp4o4d28ZOzz55z2qzdqWivQHBhxBwt7zxset+h0HpdHEkKEm7l26SZsVytiz1s58fv9OmcS+fXqlyrsPaVlYHnIXJ7iWzOzbBUfSofN9Iw25glXKDbCXJJPQJri15TipczJI1S9/oEUO8K4iRvIwVmTxUy7/Y8gUSTjU3TrzUGGEDF8BwSFvM2WzrDsXobQSSA50ReOjAc2Wqa1FOmgRMzD1jWb0xfpGACAu/QSQoaGD5g8pjWqGHyntgwuM1e8kc7dG4Woe5tsQ/mNGX5mLfC4MlmJdudG4a+zuixxoK+yNX11m/tXMTzlFLzfS2ec5J3fp6ufKLQc7HgJYQFCHp2uFc1s+SbnAAxHpxIpSSNIsAWYzKgK4ePW+PbFqpSISoAMHsq6peSiarJr80l2BVv4flzqJn4CuoA= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM4PR12MB5230.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016)(7416014)(23010399003)(4143699003)(13003099007)(22082099003)(18002099003)(3023799007)(56012099006)(6133799003)(5023799004)(11063799006);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?V1FwcElZNnB5TU83OENGbC9kMXppV2EyK1FxcGVLVkI5M0pkMHZjMTZnZHNM?= =?utf-8?B?ZUJpeGpOays4bmpNMmhtZm0yZXNnWDE2ZkJEcno0VlFtTGhIR0g3SlJhTWdB?= =?utf-8?B?bmRwR3BPY3Y2M3RCWndRTmhxK0lqSUJjRmZPcHhYZFV4c2pjZ3g0ZWhVNVla?= =?utf-8?B?ektUd1ZRd0MvaHcwa1lqL0I1UEdKUFN1ZzEzaDl0SEZvMkduUjcyWUJXMzNO?= =?utf-8?B?bEJWZGRyZmgyOWpXQkFqU2xnbGRWNnZkSGNOTGw4N2tGUUVkV0dlK0ZUYWM1?= =?utf-8?B?VTdEWWRIbkU5UlJnaFNJbytvMWtrWll3V2VrbjM3c292ZFdyem9ZMEgvMUJC?= =?utf-8?B?WGZhTFJKUzJmOWFWWERwYjBUdGhFditiaEFSWk14K3FDS0owUjAxNkhsaDdZ?= =?utf-8?B?VkpiNmh0dytqZGpzcjBZbTlReGxrV3R5Q200QjlRbHdNU3RFNGJ4ZTVwRzJ0?= =?utf-8?B?MGMwTWkyOGMvaVRVelZHb2VpT2J1TTViWDNjQ2Zna2xQblJKNExHRVRLL05G?= =?utf-8?B?WXQyNkltSVFQUlJTMHVYUXhQYldRNmdMZmY5UDZyVTg4MnJNUUQ4VzhDZnhs?= =?utf-8?B?SXM1QlduSFNWZEdtdkhvRk5yc0Q0anoyZWU3dWJIbUNYSGhydCtVTENyeVdu?= =?utf-8?B?ZGZRcVJKOG5XWmF3RFR5QlRXU25oNTdjVG9ST1E0anpQN3RwaWg1aCtablBy?= =?utf-8?B?M2FmZm5Fc0QwY3BrYmhFa2l4OGFuUmtBYTBYNzJOd3c4aEhSRTM0anpUdVBi?= =?utf-8?B?TVk3a2ZVc3dsNG56Rk9SZXN3dHh4bU10VlZpa2xHK1plQmQ5c2h1K1FicWgy?= =?utf-8?B?OWhRaE9aVzNKcnZMTitEQ2lORkhWaFowSFhZSWo2aVZxemM4U1R2d3YwbmJN?= =?utf-8?B?ZEVxNFpDaGNiaURnVHBST1ZWTi9id2xuL1VZTU9MalJBbXJmQUNNMFNTVXZU?= =?utf-8?B?RUJVY0ZRYmMzajNFb0lSYTg5QVhmeEo4QnBIT0xiU2Rlekg3V3pUNzlIZ2E2?= =?utf-8?B?T091a21KU1M0a2o0bElRRVJ4Vkh1WTRHejdhOER3ZlN6VlN2ckFiUi9pM1BG?= =?utf-8?B?azhGU2xSNk1HNm1aVTVER1ZYbHRxOHlLZXJ3NVlEdXI1cHM1dk5YV0JnOUcw?= =?utf-8?B?OGxxcG9TQnpVZHhvZXRleFptVlJxdHFMMFU2QXBNWStLeVNVWW1sK3QxWUtH?= =?utf-8?B?S3M5anlHaDlucHZ0TlZ4QU82NWJyS0N6MnJUakhLNzV3bnNOWlVESlpXQ1ZL?= =?utf-8?B?UmJIWjhDby93ZEo2NHczTE1sRFpnTEUwd1NpaW80c29OWWIzQzVxV1RIVmZQ?= =?utf-8?B?VHE5RHRoRUhrbnRHUThsMzREQTRDRGUvbVhtNWhjOHVRaW85K1NoVWdxVDRm?= =?utf-8?B?cXNadzA5VzlNMUgxdzFGaFp5VGduYmVYa0dWb1REbm10Nk45NWxkSThFNE5l?= =?utf-8?B?WXRBRTVCbHZPSjVLcEF2RHNHbXZHaFNsaE0wd2JZYUpCQWdtOXpRdGRRNEhk?= =?utf-8?B?Nmp2UlZ1SlI3dzZHbCtPckM2aFBqZ2tFZVdnbkJkUUdtNE9leC9mTmRiSVdI?= =?utf-8?B?NHVnWE8ybnRKNW1WVDNNOU9VR0JaeThYWktlZ3lkUWZraVlQNmY1T3FOVDIz?= =?utf-8?B?ZkxZdVlNTGZvMmIzNjBsbC9UM2Q1SG42YmJyWFNhbnJ4Z1ZrV3pBb2RQdW5K?= =?utf-8?B?RmEva2YwTUNBRFFlNkMrc3U1YnI3N084NVFDZWgvdldIMFZFVzZMWjlXZkR3?= =?utf-8?B?ZXJ0d3FoelpyUERFL1ArMnZEaEhhMkFoUzBqTlRpS2FFTDJvNEdxZnk1QXhl?= =?utf-8?B?d1JvUDMxMGM1aXFMaHF1TXRUcmdNNDlSanRLS2FHSzBPeXlRT1QxckpGQnBs?= =?utf-8?B?TDN2a3VVaDZnTzNEL3pDZDNrTElVK3ZFQWMvZDBXeTBjNm8vMCtZSFc3K255?= =?utf-8?B?WWNpVSs0cjhtc085emM3WDNjZzVPMW0xVVNCT0xTZkNTSWpvazVqSnZFaXpR?= =?utf-8?B?VmRTTjJMTllmTlZJa2JLSWtHeEd6Tk5URGRxbEdyS25DODlod0JHVXlWaDBz?= =?utf-8?B?aENUblo1TEZCWEpxVGgzYUJ0N0dyU1BmcFBWM1BVZEI2WjFUTEtFT3RDS3Nq?= =?utf-8?B?cS9Ua1g3OHdVd2YrT3lBOUZrUUQ0blN6bVYzVE5aT3BiN3BQMndETmlnMEVy?= =?utf-8?B?WEVhVDAydlE0L3R2R1k3ajhoa0Vkd2dDbUo2YjAwbjg0NklpbnZFTVlNaVBZ?= =?utf-8?B?bllyZ09KZnY5ZDNEeWVnb1FSREY0S0NzYy9Pam1JUHhMSFNDbmpaKzFUQmpw?= =?utf-8?B?YVdvRWVhMDUrS004YW9mMGphNnVhalZBZ0NYSHhkTmlBck02Q1FqZz09?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 49e7691c-5a93-42af-02a0-08dedec63343 X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5230.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jul 2026 20:59:58.2231 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: b+i64c4sIHs6YuiiUaIYhQYaUcSp+rBKDdbfjEfbZ8wq5BExfhljkTRpErJAq9jdLdxECn1qhKpdM3p1mbImbw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7844 Hi, Reinette, On 6/25/26 08:43, Reinette Chatre wrote: > Hi Fenghua, > > On 6/24/26 6:26 PM, Fenghua Yu wrote: >> Hi, Reinette, >> >> On 6/24/26 15:22, Reinette Chatre wrote: >>> Hi Fenghua, >>> >>> On 6/24/26 12:08 PM, Fenghua Yu wrote: >>>> Hi, Reinette, Ben, Shaopen, et al, >>>> >>>> On 5/29/26 11:06, Reinette Chatre wrote: >>>> >>>> As Shaopen and Ben mentioned earlier, we are working on two MPAM >>>> features that may need to change schemata interface. The CPU-less >>>> feature was discussed on LPC (although the interfaces will be >>>> slightly different from the LPC). >>> >>> I know. Here is where I tried to engage with you on needed interfaces after LPC: >>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/ >> >> MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform: >> 1. L3 and memory together on each processor MSC >> 2. L3 in processor MSC and memory control/monitoring in different memory MSCs. Ben said there is type 3 platform: 3. L3 cache and memory bandwidth in processor MSCs and memory bandwidth in different memory MSCs. >> >> On type 1 platform, schemata is legacy: >> MB:1=100;2=100  <-- cache id 1 and 2 as domain id >> >> On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is: >> MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18, >>                            26 as domain id >> On type 2 platform, there won't be "MB:" line. Numa 0 and 1 >> are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU >> memory nodes allocation. On type 3 platform, there could be "MB:" line for L3 cache and "MB_NODE:" for numa node. Example schemata is: MB:1=100;2=100 <-- cache id 1 and 2 as domain id MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18, 26 as domain id > > (to help make things explicit I will refer to what you call "MBN" as "MB_NODE" to make it > explicit that it is memory bandwidth allocation at node scope) > > I am trying to consider how this can be accomplished while also considering all the other > new hardware features that resctrl need to support. Consider, for example, AMD's "Global > MBA" (https://lore.kernel.org/lkml/cover.1776980182.git.babu.moger@amd.com/) that throttles > memory bandwidth at L3 scope but the user configures allocations at NODE scope. At this time > the plan is to support this with a second control associated with the MB resource that can > allocate memory bandwidth at node scope. See > https://lore.kernel.org/lkml/430ffb48-29f4-44d9-9164-9f8b743b2739@amd.com/ > > If resctrl creates a new resource for node scoped memory bandwidth allocations to support these > "type 2" systems then that will result in an inconsistent interface between architectures that > we should avoid. > > Have you been listening in on the discussions surrounding emulated controls? Considering that, > would it be possible to support the "MB" control on a type "2" system but have it be backed by > (emulated by) the underlying "MB_NODE" control? > > resctrl could expose both controls on these "type 2" systems but make it clear that "MB" > is emulated by "MB_NODE". For example: > > info/ > └── MB/ > └── resource_schemata/ > └── MB/ > └── MB_NODE/ > > User will see both controls in schemata file but when changes are made to "MB" control it > will show in the "MB_NODE" control and vice-versa. User could also disable the "MB" control > that will establish familiarity with the interface at which point resctrl can drop the > "MB" control from the schemata file on these "type 2" systems. > > Having the MB resource available with an MB control will keep resctrl backward compatible > if there are any tools that expect that. If backward compatibility is not of concern then > resctrl could initialize with the emulated control disabled by default. See discussion at > https://lore.kernel.org/lkml/5e575bc2-e67f-4696-9332-33c54023c057@intel.com/ > that describes a new resctrl capability in support of RISC-V and RDT. > With this resctrl could initialize with: > > info/ > └── MB/ > └── resource_schemata/ > ├── MB/ > │   ├── MB_NODE/ > │   │   └── status:enabled > │   └── status:disabled > └── mode:legacy [native] > > With above a "type 2" system will boot with its schemata file just containing the "MB_NODE" > control while info/MB describes the memory bandwidth resource. > On type 3 machine, schemata has both MB in legacy mode with cache id as domain id and MB_NODE with numa id as domain id. Is this directory OK? info/ └── MB/ └── resource_schemata/ ├── MB/ │ ├── MB_NODE/ │ │ └── status:disabled │ └── status:enabled ├── MB_NODE/ └── mode:node 1. MB and MB_NODE are shown in parallel in inf/MB/resource_schemata/ 2. mode is set as "node" meaning "MB" is for L3 and "MB_NODE" is for numa node 3. Emulation "MB_NODE" is disabled (or should the "MB_NODE" sub-dir be invisible?) > If AMD is ok with naming their "Global MBA" allocations "MB_NODE" then a user working with > AMD and these MPAM systems could find the controls in the same hierarchy and need not > use external knowledge to determine how to interact with resctrl fs. MB_NODE is good for me. > > >> BTW, Slow MBA (SMBA) is different from MBA Numa (MBN). SMBA still >> relies on L3 and the domain id in SMBA is still cache id. MBN >> depends on each memory controlor with numa id as domain id for both >> CPU and CPU-less memory nodes. > > ack. Similar to AMD's "Global MBA" there is also a new feature of "Global SMBA" that > needs to be supported by resctrl. > > >> On type 1 platform, there is only MB: >> >> info >> └── MB >>     └── resource_schemata >>         ├── MB >>         │   ├── max >>         │   ├── min >>         │   ├── resolution >>         │   ├── scale >>         │   ├── scope <== contains "L3" >>         │   ├── tolerance >>         │   ├── type >>         │   └── unit >> >> On type 2 platform, there is only MBN: >> info >> └── MBN >>     └── resource_schemata >>         ├── MBN >>         │   ├── max >>         │   ├── min >>         │   ├── resolution >>         │   ├── scale >>         │   ├── scope <== contains "NUMA" >>         │   ├── tolerance >>         │   ├── type >>         │   └── unit >> >> This is different from the "scope" hierarchy discussed in the link. "MB" and "MBN" won't exist on the same platform. >> >> I find it's hard (and not useful) to split "MB" for memory with CPU >> and "MBN" for CPU-less memory node. It's easier to have either "MB" >> for legacy memory with CPU or "MBN" for CPU-less memory. > > Please widen your considerations to include how resctrl can maintain backward compatibility > and how enabling of these platforms can fit well with the "similar but not identical" hardware > features from other architectures that also needs to be supported by resctrl. > > >> Any thoughts? Does this update make sense? >> >>> >>>> Hardlimit feature was not discussed yet. >>> >>> It was considered. See discussion starting at >>> https://lore.kernel.org/lkml/1c4b6b46-16f9-4887-93f5-e0f5e7f30a6f@intel.com/ >>> >>>> It's good to discuss this further in this RFC thread before the new features RFCs will be sent out. >>> >>> ack. >>> >>>> >>>> Overall, the new features can fit into this RFC well. >>>> >>>>> Hi Everybody, >>>>> >> >> [SNIP] >> >>>>> >>>>> This series can be used on an x86 system where it will show two new dummy controls >>>>> where it is possible to interact with the new controls. >>>>> For example: >>>>> >>>>> # cat schemata >>>>>           MB_MAX:0=100;1=100 >>>>>           MB_MIN:0=100;1=100 >>>>>           MB:0=100;1=100 >>>> >>>> Some platforms may support CPU-less node which is represented by numa node id, examples: >>>> 1. CXL type 2 memory node which provides CXL memory without CPU and L3 on the node >>>> 2. GPU memory node that can be accessed by all CPUs but doesn't have a local CPU and L3 bound to. >>>> etc. >>>> >>>> MPAM can allocate and monitor mem bandwidth on these memory node. >>>> Since no CPU and L3 on the node, cache id cannot be used in "MB:" line. Instead, numa ids are used to identify MB allocation and monitoring. >>>> >>>> For example, the MB allocation on CPU-less platforms could be: >>>> MB:0=100;1=100;2=100;10=100;18=100;26=100 >>>> >>>> Where: domain id 0, 1, 2, etc are numa node id shown in /sys/devices/system/node directory or by numctl. >>>> 0: socket 0, node 0, CPUs, memory >>>> 1: socket 1, node 1, CPUs, memory >>>> 2: GPU 0, node 2, no CPU, memory only >>>> 10: GPU 1, node 10, no CPU, memory only >>>> 18: GPU 3, node 18, no CPU, memory only >>>> 26: GPU 4, node 26, no CPU, memory only >>>> >>>> Arch specific driver (e.g. MPAM) detects CPU-less node. If there is any CPU-less node, use numa id in "MB:". Otherwise, fallback to legacy cache id. >>> >>> We always have to consider backward compatibility and to do so we cannot just retroactively >>> change what domain ID represents when user space interacts with the "MB" control. >>> >>> The legacy "MB" control is already defined and its domain ID represents an L3 cache ID. To >>> support these new devices resctrl would need to expose a new control. >>> >> >> Agree. Add new control "MBN" for Memory Bandwidth Allocation on numa node. See above. >> >>>> >>> >>>> There is another MPAM feature called MBW Max hardlimit which sets >>>> "MB:" allocation as hardlimit (i.e. MBW throttling percentage must >>>> be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata. >>>> It's 1:1 mapped to "MB:" to control hardlimit of MB throttling >>>> percentage on each domain. By default hardlimit is off (0) and can >>>> be turned on to set MBW Max hardlimit on a domain. >>> >>> ack. This sounds like a new control associated with the MB resource. >>> This is a boolean control as Dave highlighted in previous discussion so >>> resctrl would need to know its properties. >>> See https://lore.kernel.org/lkml/aO0Oazuxt54hQFbx@e133380.arm.com/ >>> >> >> Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is available.) >> >>>> For exmple: >>>> MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0 >>>> MB:0=100;1=100;2=80;10=100;18=100;26=100 >>>> >>>> On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node). >>>> >>>> MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle. >>>> >>> >>> ack. >>> >>>>>           L3:0=fff;1=fff >>>>> # echo 'MB_MIN:0=50' > schemata >>>>> # cat schemata >>>>>           MB_MAX:0=100;1=100 >>>>>           MB_MIN:0=50;1=100 >>>>>           MB:0=100;1=100 >>>>>           L3:0=fff;1=fff >>>>> >>>>> Writing to the dummy control will call a dummy callback that just prints to the >>>>> kernel log: >>>>> "resctrl: Updata temporary MIN control on domain 0 with user value 50" >>>>> >>>>> >>>>> Example output of info/MB/: >>>>> /sys/fs/resctrl/info/MB/thread_throttle_mode:max >>>>> /sys/fs/resctrl/info/MB/num_closids:15 >>>>> /sys/fs/resctrl/info/MB/delay_linear:1 >>>>> /sys/fs/resctrl/info/MB/min_bandwidth:10 >>>> >>>> Add two new MB info RO files: >>>> 1. /sys/fs/resctrl/info/MB/domain_id >>>> It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id. >>> >>> This proposal introduces a *global* property to the MB *resource*? It does not seem as though >>> this takes into account *anything* about how resctrl can support new hardware that has been >>> discussed before, during, or after LPC. You have not participated in these discussions and >>> now make an orthogonal proposal that does not take into account *any* of the requirements >>> that we have been struggling with for months. >>> >>> Why should this proposal be taken seriously? In your absence folks have been trying to >>> accommodate how these upcoming products and be supported and the "scope" file associated with >>> a control is intended to communicate to user space how the domain ID should be interpreted. >>> >>> Why are you proposing something entirely different here without even acknowledging current >>> approach and explaining why it does not work for you? >>> >> >> So can I change this part to adding the following files in info dirctory? >> >> 1. For numa memory bw allocation (MBN): >> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/ >> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100 >> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5 >> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar >> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10 >> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1 >> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA >> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all >> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100 > > This is not just about adding files to the info directory. The files, directories, their relationships, > and content have meaning. All I see from these proposals is an attempt to slap some new files into > resctrl without any consideration to present consistent interface to users and without consideration of > other architectures that need to be supported by resctrl. > > resctrl needs to provide a generic and consistent interface to user space irrespective of the > underlying architecture. Architectures cannot just slap some new files for their convenience. > >> >>>> 2. /sys/fs/resctrl/info/MB/max_lim >>>> It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc. >>> >>> Again this adds another *global* property to the MB resource but then above you >>> describe the new "MB_HLIM" schemata file entry that implies that it is a new control >>> for the MB resource. Having it be a new control for the MB resource matches earlier >>> discussions. To support this I thus expect it to be exposed as a new control with >>> potentially a new type if any of the existing planned types do not suffice. >>> >> >> How about adding these MB_HLIM dir and files in info? >> >> /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean >> /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0 > > This presents "MB_HLIM" as a *resource* to user space. It is not a resource > but a *control* of a resource, no? I thus expect it to instead look something like > below that makes it clear that MB_HARDMAX is a control of the MB resource. > > info > └── MB > └── resource_schemata > ├── MB > └── MB_HARDMAX Yes, this makes sense. I have changed to this hierarchy. > > > >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/resolution:100 >>>> >>>> Is it more concise to s/resource_schemata/schemata/? "resource_" seems redundant in the context "info/MB". >>> >>> We could do this, yes. >>> >>>> >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/tolerance:5 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/type:scalar >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/min:10 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/scale:1 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/scope:L3 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/unit:all >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB/max:100 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/resolution:100 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/tolerance:5 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/type:scalar >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/min:10 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scale:1 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/scope:L3 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/unit:all >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MIN/max:100 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/resolution:100 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/tolerance:5 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/type:scalar >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/min:10 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scale:1 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/scope:L3 >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/unit:all >>>>> /sys/fs/resctrl/info/MB/resource_schemata/MB_MAX/max:100 >>>>> /sys/fs/resctrl/info/MB/bandwidth_gran:10 >>>> >>>> For MBW monitoring, extend mon_data/ directory to monitor CPU-less memory node. For example, >>> >>> Here is where I attempted to discuss with you how to support monitoring on these systems: >>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/ >>> >>> Here again you respond with something completely different without acknowledging >>> the previous discussion or noting why that does not work for you. >>> >>>> >>>> On legacy platforms (i.e. L3 and memory are described in same MPAM ACPI MSC wich doesn't support CPU-less nodes): >>>> mon_data/mbm_L3_01/llc_occupancy >>>> mon_data/mbm_L3_01/mbm_total_bytes >>>> mon_data/mbm_L3_02/llc_occupancy >>>> mon_data/mbm_L3_02/mbm_total_bytes >>>> >>>> On platforms with L3 and memory in separate MPAM ACPI MSCs >>>> but there is no CPU-less node: >>>> mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 >>>> mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 >>>> mon_data/mbm_MB_00/mbm_total_bytes  <- numa node 0 (socket 0) >>>> mon_data/mbm_MB_01/mbm_total_bytes  <- numa node 1 (socket 1) >>> >>> Here too I do not find it appropriate for resctrl to retroactively >>> change its interface. "MB" is a resource and the above switches the >>> resctrl interface to imply the monitoring data of a resource can be >>> found in the mon directory that matches the resource name. This is >>> not what resctrl does today. Doing something like above will result in >>> resctrl having a confusing interface where "sometimes" memory bandwidth >>> data can be found in the L3 directory and "sometimes" memory bandwidth >>> data can be found in the MB directory. >>> >>> As I described to you in December resctrl already exposes the monitoring >>> data based on the *scope*. As you also point out above, today the memory >>> bandwidth monitoring data at L3 scope can be found in the L3 directory. >>> "L3" should thus not be interpreted as the resource L3 but the scope L3 >>> since it contains MBM data today. When viewing it as such resctrl could >>> internally be more explicit and separate monitoring scope from monitoring >>> resource and present the monitoring data based on scope to remain intuitively >>> backward compatible while obtaining support for these memory nodes. >>> >>>> >>>> On platforms with L3 and memory in separate MPAM ACPI MSCs >>>> and there are CPU-less nodes: >>>> mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 >>>> mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 >>>> mon_data/mbm_MB_00/mbm_total_bytes  <- numa node 0 (socket 0) >>>> mon_data/mbm_MB_01/mbm_total_bytes  <- numa node 1 (socket 1) >>>> mon_data/mbm_MB_02/mbm_total_bytes  <- numa node 2 (GPU 0 mem) >>>> mon_data/mbm_MB_10/mbm_total_bytes  <- numa node 10 (GPU 1 mem) >>>> mon_data/mbm_MB_18/mbm_total_bytes  <- numa node 18 (GPU 2 mem) >>>> mon_data/mbm_MB_26/mbm_total_bytes  <- numa node 26 (GPU 3 mem) >>> To be backward compatible I find it more intuitive if instead >>> this data is exposed as below: >>> >>> mon_data/mbm_L3_01/llc_occupancy    <- cache id 1 >>> mon_data/mbm_L3_02/llc_occupancy    <- cache id 2 >>> mon_data/mbm_NODE_00/mbm_total_bytes  <- numa node 0 (socket 0) >>> mon_data/mbm_NODE_01/mbm_total_bytes  <- numa node 1 (socket 1) >>> mon_data/mbm_NODE_02/mbm_total_bytes  <- numa node 2 (GPU 0 mem) >>> mon_data/mbm_NODE_10/mbm_total_bytes  <- numa node 10 (GPU 1 mem) >>> mon_data/mbm_NODE_18/mbm_total_bytes  <- numa node 18 (GPU 2 mem) >>> mon_data/mbm_NODE_26/mbm_total_bytes  <- numa node 26 (GPU 3 mem) >>> >>> When "mbm_total_bytes" move from mon_data/mbm_L3_x to >>> mon_data/mbm_NODE_x it clearly indicates that it is memory bandwidth >>> monitoring data moving from "L3" scope to "NODE" scope. >> >> The scope in mon_data shown in https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@intel.com/: >> >> mon_data >> ├── mon_L3_00       <== monitoring data at scope L3 >> │   ├── llc_occupancy >> │   ├── mbm_local_bytes >> │   └── mbm_total_bytes >> ├── mon_L3_01       <== monitoring data at scope L3 >> │   ├── llc_occupancy >> │   ├── mbm_local_bytes >> │   └── mbm_total_bytes >> ├── mon_NODE_00     <== monitoring data at scope NODE >> │   └── mbm_total_bytes >> └── mon_NODE_01     <== monitoring data at scope NODE >>     └── mbm_total_bytes >> >> On some ARM platforms, for example, socket 0 (CPUs+L3) and socket 1 >> (CPUs+L3): >> llc_occupancy is monitored through processor/L3. >> total_bytes are monitored through memory controlor. >> >> Then the above "scope" is confused: >> mon_L3_00 and mon_L3_01 only has llc_occupancy. The scope "L3" is fine here. >> But mon_NODE_00 and 01 are confused because numa node 00 and 01 have >> both L3 and memory. The name "mon_NODE_00" seems monitor both > While the node may have L3 and memory the directory only represents what is > monitored at the particular scope. > >> llc_occupany and total_bytes but it only monitor total_bytes. And on >> CPU-less platforms, a CPU-less node does only have total_bytes which >> seems match "scope NODE". >> With "scope NODE", can user tell if it has llc_occupancy or total_bytes or both? > > resctrl has a "mon_features" file associated with the monitoring scope that > informs user space which events can be expected in a resource group's monitoring > data directories. Right. > >> >> If change "mon_MB_01" to "mon_MM_01", the scope is "MM" now which >> means monitoring memory (total_bytes) in numa node 1 with "scope MM" >> (not L3)? > This sounds redundant to me since the event names already have the resource embedded. > "llc_occupancy" implies L3 occupancy > "mbm_local_bytes" ... the "mbm" implies this is memory bandwidth monitoring data. > > User space can infer the scope at which the monitoring data is collected from the > directory the event file is in. > Right. Thanks. -Fenghua