From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BD41C54E49 for ; Thu, 29 Feb 2024 18:23:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F3E386B0098; Thu, 29 Feb 2024 13:23:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EC75B6B0099; Thu, 29 Feb 2024 13:23:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D677F6B009C; Thu, 29 Feb 2024 13:23:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BFC7D6B0098 for ; Thu, 29 Feb 2024 13:23:49 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7D9BF160D9F for ; Thu, 29 Feb 2024 18:23:49 +0000 (UTC) X-FDA: 81845664978.29.A3AE5B5 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf19.hostedemail.com (Postfix) with ESMTP id E138D1A001D for ; Thu, 29 Feb 2024 18:23:47 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=TH5dNY1X; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf19.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709231028; a=rsa-sha256; cv=none; b=xmJjW1Y+ca5EdAaCfmlGfHg7/97cbo2KDys8LcFels+hVvli2sj1lbknvqsx2QOiwRGSHJ GDnfbaXXBERHPFbZpBJgrPTysW7Khnt5zVjlIN8piVF18Xmq7ZtwSQTwS0oCCqJs3ieu3w nAQx+vkLgZtEgxD10mbbJGa0wDYC7hE= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=TH5dNY1X; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf19.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709231028; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DnJ5uJgmhr9JsRLsvP+ZbK075wchIZzs2mPZUtpKtpY=; b=pbMDiIDqafAad2Urmn4AeDSq/FJPzu9r9nT1dY8ArUjo0RRCS0WAPaCqMhkr4y3OORKk0y btZk18hFtbOMKTPKsNJx9o7Ju2OQaIskvk+7Jk/tJXSrEKM93mCYRGyVWOaE12zcGuVG6h lkZDBgm/fDs8KNbOVGOWAqqdeaMOSYU= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id D542A6131F; Thu, 29 Feb 2024 18:23:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B683C43399; Thu, 29 Feb 2024 18:23:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1709231026; bh=YDp6Q///DJQ/Ci6ApMuabMLyrReTuNGT2cmn4z59lxs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TH5dNY1Xlj+vtWdXEgVZ/vgG5A9IwUljSJ6B2QetA8yJCUI+wsjWMuBZx1dKzh/Vr uNRlhvH7aQ9rE2mXHv9usXkYbqygLd4rUpufMKQTVcGVfXGzathBleJsJp42tKTcmK 1Ej8ScylvKC1QuofNrCNyTbpvCEajzzVnFntKJh/66L+seVhJfRfs3uhWVbh5/apj2 hp8IGUzIg6LmX8Y7qLVObEXkziI3zhQyLyKhxUwXkGT4clE7nWE/ljehflrJM0vBbM 2YbLILPHQuCcELVgPzDaB9AJPaz8fnghRvJ/RYv+OXSoCMdtYDgQCBMSVBdrsq9Z3N EVszKNs6FhFoA== From: SeongJae Park To: Bharata B Rao Cc: David Rientjes , John Hubbard , Zi Yan , Dave Jiang , "Aneesh Kumar K.V" , "Huang, Ying" , Alistair Popple , Christoph Lameter , Andrew Morton , Linus Torvalds , Dave Hansen , Mel Gorman , Jon Grimm , Gregory Price , Brian Morris , Wei Xu , Johannes Weiner , linux-mm@kvack.org, Adam Manzanares Subject: Re: [RFC] Memory tiering kernel alignment Date: Thu, 29 Feb 2024 10:23:43 -0800 Message-Id: <20240229182343.74268-1-sj@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <781b3a7a-082d-46c1-a31a-1484c62123ac@amd.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E138D1A001D X-Stat-Signature: h38idckycnfcpxwbw4xqywe1nef9izpx X-HE-Tag: 1709231027-301921 X-HE-Meta: U2FsdGVkX1+qa3ivyBNhEQ0y/JMU2V5UNGRufhKd5L89s/Vl/W+fhn5A4L4ZJ6GN43L0f5rYK3H78S+POsGcXt3PA4ziyjK9mKZc3bmb8HJzzEQ2uXOYuS34XuRYmu87pREYoPg25R1A/ktLBjf9Vwx+o94XjshTWrXd2yvgmOX8Pp+Z/1x9XcSKIfV33AVXCEb98oPEfRp3u+r3OKanfFHYWmmZonpaYvp00R8EIrA5VG0YnO3tXQFlaQCcAYHZiUD9AzfOVSUGmuboOPZL87o92J4SmaJ7GUt8mdHFuhHZ4hmjOKcK4ck+0tqYF50/JBQ2kuB3mfFVza7sRVFPYCVMdcfDhSIdPaDpgaKus96e8dhz4pyWb/hJpF2dQMV7Ahd01TMiYF73BSCvKw1feCe1+gJQTc9HtoUSDEWTTEyGUHC1XZT6J9qM2Y4IJtpZsI4CwJoDhCN4ICFKgCpOf+cZDjh2Ep6A+r5OHFeVovW36v+vDXjBQJmBcbD0wzmyO9IqnFAaCFLAXYVBtr5PsgSr2QxdXzqcNNIdN7rRKC0AJ4i0gJWvDZB0xLHDQ0zY7HMmKy/YUwcbjP/UCVBMZ9S7nAIvYx8ox6tg7M4JNnxNg0sbkwnmcZ2YJhhdB4Rvb7RN+kuEksSc2O3vmHPPPZQAl7bAlQh1dJELPY6N9GM9TciW+/B9dn6lI9XdUWum1ymv35ZYx8jeAxA5YVNeToOKLwf9VSNr5Z5CFduysAjv3ThqK6vLOSkaZuFQCbq2vCKcyTuQT5YCTpMoz+7XjfPUdVpra9sAQhY9nRl+PQqq/CB0RAmjV+nNEeFEw7Twhk2H+hA24slN2ypYLa60wm4IHNllWHF5lIoEcpbSbT5jwi+NZj/5aFQ5i7Ws+PUK4T7CoI0eSMYqFdDLRisGszruwUaZ+U4jhHIgrPGq9X2WZE72dyYyzs+vJ3jVOrSACsQmcJujEjh0dou+LCC cSw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, 29 Feb 2024 09:31:02 +0530 Bharata B Rao wrote: > On 29-Feb-24 7:34 AM, Davidlohr Bueso wrote: > > On Thu, 25 Jan 2024, David Rientjes wrote: > > > >> Some recent discussions have proven that there is widespread interest in > >> some very foundational topics for this technology such as: > >> > >> - Decoupling CPU balancing from memory balancing (or obsoleting CPU > >>   balancing entirely) > >> > >>   + John Hubbard notes this would be useful for GPUs: > >> > >>      a) GPUs have their own processors that are invisible to the kernel's > >>         NUMA "which tasks are active on which NUMA nodes" calculations, > >>         and > >> > >>      b) Similar to where CXL is generally going, we have already built > >>         fully memory-coherent hardware, which include memory-only NUMA > >>         nodes. > >> > >> - In-kernel hot memory abstraction, informed by hardware hinting drivers > >>   (incl some architectures like Power10), usable as a NUMA Balancing > >>   backend for promotion and other areas of the kernel like transparent > >>   hugepage utilization > > > > Regarding the hardware counters, can/will CPU vendors provide something > > better for what is currently there for PEBS/IBS - which needs a lot of > > stat crunching to make it useful for hot page detection. > > IBS works independent of PMCs and reports useful information like the > virtual and physical address of the access, precise IP, Data Source > info (like cache, DRAM, External memory/CXL etc), remote node indication > etc. Hence it doesn't really need stat crunching. > > However it captures and reports access information based on sampling > and I have seen that the best sampling interval isn't always good enough > to match the number of accesses captured by software based mechanism > like NUMA balancing. This is same for sampling methods using time interval such as DAMON. I'm trying to make a sort of automatic tuning of such intervals for DAMON based on realtime monitoring results and real request from users. For example, if the user want to find hottest memory region of X % of the total memory, we could draw hotness histogram of the memory and get some clue about if the current sampling interval is too large or small. Just a vague idea. I haven't spent time for this topic so far. Since DAMON is designed to be easy to be extended with multiple access check methods including hardware features like IBS, I think the DAMON level auto-tuning might help in this case. Thanks, SJ > > Regards, > Bharata. > >