From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BF07C05027 for ; Sun, 19 Feb 2023 20:31:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F0E7280002; Sun, 19 Feb 2023 15:31:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A131280001; Sun, 19 Feb 2023 15:31:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8685A280002; Sun, 19 Feb 2023 15:31:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 773AC280001 for ; Sun, 19 Feb 2023 15:31:44 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 2BB78C05C9 for ; Sun, 19 Feb 2023 20:31:44 +0000 (UTC) X-FDA: 80485187328.15.ACC050A Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf05.hostedemail.com (Postfix) with ESMTP id 7309B100013 for ; Sun, 19 Feb 2023 20:31:42 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=QoQ5WptN; spf=pass (imf05.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676838702; a=rsa-sha256; cv=none; b=q34aRcnBOFf/U+6hcwjcbZPqp8omFpy1UIwk47LylgbnAc/aUzDW1Cpt8vtpU5xv14wwXc 7HG5xcojGJPvyCDdsTqIkO+ceG1vpVFOqGYJa9WYh1PKKBCoSjz1ypqL4QDrCo6ePTw2fe xqGkegYN89Z0YZpqGguecH+jxvK6JqE= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=QoQ5WptN; spf=pass (imf05.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676838702; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZfTx2C+9hdvYd0i97v/uz1Gl/PQmpUdWs4c659teBqc=; b=DVmbPLEisWzhG/1wqCm61mSNXI/jUVLHvCZe+uo6jaQheWosrRM4tddbhpiHcAHkI6DV9i J7rpbcUGAhYVBzCCl+erMDMAuzHrQ1Ox1Hs/r/HlBoYjQ6VRO9wjqZogjLxPdGh5Fp7fS/ ZQBhAi1CUm1Slkh2w/+Jn0ea30UuJY4= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6379B60C3D; Sun, 19 Feb 2023 20:31:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 297BBC433D2; Sun, 19 Feb 2023 20:31:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1676838700; bh=TmMatO5jW0UB6jUPsvjwyPrdcEJXebnMyipm8Ki+mkM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QoQ5WptNN4PPOXSNeBORliFBCIwF20Aadd+IGPBlcMHpYbPXPV8svX5pdsJAh0oH4 FcUUWoDhzBOdDulK16zv40FdjBQiGpnizKWKjQOktSWRKpgEGy+YpJoijWRM0AaIK2 8N3agA7cVuE3HHfIlMAHp1n5XVXP76rRnL3/WbHQZGbahwo1ykKeHZLp9MMACqflXV rYR3L86fppEocGYTNPFMIHFu7ofzEFburLevZ78LpM/1MOV2637nztj36hwtW2o8sM 3VOd48CpOTd+UBFnX8CyadMsHxV9LroTwAGvdV/vYyWmB8vHOddC20x51Sbv/53XHJ Z9p59BjPzUlAw== From: SeongJae Park To: "Aneesh Kumar K.V" Cc: SeongJae Park , lsf-pc@lists.linux-foundation.org, Linux MM , Yu Zhao , Dave Hansen , Johannes Weiner , damon@lists.linux.dev Subject: Re: [LSF/MM/BPF TOPIC] Using hardware counters to determine hot/cold pages Date: Sun, 19 Feb 2023 20:31:38 +0000 Message-Id: <20230219203138.4873-1-sj@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <87fsb1g23o.fsf@linux.ibm.com> References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Queue-Id: 7309B100013 X-Rspamd-Server: rspam01 X-Stat-Signature: 84398hwfs7ys45ht9qjftrspi9qf55e6 X-HE-Tag: 1676838702-831294 X-HE-Meta: U2FsdGVkX1/fzOw0cFlLSrVKBO0WrPtnLRhI2/IkDTcSKPLBvUMvDx21TnYP/+TiVFoNUYyrEsn6UpatcvDGpmaYdS6o+M6V112+P1n4UdQBpgdaPjoy9ol8AkxgSRvFudVwOHrJSTSwFMNDlp9LWZOb60U5V9CBbBrre8954kvhs7HgqB1Oxd/ibiv1UDQigHrny5sYm+/BCr5sdiW8MoF880HPrK15EA5vFmbcpACOFotrwC5EVi/JmCSoG9k4YtxGyxfyY93SvmKK+d3sS87UsROR9QdnO/Ot3OGJwZp1aBb7nlKLXFH2OIrYrXia0LBi3u16fCUhySHxyNcSYxSJnXjr8RW8DE3DpyIQL7DuOoIyBAXsQebLbF2k6ayfmHeBu4JjiJWkEHJlp4hnE0WylfTE3zuzpxnQ7kHcBxbvG0XDyMDKpAIvhLxFiUue16B8XGtQyb3/KFr2RIIVaTDr5KLNl5XeKTjDTTB/J8HYCXFnBiTQbmz+pr1Iyzff4etUvRTnI2X5FkMdqjHIH/T9J+1VH1yUWOL2tFDLqzlt/T+hegwRyHp8Iwb2fyCBjah6mkjODf93TR3eNdyekWDgQdVEXz4Kl2w7LFuCkpCNYsRZvGmV+RYY8mUjy9VAkMsWbDsJ4qjEH3yuvhtzc2VuUUQzbzQrKogYfkrxC44swdOOK6XBcLq74hUV4geEIeqP6sRHqxYKEayb+XYJtExhn1Lc02hOtLa5JvOOsszZe679dr9DQv86rwIayuh4VDlish48vmzFK0HlMM1mWYqk/7bqRnbEHoEWfN0+RQTtYjswp6eNLVvP/pVOH6UiTetUEuEVrIpbgjn3msqMKbMP/SLuAhmIMxmI+dToYJEen+9rGmheY08x1WB6UMhpVHX8QmKqIAM136LtlJSmmGxCnsTVpgZo2NHVJV2ZDh34AJxILRcM0F1E3A0kOfPNNw+C39bMeaBCOkjnkjp bTtsMh80 sGkpMfC9jEDDMTgXKr7C2AAjYhRJMIzTxxs4SxzgDARj0XuUDyeV3liTuZos0DHf4lN7syofs2QDMx9DzgiE1rdmt+jAnfgGa1L+yUy25zrZ6NVOpgsyzrnp8RM2evKYuJoSsadtZWL7bp4Tb2x461zJjOTGL0Uj/tfC97NBYT3DKieFJkijoaI/4UR/xVWNHMhjqNK2W0uHE/ElAhtwXh0huvmiLzn4d6ZaEQHLJOUossutMN2ZKPV97GzVM+D+3oC0z5VxGyY9vXFn04P5SyuLMdtbrufqXCBynTHNIa44ngMFSN9oyxq23oP/bKcnZx3Y9/0lQzC4/3r27sPCUs5TVbU7kwyrVjMFV6tp3OeS/Er6iC/TimHSue2DijZaZp8mVc5Tp9SjZ0f2oQ95GD/Z3wGxyznM2mwh+FqnvKcmaMoY4ugjQic/UA3jc8qvckDs4sg/pNLtPYEmwNzMh2NHw9begyXTMifusW28zfnexAqbfpykbaghGkWjZh+KmoUDSIQ6z6mULtX0Gvs7FXndKWvF6oVk96592lbEA6Hs5VgtMvJEH+wyFkQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Aneesh, On Sun, 19 Feb 2023 19:59:47 +0530 "Aneesh Kumar K.V" wrote: > SeongJae Park writes: > > > Hi Aneesh, > > > > On Fri, 17 Feb 2023 17:28:09 +0530 Aneesh Kumar K V wrote: > > > >> PowerPC architecture (POWER10) supports a Hot/Cold page tracking > >> facility that provides access counter and access affinity details at > >> configurable page size granularity [1]. I have been looking at using > >> this counter in different areas of the kernel such as > >> > >> 1) Page reclaim/demotion > >> 2) THP utilization > >> 3) Page promotion. > >> > >> I have done some MGLRU integration and would like to discuss the > >> observation with the rest of the community. It is still not clear what > >> are the best ways to integrate these hardware counters in the Linux > >> kernel. > > > > Sounds very interesting. I think DAMON might be one another option, because it > > is designed to be easy to extended with various source of access > > information[1], and provides an abstraction layer for access temparature based > > memory management[2], namely Data Access Monitoring-based Operation Schemes > > (DAMOS). > > > >> Attached is the performance graph showing how the mongodb/ycsb > >> benchmark performs when using hardware counters with MGLRU aging. An > >> early RFC version of the code can be found at > >> https://github.com/kvaneesh/linux/commit/b472e2c8080823bb4114c286270aea3e18ffe221 > >> . I also expect we can get some numbers w.r.t THP usage before the > >> conference. > > > > I also have experimented a DAMON-based THP optimization[3], which shown > > interesting results. > > > > Hope to discuss about this with you at LSF/MM. FYI, I also proposed an LSF/MM > > topic for DAMON[4]. > > > > [1] https://docs.kernel.org/mm/damon/design.html#configurable-layers > > [2] https://docs.kernel.org/mm/damon/api.html#c.damos > > [3] https://www.amazon.science/publications/daos-data-access-aware-operating-system > > [4] https://lore.kernel.org/damon/20230214003328.55285-1-sj@kernel.org/ > > > > > > The hardware counters that are supported in the case of POWER10 are > based on physical addresses. The hardware facility will count the access > across a physical address range and there is a counter for each page > that gives the access count and also information about which node did > access the page. > > I haven't spent much time studying DAMON so I might be wrong here. The > reason I avoided using DAMON for the POC was because my goal was to > evaluate how the hardware counters measured against the pte reference > bit and I was not sure I could evaluate that using the DAMON action > facility. > > I do agree that we could add a layer similar to DAMON_PADDR and expose > the details to userspace. But I was not sure we can take action based on > that. In most cases what I wanted was to move the coldest page in the > Numa node to swap. So that is relative hotness rather than moving a page > that got a hotness value less than 10 to swap even though we can figure > out a way to make the latter similar to the former. I think you could use DAMOS quota[1]. Under the quota, DAMOS prioritizes regions based on the access pattern, and applies the action to higher-priority regions, so you can do relative coldness-based reclamation, like DAMON_RECLAIM does[2]. The interface might not very efficient for your specific case, though. I want to know such cases and improve the interface or implement new features. [1] https://docs.kernel.org/mm/damon/api.html#c.damos_quota [2] https://docs.kernel.org/admin-guide/mm/damon/reclaim.html#quota-sz > > > I will look at DAMON and see if that is the best framework for things > like this. Great, please feel free to reach out to me if you have any question or need help. Thanks, SJ > > -aneesh