From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 880BEC61DA4 for ; Sun, 19 Feb 2023 14:30:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C291B6B0072; Sun, 19 Feb 2023 09:30:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BD8FC6B0073; Sun, 19 Feb 2023 09:30:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A7A3E6B0074; Sun, 19 Feb 2023 09:30:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 930746B0072 for ; Sun, 19 Feb 2023 09:30:11 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4A94DC031C for ; Sun, 19 Feb 2023 14:30:11 +0000 (UTC) X-FDA: 80484276222.12.67015B4 Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf27.hostedemail.com (Postfix) with ESMTP id CD5114000B for ; Sun, 19 Feb 2023 14:30:08 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=nU0xzuO7; spf=pass (imf27.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676817009; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EGZ7SSY0qGjJ4cAo8CM8G6U/1RpqPcXqK25mec2kdEc=; b=UYYEYudLXdYgMPMnHhNID/cyuqBLWVaqHB/g7zxOYLGXzRJoYsNJNxZ6YbTHPUlOIe46BZ PAO0UKn4/Gqviy4S30fhI5aLqg3nRXn+XgcCTNBk7frzVYZwa56k8kKdQGrOPV10PU2RbU WRnrM4YFGpLlH753LsH2CzgI9zQqwmw= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=nU0xzuO7; spf=pass (imf27.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676817009; a=rsa-sha256; cv=none; b=scek+2xReDahCeNaUIzWFeaDqsFsVFnxU4rpWcdykKHc52lzyhvCTS5AuIlVAapQnjgO/7 r1h1SyOb6MCVFnZYTUJNsz9InBQrj/1QycbUlEIERcFcxjmJ5yg/t8ozfwLKnrr/m5fY5r XjP2RAzk89uBqcE8hsdKMgA0VzGMjKM= Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31JD412d015451; Sun, 19 Feb 2023 14:29:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : in-reply-to : references : date : message-id : content-type : mime-version; s=pp1; bh=EGZ7SSY0qGjJ4cAo8CM8G6U/1RpqPcXqK25mec2kdEc=; b=nU0xzuO77EzOmbHxhPE1vM4aOqdymf1wn++QmiY/nQ2xou++bR45sN2dIBOStOfb6Gff SYANwRomaymuknJxSczZp5r2Y1tMQAVKrTckSP88w0pZf2juSdQekNgsFI+mAiNT/p0Y MQohd2RV+IOqTlCnA02yEV0bI7zMY6uS0p4Viu7O5ROekHS9IpGh0Jc1PgNFmkEPI2pE 8yKurxK3jSTAPu2Uwc9+YdddYcB+A5xNSiat44Z1ikLzA7LPyJPgsfZu7yYoy0nVFlJG 290ElEwVTiMLMM/5ijduQmJ8WkzQxR1ze8zEZQ8vV5oP8ER6uncKEQfqEPN66XKJyD/p tg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3nu8u1u2xj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 19 Feb 2023 14:29:56 +0000 Received: from m0098416.ppops.net (m0098416.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 31JETuX1032696; Sun, 19 Feb 2023 14:29:56 GMT Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3nu8u1u2xe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 19 Feb 2023 14:29:56 +0000 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 31JDTgOO003465; Sun, 19 Feb 2023 14:29:55 GMT Received: from smtprelay03.wdc07v.mail.ibm.com ([9.208.129.113]) by ppma01dal.us.ibm.com (PPS) with ESMTPS id 3ntpa71wax-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 19 Feb 2023 14:29:55 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay03.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 31JETruq40567174 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 19 Feb 2023 14:29:53 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5706958058; Sun, 19 Feb 2023 14:29:53 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9704958057; Sun, 19 Feb 2023 14:29:50 +0000 (GMT) Received: from skywalker.linux.ibm.com (unknown [9.43.112.244]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Sun, 19 Feb 2023 14:29:50 +0000 (GMT) X-Mailer: emacs 29.0.60 (via feedmail 11-beta-1 I) From: "Aneesh Kumar K.V" To: SeongJae Park Cc: lsf-pc@lists.linux-foundation.org, Linux MM , Yu Zhao , Dave Hansen , Johannes Weiner , damon@lists.linux.dev Subject: Re: [LSF/MM/BPF TOPIC] Using hardware counters to determine hot/cold pages In-Reply-To: <20230217164211.59619-1-sj@kernel.org> References: <20230217164211.59619-1-sj@kernel.org> Date: Sun, 19 Feb 2023 19:59:47 +0530 Message-ID: <87fsb1g23o.fsf@linux.ibm.com> Content-Type: text/plain X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 1Oz75dMC7TR3xyQpi6Cxo2FtnEWEUPG3 X-Proofpoint-GUID: lLrsprXF-09XrlSEYUoHAb2KnLeREY9y X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-19_08,2023-02-17_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 clxscore=1011 priorityscore=1501 suspectscore=0 lowpriorityscore=0 impostorscore=0 mlxscore=0 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302190132 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: CD5114000B X-Stat-Signature: uupn83n1s6ujx8o9yba79i74135zn13x X-HE-Tag: 1676817008-221626 X-HE-Meta: U2FsdGVkX18/ASyjzlUK2cfuMzEGI/4SwJIjf950ZmhXBxJdTpHYu7xsoZ8SBIrjCjkfrvsywcMjC0R2dy9WlPWhyCxAdpKIjqhN/8cVrf/hklVygJ1X7fJADpv0W0Cs0bY6fE0Mlh1zRKnuEPo3hWNq6jYme/cwNLKus/koNy22JOghgQqOsRxdVKr+1Agi7ypGJIhrNLktibWJLu8toU9IWUVON+4O4TrO8IARVfOHPI1FBrUjPGefJj4E/TdCxJ2MYary+owFMpHNjU1hK/GxDmUXmcslyknQkbVlbuZS+OKkH94CRKqTngMxL1mY5FdM3NkS56jppR2cJoaGZJ2fg6urCxjtDb6EkwMeqpMiVc4H6c8sgUMxarNjKTJLVHu6LONAA7iqy5NlKGp3klPyWwI2cvqILDmoI6WLnpQ1/smRtia4dUsb7ofb5vAOW6+48gcOPAUTBA6smfmhHRaH7dpAiIBUQMf6eoo9XWZL32RdapYM/UEoV3m5K4Zp6eniDpHyBINJN+sIK+Lorylc1BmvUH1Hbow8b76hJVyK74HjEs7hfSH3a1fwvsulv+51rhrTegvXZrUcptbI9d/+pp4Y70H8tIBJlxJboZjMKCG7tmt1vhWw1f9OJOzwxFcr45v8t6YndQF4wL3ZU9IJRoe6Dq/OerYx7FdHndtY90OtdicCXMbgFpwbiVP4ogLFKI3HhGZ1xCeFjivRvk3fpcD0GsU2yT9Dn7TsW5glwkGy216G7BM9ANuUPSOkIbZC+ssLm+amfEAcdS9fArzm8LyEnY1OdV5wmxe0UAExaXr7ViA7PP++oBYbLCQ567TMbJk8giUCjaR4Wfl9Md9f02Rd4/zOjmp95cvXPcpjZJPzoQHX8CNWCVPD8QM1K+O4MWm0K+gi1s6w1gyhfMlN6u/2bANMUUP8byGGcvF1tzFrUnHKBMw3PlwjCj9jxdu7qjkFJpT8MKRSwBu OhrO7igK UaWRpBpLxMAjk90CsxipGVXTzmcFVOIKke6G+1Lur72PWO0hZcWxU4X6Cn4JRcoH0MrhxWF6lSdQMHAGaNRoXX7lXYtg7+aaVSMrG7W0IvCU7wFAmIckLZnUrKl39yp7XtFptp8LWbPrRiBNlZdQbDabhIxLo7u8ghoSArVjM2dzlmRWub4Kg75J7yO8zICwHvp0p4xnAMmyzQI9JEkPA6dZtjCdPe3M/yGnF1qrDxgc28iPneJ1IfX15kUzQ/a0nbnBnKk0yyJ2es2zV/Nm4B5aO6yqfJgYdDlyrYecfAtAO1rcq49L9jGFle8rj/HWKryR7j7UUhBDqOBPAPGk5UCnhyA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: SeongJae Park writes: > Hi Aneesh, > > On Fri, 17 Feb 2023 17:28:09 +0530 Aneesh Kumar K V wrote: > >> PowerPC architecture (POWER10) supports a Hot/Cold page tracking >> facility that provides access counter and access affinity details at >> configurable page size granularity [1]. I have been looking at using >> this counter in different areas of the kernel such as >> >> 1) Page reclaim/demotion >> 2) THP utilization >> 3) Page promotion. >> >> I have done some MGLRU integration and would like to discuss the >> observation with the rest of the community. It is still not clear what >> are the best ways to integrate these hardware counters in the Linux >> kernel. > > Sounds very interesting. I think DAMON might be one another option, because it > is designed to be easy to extended with various source of access > information[1], and provides an abstraction layer for access temparature based > memory management[2], namely Data Access Monitoring-based Operation Schemes > (DAMOS). > >> Attached is the performance graph showing how the mongodb/ycsb >> benchmark performs when using hardware counters with MGLRU aging. An >> early RFC version of the code can be found at >> https://github.com/kvaneesh/linux/commit/b472e2c8080823bb4114c286270aea3e18ffe221 >> . I also expect we can get some numbers w.r.t THP usage before the >> conference. > > I also have experimented a DAMON-based THP optimization[3], which shown > interesting results. > > Hope to discuss about this with you at LSF/MM. FYI, I also proposed an LSF/MM > topic for DAMON[4]. > > [1] https://docs.kernel.org/mm/damon/design.html#configurable-layers > [2] https://docs.kernel.org/mm/damon/api.html#c.damos > [3] https://www.amazon.science/publications/daos-data-access-aware-operating-system > [4] https://lore.kernel.org/damon/20230214003328.55285-1-sj@kernel.org/ > > The hardware counters that are supported in the case of POWER10 are based on physical addresses. The hardware facility will count the access across a physical address range and there is a counter for each page that gives the access count and also information about which node did access the page. I haven't spent much time studying DAMON so I might be wrong here. The reason I avoided using DAMON for the POC was because my goal was to evaluate how the hardware counters measured against the pte reference bit and I was not sure I could evaluate that using the DAMON action facility. I do agree that we could add a layer similar to DAMON_PADDR and expose the details to userspace. But I was not sure we can take action based on that. In most cases what I wanted was to move the coldest page in the Numa node to swap. So that is relative hotness rather than moving a page that got a hotness value less than 10 to swap even though we can figure out a way to make the latter similar to the former. I will look at DAMON and see if that is the best framework for things like this. -aneesh