From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEB79C43334 for ; Mon, 13 Jun 2022 07:05:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 450A26B0252; Mon, 13 Jun 2022 03:05:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 400156B0254; Mon, 13 Jun 2022 03:05:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C8FA6B0255; Mon, 13 Jun 2022 03:05:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1D2426B0252 for ; Mon, 13 Jun 2022 03:05:51 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E98F560E40 for ; Mon, 13 Jun 2022 07:05:50 +0000 (UTC) X-FDA: 79572327660.13.45D1613 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf21.hostedemail.com (Postfix) with ESMTP id 8D5E31C0099 for ; Mon, 13 Jun 2022 07:05:50 +0000 (UTC) Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25D4luJ0002090; Mon, 13 Jun 2022 07:05:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : date : mime-version : subject : to : cc : references : from : in-reply-to : content-type : content-transfer-encoding; s=pp1; bh=90wNVQAs7GsbSew2Laoy8zLL4m7qa0zP4QQopuuLVg8=; b=lEmfzx2eUUy/PH3xxVgz+kfn0MfolmkOByJyPu7LUtDVHWmra95aUPIYUAUxXdXLSGnM SKzcD3aAp1d/ONui0kE/GQky8U8OciPohd1vVIKPecUwzXKYyhXKd0S638FskndvXhwJ Ybddxzj816l2owJmdWiyMjvhfBMiYJc92CzW7eLDYO2u12Rm03fIb9EKtjqy5o3Ibnbn wgYkKicJpwS5e3pXiLJiCZnBGWXjFlgGKRE3C1Hopgz4CFNuMKB7PuYU+F5zRDQ3TjQr CDJz0VOiM5pDAMwDaeFIUJDGcqGxE45CWJLi+M0uNPYrvAj1JkwGgJ/W82Bp3q+2gy+S vw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3gn4qjhuvc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 13 Jun 2022 07:05:39 +0000 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 25D6Z5wf004498; Mon, 13 Jun 2022 07:05:38 GMT Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3gn4qjhuuq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 13 Jun 2022 07:05:38 +0000 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 25D6oKfB008989; Mon, 13 Jun 2022 07:05:36 GMT Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by ppma06ams.nl.ibm.com with ESMTP id 3gmjajaavr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 13 Jun 2022 07:05:36 +0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 25D75YoT22413788 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 Jun 2022 07:05:34 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 34DCDA405B; Mon, 13 Jun 2022 07:05:34 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A06F0A4060; Mon, 13 Jun 2022 07:05:28 +0000 (GMT) Received: from [9.43.31.74] (unknown [9.43.31.74]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 13 Jun 2022 07:05:28 +0000 (GMT) Message-ID: <81e5904b-73f7-2977-d135-2d27c6c216ac@linux.ibm.com> Date: Mon, 13 Jun 2022 12:35:27 +0530 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 Subject: Re: [PATCH v6 04/13] mm/demotion/dax/kmem: Set node's memory tier to MEMORY_TIER_PMEM Content-Language: en-US To: Ying Huang , linux-mm@kvack.org, akpm@linux-foundation.org Cc: Wei Xu , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes References: <20220610135229.182859-1-aneesh.kumar@linux.ibm.com> <20220610135229.182859-5-aneesh.kumar@linux.ibm.com> <193ad45f2ec47ac157a812975f3e4235fcbc061a.camel@intel.com> From: Aneesh Kumar K V In-Reply-To: <193ad45f2ec47ac157a812975f3e4235fcbc061a.camel@intel.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: BKl-RWz6saa2SjyTpViSL0jofm2AofLH X-Proofpoint-ORIG-GUID: j5fUOnyPl-nbkQoFImUumeAQy9XudaRP X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-13_02,2022-06-09_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 adultscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 spamscore=0 priorityscore=1501 phishscore=0 impostorscore=0 clxscore=1015 mlxscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206130030 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655103950; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=90wNVQAs7GsbSew2Laoy8zLL4m7qa0zP4QQopuuLVg8=; b=JbbGhZS4HuErzchjP7GzlV5QBOntPj/deoV9fwjjPCsppErc7LABJ233AVidccexofsaIr +poOC2ZVmf3VpmEPrU6XU0KMmkx/3psQCg3LEbDBnWudPT4YPYpetUdhQpu7dv5SXQJv9w MNmSpXQSbHWjrfYzAotEJqN43HdMEGA= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=lEmfzx2e; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf21.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655103950; a=rsa-sha256; cv=none; b=gazyUFhS1Grtht377BlFN1SylCFw4u762xUZKW5AFSH+niRLKC7QxkX6BDJ7/jhNMgQMF0 5SmzftXQzG8JULzrqkkRgUcb68AMethRYsjf7Y6ni6+woD416gXWZOelGOlDsSPmtWcljb Abfbt21Y1qemJdp2EmaMppq0oZ+jFjM= X-Stat-Signature: wecpcp4wwpdd8pzbsq4w8pwt7qsby4nh X-Rspamd-Queue-Id: 8D5E31C0099 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=lEmfzx2e; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf21.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1655103950-130872 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 6/13/22 12:29 PM, Ying Huang wrote: > On Fri, 2022-06-10 at 19:22 +0530, Aneesh Kumar K.V wrote: >> By default, all nodes are assigned to DEFAULT_MEMORY_TIER which >> is the memory tier designated for nodes with DRAM >> >> Set dax kmem device node's tier to MEMORY_TIER_PMEM. MEMORY_TIER_PMEM >> is assigned a default rank value of 100 and appears below DEFAULT_MEMORY_TIER >> in demotion order. >> >> Signed-off-by: Jagdish Gediya >> Signed-off-by: Aneesh Kumar K.V >> --- >>  drivers/dax/kmem.c | 4 ++ >>  include/linux/memory-tiers.h | 1 + >>  mm/memory-tiers.c | 78 ++++++++++++++++++++++++++++++++++++ >>  3 files changed, 83 insertions(+) >> >> diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c >> index a37622060fff..0cb3de3d138f 100644 >> --- a/drivers/dax/kmem.c >> +++ b/drivers/dax/kmem.c >> @@ -11,6 +11,7 @@ >>  #include >>  #include >>  #include >> +#include >>  #include "dax-private.h" >>  #include "bus.h" >> >> >> @@ -147,6 +148,9 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax) >> >> >>   dev_set_drvdata(dev, data); >> >> >> +#ifdef CONFIG_TIERED_MEMORY >> + node_create_and_set_memory_tier(numa_node, MEMORY_TIER_PMEM); >> +#endif >>   return 0; >> >> >>  err_request_mem: >> diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h >> index 44c3c3b16a36..e102ec73ab80 100644 >> --- a/include/linux/memory-tiers.h >> +++ b/include/linux/memory-tiers.h >> @@ -18,6 +18,7 @@ >>  #define MAX_MEMORY_TIERS 3 >> >> >>  extern bool numa_demotion_enabled; >> +int node_create_and_set_memory_tier(int node, int tier); >>  #else >>  #define numa_demotion_enabled false >> >> >> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c >> index c3123a457d90..00d393a5a628 100644 >> --- a/mm/memory-tiers.c >> +++ b/mm/memory-tiers.c >> @@ -67,6 +67,84 @@ static struct memory_tier *register_memory_tier(unsigned int tier, >>   return memtier; >>  } >> >> >> +static struct memory_tier *__node_get_memory_tier(int node) >> +{ >> + struct memory_tier *memtier; >> + >> + list_for_each_entry(memtier, &memory_tiers, list) { >> + if (node_isset(node, memtier->nodelist)) >> + return memtier; >> + } >> + return NULL; >> +} >> + > > I suggest to add NODE_DATA(nid)->mem_tier before this patch. That is, > part of [9/13]. That will make code much simpler and easier to > review. > I would still like to keep introducing that NODE_DATA()->memtier later, so that the everything before that can be reviewed with simpler locking. __node_get_memory_tier() get fixed in patch 9 where all the details of that lockless access is documented. This also enables one to look at the history later. > And, in addition to dax_kmem, whenever a normal node is onlined, we > need to add it to the default memory tier. I found this is done in > [5/13]. IMHO, we should move that part before this patch. > Sure will move the hotplug callback earlier. -aneesh