From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D2CE1CD1288 for ; Wed, 3 Apr 2024 17:05:42 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rs42S-00044g-5R; Wed, 03 Apr 2024 13:04:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rs42P-00044A-EU for qemu-devel@nongnu.org; Wed, 03 Apr 2024 13:04:37 -0400 Received: from frasgout.his.huawei.com ([185.176.79.56]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rs42L-0004mB-UB for qemu-devel@nongnu.org; Wed, 03 Apr 2024 13:04:37 -0400 Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4V8rfg1yVsz6D8Yh; Thu, 4 Apr 2024 01:03:07 +0800 (CST) Received: from lhrpeml500005.china.huawei.com (unknown [7.191.163.240]) by mail.maildlp.com (Postfix) with ESMTPS id 1B8041400D7; Thu, 4 Apr 2024 01:04:27 +0800 (CST) Received: from localhost (10.202.227.76) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 18:04:26 +0100 Date: Wed, 3 Apr 2024 18:04:25 +0100 To: "Ho-Ren (Jack) Chuang" CC: "Huang, Ying" , Gregory Price , , , , , Eishan Mirakhur , Vinicius Tavares Petrucci , Ravis OpenSrc , Alistair Popple , Srinivasulu Thanneeru , SeongJae Park , Dan Williams , Vishal Verma , "Dave Jiang" , Andrew Morton , , , , , "Ho-Ren (Jack) Chuang" , "Ho-Ren (Jack) Chuang" , , Hao Xiang Subject: Re: [PATCH v10 2/2] memory tier: create CPUless memory tiers after obtaining HMAT info Message-ID: <20240403180425.00003be0@Huawei.com> In-Reply-To: <20240402001739.2521623-3-horenchuang@bytedance.com> References: <20240402001739.2521623-1-horenchuang@bytedance.com> <20240402001739.2521623-3-horenchuang@bytedance.com> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.1.0 (GTK 3.24.33; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.202.227.76] X-ClientProxiedBy: lhrpeml100002.china.huawei.com (7.191.160.241) To lhrpeml500005.china.huawei.com (7.191.163.240) Received-SPF: pass client-ip=185.176.79.56; envelope-from=jonathan.cameron@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Jonathan Cameron From: Jonathan Cameron via Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org A few minor comments inline. > diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h > index a44c03c2ba3a..16769552a338 100644 > --- a/include/linux/memory-tiers.h > +++ b/include/linux/memory-tiers.h > @@ -140,12 +140,13 @@ static inline int mt_perf_to_adistance(struct access_coordinate *perf, int *adis > return -EIO; > } > > -struct memory_dev_type *mt_find_alloc_memory_type(int adist, struct list_head *memory_types) > +static inline struct memory_dev_type *mt_find_alloc_memory_type(int adist, > + struct list_head *memory_types) > { > return NULL; > } > > -void mt_put_memory_types(struct list_head *memory_types) > +static inline void mt_put_memory_types(struct list_head *memory_types) > { Why in this patch and not previous one? > > } > diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c > index 974af10cfdd8..44fa10980d37 100644 > --- a/mm/memory-tiers.c > +++ b/mm/memory-tiers.c > @@ -36,6 +36,11 @@ struct node_memory_type_map { > > static DEFINE_MUTEX(memory_tier_lock); > static LIST_HEAD(memory_tiers); > +/* > + * The list is used to store all memory types that are not created > + * by a device driver. > + */ > +static LIST_HEAD(default_memory_types); > static struct node_memory_type_map node_memory_types[MAX_NUMNODES]; > struct memory_dev_type *default_dram_type; > > @@ -108,6 +113,8 @@ static struct demotion_nodes *node_demotion __read_mostly; > > static BLOCKING_NOTIFIER_HEAD(mt_adistance_algorithms); > > +/* The lock is used to protect `default_dram_perf*` info and nid. */ > +static DEFINE_MUTEX(default_dram_perf_lock); > static bool default_dram_perf_error; > static struct access_coordinate default_dram_perf; > static int default_dram_perf_ref_nid = NUMA_NO_NODE; > @@ -505,7 +512,8 @@ static inline void __init_node_memory_type(int node, struct memory_dev_type *mem > static struct memory_tier *set_node_memory_tier(int node) > { > struct memory_tier *memtier; > - struct memory_dev_type *memtype; > + struct memory_dev_type *mtype = default_dram_type; Does the rename add anything major to the patch? If not I'd leave it alone to reduce the churn and give a more readable patch. If it is worth doing perhaps a precursor patch? > + int adist = MEMTIER_ADISTANCE_DRAM; > pg_data_t *pgdat = NODE_DATA(node); > > > @@ -514,11 +522,20 @@ static struct memory_tier *set_node_memory_tier(int node) > if (!node_state(node, N_MEMORY)) > return ERR_PTR(-EINVAL); > > - __init_node_memory_type(node, default_dram_type); > + mt_calc_adistance(node, &adist); > + if (node_memory_types[node].memtype == NULL) { > + mtype = mt_find_alloc_memory_type(adist, &default_memory_types); > + if (IS_ERR(mtype)) { > + mtype = default_dram_type; > + pr_info("Failed to allocate a memory type. Fall back.\n"); > + } > + } > + > + __init_node_memory_type(node, mtype); > > - memtype = node_memory_types[node].memtype; > - node_set(node, memtype->nodes); > - memtier = find_create_memory_tier(memtype); > + mtype = node_memory_types[node].memtype; > + node_set(node, mtype->nodes); > + memtier = find_create_memory_tier(mtype); > if (!IS_ERR(memtier)) > rcu_assign_pointer(pgdat->memtier, memtier); > return memtier; > @@ -655,6 +672,33 @@ void mt_put_memory_types(struct list_head *memory_types) > } > EXPORT_SYMBOL_GPL(mt_put_memory_types); > > +/* > + * This is invoked via `late_initcall()` to initialize memory tiers for > + * CPU-less memory nodes after driver initialization, which is > + * expected to provide `adistance` algorithms. > + */ > +static int __init memory_tier_late_init(void) > +{ > + int nid; > + > + mutex_lock(&memory_tier_lock); > + for_each_node_state(nid, N_MEMORY) > + if (node_memory_types[nid].memtype == NULL) > + /* > + * Some device drivers may have initialized memory tiers > + * between `memory_tier_init()` and `memory_tier_late_init()`, > + * potentially bringing online memory nodes and > + * configuring memory tiers. Exclude them here. > + */ Does the comment refer to this path, or to ones where memtype is set? > + set_node_memory_tier(nid); Given the large comment I would add {} to help with readability. You could flip the logic to reduce indent for_each_node_state(nid, N_MEMORY) { if (node_memory_types[nid].memtype) continue; /* * Some device drivers may have initialized memory tiers * between `memory_tier_init()` and `memory_tier_late_init()`, * potentially bringing online memory nodes and * configuring memory tiers. Exclude them here. */ set_node_memory_tier(nid); > + > + establish_demotion_targets(); > + mutex_unlock(&memory_tier_lock); > + > + return 0; > +} > +late_initcall(memory_tier_late_init); > + > static void dump_hmem_attrs(struct access_coordinate *coord, const char *prefix) > { > pr_info( > @@ -668,7 +712,7 @@ int mt_set_default_dram_perf(int nid, struct access_coordinate *perf, > { > int rc = 0; > > - mutex_lock(&memory_tier_lock); > + mutex_lock(&default_dram_perf_lock); As below, this is a classic case where guard() will help readability. > if (default_dram_perf_error) { > rc = -EIO; > goto out; > @@ -716,23 +760,30 @@ int mt_set_default_dram_perf(int nid, struct access_coordinate *perf, > } > > out: > - mutex_unlock(&memory_tier_lock); > + mutex_unlock(&default_dram_perf_lock); > return rc; > } > > int mt_perf_to_adistance(struct access_coordinate *perf, int *adist) > { > - if (default_dram_perf_error) > - return -EIO; > + int rc = 0; Looks like rc is set in all paths that reach where it isused. > > - if (default_dram_perf_ref_nid == NUMA_NO_NODE) > - return -ENOENT; > + mutex_lock(&default_dram_perf_lock); This would benefit quite a lot from guard(mutex)(&default_dram_perf_lock); and direct returns throughout. > + if (default_dram_perf_error) { > + rc = -EIO; > + goto out; > + } > > if (perf->read_latency + perf->write_latency == 0 || > - perf->read_bandwidth + perf->write_bandwidth == 0) > - return -EINVAL; > + perf->read_bandwidth + perf->write_bandwidth == 0) { > + rc = -EINVAL; > + goto out; > + } > > - mutex_lock(&memory_tier_lock); > + if (default_dram_perf_ref_nid == NUMA_NO_NODE) { > + rc = -ENOENT; > + goto out; > + } > /* > * The abstract distance of a memory node is in direct proportion to > * its memory latency (read + write) and inversely proportional to its > @@ -745,9 +796,10 @@ int mt_perf_to_adistance(struct access_coordinate *perf, int *adist) > (default_dram_perf.read_latency + default_dram_perf.write_latency) * > (default_dram_perf.read_bandwidth + default_dram_perf.write_bandwidth) / > (perf->read_bandwidth + perf->write_bandwidth); > - mutex_unlock(&memory_tier_lock); > > - return 0; > +out: > + mutex_unlock(&default_dram_perf_lock); > + return rc; > } > EXPORT_SYMBOL_GPL(mt_perf_to_adistance); > > @@ -858,7 +910,8 @@ static int __init memory_tier_init(void) > * For now we can have 4 faster memory tiers with smaller adistance > * than default DRAM tier. > */ > - default_dram_type = alloc_memory_type(MEMTIER_ADISTANCE_DRAM); > + default_dram_type = mt_find_alloc_memory_type(MEMTIER_ADISTANCE_DRAM, > + &default_memory_types); Unusual indenting. Align with just after ( > if (IS_ERR(default_dram_type)) > panic("%s() failed to allocate default DRAM tier\n", __func__); > > @@ -868,6 +921,14 @@ static int __init memory_tier_init(void) > * types assigned. > */ > for_each_node_state(node, N_MEMORY) { > + if (!node_state(node, N_CPU)) > + /* > + * Defer memory tier initialization on CPUless numa nodes. > + * These will be initialized after firmware and devices are I think this wraps at just over 80 chars. Seems silly to wrap so tightly and not quite fit under 80. (this is about 83 chars. > + * initialized. > + */ > + continue; > + > memtier = set_node_memory_tier(node); > if (IS_ERR(memtier)) > /*