From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E37B5C433EF for ; Tue, 10 May 2022 12:10:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40E2C6B0072; Tue, 10 May 2022 08:10:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BDFC6B0073; Tue, 10 May 2022 08:10:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 237816B0074; Tue, 10 May 2022 08:10:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 10B126B0072 for ; Tue, 10 May 2022 08:10:32 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id D579A61007 for ; Tue, 10 May 2022 12:10:31 +0000 (UTC) X-FDA: 79449716262.03.A026546 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf29.hostedemail.com (Postfix) with ESMTP id A122A1200B7 for ; Tue, 10 May 2022 12:10:24 +0000 (UTC) Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24ABJkNO002974; Tue, 10 May 2022 12:10:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : date : mime-version : subject : to : cc : references : from : in-reply-to : content-type : content-transfer-encoding; s=pp1; bh=mzNOXhcjmsKPzuKgqGfty3R6sY92hW9Rr5/I9r4b3Hs=; b=MiUILQ7+fJOMseVbUoKDx3af5Lij7C0/SWhIXs6K3SJmFRc6GCBcK3X/t9vQ/Pid85bv R3tl9brd+QwWzdINEhkzJwoRQADfM6fjEfSWjBARQE19xglZS8E4djiCV8UpGR5wtEhS JK2TDanT/zN69Vx6Hp6rqCAaNcrdZGiqhaH2+JsLgnimx9UQX+wDvmmm/I4i/pmYZNND SpVSBQlYp5Qd8F4o0LHN6ugn4T1Ey6GsEz0EFdO9wBvRMpGsUCN1jeiYmks0J8GNv9zw 9HKF02fYnSdy5oOwL7++3n1FupGGfVGabFl5ellKB3JYiIs4zWqk28Gn8JkPDthR/enc aQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3fyq338xsr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 10 May 2022 12:10:22 +0000 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 24AC6UFO024768; Tue, 10 May 2022 12:10:22 GMT Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3fyq338xrk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 10 May 2022 12:10:21 +0000 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 24AC2cBn029060; Tue, 10 May 2022 12:10:19 GMT Received: from b06cxnps3074.portsmouth.uk.ibm.com (d06relay09.portsmouth.uk.ibm.com [9.149.109.194]) by ppma03ams.nl.ibm.com with ESMTP id 3fwgd8v1fr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 10 May 2022 12:10:19 +0000 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 24ACAHUO50069798 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 10 May 2022 12:10:17 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1996B5204E; Tue, 10 May 2022 12:10:17 +0000 (GMT) Received: from [9.43.24.223] (unknown [9.43.24.223]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTP id 6512A52057; Tue, 10 May 2022 12:10:11 +0000 (GMT) Message-ID: Date: Tue, 10 May 2022 17:40:10 +0530 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.0 Subject: Re: RFC: Memory Tiering Kernel Interfaces Content-Language: en-US To: Hesham Almatary , Yang Shi Cc: Andrew Morton , Dave Hansen , Huang Ying , Dan Williams , Linux MM , Greg Thelen , Jagdish Gediya , Linux Kernel Mailing List , Alistair Popple , Davidlohr Bueso , Michal Hocko , Baolin Wang , Brice Goglin , Feng Tang , Tim Chen , Wei Xu References: <1642ab64-7957-e1e6-71c5-ceab9c23bf41@huawei.com> From: Aneesh Kumar K V In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: nuEGgPOXmFpGslgUksrlba9uGF1Bklcz X-Proofpoint-GUID: lhwH-Ctb-QymV8c4lcCWXWU3_v3NnIxI X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.858,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-10_01,2022-05-10_01,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 bulkscore=0 mlxlogscore=999 clxscore=1011 suspectscore=0 phishscore=0 priorityscore=1501 malwarescore=0 mlxscore=0 impostorscore=0 spamscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205100055 X-Stat-Signature: rqrjn9kznhcepkhdno96h58wzekdx94e Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=MiUILQ7+; spf=pass (imf29.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A122A1200B7 X-HE-Tag: 1652184624-170124 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 5/10/22 3:29 PM, Hesham Almatary wrote: > Hello Yang, > > On 5/10/2022 4:24 AM, Yang Shi wrote: >> On Mon, May 9, 2022 at 7:32 AM Hesham Almatary >> wrote: ... >>> >>> node 0 has a CPU and DDR memory in tier 0, node 1 has GPU and DDR memory >>> in tier 0, >>> node 2 has NVMM memory in tier 1, node 3 has some sort of bigger memory >>> (could be a bigger DDR or something) in tier 2. The distances are as >>> follows: >>> >>> --------------          -------------- >>> |   Node 0   |          |   Node 1   | >>> |  -------   |          |  -------   | >>> | |  DDR  |  |          | |  DDR  |  | >>> |  -------   |          |  -------   | >>> |            |          |            | >>> --------------          -------------- >>>          | 20               | 120    | >>>          v                  v        | >>> ----------------------------       | >>> | Node 2     PMEM          |       | 100 >>> ----------------------------       | >>>          | 100                       | >>>          v                           v >>> -------------------------------------- >>> | Node 3    Large mem                | >>> -------------------------------------- >>> >>> node distances: >>> node   0    1    2    3 >>>      0  10   20   20  120 >>>      1  20   10  120  100 >>>      2  20  120   10  100 >>>      3  120 100  100   10 >>> >>> /sys/devices/system/node/memory_tiers >>> 0-1 >>> 2 >>> 3 >>> >>> N_TOPTIER_MEMORY: 0-1 >>> >>> >>> In this case, we want to be able to "skip" the demotion path from Node 1 >>> to Node 2, >>> >>> and make demotion go directely to Node 3 as it is closer, distance wise. >>> How can >>> >>> we accommodate this scenario (or at least not rule it out as future >>> work) with the >>> >>> current RFC? >> If I remember correctly NUMA distance is hardcoded in SLIT by the >> firmware, it is supposed to reflect the latency. So I suppose it is >> the firmware's responsibility to have correct information. And the RFC >> assumes higher tier memory has better performance than lower tier >> memory (latency, bandwidth, throughput, etc), so it sounds like a >> buggy firmware to have lower tier memory with shorter distance than >> higher tier memory IMHO. > > You are correct if you're assuming the topology is all hierarchically > > symmetric, but unfortuantely, in real hardware (e.g., my example above) > > it is not. The distance/latency between two nodes in the same tier > > and a third node, is different. The firmware still provides the correct > > latency, but putting a node in a tier is up to the kernel/user, and > > is relative: e.g., Node 3 could belong to tier 1 from Node 1's > > perspective, but to tier 2 from Node 0's. > > > A more detailed example (building on my previous one) is when having > > the GPU connected to a switch: > > ---------------------------- > | Node 2     PMEM          | > ---------------------------- >       ^ >       | > --------------          -------------- > |   Node 0   |          |   Node 1   | > |  -------   |          |  -------   | > | |  DDR  |  |          | |  DDR  |  | > |  -------   |          |  -------   | > |    CPU     |          |    GPU     | > --------------          -------------- >        |                  | >        v                  v > ---------------------------- > |         Switch           | > ---------------------------- >        | >        v > -------------------------------------- > | Node 3    Large mem                | > -------------------------------------- > > Here, demoting from Node 1 to Node 3 directly would be faster as > > it only has to go through one hub, compared to demoting from Node 1 > > to Node 2, where it goes through two hubs. I hope that example > > clarifies things a little bit. > Alistair mentioned that we want to consider GPU memory to be expensive and want to demote from GPU to regular DRAM. In that case for the above case we should end up with tier 0 - > Node3 tier 1 -> Node0, Node1 tier 2 -> Node2 Hence node 0: allowed=2 node 1: allowed=2 node 2: allowed = empty node 3: allowed = 0-1 , based on fallback order 1, 0 -aneesh