From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A496C433EF for ; Tue, 5 Jul 2022 04:19:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA49B6B0071; Tue, 5 Jul 2022 00:19:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A54796B0073; Tue, 5 Jul 2022 00:19:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91B8F6B0074; Tue, 5 Jul 2022 00:19:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 827036B0071 for ; Tue, 5 Jul 2022 00:19:02 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4DC586066E for ; Tue, 5 Jul 2022 04:19:02 +0000 (UTC) X-FDA: 79651740924.18.D81EFEB Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf30.hostedemail.com (Postfix) with ESMTP id 6539A80057 for ; Tue, 5 Jul 2022 04:19:01 +0000 (UTC) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 2652pWP2027076; Tue, 5 Jul 2022 04:18:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : date : mime-version : subject : to : cc : references : from : in-reply-to : content-type : content-transfer-encoding; s=pp1; bh=EmUn+p3U8+//9srg7/zjIpTHTjdV/s90He+iwVewAAw=; b=R60tvOEeTkOlPmBe4o782e4IfmjhOa9L46BSGVUq6ioFcBA/vm8bYTgi3fLzqQoUc8ig bR3V3CnHnjVlizphFXOiGJxRxzpdwF3zR9Oskds1E8F4yqmQbVa02gjihXOqsiKOft9P P/wwwuyJQ8EesFVm3LQkreFYdn+01IW7Y030jYCTZSaUH2fBRRI9GG8SEJxcNd9DBuNv xLAwlZYkAcb/Hy3KXpHuqCdSz7NszA554z/421t6ewwVYUcKGRE0DRlY+10dOlZZ+YEH fDDjux0tv6uEFfeVstKn/stNhG5hPpjuqNDePH2xjy9PLoBdZ6gLZLjd9MeeYQmSDuhf gQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3h4cvs1g8k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 05 Jul 2022 04:18:14 +0000 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2654EuNV028738; Tue, 5 Jul 2022 04:18:13 GMT Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3h4cvs1g86-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 05 Jul 2022 04:18:13 +0000 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 26546L7p032058; Tue, 5 Jul 2022 04:18:11 GMT Received: from b06cxnps3075.portsmouth.uk.ibm.com (d06relay10.portsmouth.uk.ibm.com [9.149.109.195]) by ppma06ams.nl.ibm.com with ESMTP id 3h2d9jbf6v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 05 Jul 2022 04:18:11 +0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 2654I9BV22872558 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 5 Jul 2022 04:18:09 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E92CC4C04A; Tue, 5 Jul 2022 04:18:08 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0F1AE4C040; Tue, 5 Jul 2022 04:18:01 +0000 (GMT) Received: from [9.43.26.15] (unknown [9.43.26.15]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 5 Jul 2022 04:18:00 +0000 (GMT) Message-ID: Date: Tue, 5 Jul 2022 09:47:58 +0530 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 Subject: Re: [PATCH v8 00/12] mm/demotion: Memory tiers and demotion Content-Language: en-US To: Matthew Wilcox Cc: linux-mm@kvack.org, akpm@linux-foundation.org, Wei Xu , Huang Ying , Yang Shi , Davidlohr Bueso , Tim C Chen , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Johannes Weiner , jvgediya.oss@gmail.com References: <20220704070612.299585-1-aneesh.kumar@linux.ibm.com> From: Aneesh Kumar K V In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: wD9sbBOGKadqCB1xSyR-UMBH-fBkln82 X-Proofpoint-ORIG-GUID: qDUH9gg9wlXRr5Px6x6cRX9OoNlpNJ68 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-07-05_02,2022-06-28_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 spamscore=0 impostorscore=0 bulkscore=0 lowpriorityscore=0 mlxlogscore=999 clxscore=1011 mlxscore=0 suspectscore=0 adultscore=0 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2207050017 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656994742; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EmUn+p3U8+//9srg7/zjIpTHTjdV/s90He+iwVewAAw=; b=I0h2H8JYIhN/7hvlRxhdg+ssD+ncpV9Hp37EmwMZm04EHl3sN6Yh3YZ6oW43KnPmBh6X9X 95H9s0YP+ZdnH5BV1lT6WfRScwfWTsR78LyfFJqCkf8aP2lDzvSJMC3WCgueuxekmtRNDy XTCgI5v5YB0malHUaRQBcw2GWpW6LKY= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=R60tvOEe; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf30.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656994742; a=rsa-sha256; cv=none; b=NjqCx9ugR+sMhP2Sn/eYfXIisaXC9xVGRTnIDS/3gKYmxEYLOInEAuxz7Rt2YO/ClSt9SD qTNUzF+X928/i0/esBE7X7+8hs9qq3i2vZYz78WgNVPH0ZsBIG83KYgvJIzjaLQr9tJFNm Tu8btkuY2rR3B4WJF8N9mcdj+We+3LY= X-Rspam-User: X-Rspamd-Queue-Id: 6539A80057 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=R60tvOEe; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf30.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com X-Stat-Signature: iq5kcm5wnz7qhioomu9q3bwngzui48ab X-Rspamd-Server: rspam08 X-HE-Tag: 1656994741-317232 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 7/4/22 8:30 PM, Matthew Wilcox wrote: > On Mon, Jul 04, 2022 at 12:36:00PM +0530, Aneesh Kumar K.V wrote: >> * The current tier initialization code always initializes >> each memory-only NUMA node into a lower tier. But a memory-only >> NUMA node may have a high performance memory device (e.g. a DRAM >> device attached via CXL.mem or a DRAM-backed memory-only node on >> a virtual machine) and should be put into a higher tier. >> >> * The current tier hierarchy always puts CPU nodes into the top >> tier. But on a system with HBM (e.g. GPU memory) devices, these >> memory-only HBM NUMA nodes should be in the top tier, and DRAM nodes >> with CPUs are better to be placed into the next lower tier. > > These things that you identify as problems seem perfectly sensible to me. > Memory which is attached to this CPU has the lowest latency and should > be preferred over more remote memory, no matter its bandwidth. Allocation will prefer local memory over remote memory. Memory tiers are used during demotion and currently, the kernel demotes cold pages from DRAM memory to these special device memories because they appear as memory-only NUMA nodes. In many cases (ex: GPU) what is desired is the demotion of cold pages from GPU memory to DRAM or even slow memory. This patchset builds a framework to enable such demotion criteria. -aneesh