From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D22BC433F5 for ; Wed, 25 May 2022 11:48:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 18F6E8D0003; Wed, 25 May 2022 07:48:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 13F558D0002; Wed, 25 May 2022 07:48:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 050048D0003; Wed, 25 May 2022 07:48:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id ED5348D0002 for ; Wed, 25 May 2022 07:48:53 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C745E21292 for ; Wed, 25 May 2022 11:48:53 +0000 (UTC) X-FDA: 79504093746.04.1BE27C0 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf16.hostedemail.com (Postfix) with ESMTP id 5656C180045 for ; Wed, 25 May 2022 11:48:38 +0000 (UTC) Received: from fraeml707-chm.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4L7Tkh1kyWz67F4F; Wed, 25 May 2022 19:44:44 +0800 (CST) Received: from lhreml710-chm.china.huawei.com (10.201.108.61) by fraeml707-chm.china.huawei.com (10.206.15.35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Wed, 25 May 2022 13:48:49 +0200 Received: from localhost (10.202.226.42) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Wed, 25 May 2022 12:48:48 +0100 Date: Wed, 25 May 2022 12:48:47 +0100 From: Jonathan Cameron To: Alistair Popple CC: Wei Xu , Aneesh Kumar K.V , Dave Hansen , "Huang Ying" , Andrew Morton , "Greg Thelen" , Yang Shi , "Linux Kernel Mailing List" , Jagdish Gediya , Michal Hocko , Tim C Chen , Baolin Wang , "Feng Tang" , Davidlohr Bueso , "Dan Williams" , David Rientjes , Linux MM , Brice Goglin , "Hesham Almatary" Subject: Re: RFC: Memory Tiering Kernel Interfaces (v2) Message-ID: <20220525124847.00007a16@Huawei.com> In-Reply-To: <87h75ef3y5.fsf@nvdebian.thelocal> References: <20220512160010.00005bc4@Huawei.com> <20220518130037.00001cce@Huawei.com> <8735gzdpsx.fsf@linux.ibm.com> <87h75ef3y5.fsf@nvdebian.thelocal> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.29; i686-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.202.226.42] X-ClientProxiedBy: lhreml726-chm.china.huawei.com (10.201.108.77) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected Authentication-Results: imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com X-Rspam-User: X-Stat-Signature: og5kc4kwxyzxsykuw7y91q4n9pkz9w6e X-Rspamd-Queue-Id: 5656C180045 X-Rspamd-Server: rspam01 X-HE-Tag: 1653479318-616567 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 25 May 2022 17:47:33 +1000 Alistair Popple wrote: > Wei Xu writes: > > > On Tue, May 24, 2022 at 6:27 AM Aneesh Kumar K.V > > wrote: > >> > >> Wei Xu writes: > >> > >> > On Wed, May 18, 2022 at 5:00 AM Jonathan Cameron > >> > wrote: > >> >> > >> >> On Wed, 18 May 2022 00:09:48 -0700 > >> >> Wei Xu wrote: > >> > >> ... > >> > >> > Nice :) > >> >> > >> >> Initially I thought this was over complicated when compared to just leaving space, but > >> >> after a chat with Hesham just now you have us both convinced that this is an elegant solution. > >> >> > >> >> Few corners probably need fleshing out: > >> >> * Use of an allocator for new tiers. Flat number at startup, or new one on write of unique > >> >> value to set_memtier perhaps? Also whether to allow drivers to allocate (I think > >> >> we should). > >> >> * Multiple tiers with same rank. My assumption is from demotion path point of view you > >> >> fuse them (treat them as if they were a single tier), but keep them expressed > >> >> separately in the sysfs interface so that the rank can be changed independently. > >> >> * Some guidance on what values make sense for given rank default that might be set by > >> >> a driver. If we have multiple GPU vendors, and someone mixes them in a system we > >> >> probably don't want the default values they use to result in demotion between them. > >> >> This might well be a guidance DOC or appropriate set of #define > >> > > >> > All of these are good ideas, though I am afraid that these can make > >> > tier management too complex for what it's worth. > >> > > >> > How about an alternative tier numbering scheme that uses major.minor > >> > device IDs? For simplicity, we can just start with 3 major tiers. > >> > New tiers can be inserted in-between using minor tier IDs. > >> > >> > >> What drives the creation of a new memory tier here? Jonathan was > >> suggesting we could do something similar to writing to set_memtier for > >> creating a new memory tier. > >> > >> $ echo "memtier128" > sys/devices/system/node/node1/set_memtier > >> > >> But I am wondering whether we should implement that now. If we keep > >> "rank" concept and detach tier index (memtier0 is the memory tier with > >> index 0) separate from rank, I assume we have enough flexibility for a > >> future extension that will allow us to create a memory tier from userspace > >> and assigning it a rank value that helps the device to be placed before or > >> after DRAM in demotion order. > >> > >> ie, For now we will only have memtier0, memtier1, memtier2. We won't add > >> dynamic creation of memory tiers and the above memory tiers will have > >> rank value 0, 1, 2 according with demotion order 0 -> 1 -> 2. > > > > Great. So the consensus is to go with the "rank" approach. The above > > sounds good to me as a starting point. > > The rank approach seems good to me too. Rank is good, but I do slightly worry about accidentally defining ABI that people care about with the particular numbers used for the initial ranks. Maybe just x100 on all of them to allow things in between with no change to this initial set of 3? So 0, 100, 200 Jonathan > > - Alistair > > >> -aneesh