From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3CDEC433ED for ; Thu, 15 Apr 2021 04:07:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 40A0A61222 for ; Thu, 15 Apr 2021 04:07:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40A0A61222 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9D8B26B0036; Thu, 15 Apr 2021 00:07:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 960656B006C; Thu, 15 Apr 2021 00:07:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D9606B0070; Thu, 15 Apr 2021 00:07:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0179.hostedemail.com [216.40.44.179]) by kanga.kvack.org (Postfix) with ESMTP id 5D9D06B0036 for ; Thu, 15 Apr 2021 00:07:14 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 1D725181AEF15 for ; Thu, 15 Apr 2021 04:07:14 +0000 (UTC) X-FDA: 78033266388.28.5646DBC Received: from mail-io1-f45.google.com (mail-io1-f45.google.com [209.85.166.45]) by imf27.hostedemail.com (Postfix) with ESMTP id 1105780192D4 for ; Thu, 15 Apr 2021 04:07:01 +0000 (UTC) Received: by mail-io1-f45.google.com with SMTP id v123so16035538ioe.10 for ; Wed, 14 Apr 2021 21:07:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6AY733fGeYnXuyutjzCDjkciisfbvulWzaOHZ289MSo=; b=g+uB2jOSsHNLUINlbn5GDoqTqrYOf6elH+fG2NZo4sFIYzRUzYkbZR2L7b9BWjiyGR qCSqED8miQHqrxXw9fUSZnouWyO6e9lreI7BiUFj6rPbzmHc2JjPD/pnJTYXtfZsgnAB 7EeE+WE2jOtacH1whI5r56Ovj5UaoI2oSp59ZpxD4tlH5muut9EzH4Hl3IIKjhP42rVB Fw4wstzeYnQbhk6DaGVvgCs6KD6on+lx5ofNdYIPAvFYzerEY2g0kKbPw6zeextA14lq UgVKEJF4q4XYIyEknElyxQcde/2oI2uOP25T6xnMO8VBlLHEBscQW/1kyqsC07Db1ZRT HYHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6AY733fGeYnXuyutjzCDjkciisfbvulWzaOHZ289MSo=; b=rW6Jj3JebnFMXBop0DRNXadZ9KGWD7vNTIvjc35ZWeLMtS+WB55Na3hOTZIWfhKwKv hf00lIU6I3Cq4C/HB4IPN5penPEjOakY/BkIGPYPV+ev+aF9GSUacxfp2kOhpOKPfiS5 pCkB8sSI2Y3lddTO2UrF6POU5pTE/UkaNk3LA8FzL3kxoI65VefAdqD0EbiHsbqI9cZH 4EBjJ46lIOwPCmdWrbgoV+25eVnrRvdy/x+YTJ/ujLVnIOdhLtHMqOmEVPwoZsftYcBY s4Jh88zjZAEPphdxqbok65BxP1YNZLOBgS4uHz8QbdOU+ayZ3q8EZajVJQY6J6/RQh96 +elw== X-Gm-Message-State: AOAM531Bg+IbmFseUwF/FAnNdzgkooZEJIf9IsuePDrO7q0FYiqJCO6W dX2uWQGMg+dDI7oM9BMDGUDdYvAS6SccrFVu1UMeHA== X-Google-Smtp-Source: ABdhPJwcQgwBEVmrdjdVbeLZ/5I5D9yyyGLw9MkszxLxsK5BIx7H5Snojbzb4cyLRN70qv1qZIM8drfIumagG6aiACg= X-Received: by 2002:a5e:8c16:: with SMTP id n22mr1089398ioj.156.1618459633060; Wed, 14 Apr 2021 21:07:13 -0700 (PDT) MIME-Version: 1.0 References: <20210401183216.443C4443@viggo.jf.intel.com> <20210401183219.DC1928FA@viggo.jf.intel.com> <20210414080849.GA20886@linux> In-Reply-To: <20210414080849.GA20886@linux> From: Wei Xu Date: Wed, 14 Apr 2021 21:07:01 -0700 Message-ID: Subject: Re: [PATCH 02/10] mm/numa: automatically generate node migration order To: Oscar Salvador Cc: Dave Hansen , Linux MM , Linux Kernel Mailing List , Yang Shi , David Rientjes , Huang Ying , Dan Williams , David Hildenbrand Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 1105780192D4 X-Stat-Signature: nbz8hsjm73atj8xp15jxxjb9wfjdt35y X-Rspamd-Server: rspam02 Received-SPF: none (google.com>: No applicable sender policy available) receiver=imf27; identity=mailfrom; envelope-from=""; helo=mail-io1-f45.google.com; client-ip=209.85.166.45 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1618459621-828974 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Apr 14, 2021 at 1:08 AM Oscar Salvador wrote: > > Hi Wei Xu, > > I have some questions about it > > Fast class/memory are pictured as those nodes with CPUs, while Slow class/memory > are PMEM, right? > Then, what stands for medium class/memory? That is Dave's example. I think David's guess makes sense (HBM - fast, DRAM - medium, PMEM - slow). It may also be possible that we have DDR5 as fast, CXL-DDR4 as medium, and CXL-PMEM as slow. But the most likely use cases for now should be just two tiers: DRAM vs PMEM or other types of slower memory devices. > In Dave's example, list is created in a way that stays local to the socket, > and we go from the fast one to the slow one. > In yours, lists are created taking the fastest nodes from all sockets and > we work our way down, which means have cross-socket nodes in the list. > How much of a penalty is that? Cross-socket demotion is certainly more expensive. But because it is sequential access and can also be optimized with non-temporal stores, it may not be much slower than demotion to a local node in the next tier. The actual penalty will depend on the devices. > And while I get your point, I am not sure if that is what we pretend here. > This patchset aims to place cold pages that are about to be reclaim in slower > nodes to give them a second chance, while your design seems more to have kind > of different memory clases and be able to place applications in one of those tiers > depending on its demands or sysadmin-demand. > > Could you expand some more? Sure. What I have described has the same goal as Dave's patchset, i,e, to demote cold pages to the slower nodes when they are about to be reclaimed. The only difference is that in my suggestion the demotion target of a fast tier node is expanded from a single node to a set of nodes from the slow tier and one node in such a set can be marked as the preferred/local demotion target. This can help enable more flexible demotion policies to be configured, such as to allow a cgroup to allocate from all fast tier nodes, but only demote to a local slow tier node. Such a policy can reduce memory stranding at the fast tier (compared to if memory hardwall is used) and still allow demotion from all fast tier nodes without incurring the expensive random accesses to the demoted pages if they were demoted to remote slow tier nodes. I understand that Dave started this patchset with a simplified demotion path definition, which I agree. Meanwhile, I think this more generalized definition of demotion path is useful and can also be important for some use cases.