From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DC95C83F15 for ; Wed, 30 Aug 2023 12:08:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DCA5544014A; Wed, 30 Aug 2023 08:08:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D5248440009; Wed, 30 Aug 2023 08:08:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF2BE44014A; Wed, 30 Aug 2023 08:08:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AF724440009 for ; Wed, 30 Aug 2023 08:08:47 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 240B31405F7 for ; Wed, 30 Aug 2023 12:07:27 +0000 (UTC) X-FDA: 81180646176.04.645FDBE Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf11.hostedemail.com (Postfix) with ESMTP id D36A540061 for ; Wed, 30 Aug 2023 12:07:21 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693397242; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3T3OXDa183BVMvRVBEyyT+ximwtwxxsQGgNINW46VrA=; b=FqPGhVO4ZmJWb22pILEIiS81iYBFrVaDIDhLT2nThckSzpVLmkMlwrQft7g7Z6hDyFKECl 8Z10AdfQcNtAilT+9AarNuwXcDC62cFruvaph7Gm1zOS4NBa469WrcJd1mv+c8WDXw9sHd aZe3nRCSvCAeZeaPaog4w33Htg+v+iQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693397242; a=rsa-sha256; cv=none; b=1gmzvdOxbV++CHhtpXMZ3FNDRC9s6d+LW+IbL7rwu9WsiMT+nJm48GvPboio2Bh89c/t2D hX9KPV/jJlCHujwtIr7vP9zR+vmk+lCDwVV/zWMr8LKsGYDadKj4ueFeHOaOeMqWzwIyfL 11kSCHrVov42VeOtLdhBlwptLnOgNV4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 41E8A2F4; Wed, 30 Aug 2023 05:08:00 -0700 (PDT) Received: from [10.57.64.173] (unknown [10.57.64.173]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 4C7A53F738; Wed, 30 Aug 2023 05:07:03 -0700 (PDT) Message-ID: <5c9ba378-2920-4892-bdf0-174e47d528b7@arm.com> Date: Wed, 30 Aug 2023 13:07:01 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v5 3/5] mm: LARGE_ANON_FOLIO for improved performance Content-Language: en-GB To: "Huang, Ying" Cc: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Anshuman Khandual , Yang Shi , Zi Yan , Luis Chamberlain , Itaru Kitayama , "Kirill A. Shutemov" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org References: <20230810142942.3169679-1-ryan.roberts@arm.com> <20230810142942.3169679-4-ryan.roberts@arm.com> <87v8dg6lfu.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Ryan Roberts In-Reply-To: <87v8dg6lfu.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: D36A540061 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 5hcrg7en19ha8cnoi47me1usc9fytmqq X-HE-Tag: 1693397241-851031 X-HE-Meta: U2FsdGVkX191iJX9NZeHKaE11BuGSoiL/7UQqyeOjw8siOUa38elbNjUmYoeE5miJ/gKeXD5BGte/zptOSYpkkjAOu+e+ApwkSSUcIJ0MhuFN/NeZ5dfB2wMugA4rmkUrxVl21d5Z340HSiANLMgZKDvZYWTPc66N+A6bZjSVTxO+JMPN77ey5LZIAa2DFSM5ZMhF2VC2VTQqKX3jFVEnFJtHWpvcfAzFe/UYOzFMRbzKr5KckbzutkZXr67ZaJKvke1syv0pXoCN4ibaiQntBPMm3MHDqtq5AxpHbolqkoLHZ8XOQf1Lqw+jxIuwR5XjDAYEXgYpAEIr+WPqUaL35cC9s8LJoQF50WBO8dYjLI4h5ZJvOauAop82jL4fUIGddMjzJposDsL+YpzjroPsHKM7yK79pSzXOC6SOUsrhh0OGqrmMTx5w9PY7S7/VJu9oQIauAFCbTXsrng5RM9NoRBrtAhiopssqM2qqM7VAygXadHBqcYFvk8Xrl5npeFcK+B/E7OKjWMBs69r7808ePuPumweOl2mJolwp7olc4FjgD3sokWq41ZRbkOXemdpdkCFcFhVrEuLwjpZyUSg2O4DlCVdnpb6J+u305ORPyDvZfx8PNH/mhCwayctFxmQH99wcUxLwKvUeOU511Kj0+5tNHxD6X94N9I/qTSkA1Z0OuI/H+vksrw2yNGkypxDlnezVEnyOAbpuwNcvSxOMHZcACqyYIGqjsBDC6N7LycQpdQD1GsyLD14CERKLz7o7QRs4MnLdVGwuszR9br+3wX3WCa7GTDzyDhLKHAn/mnxeSS0Vf4Hf/QoNEz3NDMNR8QmFAjDKVdZ637Mv1nUnEA4nKT+AMgU0khk9X+n737ScxNCAEUspgeKmTA52LyWSylw+cIS35PWqta0g/BUmK+0nk++3ZWxq7ntAyn4AkvyIJr442Z2rqVA7m4EEj526bUeQCKB/fr39UCewJ iIAJwt1o Egg795OG7eKfbW8jWqb+0BqMTQB3oMJdJQGcSy4+4wIvhWzHSbM6rFP99GonIKHshVNfeIt69MfPuLf7yUPqShKZ+SwHwuKL8hFbrwGEzP/kXMZkWKk5a5SrnDEUwNKa0VhKUUnmpz8sMMmRlGDk7T8+94Z2Ot5xgtuM/kRfZlbHJeUKI6Hoi26x5tadezHwdNxbIxBDsbd4Wef3y1sLI8WmvtokxIho28Qnyn318F4u+/rVYZRs9FEmEBWzsbyrQD9UkZOBtN8N1TrRkbuBqO/1R5i7O1ICw6+ztPySgvQMFL7ApKKmrbfLRoOdgX+Gfnw5OhRwj9RtptCM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 15/08/2023 22:32, Huang, Ying wrote: > Hi, Ryan, > > Ryan Roberts writes: > >> Introduce LARGE_ANON_FOLIO feature, which allows anonymous memory to be >> allocated in large folios of a determined order. All pages of the large >> folio are pte-mapped during the same page fault, significantly reducing >> the number of page faults. The number of per-page operations (e.g. ref >> counting, rmap management lru list management) are also significantly >> reduced since those ops now become per-folio. >> >> The new behaviour is hidden behind the new LARGE_ANON_FOLIO Kconfig, >> which defaults to disabled for now; The long term aim is for this to >> defaut to enabled, but there are some risks around internal >> fragmentation that need to be better understood first. >> >> Large anonymous folio (LAF) allocation is integrated with the existing >> (PMD-order) THP and single (S) page allocation according to this policy, >> where fallback (>) is performed for various reasons, such as the >> proposed folio order not fitting within the bounds of the VMA, etc: >> >> | prctl=dis | prctl=ena | prctl=ena | prctl=ena >> | sysfs=X | sysfs=never | sysfs=madvise | sysfs=always >> ----------------|-----------|-------------|---------------|------------- >> no hint | S | LAF>S | LAF>S | THP>LAF>S >> MADV_HUGEPAGE | S | LAF>S | THP>LAF>S | THP>LAF>S >> MADV_NOHUGEPAGE | S | S | S | S > > IMHO, we should use the following semantics as you have suggested > before. > > | prctl=dis | prctl=ena | prctl=ena | prctl=ena > | sysfs=X | sysfs=never | sysfs=madvise | sysfs=always > ----------------|-----------|-------------|---------------|------------- > no hint | S | S | LAF>S | THP>LAF>S > MADV_HUGEPAGE | S | S | THP>LAF>S | THP>LAF>S > MADV_NOHUGEPAGE | S | S | S | S > > Or even, > > | prctl=dis | prctl=ena | prctl=ena | prctl=ena > | sysfs=X | sysfs=never | sysfs=madvise | sysfs=always > ----------------|-----------|-------------|---------------|------------- > no hint | S | S | S | THP>LAF>S > MADV_HUGEPAGE | S | S | THP>LAF>S | THP>LAF>S > MADV_NOHUGEPAGE | S | S | S | S > > From the implementation point of view, PTE mapped PMD-sized THP has > almost no difference with LAF (just some small sized THP). It will be > confusing to distinguish them from the interface point of view. > > So, IMHO, the real difference is the policy. For example, prefer > PMD-sized THP, prefer small sized THP, or fully auto. The sysfs > interface is used to specify system global policy. In the long term, it > can be something like below, > > never: S # disable all THP > madvise: # never by default, control via madvise() > always: THP>LAF>S # prefer PMD-sized THP in fact > small: LAF>S # prefer small sized THP > auto: # use in-kernel heuristics for THP size > > But it may be not ready to add new policies now. So, before the new > policies are ready, we can add a debugfs interface to override the > original policy in /sys/kernel/mm/transparent_hugepage/enabled. After > we have tuned enough workloads, collected enough data, we can add new > policies to the sysfs interface. I think we can all imagine many policy options. But we don't really have much evidence yet for what it best. The policy I'm currently using is intended to give some flexibility for testing (use LAF without THP by setting sysfs=never, use THP without LAF by compiling without LAF) without adding any new knobs at all. Given that, surely we can defer these decisions until we have more data? In the absence of data, your proposed solution sounds very sensible to me. But for the purposes of scaling up perf testing, I don't think its essential given the current policy will also produce the same options. If we were going to add a debugfs knob, I think the higher priority would be a knob to specify the folio order. (but again, I would rather avoid if possible). Thanks, Ryan