From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ryan Roberts Subject: Re: [PATCH v1 10/10] mm: Allocate large folios for anonymous memory Date: Tue, 27 Jun 2023 10:57:46 +0100 Message-ID: <0c98f854-b4e4-9a71-8e0c-1556bc79468c@arm.com> References: <20230626171430.3167004-1-ryan.roberts@arm.com> <20230626171430.3167004-11-ryan.roberts@arm.com> Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-ID: Content-Type: text/plain; charset="windows-1252" To: Yu Zhao Cc: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists On 27/06/2023 04:01, Yu Zhao wrote: > On Mon, Jun 26, 2023 at 11:15=E2=80=AFAM Ryan Roberts wrote: >> >> With all of the enabler patches in place, modify the anonymous memory >> write allocation path so that it opportunistically attempts to allocate >> a large folio up to `max_anon_folio_order()` size (This value is >> ultimately configured by the architecture). This reduces the number of >> page faults, reduces the size of (e.g. LRU) lists, and generally >> improves performance by batching what were per-page operations into >> per-(large)-folio operations. >> >> If CONFIG_LARGE_ANON_FOLIO is not enabled (the default) then >> `max_anon_folio_order()` always returns 0, meaning we get the existing >> allocation behaviour. >> >> Signed-off-by: Ryan Roberts >> --- >> mm/memory.c | 159 +++++++++++++++++++++++++++++++++++++++++++++++----- >> 1 file changed, 144 insertions(+), 15 deletions(-) >> >> diff --git a/mm/memory.c b/mm/memory.c >> index a8f7e2b28d7a..d23c44cc5092 100644 >> --- a/mm/memory.c >> +++ b/mm/memory.c >> @@ -3161,6 +3161,90 @@ static inline int max_anon_folio_order(struct vm_= area_struct *vma) >> return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX; >> } >> >> +/* >> + * Returns index of first pte that is not none, or nr if all are none. >> + */ >> +static inline int check_ptes_none(pte_t *pte, int nr) >> +{ >> + int i; >> + >> + for (i =3D 0; i < nr; i++) { >> + if (!pte_none(ptep_get(pte++))) >> + return i; >> + } >> + >> + return nr; >> +} >> + >> +static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order) >=20 > As suggested previously in 03/10, we can leave this for later. I disagree. This is the logic that prevents us from accidentally replacing already set PTEs, or wandering out of the VMA bounds etc. How would you cat= ch all those corener cases without this? From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0259EB64DC for ; Tue, 27 Jun 2023 10:00:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231940AbjF0KAm (ORCPT ); Tue, 27 Jun 2023 06:00:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232137AbjF0KAC (ORCPT ); Tue, 27 Jun 2023 06:00:02 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 1F05330E4; Tue, 27 Jun 2023 02:57:51 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E20E92F4; Tue, 27 Jun 2023 02:58:34 -0700 (PDT) Received: from [10.1.30.74] (C02Z41KALVDN.cambridge.arm.com [10.1.30.74]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 202713F64C; Tue, 27 Jun 2023 02:57:48 -0700 (PDT) Message-ID: <0c98f854-b4e4-9a71-8e0c-1556bc79468c@arm.com> Date: Tue, 27 Jun 2023 10:57:46 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH v1 10/10] mm: Allocate large folios for anonymous memory To: Yu Zhao Cc: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-s390@vger.kernel.org References: <20230626171430.3167004-1-ryan.roberts@arm.com> <20230626171430.3167004-11-ryan.roberts@arm.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-ia64@vger.kernel.org On 27/06/2023 04:01, Yu Zhao wrote: > On Mon, Jun 26, 2023 at 11:15 AM Ryan Roberts wrote: >> >> With all of the enabler patches in place, modify the anonymous memory >> write allocation path so that it opportunistically attempts to allocate >> a large folio up to `max_anon_folio_order()` size (This value is >> ultimately configured by the architecture). This reduces the number of >> page faults, reduces the size of (e.g. LRU) lists, and generally >> improves performance by batching what were per-page operations into >> per-(large)-folio operations. >> >> If CONFIG_LARGE_ANON_FOLIO is not enabled (the default) then >> `max_anon_folio_order()` always returns 0, meaning we get the existing >> allocation behaviour. >> >> Signed-off-by: Ryan Roberts >> --- >> mm/memory.c | 159 +++++++++++++++++++++++++++++++++++++++++++++++----- >> 1 file changed, 144 insertions(+), 15 deletions(-) >> >> diff --git a/mm/memory.c b/mm/memory.c >> index a8f7e2b28d7a..d23c44cc5092 100644 >> --- a/mm/memory.c >> +++ b/mm/memory.c >> @@ -3161,6 +3161,90 @@ static inline int max_anon_folio_order(struct vm_area_struct *vma) >> return CONFIG_LARGE_ANON_FOLIO_NOTHP_ORDER_MAX; >> } >> >> +/* >> + * Returns index of first pte that is not none, or nr if all are none. >> + */ >> +static inline int check_ptes_none(pte_t *pte, int nr) >> +{ >> + int i; >> + >> + for (i = 0; i < nr; i++) { >> + if (!pte_none(ptep_get(pte++))) >> + return i; >> + } >> + >> + return nr; >> +} >> + >> +static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order) > > As suggested previously in 03/10, we can leave this for later. I disagree. This is the logic that prevents us from accidentally replacing already set PTEs, or wandering out of the VMA bounds etc. How would you catch all those corener cases without this? From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 26A4BEB64DD for ; Tue, 27 Jun 2023 09:58:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To: Subject:MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=aKGZtlly5BesL15LKJGD6JS5Jl2vafmhvttW+I/lz3A=; b=yIQGp3WvgdBxBX Kn1yuaFEAGDksekp/cFDkVTwsxee4ImJcIWsABZ7vMc7pnZTkuDcVIt/uEi1SAVwKEsc4esl4Sjvl u16erWui+hVS7UmVlhdJsIZGj6GxT9nDhGQf0NmyZVpNxFb9K7psrXbiF7xDMuO8EzT95tQu6mwQC Le9izldJyvvvqLVEbEs/NiugQSqd+bE1kmLqrGoaQIQgcC0t6bUQm/w3lQ/sBppI7bfK0dsS6J7iQ pkc6ofVn0foTovXkGfgBwyEwK0P6p6KXLLe27ObxrElyoUJrdoFELHUdc5cB8e+X0nCPrPkrANHQQ bamT9HfyV6YaFSV9ZdaA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qE5SQ-00Ckq4-0c; Tue, 27 Jun 2023 09:57:58 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qE5SM-00CkoP-2W for linux-arm-kernel@lists.infradead.org; Tue, 27 Jun 2023 09:57:56 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E20E92F4; Tue, 27 Jun 2023 02:58:34 -0700 (PDT) Received: from [10.1.30.74] (C02Z41KALVDN.cambridge.arm.com [10.1.30.74]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 202713F64C; Tue, 27 Jun 2023 02:57:48 -0700 (PDT) Message-ID: <0c98f854-b4e4-9a71-8e0c-1556bc79468c@arm.com> Date: Tue, 27 Jun 2023 10:57:46 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH v1 10/10] mm: Allocate large folios for anonymous memory To: Yu Zhao Cc: Andrew Morton , "Matthew Wilcox (Oracle)" , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-alpha@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-s390@vger.kernel.org References: <20230626171430.3167004-1-ryan.roberts@arm.com> <20230626171430.3167004-11-ryan.roberts@arm.com> From: Ryan Roberts In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230627_025754_865831_0C878EA9 X-CRM114-Status: GOOD ( 17.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gMjcvMDYvMjAyMyAwNDowMSwgWXUgWmhhbyB3cm90ZToKPiBPbiBNb24sIEp1biAyNiwgMjAy MyBhdCAxMToxNeKAr0FNIFJ5YW4gUm9iZXJ0cyA8cnlhbi5yb2JlcnRzQGFybS5jb20+IHdyb3Rl Ogo+Pgo+PiBXaXRoIGFsbCBvZiB0aGUgZW5hYmxlciBwYXRjaGVzIGluIHBsYWNlLCBtb2RpZnkg dGhlIGFub255bW91cyBtZW1vcnkKPj4gd3JpdGUgYWxsb2NhdGlvbiBwYXRoIHNvIHRoYXQgaXQg b3Bwb3J0dW5pc3RpY2FsbHkgYXR0ZW1wdHMgdG8gYWxsb2NhdGUKPj4gYSBsYXJnZSBmb2xpbyB1 cCB0byBgbWF4X2Fub25fZm9saW9fb3JkZXIoKWAgc2l6ZSAoVGhpcyB2YWx1ZSBpcwo+PiB1bHRp bWF0ZWx5IGNvbmZpZ3VyZWQgYnkgdGhlIGFyY2hpdGVjdHVyZSkuIFRoaXMgcmVkdWNlcyB0aGUg bnVtYmVyIG9mCj4+IHBhZ2UgZmF1bHRzLCByZWR1Y2VzIHRoZSBzaXplIG9mIChlLmcuIExSVSkg bGlzdHMsIGFuZCBnZW5lcmFsbHkKPj4gaW1wcm92ZXMgcGVyZm9ybWFuY2UgYnkgYmF0Y2hpbmcg d2hhdCB3ZXJlIHBlci1wYWdlIG9wZXJhdGlvbnMgaW50bwo+PiBwZXItKGxhcmdlKS1mb2xpbyBv cGVyYXRpb25zLgo+Pgo+PiBJZiBDT05GSUdfTEFSR0VfQU5PTl9GT0xJTyBpcyBub3QgZW5hYmxl ZCAodGhlIGRlZmF1bHQpIHRoZW4KPj4gYG1heF9hbm9uX2ZvbGlvX29yZGVyKClgIGFsd2F5cyBy ZXR1cm5zIDAsIG1lYW5pbmcgd2UgZ2V0IHRoZSBleGlzdGluZwo+PiBhbGxvY2F0aW9uIGJlaGF2 aW91ci4KPj4KPj4gU2lnbmVkLW9mZi1ieTogUnlhbiBSb2JlcnRzIDxyeWFuLnJvYmVydHNAYXJt LmNvbT4KPj4gLS0tCj4+ICBtbS9tZW1vcnkuYyB8IDE1OSArKysrKysrKysrKysrKysrKysrKysr KysrKysrKysrKysrKysrKysrKysrKysrKy0tLS0tCj4+ICAxIGZpbGUgY2hhbmdlZCwgMTQ0IGlu c2VydGlvbnMoKyksIDE1IGRlbGV0aW9ucygtKQo+Pgo+PiBkaWZmIC0tZ2l0IGEvbW0vbWVtb3J5 LmMgYi9tbS9tZW1vcnkuYwo+PiBpbmRleCBhOGY3ZTJiMjhkN2EuLmQyM2M0NGNjNTA5MiAxMDA2 NDQKPj4gLS0tIGEvbW0vbWVtb3J5LmMKPj4gKysrIGIvbW0vbWVtb3J5LmMKPj4gQEAgLTMxNjEs NiArMzE2MSw5MCBAQCBzdGF0aWMgaW5saW5lIGludCBtYXhfYW5vbl9mb2xpb19vcmRlcihzdHJ1 Y3Qgdm1fYXJlYV9zdHJ1Y3QgKnZtYSkKPj4gICAgICAgICAgICAgICAgIHJldHVybiBDT05GSUdf TEFSR0VfQU5PTl9GT0xJT19OT1RIUF9PUkRFUl9NQVg7Cj4+ICB9Cj4+Cj4+ICsvKgo+PiArICog UmV0dXJucyBpbmRleCBvZiBmaXJzdCBwdGUgdGhhdCBpcyBub3Qgbm9uZSwgb3IgbnIgaWYgYWxs IGFyZSBub25lLgo+PiArICovCj4+ICtzdGF0aWMgaW5saW5lIGludCBjaGVja19wdGVzX25vbmUo cHRlX3QgKnB0ZSwgaW50IG5yKQo+PiArewo+PiArICAgICAgIGludCBpOwo+PiArCj4+ICsgICAg ICAgZm9yIChpID0gMDsgaSA8IG5yOyBpKyspIHsKPj4gKyAgICAgICAgICAgICAgIGlmICghcHRl X25vbmUocHRlcF9nZXQocHRlKyspKSkKPj4gKyAgICAgICAgICAgICAgICAgICAgICAgcmV0dXJu IGk7Cj4+ICsgICAgICAgfQo+PiArCj4+ICsgICAgICAgcmV0dXJuIG5yOwo+PiArfQo+PiArCj4+ ICtzdGF0aWMgaW50IGNhbGNfYW5vbl9mb2xpb19vcmRlcl9hbGxvYyhzdHJ1Y3Qgdm1fZmF1bHQg KnZtZiwgaW50IG9yZGVyKQo+IAo+IEFzIHN1Z2dlc3RlZCBwcmV2aW91c2x5IGluIDAzLzEwLCB3 ZSBjYW4gbGVhdmUgdGhpcyBmb3IgbGF0ZXIuCgpJIGRpc2FncmVlLiBUaGlzIGlzIHRoZSBsb2dp YyB0aGF0IHByZXZlbnRzIHVzIGZyb20gYWNjaWRlbnRhbGx5IHJlcGxhY2luZwphbHJlYWR5IHNl dCBQVEVzLCBvciB3YW5kZXJpbmcgb3V0IG9mIHRoZSBWTUEgYm91bmRzIGV0Yy4gSG93IHdvdWxk IHlvdSBjYXRjaAphbGwgdGhvc2UgY29yZW5lciBjYXNlcyB3aXRob3V0IHRoaXM/CgpfX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1hcm0ta2VybmVs IG1haWxpbmcgbGlzdApsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDov L2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1hcm0ta2VybmVsCg==