From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54465C61D9C for ; Wed, 22 Nov 2023 09:22:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C754B6B0582; Wed, 22 Nov 2023 04:22:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C25246B0583; Wed, 22 Nov 2023 04:22:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AEC796B0584; Wed, 22 Nov 2023 04:22:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9FB3B6B0582 for ; Wed, 22 Nov 2023 04:22:49 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 70DE31A0926 for ; Wed, 22 Nov 2023 09:22:49 +0000 (UTC) X-FDA: 81485050458.11.7E6EB60 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id B39F8100009 for ; Wed, 22 Nov 2023 09:22:46 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700644967; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y+s+SaX/slE1rs9PVHeOu+WZBD5yj43We3O32RnMRtw=; b=H4wI64vcvIdNE3McFZ2GvKFqiqB6O9HrAck4s2ItPdmmCaR02TRz0enTLTkqvxMLJ8W9k9 4GfBNwry6+4OLU/Y3dBjD1k3m3lQCaMxkmJNXF/peUGhKB90dpe96mZ9XPcGZ/mxfQgbZI V5hezeU2dMpbdM/0jzU57v4/sfucu1A= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700644967; a=rsa-sha256; cv=none; b=cwKd92FJHq33Y3YwClNdguDWkuSQOVewRPUVoIcqBD2hUAT+GoguQQeC7bDvo8Tc37z1OJ uIK/p2+/9IirDviCedKw3/+h5PgD9DS1R/wvipj+ZUvznmIhQf/Ki05JZ/ehiyW6WHXJY+ zczaG8FE8RJDNxncKzBBLWDYIR3/oSk= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 42BE01595; Wed, 22 Nov 2023 01:23:32 -0800 (PST) Received: from arm.com (e121798.cambridge.arm.com [10.1.197.44]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3C2073F73F; Wed, 22 Nov 2023 01:22:41 -0800 (PST) Date: Wed, 22 Nov 2023 09:22:38 +0000 From: Alexandru Elisei To: Peter Collingbourne Cc: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: Re: [PATCH RFC v2 20/27] mm: hugepage: Handle huge page fault on access Message-ID: References: <20231119165721.9849-1-alexandru.elisei@arm.com> <20231119165721.9849-21-alexandru.elisei@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: B39F8100009 X-Stat-Signature: rcduy1pfhbkijxxusrhqugprighd6mck X-HE-Tag: 1700644966-787866 X-HE-Meta: U2FsdGVkX18zA0Hvk7BjIeRL0rbaDBMGVMersEp55L6oX/wLHURlWXUdF5OBqxe4V65PIx3r5XUVxG+a2naJT9qAv4JZ26Z8ud+712tLUOkfQ/P+kooW4y5IOjI2tsy355QKPYhjm6JRI/I+Rzic3f39RCWDXZJKESf6dohd3vaEAmclT5V6HHKhrNRKm9W3xsFD+rws+eTGrZguOPyRKeQBPEwAhk+wLoye5LeUzMaJnkyX/u0uby1pFqmoCDThiddLaHXT9Xyjk+IBm6KAQdst+x8Nx9JW2k3sjDM/wYXrus99Y+oouICJKuJsnH/Nyxdw/MKQBQbL2+x7xJ3W+I8En3XtRZMU5vSZ+goopGaN5c8tw/HOB5PuRn4OFqO73dsQRcZ7cmLOzx9WdjMR5S4x97Nwr8v0ZrxAeyqdoyZHttbfTtpooYGWIE8JnHFZUj4GmtE4tOxHpKly8zkg5tdnpBYeO0fiBNsGyoNZOF5ZjvzoS775fYhz8xh6zm5NS05mptgLjCaSbXiBGPHXO8u0KJFsB9iRidvGe6YFb9Ulp4vhJdpUo3qin25Xb2hdtuqAl4WwR5Jds4hLm7e6D2Yu+KkvliDkrzZHdluDSNwEKwJ8SwPqOBG5m8PSHvUYU5NivskvKzcN9NzELjqXnEYCuy8Cq9at/BV6+CRi5LVycSCEzHbfeNcRd8xORCyqhyc9XA/1ZBTm7ZXwA1k4UkDSAb7yqQillITDIlDHIu/L0VJu1ihHSJmaB8qgeLwmO42WxgbwPBlo4h58E0vosefngQ6yinjzGxNWIYLQbAAALwLfGxe2rdvlCbp2qxUs5DS/MEObPIMUs7INWhOOqpH9/AMAloBWe4cUxsNjnFNQulaZqRObPTdsRVybG/g8zpKKsn6zC1BT89vM+mkyCsClXbg78cEFaU/O3PXX62oNwmR3rG9TVQ4HHptuOrJQy9RAmNUrVv0C6775ojv ddOxDXsv ui6dQBAMYW1BGtE+oTietn7QdfKYlOVeqJ5CNqh10RlTk24+GKTVB+hxAunn3QR5ggRnV+/xXxmwttC19fXecwIvOEsNLNwQjcmOk0gTrKy8O0oco1CeswvHlcyB+9esTD4ogE6izIyX3Al6PgcrR880kXlxld6Wzj11avfmzNCJmzj0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Peter, On Tue, Nov 21, 2023 at 05:28:49PM -0800, Peter Collingbourne wrote: > On Sun, Nov 19, 2023 at 8:59 AM Alexandru Elisei > wrote: > > > > Handle PAGE_FAULT_ON_ACCESS faults for huge pages in a similar way to > > regular pages. > > > > Signed-off-by: Alexandru Elisei > > --- > > arch/arm64/include/asm/mte_tag_storage.h | 1 + > > arch/arm64/include/asm/pgtable.h | 7 ++ > > arch/arm64/mm/fault.c | 81 ++++++++++++++++++++++++ > > include/linux/huge_mm.h | 2 + > > include/linux/pgtable.h | 5 ++ > > mm/huge_memory.c | 4 +- > > mm/memory.c | 3 + > > 7 files changed, 101 insertions(+), 2 deletions(-) > > > > diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h > > index c70ced60a0cd..b97406d369ce 100644 > > --- a/arch/arm64/include/asm/mte_tag_storage.h > > +++ b/arch/arm64/include/asm/mte_tag_storage.h > > @@ -35,6 +35,7 @@ void free_tag_storage(struct page *page, int order); > > bool page_tag_storage_reserved(struct page *page); > > > > vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf); > > +vm_fault_t handle_huge_page_missing_tag_storage(struct vm_fault *vmf); > > #else > > static inline bool tag_storage_enabled(void) > > { > > diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h > > index 8cc135f1c112..1704411c096d 100644 > > --- a/arch/arm64/include/asm/pgtable.h > > +++ b/arch/arm64/include/asm/pgtable.h > > @@ -477,6 +477,13 @@ static inline vm_fault_t arch_do_page_fault_on_access(struct vm_fault *vmf) > > return handle_page_missing_tag_storage(vmf); > > return VM_FAULT_SIGBUS; > > } > > + > > +static inline vm_fault_t arch_do_huge_page_fault_on_access(struct vm_fault *vmf) > > +{ > > + if (tag_storage_enabled()) > > + return handle_huge_page_missing_tag_storage(vmf); > > + return VM_FAULT_SIGBUS; > > +} > > #endif /* CONFIG_ARCH_HAS_FAULT_ON_ACCESS */ > > > > #define pmd_present_invalid(pmd) (!!(pmd_val(pmd) & PMD_PRESENT_INVALID)) > > diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c > > index f5fa583acf18..6730a0812a24 100644 > > --- a/arch/arm64/mm/fault.c > > +++ b/arch/arm64/mm/fault.c > > @@ -1041,6 +1041,87 @@ vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf) > > > > return 0; > > > > +out_retry: > > + put_page(page); > > + if (vmf->flags & FAULT_FLAG_VMA_LOCK) > > + vma_end_read(vma); > > + if (fault_flag_allow_retry_first(vmf->flags)) { > > + err = VM_FAULT_RETRY; > > + } else { > > + /* Replay the fault. */ > > + err = 0; > > + } > > + return err; > > +} > > + > > +vm_fault_t handle_huge_page_missing_tag_storage(struct vm_fault *vmf) > > +{ > > + unsigned long haddr = vmf->address & HPAGE_PMD_MASK; > > + struct vm_area_struct *vma = vmf->vma; > > + pmd_t old_pmd, new_pmd; > > + bool writable = false; > > + struct page *page; > > + vm_fault_t err; > > + int ret; > > + > > + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); > > + if (unlikely(!pmd_same(vmf->orig_pmd, *vmf->pmd))) { > > + spin_unlock(vmf->ptl); > > + return 0; > > + } > > + > > + old_pmd = vmf->orig_pmd; > > + new_pmd = pmd_modify(old_pmd, vma->vm_page_prot); > > + > > + /* > > + * Detect now whether the PMD could be writable; this information > > + * is only valid while holding the PT lock. > > + */ > > + writable = pmd_write(new_pmd); > > + if (!writable && vma_wants_manual_pte_write_upgrade(vma) && > > + can_change_pmd_writable(vma, vmf->address, new_pmd)) > > + writable = true; > > + > > + page = vm_normal_page_pmd(vma, haddr, new_pmd); > > + if (!page) > > + goto out_map; > > + > > + if (!(vma->vm_flags & VM_MTE)) > > + goto out_map; > > + > > + get_page(page); > > + vma_set_access_pid_bit(vma); > > + > > + spin_unlock(vmf->ptl); > > + writable = false; > > + > > + if (unlikely(is_migrate_isolate_page(page))) > > + goto out_retry; > > + > > + ret = reserve_tag_storage(page, HPAGE_PMD_ORDER, GFP_HIGHUSER_MOVABLE); > > + if (ret) > > + goto out_retry; > > + > > + put_page(page); > > + > > + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); > > + if (unlikely(!pmd_same(old_pmd, *vmf->pmd))) { > > + spin_unlock(vmf->ptl); > > + return 0; > > + } > > + > > +out_map: > > + /* Restore the PMD */ > > + new_pmd = pmd_modify(old_pmd, vma->vm_page_prot); > > + new_pmd = pmd_mkyoung(new_pmd); > > + if (writable) > > + new_pmd = pmd_mkwrite(new_pmd, vma); > > + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, new_pmd); > > + update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); > > + spin_unlock(vmf->ptl); > > + > > + return 0; > > + > > out_retry: > > put_page(page); > > if (vmf->flags & FAULT_FLAG_VMA_LOCK) > > diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h > > index fa0350b0812a..bb84291f9231 100644 > > --- a/include/linux/huge_mm.h > > +++ b/include/linux/huge_mm.h > > @@ -36,6 +36,8 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, > > int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, > > pmd_t *pmd, unsigned long addr, pgprot_t newprot, > > unsigned long cp_flags); > > +bool can_change_pmd_writable(struct vm_area_struct *vma, unsigned long addr, > > + pmd_t pmd); > > > > vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write); > > vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write); > > diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h > > index e2c761dd6c41..de45f475bf8d 100644 > > --- a/include/linux/pgtable.h > > +++ b/include/linux/pgtable.h > > @@ -1473,6 +1473,11 @@ static inline vm_fault_t arch_do_page_fault_on_access(struct vm_fault *vmf) > > { > > return VM_FAULT_SIGBUS; > > } > > + > > +static inline vm_fault_t arch_do_huge_page_fault_on_access(struct vm_fault *vmf) > > +{ > > + return VM_FAULT_SIGBUS; > > +} > > #endif > > > > #endif /* CONFIG_MMU */ > > diff --git a/mm/huge_memory.c b/mm/huge_memory.c > > index 9beead961a65..d1402b43ea39 100644 > > --- a/mm/huge_memory.c > > +++ b/mm/huge_memory.c > > @@ -1406,8 +1406,8 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) > > return VM_FAULT_FALLBACK; > > } > > > > -static inline bool can_change_pmd_writable(struct vm_area_struct *vma, > > - unsigned long addr, pmd_t pmd) > > +inline bool can_change_pmd_writable(struct vm_area_struct *vma, > > Remove inline keyword here. Indeed, as it does nothing now that the function is not static. Thanks, Alex > > Peter > > > + unsigned long addr, pmd_t pmd) > > { > > struct page *page; > > > > diff --git a/mm/memory.c b/mm/memory.c > > index a04a971200b9..46b926625503 100644 > > --- a/mm/memory.c > > +++ b/mm/memory.c > > @@ -5168,6 +5168,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, > > return 0; > > } > > if (pmd_trans_huge(vmf.orig_pmd) || pmd_devmap(vmf.orig_pmd)) { > > + if (fault_on_access_pmd(vmf.orig_pmd) && vma_is_accessible(vma)) > > + return arch_do_huge_page_fault_on_access(&vmf); > > + > > if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma)) > > return do_huge_pmd_numa_page(&vmf); > > > > -- > > 2.42.1 > >