From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 452C0E94115 for ; Fri, 6 Oct 2023 20:15:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233365AbjJFUPI (ORCPT ); Fri, 6 Oct 2023 16:15:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233167AbjJFUPG (ORCPT ); Fri, 6 Oct 2023 16:15:06 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EFAB2BD for ; Fri, 6 Oct 2023 13:15:04 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 28876C433C7; Fri, 6 Oct 2023 20:15:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1696623304; bh=MugWodMC+3nGqUIVKFN7TLIMgO8tPJVykLvL7/Vx+8o=; h=Date:To:From:Subject:From; b=nw+YlbBfO6tEmPPum4m8FWbcb+DMiNyXwYmaAkLhaHzCZPwN1vAt+zwUvkxpomB6b TMxyCJVzjhnn8Br+eBrGbNCE5Q9vPKL8LSHjwlo0aDkhM+W4Ffv9+hd/RVsga7x/4f K1zFovTy/ihQp5hH/XzkIYtNZOrcwi2FniNPUBzo= Date: Fri, 06 Oct 2023 13:15:01 -0700 To: mm-commits@vger.kernel.org, surenb@google.com, willy@infradead.org, akpm@linux-foundation.org From: Andrew Morton Subject: + mm-call-wp_page_copy-under-the-vma-lock.patch added to mm-unstable branch Message-Id: <20231006201504.28876C433C7@smtp.kernel.org> Precedence: bulk Reply-To: linux-kernel@vger.kernel.org List-ID: X-Mailing-List: mm-commits@vger.kernel.org The patch titled Subject: mm: call wp_page_copy() under the VMA lock has been added to the -mm mm-unstable branch. Its filename is mm-call-wp_page_copy-under-the-vma-lock.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-call-wp_page_copy-under-the-vma-lock.patch This patch will later appear in the mm-unstable branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via the mm-everything branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there every 2-3 working days ------------------------------------------------------ From: "Matthew Wilcox (Oracle)" Subject: mm: call wp_page_copy() under the VMA lock Date: Fri, 6 Oct 2023 20:53:14 +0100 It is usually safe to call wp_page_copy() under the VMA lock. The only unsafe situation is when no anon_vma has been allocated for this VMA, and we have to look at adjacent VMAs to determine if their anon_vma can be shared. Since this happens only for the first COW of a page in this VMA, the majority of calls to wp_page_copy() do not need to fall back to the mmap_sem. Add vmf_anon_prepare() as an alternative to anon_vma_prepare() which will return RETRY if we currently hold the VMA lock and need to allocate an anon_vma. This lets us drop the check in do_wp_page(). Link: https://lkml.kernel.org/r/20231006195318.4087158-3-willy@infradead.org Signed-off-by: Matthew Wilcox (Oracle) Cc: Suren Baghdasaryan Signed-off-by: Andrew Morton --- mm/memory.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-) --- a/mm/memory.c~mm-call-wp_page_copy-under-the-vma-lock +++ a/mm/memory.c @@ -3045,6 +3045,21 @@ static inline void wp_page_reuse(struct count_vm_event(PGREUSE); } +static vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + + if (likely(vma->anon_vma)) + return 0; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + if (__anon_vma_prepare(vma)) + return VM_FAULT_OOM; + return 0; +} + /* * Handle the case of a page which we actually need to copy to a new page, * either due to COW or unsharing. @@ -3072,27 +3087,29 @@ static vm_fault_t wp_page_copy(struct vm pte_t entry; int page_copied = 0; struct mmu_notifier_range range; - int ret; + vm_fault_t ret; delayacct_wpcopy_start(); if (vmf->page) old_folio = page_folio(vmf->page); - if (unlikely(anon_vma_prepare(vma))) - goto oom; + ret = vmf_anon_prepare(vmf); + if (unlikely(ret)) + goto out; if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); if (!new_folio) goto oom; } else { + int err; new_folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false); if (!new_folio) goto oom; - ret = __wp_page_copy_user(&new_folio->page, vmf->page, vmf); - if (ret) { + err = __wp_page_copy_user(&new_folio->page, vmf->page, vmf); + if (err) { /* * COW failed, if the fault was solved by other, * it's fine. If not, userspace would re-fault on @@ -3105,7 +3122,7 @@ static vm_fault_t wp_page_copy(struct vm folio_put(old_folio); delayacct_wpcopy_end(); - return ret == -EHWPOISON ? VM_FAULT_HWPOISON : 0; + return err == -EHWPOISON ? VM_FAULT_HWPOISON : 0; } kmsan_copy_page_meta(&new_folio->page, vmf->page); } @@ -3215,11 +3232,13 @@ static vm_fault_t wp_page_copy(struct vm oom_free_new: folio_put(new_folio); oom: + ret = VM_FAULT_OOM; +out: if (old_folio) folio_put(old_folio); delayacct_wpcopy_end(); - return VM_FAULT_OOM; + return ret; } /** @@ -3461,12 +3480,6 @@ reuse: return 0; } copy: - if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma->anon_vma) { - pte_unmap_unlock(vmf->pte, vmf->ptl); - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } - /* * Ok, we need to copy. Oh, well.. */ _ Patches currently in -mm which might be from willy@infradead.org are mm-make-lock_folio_maybe_drop_mmap-vma-lock-aware.patch mm-call-wp_page_copy-under-the-vma-lock.patch mm-handle-shared-faults-under-the-vma-lock.patch mm-handle-cow-faults-under-the-vma-lock.patch mm-handle-read-faults-under-the-vma-lock.patch mm-handle-write-faults-to-ro-pages-under-the-vma-lock.patch iomap-hold-state_lock-over-call-to-ifs_set_range_uptodate.patch iomap-protect-read_bytes_pending-with-the-state_lock.patch mm-add-folio_end_read.patch ext4-use-folio_end_read.patch buffer-use-folio_end_read.patch iomap-use-folio_end_read.patch bitops-add-xor_unlock_is_negative_byte.patch alpha-implement-xor_unlock_is_negative_byte.patch m68k-implement-xor_unlock_is_negative_byte.patch mips-implement-xor_unlock_is_negative_byte.patch powerpc-implement-arch_xor_unlock_is_negative_byte-on-32-bit.patch riscv-implement-xor_unlock_is_negative_byte.patch s390-implement-arch_xor_unlock_is_negative_byte.patch mm-delete-checks-for-xor_unlock_is_negative_byte.patch mm-add-folio_xor_flags_has_waiters.patch mm-make-__end_folio_writeback-return-void.patch mm-use-folio_xor_flags_has_waiters-in-folio_end_writeback.patch