From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C4B3C54EBE for ; Mon, 16 Jan 2023 19:18:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E39776B0074; Mon, 16 Jan 2023 14:18:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DEB506B0075; Mon, 16 Jan 2023 14:18:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8AA66B0078; Mon, 16 Jan 2023 14:18:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B6E6B6B0074 for ; Mon, 16 Jan 2023 14:18:17 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5710114014B for ; Mon, 16 Jan 2023 19:18:17 +0000 (UTC) X-FDA: 80361623034.17.85BEC92 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id D38BF100009 for ; Mon, 16 Jan 2023 19:18:15 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=fFm5rp76; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673896696; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MmiA8IG8NeTUdNNUNyNzEP348XkHSIXqlo8iyVbxHxw=; b=6simfU1499/DSlzA5OlQlnkfTToYhoynKkmetM1HoNmRHmFYbx7iL6iVB9bX7X/e6e10Vs aLLYEG8DfHUHHPyLIRzomMXF25Foe/qjsYnZuOXnS2sjKcM1VbGLJ2lo6lVdxNUh7frbXL Ct4+MXpAiv2KpUv+VWYQmUb+eSwQFm0= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=fFm5rp76; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673896696; a=rsa-sha256; cv=none; b=zNlaAOP/2oGXKnMJQAf4pmhme17bHUrS04WGQYrJ+mdD5JjcS6m47tDW/TNzTeeuTL4rCE KTy4C9ixMLZJ4BxBMmoe8lJIlbv6n1lJ8uAzi65fppx4bE6QB3GsldR0lSKSVhmTslIonc 3A8u7NNWH8gt2/lPfuzgH1qRs1PCMe0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=MmiA8IG8NeTUdNNUNyNzEP348XkHSIXqlo8iyVbxHxw=; b=fFm5rp76SWD5TTev7viPiHiRmF /cns+0QnF/1Hs51ZFg4ujGwVcKzgupbnrYNM0digkGBhbgF2DZi7XwOtre9+6zzmbvYHLueEj5iyP zQ5A86RK0egszDqIrqhNAW7h2NWjKPBbmQBxXh9R6WUQNIA4q0Pn+unh+YmQRj5kaImrZy1myvy6a QuhySlBfs3IzWMWQaHGyp8gDSuZbGTxP//fm5GspAgl096zJ45ZQTmhyXl+/1NaXQjIis/YAT2cTO rQrLiZq+RsW6aTEZFAh3Hv6V/oE6ypN3EMx5zhc6sa7RsR2Bflk9nSUE3nQhnsL52o23owxOh1/Ti mC/PbFZA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pHUzm-00904l-Rc; Mon, 16 Jan 2023 19:18:14 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org, Andrew Morton Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH 3/5] mm: Convert wp_page_copy() to use folios Date: Mon, 16 Jan 2023 19:18:11 +0000 Message-Id: <20230116191813.2145215-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230116191813.2145215-1-willy@infradead.org> References: <20230116191813.2145215-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: D38BF100009 X-Stat-Signature: nudurdo43hsoppi6oucjo7h5ex7mauu5 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1673896695-55684 X-HE-Meta: U2FsdGVkX19QI4xrGw34ANhaTaaShMfr+D5q41QW+oD2eOr+I5YcPSSI2kEXYWVpdSpE15kd0pth0HO0+DzY87fGeqnzakbu55vUI1ZzCfNb5OJFC8yCljMgacMYpOtD/vaiCVYskwKyQgwSlCRZBj6VlNVf7AoTq09vt/JS45tFP1YLXT8gt59rMWzCIdxO86MUSDhWrTnojHs5sOwf5E4efHrjoZUOm6hX//MkEcI6y4GIfI+ReL8Ci66SnbGOCuAgnjlxWakFu5wAVC64uJ4gw1IXHU9bqNibFbOtqeghq3LPDQpT4yvQuOi+ImCLMRGE2eHORYLJA2mwv6sHdaOqH9JhSrZn3AMKq+/JRomHRADcQIpR7blVjUikggylm5IKSMm5+Z7RSHSvTewWu9yv1zYRZDAyLHRV3/tl9lD/O36CpNOrZIK8XC0zA1y3G/lWj6sFE5TVP0Skmng6sy89ssX4oPJaPn92ZDnCHxJPQ4TMyPizxlE9xjls7KK1MvWMggJVkMh4dz5iYPpngnSr+fquHkdUzK7IZCX1yTCKtT0YJ4+x7an9gjJoph7tmpmI7jaGmO5/uXHzOSiuONsg8hnwrCr1QORG81x2mABGqqzJtSvR3vPYo8Xu8y1sgVFH/+Lln4qhr2u6+2hWDRaaVqfFM9ynJNDCs4rS3qJdtWCrVgU7G2mOdM2T4angvOHvF+6CUMo1Z/qVVD2qYnyJppCO4LU0tP3FGwN4ziYoZ/E6ctLTpkwMCUeUrFS427PH0rPSbaleTzuLCzwNnhfy5aryco++xnmqDNyXf94g1w/piWfD87E/0GUOkKhDMbSA+yu/tANBTA1KJaZOpFFQ1X8mJMjineI3c1XsGCKS2PU/2dxYF/uXRelAC+CkWxs6WEaRwU2n0nwm+biqMwDXs796NGJbkwP5CzevlEYa3RcRFLH1FQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use new_folio instead of new_page throughout, because we allocated it and know it's an order-0 folio. Most old_page uses become old_folio, but use vmf->page where we need the precise page. --- mm/memory.c | 65 ++++++++++++++++++++++++++--------------------------- 1 file changed, 32 insertions(+), 33 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6099c786b5f9..dc8a6fd45958 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3043,8 +3043,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; struct vm_area_struct *vma = vmf->vma; struct mm_struct *mm = vma->vm_mm; - struct page *old_page = vmf->page; - struct page *new_page = NULL; + struct folio *old_folio = NULL; + struct folio *new_folio = NULL; pte_t entry; int page_copied = 0; struct mmu_notifier_range range; @@ -3052,23 +3052,22 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) delayacct_wpcopy_start(); + if (vmf->page) + old_folio = page_folio(vmf->page); if (unlikely(anon_vma_prepare(vma))) goto oom; if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { - struct folio *new_folio; - new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); if (!new_folio) goto oom; - new_page = &new_folio->page; } else { - new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, - vmf->address); - if (!new_page) + new_folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, + vmf->address, false); + if (!new_folio) goto oom; - ret = __wp_page_copy_user(new_page, old_page, vmf); + ret = __wp_page_copy_user(&new_folio->page, vmf->page, vmf); if (ret) { /* * COW failed, if the fault was solved by other, @@ -3077,21 +3076,21 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * from the second attempt. * The -EHWPOISON case will not be retried. */ - put_page(new_page); - if (old_page) - put_page(old_page); + folio_put(new_folio); + if (old_folio) + folio_put(old_folio); delayacct_wpcopy_end(); return ret == -EHWPOISON ? VM_FAULT_HWPOISON : 0; } - kmsan_copy_page_meta(new_page, old_page); + kmsan_copy_page_meta(&new_folio->page, vmf->page); } - if (mem_cgroup_charge(page_folio(new_page), mm, GFP_KERNEL)) + if (mem_cgroup_charge(new_folio, mm, GFP_KERNEL)) goto oom_free_new; - cgroup_throttle_swaprate(new_page, GFP_KERNEL); + cgroup_throttle_swaprate(&new_folio->page, GFP_KERNEL); - __SetPageUptodate(new_page); + __folio_mark_uptodate(new_folio); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, vmf->address & PAGE_MASK, @@ -3103,16 +3102,16 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) */ vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); if (likely(pte_same(*vmf->pte, vmf->orig_pte))) { - if (old_page) { - if (!PageAnon(old_page)) { - dec_mm_counter(mm, mm_counter_file(old_page)); + if (old_folio) { + if (!folio_test_anon(old_folio)) { + dec_mm_counter(mm, mm_counter_file(&old_folio->page)); inc_mm_counter(mm, MM_ANONPAGES); } } else { inc_mm_counter(mm, MM_ANONPAGES); } flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); - entry = mk_pte(new_page, vma->vm_page_prot); + entry = mk_pte(&new_folio->page, vma->vm_page_prot); entry = pte_sw_mkyoung(entry); if (unlikely(unshare)) { if (pte_soft_dirty(vmf->orig_pte)) @@ -3131,8 +3130,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * some TLBs while the old PTE remains in others. */ ptep_clear_flush_notify(vma, vmf->address, vmf->pte); - page_add_new_anon_rmap(new_page, vma, vmf->address); - lru_cache_add_inactive_or_unevictable(new_page, vma); + folio_add_new_anon_rmap(new_folio, vma, vmf->address); + folio_add_lru_vma(new_folio, vma); /* * We call the notify macro here because, when using secondary * mmu page tables (such as kvm shadow page tables), we want the @@ -3141,7 +3140,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) BUG_ON(unshare && pte_write(entry)); set_pte_at_notify(mm, vmf->address, vmf->pte, entry); update_mmu_cache(vma, vmf->address, vmf->pte); - if (old_page) { + if (old_folio) { /* * Only after switching the pte to the new page may * we remove the mapcount here. Otherwise another @@ -3164,18 +3163,18 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * mapcount is visible. So transitively, TLBs to * old page will be flushed before it can be reused. */ - page_remove_rmap(old_page, vma, false); + page_remove_rmap(vmf->page, vma, false); } /* Free the old page.. */ - new_page = old_page; + new_folio = old_folio; page_copied = 1; } else { update_mmu_tlb(vma, vmf->address, vmf->pte); } - if (new_page) - put_page(new_page); + if (new_folio) + folio_put(new_folio); pte_unmap_unlock(vmf->pte, vmf->ptl); /* @@ -3183,19 +3182,19 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * the above ptep_clear_flush_notify() did already call it. */ mmu_notifier_invalidate_range_only_end(&range); - if (old_page) { + if (old_folio) { if (page_copied) - free_swap_cache(old_page); - put_page(old_page); + free_swap_cache(&old_folio->page); + folio_put(old_folio); } delayacct_wpcopy_end(); return 0; oom_free_new: - put_page(new_page); + folio_put(new_folio); oom: - if (old_page) - put_page(old_page); + if (old_folio) + folio_put(old_folio); delayacct_wpcopy_end(); return VM_FAULT_OOM; -- 2.35.1