From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76F58C0219B for ; Tue, 11 Feb 2025 11:15:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2DD56B0095; Tue, 11 Feb 2025 06:15:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EDD336B0096; Tue, 11 Feb 2025 06:15:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA4146B0098; Tue, 11 Feb 2025 06:15:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BDBE56B0095 for ; Tue, 11 Feb 2025 06:15:07 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7F5221A15DE for ; Tue, 11 Feb 2025 11:15:07 +0000 (UTC) X-FDA: 83107407054.13.0903CCF Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id B19EE40006 for ; Tue, 11 Feb 2025 11:15:05 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739272505; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YE5yRynr1Dp9nNko1h3HDsMNEVJMytTso3JYAphAGZQ=; b=svZIPqfKFiGjh7yA4m4OhJPW96Sb7tfTsLVos3F5G7wjkkbcMh1TDkYH1+aOGdHXOTPcDh 8ucFENzUBmhsMmmBasbxDcR0geLErE06BYYZbWLtVDhKY42m/KSIjh/RxGAhiBnUWalST0 rXTqA7hk8v/OYKLG+2iB14iukoxgvf0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739272505; a=rsa-sha256; cv=none; b=6JoOvDtp830ZAnJr9b1NWZ0OUmvLQsMMqMdKJx+M66V9uciWLdvgDHgwqS/QSxzReTv0VZ 2ul8u3q2GovBIhbF1FBUDi9VZpqd+D7wtmtHHPerQ+cPTlxxGeSETPmj+jF2CB4m8HzmPP EIZ6UuSF6nfaMvHt4OvUqXNQkRcN0YE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7967113D5; Tue, 11 Feb 2025 03:15:26 -0800 (PST) Received: from K4MQJ0H1H2.emea.arm.com (K4MQJ0H1H2.blr.arm.com [10.162.40.80]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id F17C83F5A1; Tue, 11 Feb 2025 03:14:54 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: npache@redhat.com, ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [PATCH v2 08/17] khugepaged: Introduce vma_collapse_anon_folio() Date: Tue, 11 Feb 2025 16:43:17 +0530 Message-Id: <20250211111326.14295-9-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250211111326.14295-1-dev.jain@arm.com> References: <20250211111326.14295-1-dev.jain@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Queue-Id: B19EE40006 X-Rspamd-Server: rspam07 X-Stat-Signature: xgkefkitk9wae3bn7zcoomz7t1m6cgxt X-HE-Tag: 1739272505-623954 X-HE-Meta: U2FsdGVkX19ffppdUbaZg3t1jB5Gzp+YFEU95J3qu9riWRRV+eB0z4fInyRlnoa9Karnim+D9kprR56yxHfvD5wX8VgzMzemhVoM5ykJZKKqnIcq75llVMIE9VvgJiArIggkFRmoJlfluKROEBk8tP6JXyrRctx29PBAWs0q27a1zXemlN09TR2Ju9na3Aje8VfUi0wGDOtg9SEewxmO5Q058c+HdWjdrqUQ2IT4buqCobNSfhPZSVboczLiO/18WefDgiYKmnx6znSqoM/MSNe719wpmic6qe6bJSXZaua6iPRYc6BK6VplmLPVHpXCC8929KpNDbyoxcjJJrHp4RNJG4kZhf00DAfDVQ4R/NW5dUPcADwTmk33UUiuybhEUla0wOf2H1dF1BHG9NmaooFQaeELlbxjakxZ8aIq9BgCUkxlUbsqCZn6lvdKqz622jEQWJcdTzOTMoQaovqECke/Wp9ZGfh7UhAzUPMVaIdfmuP/N5bZOsBeu+syJyUo6X2PnzOBIuERPgDjp7F19LLAfcD2gjgiw7kfzL+Nglng+hReh7goli6Ty2Mzzk71nXVppzWmx/6qbphMGtC81FpkTWELTV66X6g07VXZvijsBjPHObBgid68Jdc0tVotODLAwwS/YqR4eao83kKJAyXGk42JAKUVRFNeVpl/o7FflApmcNWXxbWawJ2jIpyPnuhQePYLRyPOQ6VOvvY5XyWSzL9aKTyLloQDEftR2UWeO1cR4I4OZwuZ1x+oS3l0QxjgWT2hDil5aHhKJwQHt1a+iEE1YI2gGs6rwfANVHYx3aNDIYEgjzMBBf8j3CtfYv3pPX6aZw0Zd12fW7Y7kgVWsQqZ1CCrRMD1YUCHOxJpzOlxImgykPYtzgGAKNWZywnXLiwvZvTWQKWRKdbV7bIoQn5L8Hysz+AVeX/YVVykNOzKnWv6hUkw0so0bwIGsjVGzSkRRLNqHq5SD9a 2gnxcY77 vKqxuiy6vBU2zU+d+o6VeZD7xSj2jFxwte/r6DZXwq+YyWofIQd1g20E7A2Uxvi1E0RLdHNTqoBDX943CNfBgszj6LFCZMT1ReXIVwZuudpMFOOc+0emKb3zuT+m4I116SyFR9KDnpJm2FITxEGX4JnjPqwmPyMx26jAAu+VZEw2kFaF9IGGdwPIn5WnC53Ozoe2SoWFeeZYrSILwpr3KFxr+rN7x33DgCP31nNM/3MZ9JaPZTGnW7oja8ZR3REx3dW6hY45ErnK0xt7EhsqfUqg10w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similar to PMD collapse, take the write locks to stop pagetable walking. Copy page contents, clear the PTEs, remove folio pins, and (try to) unmap the old folios. Set the PTEs to the new folio using the set_ptes() API. Signed-off-by: Dev Jain --- mm/khugepaged.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index fbfd8a78ef51..a674014b6563 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1217,6 +1217,96 @@ static int vma_collapse_anon_folio_pmd(struct mm_struct *mm, unsigned long addre return result; } +/* Similar to the PMD case except we have to batch set the PTEs */ +static int vma_collapse_anon_folio(struct mm_struct *mm, unsigned long address, + struct vm_area_struct *vma, struct collapse_control *cc, pmd_t *pmd, + struct folio *folio, int order) +{ + LIST_HEAD(compound_pagelist); + spinlock_t *pmd_ptl, *pte_ptl; + int result = SCAN_FAIL; + struct mmu_notifier_range range; + pmd_t _pmd; + pte_t *pte; + pte_t entry; + int nr_pages = folio_nr_pages(folio); + unsigned long haddress = address & HPAGE_PMD_MASK; + + VM_BUG_ON(address & ((PAGE_SIZE << order) - 1));; + + result = hugepage_vma_revalidate(mm, address, true, &vma, order, cc); + if (result != SCAN_SUCCEED) + goto out; + result = check_pmd_still_valid(mm, address, pmd); + if (result != SCAN_SUCCEED) + goto out; + + vma_start_write(vma); + anon_vma_lock_write(vma->anon_vma); + + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, haddress, + haddress + HPAGE_PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); + + pmd_ptl = pmd_lock(mm, pmd); + _pmd = pmdp_collapse_flush(vma, haddress, pmd); + spin_unlock(pmd_ptl); + mmu_notifier_invalidate_range_end(&range); + tlb_remove_table_sync_one(); + + pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl); + if (pte) { + result = __collapse_huge_page_isolate(vma, address, pte, cc, + &compound_pagelist, order); + spin_unlock(pte_ptl); + } else { + result = SCAN_PMD_NULL; + } + + if (unlikely(result != SCAN_SUCCEED)) { + if (pte) + pte_unmap(pte); + spin_lock(pmd_ptl); + BUG_ON(!pmd_none(*pmd)); + pmd_populate(mm, pmd, pmd_pgtable(_pmd)); + spin_unlock(pmd_ptl); + anon_vma_unlock_write(vma->anon_vma); + goto out; + } + + anon_vma_unlock_write(vma->anon_vma); + + result = __collapse_huge_page_copy(pte, folio, pmd, *pmd, + vma, address, pte_ptl, + &compound_pagelist, order); + pte_unmap(pte); + if (unlikely(result != SCAN_SUCCEED)) + goto out; + + __folio_mark_uptodate(folio); + entry = mk_pte(&folio->page, vma->vm_page_prot); + entry = maybe_mkwrite(pte_mkdirty(entry), vma); + + spin_lock(pte_ptl); + folio_ref_add(folio, nr_pages - 1); + folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE); + folio_add_lru_vma(folio, vma); + set_ptes(mm, address, pte, entry, nr_pages); + spin_unlock(pte_ptl); + spin_lock(pmd_ptl); + + /* See pmd_install() */ + smp_wmb(); + BUG_ON(!pmd_none(*pmd)); + pmd_populate(mm, pmd, pmd_pgtable(_pmd)); + update_mmu_cache_pmd(vma, haddress, pmd); + spin_unlock(pmd_ptl); + + result = SCAN_SUCCEED; +out: + return result; +} + static int collapse_huge_page(struct mm_struct *mm, unsigned long address, int referenced, int unmapped, int order, struct collapse_control *cc) @@ -1276,6 +1366,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (order == HPAGE_PMD_ORDER) result = vma_collapse_anon_folio_pmd(mm, address, vma, cc, pmd, folio); + else + result = vma_collapse_anon_folio(mm, address, vma, cc, pmd, folio, order); mmap_write_unlock(mm); -- 2.30.2