From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20C6A41C63 for ; Thu, 17 Apr 2025 23:22:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744932136; cv=none; b=rp8SVpqrc9cdIJJY8N+RRkaiHHqu6BKn8AeqSQMLwqnfYo2ps5KuiZ20SQnaI9B3dQN8amMRAHG9FLNHTjjyqM3G275rGBQAxoaYqbyE5UyN8Y2os7/oixHWijtdwrb/PqEFmzAzKhV6wgrmX0ob6XiseOhgOUdeSU0SV/kv/Fc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744932136; c=relaxed/simple; bh=m6H07gxpS1FSn0vUeR4KV++7V6CbxDcKIPC9gccDC0I=; h=Date:To:From:Subject:Message-Id; b=ey3Z8qyV1XEVFWPGszAJOS/gEU3h/a/G/Er5vr3GSEZ3p1QAd/0ukH4T7/S6JVqfQCljmDWoEEaPUV5Eg1hQp9lfT7sD1TZzmuJI3XQEVosiw1G5WelnJlO8ULtP4D4c1Az/cVN5785/w64h9GOnvYA99FxK4pLAXcDnfdfdbZc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=EYaWCIn1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="EYaWCIn1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 42AFEC4CEE4; Thu, 17 Apr 2025 23:22:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1744932135; bh=m6H07gxpS1FSn0vUeR4KV++7V6CbxDcKIPC9gccDC0I=; h=Date:To:From:Subject:From; b=EYaWCIn1P3nseb+e1E5Jq8hN6p57rtf9Xk1AYQ9xxzWDnbGY20rHWetEEUh/EiL/D 88EPJRvYGAJqjS9pH29whvEChQovq6ru7E/gjZLlvwMXfdRbHZDT6sx2Mp4dD6oJSn fnY6sgLYijx0/jBTM3iCMhgO6EmmDrDSggFg8uhg= Date: Thu, 17 Apr 2025 16:22:13 -0700 To: mm-commits@vger.kernel.org,zokeefe@google.com,ziy@nvidia.com,yang@os.amperecomputing.com,willy@infradead.org,will@kernel.org,wangkefeng.wang@huawei.com,vishal.moola@gmail.com,usamaarif642@gmail.com,tiwai@suse.de,thomas.hellstrom@linux.intel.com,surenb@google.com,sunnanyong@huawei.com,shuah@kernel.org,ryan.roberts@arm.com,rostedt@goodmis.org,rientjes@google.com,rdunlap@infradead.org,raquini@redhat.com,peterx@redhat.com,mhocko@suse.com,mhiramat@kernel.org,mathieu.desnoyers@efficios.com,kirill.shutemov@linux.intel.com,jack@suse.cz,hannes@cmpxchg.org,dev.jain@arm.com,david@redhat.com,corbet@lwn.net,cl@gentwo.org,catalin.marinas@arm.com,baolin.wang@linux.alibaba.com,baohua@kernel.org,anshuman.khandual@arm.com,aarcange@redhat.com,npache@redhat.com,akpm@linux-foundation.org From: Andrew Morton Subject: + introduce-khugepaged_collapse_single_pmd-to-unify-khugepaged-and-madvise_collapse.patch added to mm-new branch Message-Id: <20250417232215.42AFEC4CEE4@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The patch titled Subject: introduce khugepaged_collapse_single_pmd to unify khugepaged and madvise_collapse has been added to the -mm mm-new branch. Its filename is introduce-khugepaged_collapse_single_pmd-to-unify-khugepaged-and-madvise_collapse.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/introduce-khugepaged_collapse_single_pmd-to-unify-khugepaged-and-madvise_collapse.patch This patch will later appear in the mm-new branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via the mm-everything branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there every 2-3 working days ------------------------------------------------------ From: Nico Pache Subject: introduce khugepaged_collapse_single_pmd to unify khugepaged and madvise_collapse Date: Wed, 16 Apr 2025 18:02:27 -0600 Patch series "khugepaged: mTHP support", v4. The following series provides khugepaged and madvise collapse with the capability to collapse regions to mTHPs. To achieve this we generalize the khugepaged functions to no longer depend on PMD_ORDER. Then during the PMD scan, we keep track of chunks of pages (defined by KHUGEPAGED_MTHP_MIN_ORDER) that are utilized. This info is tracked using a bitmap. After the PMD scan is done, we do binary recursion on the bitmap to find the optimal mTHP sizes for the PMD range. The restriction on max_ptes_none is removed during the scan, to make sure we account for the whole PMD range. When no mTHP size is enabled, the legacy behavior of khugepaged is maintained. max_ptes_none will be scaled by the attempted collapse order to determine how full a THP must be to be eligible. If a mTHP collapse is attempted, but contains swapped out, or shared pages, we dont perform the collapse. With the default max_ptes_none=511, the code should keep its most of its original behavior. To exercise mTHP collapse we need to set max_ptes_none<=255. With max_ptes_none > HPAGE_PMD_NR/2 you will experience collapse "creep" and constantly promote mTHPs to the next available size. Patch 1: Some refactoring to combine madvise_collapse and khugepaged Patch 2: Refactor/rename hpage_collapse Patch 3-5: Generalize khugepaged functions for arbitrary orders Patch 6-9: The mTHP patches Patch 10-11: Tracing/stats Patch 12: Documentation This patch (of 12): The khugepaged daemon and madvise_collapse have two different implementations that do almost the same thing. Create khugepaged_collapse_single_pmd to increase code reuse and create an entry point for future khugepaged changes. Refactor madvise_collapse and khugepaged_scan_mm_slot to use the new khugepaged_collapse_single_pmd function. Link: https://lkml.kernel.org/r/20250417000238.74567-1-npache@redhat.com Link: https://lkml.kernel.org/r/20250417000238.74567-2-npache@redhat.com Signed-off-by: Nico Pache Cc: Andrea Arcangeli Cc: Anshuman Khandual Cc: Baolin Wang Cc: Barry Song Cc: Catalin Marinas Cc: Christoph Lameter (Ampere) Cc: David Hildenbrand Cc: David Rientjes Cc: Dev Jain Cc: Jan Kara Cc: Johannes Weiner Cc: Jonathan Corbet Cc: Kefeng Wang Cc: Kirill A. Shuemov Cc: "Masami Hiramatsu (Google)" Cc: Mathieu Desnoyers Cc: Matthew Wilcox (Oracle) Cc: Michal Hocko Cc: Nanyong Sun Cc: Peter Xu Cc: Rafael Aquini Cc: Randy Dunlap Cc: Reported-by:Takashi Iwai Cc: Ryan Roberts Cc: Steven Rostedt Cc: Suren Baghdasaryan Cc: Thomas Hellstrom Cc: Usama Arif Cc: Vishal Moola (Oracle) Cc: Will Deacon Cc: Yang Shi Cc: Zach O'Keefe Cc: Zi Yan Cc: Shuah Khan Signed-off-by: Andrew Morton --- mm/khugepaged.c | 92 +++++++++++++++++++++++----------------------- 1 file changed, 46 insertions(+), 46 deletions(-) --- a/mm/khugepaged.c~introduce-khugepaged_collapse_single_pmd-to-unify-khugepaged-and-madvise_collapse +++ a/mm/khugepaged.c @@ -2363,6 +2363,48 @@ static int hpage_collapse_scan_file(stru } #endif +/* + * Try to collapse a single PMD starting at a PMD aligned addr, and return + * the results. + */ +static int khugepaged_collapse_single_pmd(unsigned long addr, + struct vm_area_struct *vma, bool *mmap_locked, + struct collapse_control *cc) +{ + int result = SCAN_FAIL; + struct mm_struct *mm = vma->vm_mm; + unsigned long tva_flags = cc->is_khugepaged ? TVA_ENFORCE_SYSFS : 0; + + if (thp_vma_allowable_order(vma, vma->vm_flags, + tva_flags, PMD_ORDER)) { + if (IS_ENABLED(CONFIG_SHMEM) && !vma_is_anonymous(vma)) { + struct file *file = get_file(vma->vm_file); + pgoff_t pgoff = linear_page_index(vma, addr); + + mmap_read_unlock(mm); + *mmap_locked = false; + result = hpage_collapse_scan_file(mm, addr, file, pgoff, + cc); + fput(file); + if (result == SCAN_PTE_MAPPED_HUGEPAGE) { + mmap_read_lock(mm); + if (hpage_collapse_test_exit_or_disable(mm)) + goto end; + result = collapse_pte_mapped_thp(mm, addr, + !cc->is_khugepaged); + mmap_read_unlock(mm); + } + } else { + result = hpage_collapse_scan_pmd(mm, vma, addr, + mmap_locked, cc); + } + if (cc->is_khugepaged && result == SCAN_SUCCEED) + ++khugepaged_pages_collapsed; + } +end: + return result; +} + static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) @@ -2437,33 +2479,9 @@ skip: VM_BUG_ON(khugepaged_scan.address < hstart || khugepaged_scan.address + HPAGE_PMD_SIZE > hend); - if (IS_ENABLED(CONFIG_SHMEM) && !vma_is_anonymous(vma)) { - struct file *file = get_file(vma->vm_file); - pgoff_t pgoff = linear_page_index(vma, - khugepaged_scan.address); - mmap_read_unlock(mm); - mmap_locked = false; - *result = hpage_collapse_scan_file(mm, - khugepaged_scan.address, file, pgoff, cc); - fput(file); - if (*result == SCAN_PTE_MAPPED_HUGEPAGE) { - mmap_read_lock(mm); - if (hpage_collapse_test_exit_or_disable(mm)) - goto breakouterloop; - *result = collapse_pte_mapped_thp(mm, - khugepaged_scan.address, false); - if (*result == SCAN_PMD_MAPPED) - *result = SCAN_SUCCEED; - mmap_read_unlock(mm); - } - } else { - *result = hpage_collapse_scan_pmd(mm, vma, - khugepaged_scan.address, &mmap_locked, cc); - } - - if (*result == SCAN_SUCCEED) - ++khugepaged_pages_collapsed; + *result = khugepaged_collapse_single_pmd(khugepaged_scan.address, + vma, &mmap_locked, cc); /* move to next address */ khugepaged_scan.address += HPAGE_PMD_SIZE; @@ -2783,36 +2801,18 @@ int madvise_collapse(struct vm_area_stru mmap_assert_locked(mm); memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); - if (IS_ENABLED(CONFIG_SHMEM) && !vma_is_anonymous(vma)) { - struct file *file = get_file(vma->vm_file); - pgoff_t pgoff = linear_page_index(vma, addr); - mmap_read_unlock(mm); - mmap_locked = false; - result = hpage_collapse_scan_file(mm, addr, file, pgoff, - cc); - fput(file); - } else { - result = hpage_collapse_scan_pmd(mm, vma, addr, - &mmap_locked, cc); - } + result = khugepaged_collapse_single_pmd(addr, vma, &mmap_locked, cc); + if (!mmap_locked) *prev = NULL; /* Tell caller we dropped mmap_lock */ -handle_result: switch (result) { case SCAN_SUCCEED: case SCAN_PMD_MAPPED: ++thps; break; case SCAN_PTE_MAPPED_HUGEPAGE: - BUG_ON(mmap_locked); - BUG_ON(*prev); - mmap_read_lock(mm); - result = collapse_pte_mapped_thp(mm, addr, true); - mmap_read_unlock(mm); - goto handle_result; - /* Whitelisted set of results where continuing OK */ case SCAN_PMD_NULL: case SCAN_PTE_NON_PRESENT: case SCAN_PTE_UFFD_WP: _ Patches currently in -mm which might be from npache@redhat.com are introduce-khugepaged_collapse_single_pmd-to-unify-khugepaged-and-madvise_collapse.patch khugepaged-rename-hpage_collapse_-to-khugepaged_.patch khugepaged-generalize-hugepage_vma_revalidate-for-mthp-support.patch khugepaged-generalize-__collapse_huge_page_-for-mthp-support.patch khugepaged-introduce-khugepaged_scan_bitmap-for-mthp-support.patch khugepaged-add-mthp-support.patch khugepaged-skip-collapsing-mthp-to-smaller-orders.patch khugepaged-avoid-unnecessary-mthp-collapse-attempts.patch khugepaged-improve-tracepoints-for-mthp-orders.patch khugepaged-add-per-order-mthp-khugepaged-stats.patch documentation-mm-update-the-admin-guide-for-mthp-collapse.patch mm-defer-thp-insertion-to-khugepaged.patch mm-document-mthp-defer-usage.patch khugepaged-add-defer-option-to-mthp-options.patch selftests-mm-add-defer-to-thp-setting-parser.patch