From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6953CD37B1 for ; Fri, 15 Sep 2023 23:28:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233859AbjIOX2P (ORCPT ); Fri, 15 Sep 2023 19:28:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237488AbjIOX1s (ORCPT ); Fri, 15 Sep 2023 19:27:48 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 433731FDE for ; Fri, 15 Sep 2023 16:27:43 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D2DFBC433CA; Fri, 15 Sep 2023 23:27:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1694820462; bh=EobELYb2fokRDw+vajr0EMs/UBP1ikR+Re0twwqRYl0=; h=Date:To:From:Subject:From; b=huz2Iay8jhslJ7FeohHOTV2xwfO3v0j1SfovZ2+ySlIRqTTiIespyfRTVqGLxUzQV wB3z5g+cK08ndINSvW9rwN2VQB1mO2euW3PRr4e0nCRDGHkXEmYisDEkvDdJ0g22mC 4+hQVYhuSKVCqfEEp8WwOxKbRUP+BhA/mlH+QJhQ= Date: Fri, 15 Sep 2023 16:27:42 -0700 To: mm-commits@vger.kernel.org, willy@infradead.org, songmuchun@bytedance.com, sidhartha.kumar@oracle.com, rientjes@google.com, osalvador@suse.de, naoya.horiguchi@linux.dev, mhocko@suse.com, linmiaohe@huawei.com, jthoughton@google.com, joao.m.martins@oracle.com, duanxiongchun@bytedance.com, david@redhat.com, anshuman.khandual@arm.com, mike.kravetz@oracle.com, akpm@linux-foundation.org From: Andrew Morton Subject: + hugetlb-batch-tlb-flushes-when-restoring-vmemmap.patch added to mm-unstable branch Message-Id: <20230915232742.D2DFBC433CA@smtp.kernel.org> Precedence: bulk Reply-To: linux-kernel@vger.kernel.org List-ID: X-Mailing-List: mm-commits@vger.kernel.org The patch titled Subject: hugetlb: batch TLB flushes when restoring vmemmap has been added to the -mm mm-unstable branch. Its filename is hugetlb-batch-tlb-flushes-when-restoring-vmemmap.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/hugetlb-batch-tlb-flushes-when-restoring-vmemmap.patch This patch will later appear in the mm-unstable branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via the mm-everything branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there every 2-3 working days ------------------------------------------------------ From: Mike Kravetz Subject: hugetlb: batch TLB flushes when restoring vmemmap Date: Fri, 15 Sep 2023 15:15:45 -0700 Update the internal hugetlb restore vmemmap code path such that TLB flushing can be batched. Use the existing mechanism of passing the VMEMMAP_REMAP_NO_TLB_FLUSH flag to indicate flushing should not be performed for individual pages. The routine hugetlb_vmemmap_restore_folios is the only user of this new mechanism, and it will perform a global flush after all vmemmap is restored. Link: https://lkml.kernel.org/r/20230915221548.552084-13-mike.kravetz@oracle.com Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz Cc: Anshuman Khandual Cc: David Hildenbrand Cc: David Rientjes Cc: James Houghton Cc: "Matthew Wilcox (Oracle)" Cc: Miaohe Lin Cc: Michal Hocko Cc: Muchun Song Cc: Naoya Horiguchi Cc: Oscar Salvador Cc: Sidhartha Kumar Cc: Xiongchun Duan Signed-off-by: Andrew Morton --- mm/hugetlb_vmemmap.c | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) --- a/mm/hugetlb_vmemmap.c~hugetlb-batch-tlb-flushes-when-restoring-vmemmap +++ a/mm/hugetlb_vmemmap.c @@ -460,18 +460,19 @@ out: * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @flags: modify behavior for bulk operations * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, unsigned long flags) { LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, - .flags = 0, + .flags = flags, }; /* See the comment in the vmemmap_remap_free(). */ @@ -493,17 +494,7 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_k static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); -/** - * hugetlb_vmemmap_restore - restore previously optimized (by - * hugetlb_vmemmap_optimize()) vmemmap pages which - * will be reallocated and remapped. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be restored. - * - * Return: %0 if @head's vmemmap pages have been reallocated and remapped, - * negative error code otherwise. - */ -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +static int __hugetlb_vmemmap_restore(const struct hstate *h, struct page *head, unsigned long flags) { int ret; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; @@ -524,7 +515,7 @@ int hugetlb_vmemmap_restore(const struct * When a HugeTLB page is freed to the buddy allocator, previously * discarded vmemmap pages must be allocated and remapping. */ - ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse); + ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, flags); if (!ret) { ClearHPageVmemmapOptimized(head); static_branch_dec(&hugetlb_optimize_vmemmap_key); @@ -534,6 +525,21 @@ int hugetlb_vmemmap_restore(const struct } /** + * hugetlb_vmemmap_restore - restore previously optimized (by + * hugetlb_vmemmap_optimize()) vmemmap pages which + * will be reallocated and remapped. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be restored. + * + * Return: %0 if @head's vmemmap pages have been reallocated and remapped, + * negative error code otherwise. + */ +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +{ + return __hugetlb_vmemmap_restore(h, head, 0); +} + +/** * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list. * @h: struct hstate. * @folio_list: list of folios. @@ -557,7 +563,8 @@ int hugetlb_vmemmap_restore_folios(const num_restored = 0; list_for_each_entry(folio, folio_list, lru) { if (folio_test_hugetlb_vmemmap_optimized(folio)) { - t_ret = hugetlb_vmemmap_restore(h, &folio->page); + t_ret = __hugetlb_vmemmap_restore(h, &folio->page, + VMEMMAP_REMAP_NO_TLB_FLUSH); if (t_ret) ret = t_ret; else @@ -565,6 +572,8 @@ int hugetlb_vmemmap_restore_folios(const } } + flush_tlb_all(); + if (*restored) *restored = num_restored; return ret; _ Patches currently in -mm which might be from mike.kravetz@oracle.com are hugetlb-set-hugetlb-page-flag-before-optimizing-vmemmap.patch hugetlb-optimize-update_and_free_pages_bulk-to-avoid-lock-cycles.patch hugetlb-restructure-pool-allocations.patch hugetlb-perform-vmemmap-optimization-on-a-list-of-pages.patch hugetlb-perform-vmemmap-restoration-on-a-list-of-pages.patch hugetlb-batch-freeing-of-vmemmap-pages.patch hugetlb-batch-tlb-flushes-when-restoring-vmemmap.patch