From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B4FB37189A for ; Sat, 1 Aug 2026 03:15:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785554164; cv=none; b=qSr8I8XXq0opcISPVkDJbxDM2PBusIQ1e92z3rIcQ2b2NgBbKY5I3FVeiVu3usFTT7GMq4KUaF74Ygzp3aLYexXTVxlV5H+cVjP4nXj17d38Vk9f9exLq36iGmXPiQ5WARXB3MI2WxQW2IhkOT6xxfRLD512OliFMNTJl4XGQYI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785554164; c=relaxed/simple; bh=WZLHi8BIDSVegBGGdKjgYuz8oQcB86ywFgYzEZiIc9k=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=Bwc+ceGmUPsxp5uD+pWdf3mVjRfkVQhqe4Jos0kp0Mx1+IHaUFDVn/mvIFhjWs+LhewL43t63eNCxzkwxabjuCwC23WWiX0gl21zIUyvGmtisyGqRgDnZUCKdl23Vw2drENpERuTPJe4emmdjgaR5CQiyJqDQp8decouxsblpc8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=LTKS8w+M; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="LTKS8w+M" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:Message-ID:Date:Subject:Cc :To:From:Sender:Reply-To:Content-Type:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=FGMAMYIv+aQ30Fi804H0kZD96vsuWwwShutLmJS318I=; b=LTKS8w+MQCT2aEvfy7YaJgFaQi quZvwa/RryQXj2gmrbJXTSjThva6OU+diwnQpJzz8L+zLVScwFvwxvo54jKK1kTsMTXwn8ooV0XlZ yO/NdW2FqkBSVQejev2UoYjcVoULp7FXaWc94dPdMUbbcmLYRD8s0nN1g2cXbzGCorGeDlDsi2cyr s+daHwB0Ueniea5W72iCX83D/F5u0hf4/0ZrJikUa9UCP7ZnM/8/Bs19vOZO/C+3f0M9AbftWOMCY N6RPMlzd/IAxzqurEoEqK+K+hzvHlcsvc/8GqbdRhiCoE5Wy+4p/CYEQL7u/miLlkSYBzwUtmRlEc B5OrlwNg==; Received: from [96.67.55.146] (helo=fangorn.surriel.com) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wq0CU-000000001Q5-3Wwz; Fri, 31 Jul 2026 23:15:50 -0400 From: Rik van Riel To: linux-kernel@vger.kernel.org, Andrew Morton , David Hildenbrand Cc: kernel-team@meta.com, Rik van Riel , Jason Gunthorpe , John Hubbard , Peter Xu , linux-mm@kvack.org, Lorenzo Stoakes Subject: [PATCH 0/5] mm/gup: batch contiguous pages in follow_page_mask() Date: Fri, 31 Jul 2026 23:15:35 -0400 Message-ID: <20260801031540.2742891-1-riel@surriel.com> X-Mailer: git-send-email 2.54.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit follow_page_mask() walks the page tables one page at a time, even when the caller asked for a whole run of contiguous pages. Every page of a large folio re-enters the pmd/pud/pte walk and re-takes the page table lock. This series changes follow_page_mask() to return a page count and a new @end argument bounding how many pages remain, instead of a single struct page, so a walker can hand back more than one page per call. Patch 1 converts the follow_page_mask()/follow_p4d_mask()/ follow_pud_mask()/follow_pmd_mask()/follow_page_pte() call chain to return a long instead of a struct page pointer or ERR_PTR(), with no functional change: every path still handles exactly one page. Patch 2 is pure code motion, splitting the "commit to a resolved page" tail of follow_page_pte() into its own follow_page_pte_commit(), no functional change. Patch 3 adds gup_fill_pages(), a small helper that fills pages[] and flushes caches for a run of subpages, and converts the three existing per-page call sites to use it with nr == 1, no functional change. Patch 4 has follow_huge_pud()/follow_huge_pmd() report the huge page's real subpage count instead of a separate *page_mask output, and retires *page_mask and __get_user_pages()'s dead second try_grab_folio() call and subpage loop. It also defers gup_fill_pages() past the pud/pmd unlock, so a 1 GB PUD-mapped folio doesn't hold that lock for a full array fill and cache flush. Patch 5 adds follow_pte_batch() and has follow_page_pte() call it once per contiguous same-folio run instead of once per page, so a PTE-mapped mTHP no longer restarts the walk and re-takes the PTE lock per subpage. This is the only patch that changes the number of page table walks or lock acquisitions. Benchmarked with mm/gup_test.c (PIN_LONGTERM_BENCHMARK, pin_user_pages + FOLL_LONGTERM, 256 MB region, median of 16 runs, folio formation verified via the per-size anon_fault_alloc counter): before after 4 kB base pages 2721 us 1198 us (2.3x) 64 kB mTHP 2929 us 201 us (14.6x) 2 MB THP 73 us 69 us (flat) The 4 kB result comes entirely from patch 5 merging two separate try_grab_folio() calls and lock acquisitions into one; folio size and PTE batching play no part in it. 64 kB mTHP adds the walk-restart avoidance on top. 2 MB THP is unaffected, since follow_huge_pmd() already handled it in one call. Patch 4's lock-hold-time change is a scalability argument, not a measured one -- it is not visible in this single-threaded benchmark. Suggested-by: David Hildenbrand Rik van Riel (5): mm/gup: convert follow_page_mask() to return a long mm/gup: split follow_page_pte_commit() out of follow_page_pte() mm/gup: add gup_fill_pages() and use it mm/gup: return a huge page's full count from follow_page_mask() mm/gup: walk multiple PTEs per follow_page_pte() call mm/gup.c | 532 +++++++++++++++++++++++++++++++++---------------------- 1 file changed, 322 insertions(+), 210 deletions(-) base-commit: fc02acf6ac0c -- 2.53.0-Meta