From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E26C2FFF8B for ; Tue, 11 Aug 2026 02:53:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786416799; cv=none; b=hdbhJ+nOgjc/Ej5RK6v1U0vEiAmQFI8aA6Co2epNYlT13HnMUbEyYb1brStNKzVVAwFzBYkuo/B8Kf0YOYvjWTdjYd4oNthJDC5M86K97tR7QA3oqppAQs0jozqBgAHbJRr/SmmCpkP8qE8GeZxeOVrH+VTiRFcSltbmVgcFHIw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786416799; c=relaxed/simple; bh=klTwV8iCBH6C2vFyyS8IV1gZt93cFsDxgUR/y/wr9ao=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=Z6UfFx6NRJK43H1weuyZgZVJb7Zxy4OCulqLCal3XnNM29WgHS4fEEijaHqTMv7obglv6bFSC9TB8GF0CcoWx/uISV7Xy6lC9h+rsIlGlj6i3COYcQWUTVKqf72lsz5sABegmYU3jc76KiT5i9ZDPKUaV1tkHmKwrVIfg7jCYDw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=DVcvSo+4; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="DVcvSo+4" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:Message-ID:Date:Subject:Cc :To:From:Sender:Reply-To:Content-Type:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=LGXcCPY7kG2CVFlPckufNlkx0xRe8qRFiK0PwEOL4G4=; b=DVcvSo+4a22PxOsL4f2RdSb5p+ PXzLlheA1AKKk5NvLrHrl+lgRhdkPTM/bSTa3WM1eD/W/vA5PXBYNKSz4sho7mUcLsMnI03RxWBPV iCFJ5QiutVi7UsF/T91Lgc1OiX8+jOj5domYrTkhGkw7c0i7zjP9d/dEtj5y5Wsb9RyVBt/6KpCBo Q4wkBHy6vs58emxRVJ2A+aCBeAVxZC0RTFwAaaAURjtfICLemOqrvMmKkMiOc8seIgWeo+izo6PJa 9D5BLli/svS2BXr0zxuByLCbwVNGgPmoW3CI/8hsc+PpXEeD7QtZUAwpMTcyBV71VYccbElfF24xQ GVgNVN2Q==; Received: from [96.67.55.146] (helo=fangorn.surriel.com) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wtcb6-000000007kZ-2Cpo; Mon, 10 Aug 2026 22:52:12 -0400 From: Rik van Riel To: linux-kernel@vger.kernel.org Cc: kernel-team@meta.com, Rik van Riel , Andrew Morton , David Hildenbrand , Jason Gunthorpe , John Hubbard , Peter Xu , linux-mm@kvack.org Subject: [RFC PATCH v3 0/8] batch lookups in follow_page_mask() Date: Mon, 10 Aug 2026 22:51:49 -0400 Message-ID: <20260811025157.1632867-1-riel@surriel.com> X-Mailer: git-send-email 2.55.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit follow_page_mask() walks the page tables one page at a time, even when the caller asked for a whole run of contiguous pages. Every page of a large folio re-enters the pmd/pud/pte walk and re-takes the page table lock. This series changes follow_page_mask() to return a page count and a fill an array of pages, instead of a single struct page, so a walker can hand back more than one page per call. Patches 1 to 4 are preparation, no functional change: 1: move __get_user_pages()'s open-coded pages[] fill and cache flush into a gup_fill_pages() helper, which the rest of the series reuses. 2: convert the follow_page_mask()/follow_p4d_mask()/follow_pud_mask()/ follow_pmd_mask()/follow_page_pte() call chain to return a long instead of a struct page pointer or ERR_PTR(). Every path still handles one page. 3: split the "commit to a resolved page" tail of follow_page_pte() into follow_page_pte_commit(). 4: split the "work out which page this PTE maps" half of follow_page_pte() into follow_one_pte(), leaving one unlock and one exit. Patch 5 has the huge page paths store the page and leave the array fill to follow_pud_mask()/follow_pmd_mask() after they unlock, so the cache flushes happen outside the pud/pmd critical section. Patch 6 has follow_huge_pud()/follow_huge_pmd() report the huge page's real subpage count instead of a separate *page_mask output, and retires *page_mask and __get_user_pages()'s try_grab_folio() call and subpage loop. Patch 7 walks every PTE in a page table in one follow_page_pte() call instead of one per page. Patch 8 adds follow_pte_batch() so a contiguous same-folio run is committed with one refcount grab. This is the only patch whose benefit depends on folio size; patch 7 alone covers plain base pages. Patches 7 and 8 carry their own benchmark tables, both measured against the base of the series, so the split between the two mechanisms is visible: the single-call walk is worth 2.3x on base pages and 2.2x on 64 kB mTHP, and refcount batching adds a further 5.9x on the mTHP case. v3: - split up the series into 8 much smaller patches (David & Lorenzo) - shorten changelogs where they were too long (Lorenzo) - fix FOLL_WRITE folio dirtying by gathering dirty bits from all PTEs Link: https://lore.kernel.org/r/20260730035350.1fc95dd8@fangorn/ [RFC] Link: https://lore.kernel.org/r/20260801031540.2742891-1-riel@surriel.com/ [RFC v2] Suggested-by: David Hildenbrand Rik van Riel (8): mm/gup: break out gup_fill_pages() helper mm/gup: convert follow_page_mask() to return a long mm/gup: split follow_page_pte_commit() out of follow_page_pte() mm/gup: break out follow_one_pte() helper mm/gup: fill the pages array outside the pud/pmd lock mm/gup: return a huge page's full count from follow_page_mask() mm/gup: walk multiple PTEs per follow_page_pte() call mm/gup: batch contiguous same-folio PTEs into one refcount grab mm/gup.c | 515 +++++++++++++++++++++++++++++++++---------------------- 1 file changed, 308 insertions(+), 207 deletions(-) -- 2.55.0