From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 097C83DA5A8 for ; Tue, 16 Jun 2026 19:03:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781636638; cv=none; b=l0Y+PHLlg+c1SEAaHAzwAxQHXtva19HUbEU2/oi5lAVbfroOxv+6jWexTY65pzUXSDsCWRWt8uWQdLU14xjloWkq8YR6GvwpcaYruHnGiWbhAlnEjSeXXlj+SZ4nkTuq8IM9VrXLolohwh+tosz9tGN4/tqBtbOaQbw65yD2Trk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781636638; c=relaxed/simple; bh=6q/gkR1YW13Dc8InkFK2CqGh+zCIIBcNHIQE3xqAEbg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qhcYzGHQ9i6PqqtXvmtfnL/03IUFfaRytvgAWhqpPihjlz+ePOOtsuid7cg+K/5/FUJuDijM3VxQU8AmCk62zv8v3noWBeO+HTydOtbOqCQyCp05+xYxx+EfeQUn1R1LSRP2QbBYb0IVvnorMMlrF49LBhCyZfgFWGnLDFHizvc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=WjnH4X4b; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="WjnH4X4b" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=DjDE7wKN3rUmIZIqeyfahm1B1o9cqGnY+Db4wh52/Bw=; b=WjnH4X4boIaoBJMZdUKpm/czgi NdqdncMtB88Ww31D+EtWTTXRkVJ5/dWz0Otdu66yetupfgZpCFXIn8SSLdb0yr/X1tpnRVWPUlSG4 gthnRjzZh19EI99mbhokNzYwDYT2SxxGTqTldW6alArbjMLBHJ3wSCe+BFqJsBQzFR151OBsuVEYI gTarx3ziZnYlv7/k4uXn4RIJIUqCcGmX4zciwcu5RY/t7lDmPih4my/1/4b7LlHipZjL9+M/osrzP ceogbzPn1HUMOK+zB2lm5nMSxDtsTYpKIKG/Q/HuClWXlSOtP4Iblkqjf3hf743zi2ACrMkEs3Adl ZVlnrolg==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wZZ4S-000000005GL-1Fcw; Tue, 16 Jun 2026 15:03:36 -0400 From: Rik van Riel To: linux-kernel@vger.kernel.org Cc: Rik van Riel , x86@kernel.org, linux-mm@kvack.org, "Thomas Gleixner" , "Ingo Molnar" , "Dmitry Ilvokhin" , "Borislav Petkov" , "Dave Hansen" , "Andrew Morton" , "David Hildenbrand" , "Lorenzo Stoakes" , "Liam R. Howlett" , "Vlastimil Babka" , "Suren Baghdasaryan" Subject: [PATCH 2/3] mm/pagewalk: let folio_walk_start() run under the per-VMA lock Date: Tue, 16 Jun 2026 15:02:59 -0400 Message-ID: <20260616190300.1509639-3-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260616190300.1509639-1-riel@surriel.com> References: <20260616190300.1509639-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit folio_walk_start() asserts that the mmap lock is held. For callers that only need to read a single, already-present page, the mmap lock is a heavy and often badly contended hammer: the VMA can instead be stabilized with the per-VMA lock, and the page table pages that are walked are kept alive by RCU page-table freeing (CONFIG_MMU_GATHER_RCU_TABLE_FREE). Add an FW_VMA_LOCKED flag. When passed, folio_walk_start() asserts the per-VMA lock instead of the mmap lock, requires RCU-freed page tables, and refuses hugetlb VMAs (PMD sharing cannot be walked safely this way). Everything else folio_walk_start() relies on -- the page table locks, pmdp_get_lockless() and pte_offset_map_lock() -- is already safe without the mmap lock, mirroring the per-VMA lock page fault path. No existing caller passes FW_VMA_LOCKED, so behaviour is unchanged. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Rik van Riel --- include/linux/pagewalk.h | 5 +++++ mm/pagewalk.c | 18 ++++++++++++++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h index b41d7265c01b..84dd0d68f747 100644 --- a/include/linux/pagewalk.h +++ b/include/linux/pagewalk.h @@ -150,6 +150,11 @@ typedef int __bitwise folio_walk_flags_t; /* Walk shared zeropages (small + huge) as well. */ #define FW_ZEROPAGE ((__force folio_walk_flags_t)BIT(0)) +/* + * The caller holds the per-VMA lock instead of the mmap lock. Only valid with + * RCU-freed page tables (CONFIG_MMU_GATHER_RCU_TABLE_FREE) and not for hugetlb. + */ +#define FW_VMA_LOCKED ((__force folio_walk_flags_t)BIT(1)) enum folio_walk_level { FW_LEVEL_PTE, diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 3ae2586ff45b..c85364b73e12 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -890,7 +890,9 @@ int walk_page_mapping(struct address_space *mapping, pgoff_t first_index, * huge_ptep_set_*, ...). Note that the page table entry stored in @fw might * not correspond to the first physical entry of a logical hugetlb entry. * - * The mmap lock must be held in read mode. + * The mmap lock must be held in read mode. Alternatively, if @FW_VMA_LOCKED is + * passed, the VMA's per-VMA lock must be held (only supported with RCU-freed + * page tables, i.e. CONFIG_MMU_GATHER_RCU_TABLE_FREE, and not for hugetlb). * * Return: folio pointer on success, otherwise NULL. */ @@ -908,7 +910,19 @@ struct folio *folio_walk_start(struct folio_walk *fw, pgd_t *pgdp; p4d_t *p4dp; - mmap_assert_locked(vma->vm_mm); + if (flags & FW_VMA_LOCKED) { + /* + * Lockless walk: the per-VMA lock keeps the VMA stable, and + * RCU-freed page tables keep the walked page table pages alive + * across the lockless upper-level walk and pte_offset_map_lock(). + * Hugetlb (PMD sharing) is not supported on this path. + */ + VM_WARN_ON_ONCE(!IS_ENABLED(CONFIG_MMU_GATHER_RCU_TABLE_FREE)); + VM_WARN_ON_ONCE(is_vm_hugetlb_page(vma)); + vma_assert_locked(vma); + } else { + mmap_assert_locked(vma->vm_mm); + } vma_pgtable_walk_begin(vma); if (WARN_ON_ONCE(addr < vma->vm_start || addr >= vma->vm_end)) -- 2.53.0-Meta