From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1B63470E8D for ; Wed, 29 Jul 2026 11:13:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785323599; cv=none; b=YID8RLxCWHXOPTnZAV9FfU2zJRODbj16uKybGIgUj4YcYPHErgBKYJViLa9AyCKj5C9kh/ppmyl7IuTk3LNIziTcJpN4+E5P+OPk/UErwKJbgBNsQfSKWslBPBqLX1HZ0dUV6438AJMIqMi6KIMr/bCYHx4PJ4ZXhi9+HVMeikE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785323599; c=relaxed/simple; bh=yfb5RzYduMLwqX1QKi96MC81ldNG1RPhjWZT+tGYdOY=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=PYLx9nJtJdxcQMJ+Mq0oHR/BmjfINkBgbMyK61pEVdW2S/hUl/cvSZpNHtI5OdPy6dxWqXMJp1/BWAifOBnQEN0iHPnRY5Zppq7bL0p0C8iaofmXqUiwgZrXP2QzvsGKUE3F63fxi1RXHI3w+kV90PaDO42Qgc43AVmk34oJImY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=WBSvK0LQ; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="WBSvK0LQ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=CU2s/2Ix5XgOBgId84uXukZLT3PuCIP33pHN4f3NJGM=; b=WBSvK0LQO1hnt81jRxC3jzx3xF fTdR9xymko5IryEGyyLIuhMA09o7hjGGbMh9Ggu2AUwPjqZT1E6wmKfzWq1Ukn2QfvN04Qi5onNYr t+pliVh8tVmi4Lcl6XOIzdZWAp/KQX0ZIu3ErdViLjdX/2ZlKO1o50OznKIb46VIAJf9RV/4QS6Pq CbZNxAFZTWOqFZlnnuW5hhWUm7mi4KHWO3aoMcgi4PmJCgL3py1tBOVOk8j0QpZaPsxvNM3yfW4/K dwbYInszSpVLn/QPx/hr94z92fkSXp3diFPGN8GzWgLv09D6Dn2oJYVodp5gB7arDH4uA19wOwSTE Up++3atw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.99.1 #2 (Red Hat Linux)) id 1wp2CU-00000001xXv-3kmm; Wed, 29 Jul 2026 11:11:54 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id BD9733011E4; Wed, 29 Jul 2026 13:11:40 +0200 (CEST) Message-ID: <20260729111119.604452135@infradead.org> User-Agent: quilt/0.68 Date: Wed, 29 Jul 2026 13:08:10 +0200 From: Peter Zijlstra To: Dave Hansen Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, Andy Lutomirski , Borislav Petkov , David Hildenbrand , Ingo Molnar , Jason Gunthorpe , Juergen Gross , Kevin Tian , Kiryl Shutsemau , "Liam R. Howlett" , Lorenzo Stoakes , Lu Baolu , Mike Rapoport , "H. Peter Anvin" , Shakeel Butt , Suren Baghdasaryan , Thomas Gleixner , Toshi Kani , Vlastimil Babka , Will Deacon , linux-mm@kvack.org, x86@kernel.org Subject: [PATCH 3/3] x86/mm: Fix and document DEBUG_PAGEALLOC References: <20260729110807.797920433@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 It turns out that commit 5fce67641a3e ("x86/mm/pat: Don't gate cpa_lock on debug_pagealloc_enabled()") was a little too quick to remove the debug_pagealloc exception for cpa_lock. Notably __kernel_map_pages() is used by the page-allocator from any context the page-allocator itself is used, which violates the cpa_lock rules. Re-instate the exception, except make it specific to the __kernel_map_pages() such that any other cpa() usage is still fully serialized by cpa_lock. Also note that since cpa() should not be used on memory that isn't allocated, the page-allocator locking and cpa are infact mutually exclusive and all cpa usage in fully serialized. Add a comment explaining this and other 'funnies' surrounding DEBUG_PAGEALLOC, including how pgd_lock is not affected and the TLB trickery. Fixes: 5fce67641a3e ("x86/mm/pat: Don't gate cpa_lock on debug_pagealloc_enabled()") Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/mm/pat/set_memory.c | 80 +++++++++++++++++++++++++++++++------------ 1 file changed, 58 insertions(+), 22 deletions(-) --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -68,11 +68,12 @@ static const int cpa_warn_level = CPA_PR */ static DEFINE_SPINLOCK(cpa_lock); -#define CPA_FLUSHTLB 1 -#define CPA_ARRAY 2 -#define CPA_PAGES_ARRAY 4 -#define CPA_NO_CHECK_ALIAS 8 /* Do not search for aliases */ -#define CPA_COLLAPSE 16 /* try to collapse large pages */ +#define CPA_FLUSHTLB 0x01 +#define CPA_ARRAY 0x02 +#define CPA_PAGES_ARRAY 0x04 +#define CPA_NO_CHECK_ALIAS 0x08 /* Do not search for aliases */ +#define CPA_COLLAPSE 0x10 /* try to collapse large pages */ +#define CPA_DEBUG_PAGEALLOC 0x20 static inline pgprot_t cachemode2pgprot(enum page_cache_mode pcm) { @@ -1989,6 +1990,7 @@ static int __change_page_attr_set_clr(st { unsigned long numpages = cpa->numpages; unsigned long rempages = numpages; + bool lock = true; int ret = 0; /* @@ -1998,6 +2000,29 @@ static int __change_page_attr_set_clr(st !cpa->force_split) return ret; + /* + * DEBUG_PAGEALLOC is special; it is called from any context the + * page-allocator is, which violates the normal cpa_lock locking + * rules. + * + * However, since it is part of the page-allocator, things are still + * properly serialized by the page-allocator locking and the fact that + * when a page is owned by the page-allocator, it isn't owned by + * anybody else. That is, you *SHOULD* not be calling cpa() on memory + * that isn't allocated. + * + * Additionally, DEBUG_PAGEALLOC ensures (per probe_page_size_mask()) + * that the kernel mapping is 4k pages, therefore there are no large + * pages to split/collapse. + * + * Furthermore, the page-allocator strictly manages pages that + * *exist*, avoiding pgd_lock. + * + * Therefore, it is safe to not take cpa_lock. + */ + if (debug_pagealloc_enabled() && (cpa->flags & CPA_DEBUG_PAGEALLOC)) + lock = false; + while (rempages) { /* * Store the remaining nr of pages for the large page @@ -2008,9 +2033,12 @@ static int __change_page_attr_set_clr(st if (cpa->flags & (CPA_ARRAY | CPA_PAGES_ARRAY)) cpa->numpages = 1; - spin_lock(&cpa_lock); - ret = __change_page_attr(cpa, primary); - spin_unlock(&cpa_lock); + if (lock) { + guard(spinlock)(&cpa_lock); + ret = __change_page_attr(cpa, primary); + } else { + ret = __change_page_attr(cpa, primary); + } if (ret) goto out; @@ -2589,7 +2617,7 @@ int set_pages_rw(struct page *page, int return set_memory_rw(addr, numpages); } -static int __set_pages_p(struct page *page, int numpages) +static int __set_pages_p(struct page *page, int numpages, unsigned int cpa_flags) { unsigned long tempaddr = (unsigned long) page_address(page); struct cpa_data cpa = { .vaddr = &tempaddr, @@ -2597,7 +2625,7 @@ static int __set_pages_p(struct page *pa .numpages = numpages, .mask_set = __pgprot(_PAGE_PRESENT | _PAGE_RW), .mask_clr = __pgprot(0), - .flags = CPA_NO_CHECK_ALIAS }; + .flags = CPA_NO_CHECK_ALIAS | cpa_flags }; /* * No alias checking needed for setting present flag. otherwise, @@ -2608,7 +2636,7 @@ static int __set_pages_p(struct page *pa return __change_page_attr_set_clr(&cpa, 1); } -static int __set_pages_np(struct page *page, int numpages) +static int __set_pages_np(struct page *page, int numpages, unsigned int cpa_flags) { unsigned long tempaddr = (unsigned long) page_address(page); struct cpa_data cpa = { .vaddr = &tempaddr, @@ -2616,7 +2644,7 @@ static int __set_pages_np(struct page *p .numpages = numpages, .mask_set = __pgprot(0), .mask_clr = __pgprot(_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY), - .flags = CPA_NO_CHECK_ALIAS }; + .flags = CPA_NO_CHECK_ALIAS | cpa_flags }; /* * No alias checking needed for setting not present flag. otherwise, @@ -2629,20 +2657,20 @@ static int __set_pages_np(struct page *p int set_direct_map_invalid_noflush(struct page *page) { - return __set_pages_np(page, 1); + return __set_pages_np(page, 1, 0); } int set_direct_map_default_noflush(struct page *page) { - return __set_pages_p(page, 1); + return __set_pages_p(page, 1, 0); } int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid) { if (valid) - return __set_pages_p(page, nr); + return __set_pages_p(page, nr, 0); - return __set_pages_np(page, nr); + return __set_pages_np(page, nr, 0); } #ifdef CONFIG_DEBUG_PAGEALLOC @@ -2661,15 +2689,23 @@ void __kernel_map_pages(struct page *pag * and hence no memory allocations during large page split. */ if (enable) - __set_pages_p(page, numpages); + __set_pages_p(page, numpages, CPA_DEBUG_PAGEALLOC); else - __set_pages_np(page, numpages); + __set_pages_np(page, numpages, CPA_DEBUG_PAGEALLOC); /* - * We should perform an IPI and flush all tlbs, - * but that can deadlock->flush only current cpu. - * Preemption needs to be disabled around __flush_tlb_all() due to - * CR3 reload in __native_flush_tlb(). + * We should perform an IPI and flush all tlbs, but that can + * deadlock->flush only current cpu. + * + * Not doing a global TLB flush means that remote CPUs will retain + * stale TLB entries. In case of P->NP (on free) this means the remote + * CPUs will not take the faults, making the debug scheme less + * reliable. On the NP->P (on alloc) this means the remote CPUs can + * take a spurious fault. However spurious_kernel_fault() will observe + * *_present() and fix it up. + * + * Preemption needs to be disabled around __flush_tlb_all() due to CR3 + * reload in __native_flush_tlb(). */ preempt_disable(); __flush_tlb_all();