[PATCH] x86/mm: Revert INVLPGB optimization for set_memory code

[Dave Hansen <dave.hansen@linux.intel.com>]

From: Dave Hansen <dave.hansen@linux.intel.com>

tl;dr: Revert an INVLPGB optimization that did not properly handle
discontiguous virtual addresses.

Full story:

I got a report from some graphics (i915) folks that bisected a
regression in their test suite to 86e6815b316e ("x86/mm: Change
cpa_flush() to call flush_kernel_range() directly").  There was a bit
of flip-flopping on the exact bisect, but the code here does seem
wrong to me. The i915 folks were calling set_pages_array_wc(), so
using the CPA_PAGES_ARRAY mode.

Basically, the 'struct cpa_data' can wrap up all kinds of page table
changes.  Some of these are virtually contiguous, but some are very
much not which is one reason why there are ->vaddr and ->pages arrays.

86e6815b316e made the mistake of assuming that the virtual addresses
in the cpa_data are always contiguous. It got things right when neither
CPA_ARRAY/CPA_PAGES_ARRAY is used, but theoretically wrong when either
of those is used.

In the i915 case, it probably failed to flush some WB TLB entries and
install WC ones, leaving some data in the caches and not flushing it
out to where the device could see it. That eventually caused graphics
problems.

Revert the INVLPGB optimization. It can be reintroduced later, but it
will need to be a bit careful about the array modes.

Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Reported-by: Cui, Ling <ling.cui@intel.com>
Cc: Yu-cheng Yu <yu-cheng.yu@intel.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Hellstrom, Thomas <thomas.hellstrom@intel.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: x86@kernel.org
---

 b/arch/x86/mm/pat/set_memory.c |   20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff -puN arch/x86/mm/pat/set_memory.c~revert-86e6815b316ec0ea8 arch/x86/mm/pat/set_memory.c
--- a/arch/x86/mm/pat/set_memory.c~revert-86e6815b316ec0ea8	2026-04-21 06:56:29.208686362 -0700
+++ b/arch/x86/mm/pat/set_memory.c	2026-04-21 06:57:54.946011984 -0700
@@ -399,6 +399,15 @@ static void cpa_flush_all(unsigned long
 	on_each_cpu(__cpa_flush_all, (void *) cache, 1);
 }
 
+static void __cpa_flush_tlb(void *data)
+{
+	struct cpa_data *cpa = data;
+	unsigned int i;
+
+	for (i = 0; i < cpa->numpages; i++)
+		flush_tlb_one_kernel(fix_addr(__cpa_addr(cpa, i)));
+}
+
 static int collapse_large_pages(unsigned long addr, struct list_head *pgtables);
 
 static void cpa_collapse_large_pages(struct cpa_data *cpa)
@@ -435,7 +444,6 @@ static void cpa_collapse_large_pages(str
 
 static void cpa_flush(struct cpa_data *cpa, int cache)
 {
-	unsigned long start, end;
 	unsigned int i;
 
 	BUG_ON(irqs_disabled() && !early_boot_irqs_disabled);
@@ -445,12 +453,10 @@ static void cpa_flush(struct cpa_data *c
 		goto collapse_large_pages;
 	}
 
-	start = fix_addr(__cpa_addr(cpa, 0));
-	end =   start + cpa->numpages * PAGE_SIZE;
-	if (cpa->force_flush_all)
-		end = TLB_FLUSH_ALL;
-
-	flush_tlb_kernel_range(start, end);
+	if (cpa->force_flush_all || cpa->numpages > tlb_single_page_flush_ceiling)
+		flush_tlb_all();
+	else
+		on_each_cpu(__cpa_flush_tlb, cpa, 1);
 
 	if (!cache)
 		goto collapse_large_pages;
_