[PATCH v3] mm/khugepaged: fix inconsistent MMF_VM_HUGEPAGE flag due to allocation failure order

[Ye Liu <ye.liu@linux.dev>]

From: Ye Liu <liuye@kylinos.cn>

__khugepaged_enter() sets MMF_VM_HUGEPAGE before allocating the
corresponding mm_slot.  If mm_slot_alloc() fails, the function
returns with the flag set but without inserting the mm into the
khugepaged tracking structures, leaving the mm in an inconsistent
state where future registration attempts are skipped.

Fix this by reordering: allocate the mm_slot first, then check and
set the flag.  If the flag is already set, free the allocated slot
and return.  This ensures the flag is only set when the mm is
successfully registered in the khugepaged tracking structures.

Fixes: 16618670276a ("mm: khugepaged: avoid pointless allocation for "struct mm_slot"")
Suggested-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Ye Liu <liuye@kylinos.cn>
---
Changes since v2:
- Reorder to allocate mm_slot first, free it when flag already set,
  as suggested by David, Dev Jain and Lance Yang
- Update the subject line to better match the patch.
- Link: https://lore.kernel.org/all/20260506012130.9306-1-ye.liu@linux.dev/

Changes since v1:
- Add Fixes tag as suggested by Dev Jain and Lance Yang
- Link: https://lore.kernel.org/all/20260501075708.327217-1-ye.liu@linux.dev/

 mm/khugepaged.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 5f4e009593e0..78735f34250a 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -437,13 +437,16 @@ void __khugepaged_enter(struct mm_struct *mm)
 
 	/* __khugepaged_exit() must not run from under us */
 	VM_BUG_ON_MM(collapse_test_exit(mm), mm);
-	if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm)))
-		return;
 
 	slot = mm_slot_alloc(mm_slot_cache);
 	if (!slot)
 		return;
 
+	if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm))) {
+		mm_slot_free(mm_slot_cache, slot);
+		return;
+	}
+
 	spin_lock(&khugepaged_mm_lock);
 	mm_slot_insert(mm_slots_hash, mm, slot);
 	/*
-- 
2.43.0