* [PATCH] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure @ 2026-05-01 7:57 Ye Liu 2026-05-01 10:40 ` Dev Jain 0 siblings, 1 reply; 4+ messages in thread From: Ye Liu @ 2026-05-01 7:57 UTC (permalink / raw) To: Andrew Morton, David Hildenbrand, Lorenzo Stoakes Cc: Ye Liu, Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts, Dev Jain, Barry Song, Lance Yang, linux-mm, linux-kernel From: Ye Liu <liuye@kylinos.cn> __khugepaged_enter() sets MMF_VM_HUGEPAGE before allocating the corresponding mm_slot. If mm_slot_alloc() fails, the function returns with the flag set but without inserting the mm into the khugepaged tracking structures. This leaves the mm in an inconsistent state: it is marked as registered (MMF_VM_HUGEPAGE set), but will never be scanned by khugepaged. Future attempts to register the mm are skipped since khugepaged_enter_vma() checks the flag and returns early. Fix this by clearing MMF_VM_HUGEPAGE when mm_slot_alloc() fails, restoring the ability to retry registration later. Signed-off-by: Ye Liu <liuye@kylinos.cn> --- mm/khugepaged.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 7d48d4fbd5f3..60ab7c1b61dd 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm) return; slot = mm_slot_alloc(mm_slot_cache); - if (!slot) + if (!slot) { + mm_flags_clear(MMF_VM_HUGEPAGE, mm); return; + } spin_lock(&khugepaged_mm_lock); mm_slot_insert(mm_slots_hash, mm, slot); -- 2.43.0 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure 2026-05-01 7:57 [PATCH] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure Ye Liu @ 2026-05-01 10:40 ` Dev Jain 2026-05-01 13:24 ` Lance Yang 0 siblings, 1 reply; 4+ messages in thread From: Dev Jain @ 2026-05-01 10:40 UTC (permalink / raw) To: Ye Liu, Andrew Morton, David Hildenbrand, Lorenzo Stoakes Cc: Ye Liu, Zi Yan, Baolin Wang, Liam R. Howlett, Nico Pache, Ryan Roberts, Barry Song, Lance Yang, linux-mm, linux-kernel On 01/05/26 1:27 pm, Ye Liu wrote: > From: Ye Liu <liuye@kylinos.cn> > > __khugepaged_enter() sets MMF_VM_HUGEPAGE before allocating the > corresponding mm_slot. If mm_slot_alloc() fails, the function > returns with the flag set but without inserting the mm into the > khugepaged tracking structures. > > This leaves the mm in an inconsistent state: it is marked as > registered (MMF_VM_HUGEPAGE set), but will never be scanned by > khugepaged. Future attempts to register the mm are skipped since > khugepaged_enter_vma() checks the flag and returns early. > > Fix this by clearing MMF_VM_HUGEPAGE when mm_slot_alloc() fails, > restoring the ability to retry registration later. > > Signed-off-by: Ye Liu <liuye@kylinos.cn> > --- > mm/khugepaged.c | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/mm/khugepaged.c b/mm/khugepaged.c > index 7d48d4fbd5f3..60ab7c1b61dd 100644 > --- a/mm/khugepaged.c > +++ b/mm/khugepaged.c > @@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm) > return; > > slot = mm_slot_alloc(mm_slot_cache); > - if (!slot) > + if (!slot) { > + mm_flags_clear(MMF_VM_HUGEPAGE, mm); > return; > + } You could just move the test_and_set() after this no? So if slot allocation fails then MMF_VM_HUGEPAGE is never set. Fixes tag is probably required but not sure about Ccing stable. > > spin_lock(&khugepaged_mm_lock); > mm_slot_insert(mm_slots_hash, mm, slot); ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure 2026-05-01 10:40 ` Dev Jain @ 2026-05-01 13:24 ` Lance Yang 2026-05-05 4:17 ` Dev Jain 0 siblings, 1 reply; 4+ messages in thread From: Lance Yang @ 2026-05-01 13:24 UTC (permalink / raw) To: dev.jain, ye.liu Cc: akpm, david, ljs, liuye, ziy, baolin.wang, liam, npache, ryan.roberts, baohua, lance.yang, linux-mm, linux-kernel On Fri, May 01, 2026 at 04:10:58PM +0530, Dev Jain wrote: > > >On 01/05/26 1:27 pm, Ye Liu wrote: >> From: Ye Liu <liuye@kylinos.cn> >> >> __khugepaged_enter() sets MMF_VM_HUGEPAGE before allocating the >> corresponding mm_slot. If mm_slot_alloc() fails, the function >> returns with the flag set but without inserting the mm into the >> khugepaged tracking structures. >> >> This leaves the mm in an inconsistent state: it is marked as >> registered (MMF_VM_HUGEPAGE set), but will never be scanned by >> khugepaged. Future attempts to register the mm are skipped since >> khugepaged_enter_vma() checks the flag and returns early. >> >> Fix this by clearing MMF_VM_HUGEPAGE when mm_slot_alloc() fails, >> restoring the ability to retry registration later. >> >> Signed-off-by: Ye Liu <liuye@kylinos.cn> >> --- >> mm/khugepaged.c | 4 +++- >> 1 file changed, 3 insertions(+), 1 deletion(-) >> >> diff --git a/mm/khugepaged.c b/mm/khugepaged.c >> index 7d48d4fbd5f3..60ab7c1b61dd 100644 >> --- a/mm/khugepaged.c >> +++ b/mm/khugepaged.c >> @@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm) >> return; >> >> slot = mm_slot_alloc(mm_slot_cache); >> - if (!slot) >> + if (!slot) { >> + mm_flags_clear(MMF_VM_HUGEPAGE, mm); >> return; >> + } > >You could just move the test_and_set() after this no? Yep, that sounds better :) Just one small thing: if we move the test_and_set(), after mm_slot_alloc(), we need to free the slot when test_and_set() says that the flag was already set. Otherwise the racing caller that loses would leak the slot :) Something like: slot = mm_slot_alloc(mm_slot_cache); if (!slot) return; if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm))) { mm_slot_free(mm_slot_cache, slot); return; } >So if slot allocation fails then MMF_VM_HUGEPAGE is never set. > >Fixes tag is probably required but not sure about Ccing stable. Yes, Fixes sounds good. khugepaged collapse is best-effort though, anyway :) So I'm also not sure about Ccing stable. >> >> spin_lock(&khugepaged_mm_lock); >> mm_slot_insert(mm_slots_hash, mm, slot); Cheers, Lance ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure 2026-05-01 13:24 ` Lance Yang @ 2026-05-05 4:17 ` Dev Jain 0 siblings, 0 replies; 4+ messages in thread From: Dev Jain @ 2026-05-05 4:17 UTC (permalink / raw) To: Lance Yang, ye.liu Cc: akpm, david, ljs, liuye, ziy, baolin.wang, liam, npache, ryan.roberts, baohua, linux-mm, linux-kernel On 01/05/26 6:54 pm, Lance Yang wrote: > > On Fri, May 01, 2026 at 04:10:58PM +0530, Dev Jain wrote: >> >> >> On 01/05/26 1:27 pm, Ye Liu wrote: >>> From: Ye Liu <liuye@kylinos.cn> >>> >>> __khugepaged_enter() sets MMF_VM_HUGEPAGE before allocating the >>> corresponding mm_slot. If mm_slot_alloc() fails, the function >>> returns with the flag set but without inserting the mm into the >>> khugepaged tracking structures. >>> >>> This leaves the mm in an inconsistent state: it is marked as >>> registered (MMF_VM_HUGEPAGE set), but will never be scanned by >>> khugepaged. Future attempts to register the mm are skipped since >>> khugepaged_enter_vma() checks the flag and returns early. >>> >>> Fix this by clearing MMF_VM_HUGEPAGE when mm_slot_alloc() fails, >>> restoring the ability to retry registration later. >>> >>> Signed-off-by: Ye Liu <liuye@kylinos.cn> >>> --- >>> mm/khugepaged.c | 4 +++- >>> 1 file changed, 3 insertions(+), 1 deletion(-) >>> >>> diff --git a/mm/khugepaged.c b/mm/khugepaged.c >>> index 7d48d4fbd5f3..60ab7c1b61dd 100644 >>> --- a/mm/khugepaged.c >>> +++ b/mm/khugepaged.c >>> @@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm) >>> return; >>> >>> slot = mm_slot_alloc(mm_slot_cache); >>> - if (!slot) >>> + if (!slot) { >>> + mm_flags_clear(MMF_VM_HUGEPAGE, mm); >>> return; >>> + } >> >> You could just move the test_and_set() after this no? > > Yep, that sounds better :) > > Just one small thing: if we move the test_and_set(), after > mm_slot_alloc(), we need to free the slot when test_and_set() says that > the flag was already set. Otherwise the racing caller that loses would > leak the slot :) > > Something like: > > slot = mm_slot_alloc(mm_slot_cache); > if (!slot) > return; > > if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm))) { > mm_slot_free(mm_slot_cache, slot); > return; > } My bad, I missed the fact that test-and-set may also fail. So I think Ye's patch is also fine. > >> So if slot allocation fails then MMF_VM_HUGEPAGE is never set. >> >> Fixes tag is probably required but not sure about Ccing stable. > > Yes, Fixes sounds good. khugepaged collapse is best-effort though, > anyway :) > > So I'm also not sure about Ccing stable. > >>> >>> spin_lock(&khugepaged_mm_lock); >>> mm_slot_insert(mm_slots_hash, mm, slot); > > Cheers, Lance ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-05-05 4:17 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-05-01 7:57 [PATCH] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure Ye Liu 2026-05-01 10:40 ` Dev Jain 2026-05-01 13:24 ` Lance Yang 2026-05-05 4:17 ` Dev Jain
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox