[PATCH v2] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure

[PATCH v2] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure

[Ye Liu]

From: Ye Liu <liuye@kylinos.cn>

__khugepaged_enter() sets MMF_VM_HUGEPAGE before allocating the
corresponding mm_slot.  If mm_slot_alloc() fails, the function
returns with the flag set but without inserting the mm into the
khugepaged tracking structures.

This leaves the mm in an inconsistent state: it is marked as
registered (MMF_VM_HUGEPAGE set), but will never be scanned by
khugepaged.  Future attempts to register the mm are skipped since
khugepaged_enter_vma() checks the flag and returns early.

Fix this by clearing MMF_VM_HUGEPAGE when mm_slot_alloc() fails,
restoring the ability to retry registration later.

Fixes: 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")
Signed-off-by: Ye Liu <liuye@kylinos.cn>
---
Changes since v1:
- Add Fixes tag as suggested by Dev Jain and Lance Yang

 mm/khugepaged.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 7d48d4fbd5f3..60ab7c1b61dd 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm)
 		return;
 
 	slot = mm_slot_alloc(mm_slot_cache);
-	if (!slot)
+	if (!slot) {
+		mm_flags_clear(MMF_VM_HUGEPAGE, mm);
 		return;
+	}
 
 	spin_lock(&khugepaged_mm_lock);
 	mm_slot_insert(mm_slots_hash, mm, slot);
-- 
2.43.0

Re: [PATCH v2] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure

[Lance Yang]

On Wed, May 06, 2026 at 09:21:30AM +0800, Ye Liu wrote:
>From: Ye Liu <liuye@kylinos.cn>
>
>__khugepaged_enter() sets MMF_VM_HUGEPAGE before allocating the
>corresponding mm_slot.  If mm_slot_alloc() fails, the function
>returns with the flag set but without inserting the mm into the
>khugepaged tracking structures.
>
>This leaves the mm in an inconsistent state: it is marked as
>registered (MMF_VM_HUGEPAGE set), but will never be scanned by
>khugepaged.  Future attempts to register the mm are skipped since
>khugepaged_enter_vma() checks the flag and returns early.
>
>Fix this by clearing MMF_VM_HUGEPAGE when mm_slot_alloc() fails,
>restoring the ability to retry registration later.
>
>Fixes: 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")

Nit: the title does not exactly match the original commit subject.

It should be

Fixes: 16618670276a ("mm: khugepaged: avoid pointless allocation for "struct mm_slot"")

which includes quotes around "struct mm_slot".

No need to resend; I think Andrew can fix it up when applying :)

>Signed-off-by: Ye Liu <liuye@kylinos.cn>
>---

Thanks, LGTM.
Reviewed-by: Lance Yang <lance.yang@linux.dev>

Re: [PATCH v2] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure

[Baolin Wang]

On 5/6/26 9:21 AM, Ye Liu wrote:
> From: Ye Liu <liuye@kylinos.cn>
> 
> __khugepaged_enter() sets MMF_VM_HUGEPAGE before allocating the
> corresponding mm_slot.  If mm_slot_alloc() fails, the function
> returns with the flag set but without inserting the mm into the
> khugepaged tracking structures.
> 
> This leaves the mm in an inconsistent state: it is marked as
> registered (MMF_VM_HUGEPAGE set), but will never be scanned by
> khugepaged.  Future attempts to register the mm are skipped since
> khugepaged_enter_vma() checks the flag and returns early.
> 
> Fix this by clearing MMF_VM_HUGEPAGE when mm_slot_alloc() fails,
> restoring the ability to retry registration later.
> 
> Fixes: 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")
> Signed-off-by: Ye Liu <liuye@kylinos.cn>
> ---

Good catch. Feel free to add:
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>

Re: [PATCH v2] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure

[Dev Jain]

On 06/05/26 6:51 am, Ye Liu wrote:
> From: Ye Liu <liuye@kylinos.cn>
> 
> __khugepaged_enter() sets MMF_VM_HUGEPAGE before allocating the
> corresponding mm_slot.  If mm_slot_alloc() fails, the function
> returns with the flag set but without inserting the mm into the
> khugepaged tracking structures.
> 
> This leaves the mm in an inconsistent state: it is marked as
> registered (MMF_VM_HUGEPAGE set), but will never be scanned by
> khugepaged.  Future attempts to register the mm are skipped since
> khugepaged_enter_vma() checks the flag and returns early.
> 
> Fix this by clearing MMF_VM_HUGEPAGE when mm_slot_alloc() fails,
> restoring the ability to retry registration later.
> 
> Fixes: 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")
> Signed-off-by: Ye Liu <liuye@kylinos.cn>
> ---
> Changes since v1:
> - Add Fixes tag as suggested by Dev Jain and Lance Yang
> 
>  mm/khugepaged.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 7d48d4fbd5f3..60ab7c1b61dd 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm)
>  		return;
>  
>  	slot = mm_slot_alloc(mm_slot_cache);
> -	if (!slot)
> +	if (!slot) {
> +		mm_flags_clear(MMF_VM_HUGEPAGE, mm);
>  		return;
> +	}

Note that, a racing khugepaged_enter_vma() may back off
when it sees that MMF_VM_HUGEPAGE is set, but then the above
clears the flag after slot alloc failure. So we end up not
registering the mm with khugepaged. But I am sure no one
cares, we are in much big trouble if slot alloc is failing.

Although one could argue the same about this patch, I will
still say it is important to fix "flag is set but not registered
with khugepaged" because that just feels wrong.

Reviewed-by: Dev Jain <dev.jain@arm.com>


>  
>  	spin_lock(&khugepaged_mm_lock);
>  	mm_slot_insert(mm_slots_hash, mm, slot);

Re: [PATCH v2] mm/khugepaged: clear MMF_VM_HUGEPAGE on mm_slot_alloc() failure

[Lance Yang]

On Wed, May 06, 2026 at 12:16:35PM +0530, Dev Jain wrote:
>
>
>On 06/05/26 6:51 am, Ye Liu wrote:
>> From: Ye Liu <liuye@kylinos.cn>
>> 
>> __khugepaged_enter() sets MMF_VM_HUGEPAGE before allocating the
>> corresponding mm_slot.  If mm_slot_alloc() fails, the function
>> returns with the flag set but without inserting the mm into the
>> khugepaged tracking structures.
>> 
>> This leaves the mm in an inconsistent state: it is marked as
>> registered (MMF_VM_HUGEPAGE set), but will never be scanned by
>> khugepaged.  Future attempts to register the mm are skipped since
>> khugepaged_enter_vma() checks the flag and returns early.
>> 
>> Fix this by clearing MMF_VM_HUGEPAGE when mm_slot_alloc() fails,
>> restoring the ability to retry registration later.
>> 
>> Fixes: 16618670276a ("mm: khugepaged: avoid pointless allocation for struct mm_slot")
>> Signed-off-by: Ye Liu <liuye@kylinos.cn>
>> ---
>> Changes since v1:
>> - Add Fixes tag as suggested by Dev Jain and Lance Yang
>> 
>>  mm/khugepaged.c | 4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>> 
>> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
>> index 7d48d4fbd5f3..60ab7c1b61dd 100644
>> --- a/mm/khugepaged.c
>> +++ b/mm/khugepaged.c
>> @@ -559,8 +559,10 @@ void __khugepaged_enter(struct mm_struct *mm)
>>  		return;
>>  
>>  	slot = mm_slot_alloc(mm_slot_cache);
>> -	if (!slot)
>> +	if (!slot) {
>> +		mm_flags_clear(MMF_VM_HUGEPAGE, mm);
>>  		return;
>> +	}
>
>Note that, a racing khugepaged_enter_vma() may back off
>when it sees that MMF_VM_HUGEPAGE is set, but then the above
>clears the flag after slot alloc failure. So we end up not
>registering the mm with khugepaged. But I am sure no one
>cares, we are in much big trouble if slot alloc is failing.

Right. A racing khugepaged_enter_vma() can see MMF_VM_HUGEPAGE is set
and return, then !slot clears it again. If there is no later
khugepaged_enter_vma(), the mm still wouldn't get registered :)

>Although one could argue the same about this patch, I will
>still say it is important to fix "flag is set but not registered
>with khugepaged" because that just feels wrong.

+1 that is still better then keeping MMF_VM_HUGEPAGE set forever: any
later attempt can retry and install the mm_slot :)

Cheers, Lance