Re: [PATCH v2 4/5] mm: hugetlb: Return -ENOSPC on memcg charge failure

[Joshua Hahn <joshua.hahnjy@gmail.com>]

On Wed, 08 Jul 2026 15:12:52 -0700 Ackerley Tng via B4 Relay <devnull+ackerleytng.google.com@kernel.org> wrote:

> From: Ackerley Tng <ackerleytng@google.com>
> 
> When mem_cgroup_charge_hugetlb() fails with -ENOMEM, alloc_hugetlb_folio()
> currently propagates this error. This results in the page fault handler
> returning VM_FAULT_OOM.
> 
> Because HugeTLB allocations are high-order and use __GFP_RETRY_MAYFAIL,
> they bypass the OOM killer. Returning VM_FAULT_OOM to the #PF handler
> without triggering the OOM killer (or having it make progress) leads to
> an infinite loop of retrying the fault.
> 
> Avoid this loop by returning -ENOSPC when charging fails, which maps to
> VM_FAULT_SIGBUS, terminating the process cleanly.
> 
> Make mem_cgroup_charge_hugetlb() fault handling use a common error handling
> path, the same handling used for hugetlb_cgroup_uncharge_cgroup{,_rsvd}(),
> which also don't trigger the OOM killer and hence opt to terminate the
> process with a SIGBUS.
> 
> Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
> Cc: stable@vger.kernel.org

Tested-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>

> Signed-off-by: Ackerley Tng <ackerleytng@google.com>
> Reviewed-by: Muchun Song <muchun.song@linux.dev>
> ---
>  mm/hugetlb.c | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 1f3f4b964b153..3e1d99f03c70e 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -2991,7 +2991,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
>  
>  	if (ret == -ENOMEM) {
>  		folio_put(folio);
> -		return ERR_PTR(-ENOMEM);
> +		goto err;
>  	}
>  
>  	return folio;
> @@ -3014,6 +3014,17 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
>  out_end_reservation:
>  	if (map_chg != MAP_CHG_ENFORCED)
>  		vma_end_reservation(h, vma, addr);

NIT: is there a reason why the comment block below doesn't go all the
way up to 80 columns? I also think that we can achieve the same effect
with a shorter comment block : -) I think the essence is to answer
"Why -ENOSPC and not -ENOMEM?"

> +err:
> +	/*
> +	 * Return -ENOSPC when this function fails to allocate or
> +	 * charge a huge page. If a standard (PAGE_SIZE) page
> +	 * allocation fails, the OOM killer is given a chance to run,
> +	 * which may resolve the failure on retry. However, for
> +	 * HugeTLB allocations, the OOM killer is not triggered.
> +	 * Returning -ENOMEM (or anything resulting in VM_FAULT_OOM)
> +	 * would leak to the #PF handler, causing it to loop
> +	 * indefinitely retrying the fault.
> +	 */
>  	return ERR_PTR(-ENOSPC);
>  }

But this fix looks good otherwise and I don't think this a big deal : -)

Thanks Ackerley!
Joshua