From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 26ABA41229C; Wed, 22 Jul 2026 23:41:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784763680; cv=none; b=M9bjkrD++wRpw+WSmVdvGKN1gOrHg3CQjbY3yvGgvBamwNpaQjVOPuo4yleKoIA1Zfv1ya1PXWPCta0UUbYABN2nfjJdP9uSowqAcIWIEZHEYVOY7TzLlEXZpb10WFcmIeIOE7dKVSj1E2a5KiIOPw/yBJKKue3Zlc/Qsc7L+Y4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784763680; c=relaxed/simple; bh=asDu53szvZ8BxoP3xBFFR39bZX7xQtE305TUvB0bAIE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=mqtyfU1yj/pCZAyPIwNw3oMrM4fYltsYXVMEHH9snRDoFFV0vnQ9msaQmjDvZ17BTkOygECfuXwtkAwRml5RakOsc8hf5NcUi/VjTSz7j/rcra0z1PCuOvkdbF65Y4P51Jhfwl13wG5NbR3ZEQlh77xYwJgTfuFR+RgayJWVaXY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lxknJXe7; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lxknJXe7" Received: by smtp.kernel.org (Postfix) with ESMTPS id C6518C2BCC6; Wed, 22 Jul 2026 23:41:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1784763679; bh=asDu53szvZ8BxoP3xBFFR39bZX7xQtE305TUvB0bAIE=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=lxknJXe7l9WWVl1nrdgPiDECtcTR/T0+0QyM5xKGNuyOhOYaOFp6SdHkZle3Cmsoc Fu59LGcdYS7jLw/n7M7E+GtnE/sUYF6xkkhiOcgzVvfurCiYerWm/ECSetMhR0HqYr Vt6Bv0du2qmOM8DxocCyXhex9yRokvfeENUjVAQTSasaITQs72EP77EzOpDWhpXXkR 94vYDqDsQnLa2Sk7FG+RZjISvTQeN0duxCqh3Xsi3DwBYJQLDubjz3HTJwCTv3rUre U5qOfF9SoXIeceLDrvZUJ6fUkWdQNrsvoMBcwtHHgV2nJOwwzGdHCEvowgbkSxYNW0 YyTr1Bi9zTKWw== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A49A3C44536; Wed, 22 Jul 2026 23:41:19 +0000 (UTC) From: Ackerley Tng via B4 Relay Date: Wed, 22 Jul 2026 16:41:10 -0700 Subject: [PATCH v4 02/16] mm: hugetlb: Return -ENOSPC on memcg charge failure Precedence: bulk X-Mailing-List: cgroups@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260722-hugetlb-alloc-failure-fixes-v4-2-88e8b81970dc@google.com> References: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com> In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com> To: Muchun Song , Oscar Salvador , David Hildenbrand , Joshua Hahn , Shakeel Butt , Nhat Pham , Andrew Morton , Peter Xu , Wupeng Ma , fvdl@google.com, rientjes@google.com, jthoughton@google.com, Mike Kravetz , Johannes Weiner , Michal Hocko , Roman Gushchin , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Jonathan Corbet , Shuah Khan , Alex Shi , Yanteng Si , Dongliang Mu , Hongxiang Lou , Miaohe Lin Cc: vannapurve@google.com, erdemaktas@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, Ackerley Tng , stable@vger.kernel.org X-Mailer: b4 0.14.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1784763678; l=2279; i=ackerleytng@google.com; s=20260225; h=from:subject:message-id; bh=wmoXONTSMCE0pZEPoT08cRwobUPLPpiUr/LRWZQPS8U=; b=KDHlw7WdJmh8mcn+h2AhBlYR5bdH3uYOuVSgl+nfxzJXdfEHSQ3u3Zm+FIYqoIRms5b4sL//9 RkxUzY2QUxjAzwB8bEk+AgPBG+by/l/X50vgdac5sR6PTRn695EIGVh X-Developer-Key: i=ackerleytng@google.com; a=ed25519; pk=sAZDYXdm6Iz8FHitpHeFlCMXwabodTm7p8/3/8xUxuU= X-Endpoint-Received: by B4 Relay for ackerleytng@google.com/20260225 with auth_id=649 X-Original-From: Ackerley Tng Reply-To: ackerleytng@google.com From: Ackerley Tng When mem_cgroup_charge_hugetlb() fails with -ENOMEM, alloc_hugetlb_folio() currently propagates this error. This results in the page fault handler returning VM_FAULT_OOM. Because HugeTLB allocations are high-order and use __GFP_RETRY_MAYFAIL, they bypass the OOM killer. Returning VM_FAULT_OOM to the #PF handler without triggering the OOM killer (or having it make progress) leads to an infinite loop of retrying the fault. Avoid this loop by returning -ENOSPC when charging fails, which maps to VM_FAULT_SIGBUS, terminating the process cleanly. Make mem_cgroup_charge_hugetlb() fault handling use a common error handling path, the same handling used for hugetlb_cgroup_uncharge_cgroup{,_rsvd}(), which also don't trigger the OOM killer and hence opt to terminate the process with a SIGBUS. Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb") Cc: stable@vger.kernel.org Signed-off-by: Ackerley Tng Reviewed-by: Muchun Song Tested-by: Joshua Hahn Reviewed-by: Joshua Hahn --- mm/hugetlb.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 36fa3fb3945d8..15f9c5a9f75e2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3010,7 +3010,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, if (ret == -ENOMEM) { free_huge_folio(folio); - return ERR_PTR(-ENOMEM); + goto err; } return folio; @@ -3035,6 +3035,16 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, out_end_reservation: if (map_chg != MAP_CHG_ENFORCED) vma_end_reservation(h, vma, addr); +err: + /* + * Return -ENOSPC when this function fails to allocate or charge a huge + * page. If a standard (PAGE_SIZE) page allocation fails, the OOM killer + * is given a chance to run, which may resolve the failure on + * retry. However, for HugeTLB allocations, the OOM killer is not + * triggered. Returning -ENOMEM (or anything resulting in VM_FAULT_OOM) + * would leak to the #PF handler, causing it to loop indefinitely + * retrying the fault. + */ return ERR_PTR(-ENOSPC); } -- 2.55.0.229.g6434b31f56-goog