[PATCH v2] mm/hugetlb: fix memory offline failure due to hwpoisoned file hugetlb

[PATCH v2] mm/hugetlb: fix memory offline failure due to hwpoisoned file hugetlb

[Jinjiang Tu]

When a file hugetlb folio triggers UCE, me_huge_page() will keep the
hugetlb folio in pagcahe with refcount increased and PG_hwpoison set. Even
after the hugetlb file is deleted, the hugetlb folio is still leaked.

If we want to offline the memory block that the hwpoisoned hugetlb folio
belongs to, it fails in dissolve_free_hugetlb_folios() due to the
hwpoisoned hugetlb folio isn't free.

I can reproduce this issue with the following steps in qemu:
 1) echo offline >/sys/devices/system/memory/auto_online_blocks
 2) in qemu monitor:
       object_add memory-backend-ram,id=mem10,size=1G
       device_add pc-dimm,id=dimm1,memdev=mem10,node=2
 3) echo online_movable > /sys/devices/system/node/node2/memory136/state
 4) echo 5 > /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages
 5) run ./hugetlb_file. This process will receive SIGBUS.
 6) remove the hugetlbfs file.
 7) echo offline > /sys/devices/system/node/node2/memory136/state

hugetlb_file.c:
  fd = open("/dev/hugepages/my_hugepage_file", O_CREAT | O_RDWR, 0755);
  fallocate(fd, 0, 0, HUGEPAGE_SIZE * 2);
  addr = mmap(NULL, HUGEPAGE_SIZE * 2, PROT_READ | PROT_WRITE,
		MAP_SHARED | MAP_HUGETLB, fd, 0);
  memset(addr, 0xaa, HUGEPAGE_SIZE * 2);
  madvise(addr, HUGEPAGE_SIZE, MADV_HWPOISON);

To fix it, force to put ref of hwpoisoned hugetlb in memory offline, the
hwpoisoned hugetlb will be freed and succeeds to be dissolved. We couldn't
avoid races here, just like commit b023f46813cd ("memory-hotplug: skip
HWPoisoned page when offlining pages"), which force to skip hwpoisoned
page regardless of refcount.

Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
---
 mm/hugetlb.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 327eaa4074d3..b7d6c905b4b1 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2115,6 +2115,15 @@ int dissolve_free_hugetlb_folios(unsigned long start_pfn, unsigned long end_pfn)
 
 	for (pfn = start_pfn; pfn < end_pfn; pfn += 1 << order) {
 		folio = pfn_folio(pfn);
+
+		/*
+		 * For hwpoisoned hugetlb, put the refcount increaed by
+		 * memory-failure, make it succeed to dissolve.
+		 */
+		if (unlikely(folio_test_hwpoison(folio) && folio_test_hugetlb(folio)
+				&& folio_ref_count(folio)))
+			folio_put(folio);
+
 		rc = dissolve_free_hugetlb_folio(folio);
 		if (rc)
 			break;
-- 
2.43.0

Re: [PATCH v2] mm/hugetlb: fix memory offline failure due to hwpoisoned file hugetlb

[Andrew Morton]

On Sat, 21 Mar 2026 10:10:31 +0800 Jinjiang Tu <tujinjiang@huawei.com> wrote:

> When a file hugetlb folio triggers UCE, me_huge_page() will keep the
> hugetlb folio in pagcahe with refcount increased and PG_hwpoison set. Even
> after the hugetlb file is deleted, the hugetlb folio is still leaked.
> 
> If we want to offline the memory block that the hwpoisoned hugetlb folio
> belongs to, it fails in dissolve_free_hugetlb_folios() due to the
> hwpoisoned hugetlb folio isn't free.
> 
> I can reproduce this issue with the following steps in qemu:
>  1) echo offline >/sys/devices/system/memory/auto_online_blocks
>  2) in qemu monitor:
>        object_add memory-backend-ram,id=mem10,size=1G
>        device_add pc-dimm,id=dimm1,memdev=mem10,node=2
>  3) echo online_movable > /sys/devices/system/node/node2/memory136/state
>  4) echo 5 > /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages
>  5) run ./hugetlb_file. This process will receive SIGBUS.
>  6) remove the hugetlbfs file.
>  7) echo offline > /sys/devices/system/node/node2/memory136/state
> 
> hugetlb_file.c:
>   fd = open("/dev/hugepages/my_hugepage_file", O_CREAT | O_RDWR, 0755);
>   fallocate(fd, 0, 0, HUGEPAGE_SIZE * 2);
>   addr = mmap(NULL, HUGEPAGE_SIZE * 2, PROT_READ | PROT_WRITE,
> 		MAP_SHARED | MAP_HUGETLB, fd, 0);
>   memset(addr, 0xaa, HUGEPAGE_SIZE * 2);
>   madvise(addr, HUGEPAGE_SIZE, MADV_HWPOISON);
> 
> To fix it, force to put ref of hwpoisoned hugetlb in memory offline, the
> hwpoisoned hugetlb will be freed and succeeds to be dissolved. We couldn't
> avoid races here, just like commit b023f46813cd ("memory-hotplug: skip
> HWPoisoned page when offlining pages"), which force to skip hwpoisoned
> page regardless of refcount.
> 

It would be nice to know what changed since v1 and why, although the
patch is super-tiny so I guess that doesn't matter much.

AI review has a question regarding the v1 patch:
	https://sashiko.dev/#/patchset/20260318020711.3596947-1-tujinjiang@huawei.com

Re: [PATCH v2] mm/hugetlb: fix memory offline failure due to hwpoisoned file hugetlb

[David Hildenbrand (Arm)]

On 3/21/26 03:10, Jinjiang Tu wrote:
> When a file hugetlb folio triggers UCE, me_huge_page() will keep the
> hugetlb folio in pagcahe with refcount increased and PG_hwpoison set. Even
> after the hugetlb file is deleted, the hugetlb folio is still leaked.
> 
> If we want to offline the memory block that the hwpoisoned hugetlb folio
> belongs to, it fails in dissolve_free_hugetlb_folios() due to the
> hwpoisoned hugetlb folio isn't free.
> 
> I can reproduce this issue with the following steps in qemu:
>  1) echo offline >/sys/devices/system/memory/auto_online_blocks
>  2) in qemu monitor:
>        object_add memory-backend-ram,id=mem10,size=1G
>        device_add pc-dimm,id=dimm1,memdev=mem10,node=2
>  3) echo online_movable > /sys/devices/system/node/node2/memory136/state
>  4) echo 5 > /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages
>  5) run ./hugetlb_file. This process will receive SIGBUS.
>  6) remove the hugetlbfs file.
>  7) echo offline > /sys/devices/system/node/node2/memory136/state
> 
> hugetlb_file.c:
>   fd = open("/dev/hugepages/my_hugepage_file", O_CREAT | O_RDWR, 0755);
>   fallocate(fd, 0, 0, HUGEPAGE_SIZE * 2);
>   addr = mmap(NULL, HUGEPAGE_SIZE * 2, PROT_READ | PROT_WRITE,
> 		MAP_SHARED | MAP_HUGETLB, fd, 0);
>   memset(addr, 0xaa, HUGEPAGE_SIZE * 2);
>   madvise(addr, HUGEPAGE_SIZE, MADV_HWPOISON);
> 
> To fix it, force to put ref of hwpoisoned hugetlb in memory offline, the
> hwpoisoned hugetlb will be freed and succeeds to be dissolved. We couldn't
> avoid races here, just like commit b023f46813cd ("memory-hotplug: skip
> HWPoisoned page when offlining pages"), which force to skip hwpoisoned
> page regardless of refcount.

I always considered that handling quite dubious. Just because a page has
hwpoisoned set doesn't mean that we can just offline it.

I think that mus be cleaned up at some point.

But not sure how to do this cleanly.

Why do we even care about offlining memory with hwpoisoned pages? What
is the use case for your change?

I know, it's very desirable to do it, but I much rather have it not
working then having something that is likely mostly broken and actually
might cause harm.

-- 
Cheers,

David

Re: [PATCH v2] mm/hugetlb: fix memory offline failure due to hwpoisoned file hugetlb

[Jinjiang Tu]

在 2026/3/23 20:14, David Hildenbrand (Arm) 写道:
> On 3/21/26 03:10, Jinjiang Tu wrote:
>> When a file hugetlb folio triggers UCE, me_huge_page() will keep the
>> hugetlb folio in pagcahe with refcount increased and PG_hwpoison set. Even
>> after the hugetlb file is deleted, the hugetlb folio is still leaked.
>>
>> If we want to offline the memory block that the hwpoisoned hugetlb folio
>> belongs to, it fails in dissolve_free_hugetlb_folios() due to the
>> hwpoisoned hugetlb folio isn't free.
>>
>> I can reproduce this issue with the following steps in qemu:
>>   1) echo offline >/sys/devices/system/memory/auto_online_blocks
>>   2) in qemu monitor:
>>         object_add memory-backend-ram,id=mem10,size=1G
>>         device_add pc-dimm,id=dimm1,memdev=mem10,node=2
>>   3) echo online_movable > /sys/devices/system/node/node2/memory136/state
>>   4) echo 5 > /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages
>>   5) run ./hugetlb_file. This process will receive SIGBUS.
>>   6) remove the hugetlbfs file.
>>   7) echo offline > /sys/devices/system/node/node2/memory136/state
>>
>> hugetlb_file.c:
>>    fd = open("/dev/hugepages/my_hugepage_file", O_CREAT | O_RDWR, 0755);
>>    fallocate(fd, 0, 0, HUGEPAGE_SIZE * 2);
>>    addr = mmap(NULL, HUGEPAGE_SIZE * 2, PROT_READ | PROT_WRITE,
>> 		MAP_SHARED | MAP_HUGETLB, fd, 0);
>>    memset(addr, 0xaa, HUGEPAGE_SIZE * 2);
>>    madvise(addr, HUGEPAGE_SIZE, MADV_HWPOISON);
>>
>> To fix it, force to put ref of hwpoisoned hugetlb in memory offline, the
>> hwpoisoned hugetlb will be freed and succeeds to be dissolved. We couldn't
>> avoid races here, just like commit b023f46813cd ("memory-hotplug: skip
>> HWPoisoned page when offlining pages"), which force to skip hwpoisoned
>> page regardless of refcount.
> I always considered that handling quite dubious. Just because a page has
> hwpoisoned set doesn't mean that we can just offline it.
>
> I think that mus be cleaned up at some point.
>
> But not sure how to do this cleanly.
>
> Why do we even care about offlining memory with hwpoisoned pages? What
> is the use case for your change?

Considering CXL memory device and we hotplug the memory as NUMA. If the
device is disconnected, accessing the CXL memory will trigger memory-failure.

We still want to offline the memory, so that we can reonline and use the
memory again when CXL memory device is reconnected.

> I know, it's very desirable to do it, but I much rather have it not
> working then having something that is likely mostly broken and actually
> might cause harm.
>

Re: [PATCH v2] mm/hugetlb: fix memory offline failure due to hwpoisoned file hugetlb

[David Hildenbrand (Arm)]

On 3/24/26 07:41, Jinjiang Tu wrote:
> 
> 在 2026/3/23 20:14, David Hildenbrand (Arm) 写道:
>> On 3/21/26 03:10, Jinjiang Tu wrote:
>>> When a file hugetlb folio triggers UCE, me_huge_page() will keep the
>>> hugetlb folio in pagcahe with refcount increased and PG_hwpoison set.
>>> Even
>>> after the hugetlb file is deleted, the hugetlb folio is still leaked.
>>>
>>> If we want to offline the memory block that the hwpoisoned hugetlb folio
>>> belongs to, it fails in dissolve_free_hugetlb_folios() due to the
>>> hwpoisoned hugetlb folio isn't free.
>>>
>>> I can reproduce this issue with the following steps in qemu:
>>>   1) echo offline >/sys/devices/system/memory/auto_online_blocks
>>>   2) in qemu monitor:
>>>         object_add memory-backend-ram,id=mem10,size=1G
>>>         device_add pc-dimm,id=dimm1,memdev=mem10,node=2
>>>   3) echo online_movable > /sys/devices/system/node/node2/memory136/
>>> state
>>>   4) echo 5 > /sys/devices/system/node/node2/hugepages/
>>> hugepages-2048kB/nr_hugepages
>>>   5) run ./hugetlb_file. This process will receive SIGBUS.
>>>   6) remove the hugetlbfs file.
>>>   7) echo offline > /sys/devices/system/node/node2/memory136/state
>>>
>>> hugetlb_file.c:
>>>    fd = open("/dev/hugepages/my_hugepage_file", O_CREAT | O_RDWR, 0755);
>>>    fallocate(fd, 0, 0, HUGEPAGE_SIZE * 2);
>>>    addr = mmap(NULL, HUGEPAGE_SIZE * 2, PROT_READ | PROT_WRITE,
>>>         MAP_SHARED | MAP_HUGETLB, fd, 0);
>>>    memset(addr, 0xaa, HUGEPAGE_SIZE * 2);
>>>    madvise(addr, HUGEPAGE_SIZE, MADV_HWPOISON);
>>>
>>> To fix it, force to put ref of hwpoisoned hugetlb in memory offline, the
>>> hwpoisoned hugetlb will be freed and succeeds to be dissolved. We
>>> couldn't
>>> avoid races here, just like commit b023f46813cd ("memory-hotplug: skip
>>> HWPoisoned page when offlining pages"), which force to skip hwpoisoned
>>> page regardless of refcount.
>> I always considered that handling quite dubious. Just because a page has
>> hwpoisoned set doesn't mean that we can just offline it.
>>
>> I think that mus be cleaned up at some point.
>>
>> But not sure how to do this cleanly.
>>
>> Why do we even care about offlining memory with hwpoisoned pages? What
>> is the use case for your change?
> 
> Considering CXL memory device and we hotplug the memory as NUMA. If the
> device is disconnected, accessing the CXL memory will trigger memory-
> failure.
> 
> We still want to offline the memory, so that we can reonline and use the
> memory again when CXL memory device is reconnected.

Disconnecting a CXL device while the memory is still exposed to Linux,
and using memory-faults for protecting from that?

Oh my, that is horrible!

That's not what memory-failure handling is supposed to be used for. It's
supposed to be used for, you know "memory failure", not disconnected
devices :)

-- 
Cheers,

David