[RFC PATCH 0/2] mm/memory_hotplug: bound offline retry loops with a configurable limit

[Aboorva Devarajan <aboorvad@linux.ibm.com>]

Memory offlining can loop forever when a memory block holds a page that
can never be migrated or freed.  This series adds an opt-in retry limit
that lets offline_pages() bail out with -EBUSY instead of retrying
forever.  The default (0) preserves today's behaviour exactly.

Patch 1 is the fix; patch 2 is an in-tree selftest that reproduces the
hang from an unsignalable kworker and verifies the limit recovers it.

1. The problem
==============

offline_pages() repeats two nested steps until the whole range is
isolated:

  1. Inner loop: find and migrate movable pages out of the range
     (scan_movable_pages() + do_migrate_range()).
  2. Outer loop: re-check isolation (test_pages_isolated()); if not
     fully isolated, go back to step 1.

Neither loop has a termination condition.  When a page can never be
migrated, scan_movable_pages() keeps returning the same pfn,
do_migrate_range() keeps failing on it, and control never even reaches
the outer isolation re-check - the inner loop spins forever.  The admin
guide acknowledges this:

  "Further, memory offlining might retry for a long time (or even
   forever), until aborted by the user."

The single escape in the loop is signal_pending(current):

    offline_pages(pfn, end_pfn):
        do {                                    /* outer: isolation */
            do {                                /* inner: migration  */
                if (signal_pending(current))    /* <-- ONLY escape   */
                        goto failed_removal;
                pfn = scan_movable_pages(pfn, end_pfn);
                do_migrate_range(pfn, end_pfn); /* may never succeed */
            } while (pfn < end_pfn);
        } while (test_pages_isolated(...));


2. Why the kernel itself should be able to bail
===============================================

We keep hitting this during memory hot-remove operations: a single
stuck page can block the whole operation.  This was also reported in
[4] earlier; one example, where offlining kept failing to migrate a
busy block-device page-cache page (aops:def_blk_aops) in a normal zone:

  [10880.889199] page dumped because: migration failure
  [10880.889232] migrating pfn 2a87b failed ret:1
  [10880.889235] page: refcount:3 mapcount:0 mapping:00000000718ec5a6 index:0x857 pfn:0x2a87b
  [10880.889241] aops:def_blk_aops ino:800003 dentry name(?):""
  [10880.889245] flags: 0x33ffffe00004104(referenced|active|private|node=3|zone=0|lastcpupid=0x1fffff)
  ...
  [10880.889291] migrating pfn 2a87b failed ret:1

A userspace-driven abort (timeout + signal) covers the common case
where the offline is requested from process context, e.g. a tool
writing to sysfs or an "echo offline > .../state".  There may
still be scenarios where handling this in the kernel itself is helpful
- for instance, hotplug requests that are processed entirely from
kernel thread context.  An ACPI DIMM eject (kacpi_hotplug_wq) runs
offline_pages() on an ordered workqueue, where there is no task to
receive the signal, so the loop's one escape (signal_pending()) can
never fire and a stuck page also blocks every hotplug event queued
behind it.  For example, an ACPI DIMM eject stuck in the offline loop
looks like:

  task: kworker/u16:3   Workqueue: kacpi_hotplug acpi_hotplug_work_fn
  Call Trace:
   offline_pages+0x...
   memory_subsys_offline+0x...
   device_offline+0x...
   acpi_bus_offline+0x...
   acpi_scan_hot_remove+0x...
   acpi_device_hotplug+0x...
   acpi_hotplug_work_fn+0x...
   process_one_work+0x...
   worker_thread+0x...
   kthread+0x...


3. Prior reports and proposals
==============================

Variants of this have surfaced repeatedly:

[1] 2020: zswap/z3fold pages made offlining loop forever;
    isolate_movable_page() permanently failed and the same pfn was
    retried indefinitely (same "isolation failed" signature as [5]):
    https://lore.kernel.org/all/D90B73BA-22EC-407E-838F-2BA646C60DE0@lca.pw/

[2] 2023: a proposal to bail after 3 consecutive migration failures on
    the same pfn.  Nacked: any hard-coded retry limit leads to premature
    failures; pointer to userspace-driven termination:
    https://lore.kernel.org/all/20230428100846.95535-1-yajun.deng@linux.dev/

[3] 2024: a proposal for a hard-coded limit of 5 migration retries
    (fuse temp-page series v4 4/6); dropped in v5 after review:
    https://lore.kernel.org/linux-mm/20241107235614.3637221-5-joannelkoong@gmail.com/

[4] 2025: ppc64 hot-unplug hang on aops:def_blk_aops migration failures:
    https://lore.kernel.org/all/7ed99822-9d63-4c6c-a492-d4820abebe18@linux.vnet.ibm.com/

[5] 2025: ppc64 hot-unplug hang 20+ hours on a single slab page failing
    isolation, also blocking pcp_batch_high_lock users:
    https://lore.kernel.org/all/d35eca2bbdf8675c43d528571bb61c7520e669cb.camel@linux.ibm.com/

The common failure mode is the same: when migration cannot succeed,
offline_pages() has no way to give up on its own.


4. Why a configurable limit, not a hard-coded one
=================================================

The kernel used to have exactly this bound with a hard-coded limit of 5
migration retries plus a 120s timeout removed by commit 72b39cfc4d75
("mm, memory_hotplug: do not fail offlining too early") because
hard-coded policy caused premature failures on large machines.
That commit explicitly left the door open:

  "If we need some upper bound - e.g. timeout based - then we should
   have a proper and user defined policy for that.  In any case there
   should be a clear use case when introducing it."

In general, it would be good to have this handled in the kernel as
well, by providing a configuration option: a user-defined policy, off
by default, that lets the kernel bound the retries on its own.  This
helps the kworker paths above, which have no userspace to time out or
signal them, but it is not limited to them: it also gives ordinary
process-context offlines a way out when the userspace driver does not
implement its own timeout/abort, so a single stuck page need not wedge
the operation indefinitely.


5. The fix (patch 1)
====================

A single counter at the top of the inner migration loop bounds the
number of migration passes:

    max_passes = READ_ONCE(offline_migrate_max_passes);
    if (max_passes && pass++ >= max_passes) {
            dump_page(stuck page);
            ret = -EBUSY;
            goto failed_removal_isolated;
    }

Both unbounded cases pass through this point: a stuck page spins in the
inner loop without ever reaching the outer test_pages_isolated()
re-check (so an outer-loop counter would never fire), and every
outer-loop restart re-enters the inner loop.  Either way the counter
only grows, so the limit eventually trips.

Because the value is re-read every pass, writing the parameter aborts an
offline that is already stuck - an admin can rescue a hung kworker
without rebooting.  On bailout the existing failure path (shared with
signal backoff and unmovable pages) un-isolates the range, sends
MEM_CANCEL_OFFLINE and returns -EBUSY; the block stays online and
usable, and dump_page() dumps the offending page.

It is a module parameter, default 0 (unlimited = today's behaviour); at
0 the check short-circuits and there is no functional change:
    boot:    memory_hotplug.offline_migrate_max_passes=
    runtime: /sys/module/memory_hotplug/parameters/offline_migrate_max_passes

Alternative implementation: stall timeout
-----------------------------------------

Instead of counting passes, the bound could be a stall timeout: give
up only after some time has elapsed with no forward progress.  Any
successfully migrated page resets the clock, so it fires only on an
offline that is genuinely stuck, not one that is merely slow:

    /* give up only after X ms with ZERO forward progress */
    if (migrated_a_page)
            last_progress = jiffies;        /* any progress resets */
    if (stall_ms && time_after(jiffies,
            last_progress + msecs_to_jiffies(stall_ms)))
            goto failed_removal_isolated;   /* -EBUSY */

We are happy to send v2 in this shape if that or another approach is
preferred.


6. Testing
==========

Patch 2 adds tools/testing/selftests/mm/vmtest_memory_hotplug.sh, a
self-contained VM test (virtme-ng + qemu, boots the just-built kernel,
no rootfs/QMP).  It cold-plugs a pc-dimm, onlines it ZONE_MOVABLE,
vmsplice-pins one page, and ejects it from inside the guest so the
offline runs on the kacpi_hotplug_wq kworker; this reproduces the hang
and then rescues it live.  The vmsplice pin is a deliberate, targeted
reproducer used only to make the hang path reproducible in a test VM.

Actual run (kernel log timestamps trimmed):

    TAP version 13
    1..3
    # DIMM memory blocks: 32 33
    # pinned one page in block memory33
    # ejecting /sys/bus/acpi/devices/PNP0C80:00
    # block memory33 going-offline, holding 10s
    # do_migrate_range+0x1eb/0x260
    # offline_pages+0x361/0x520
    # memory_subsys_offline+0xdb/0x180
    # device_offline+0xd0/0x130
    # acpi_bus_offline+0x11f/0x190
    # acpi_device_hotplug+0x1e8/0x3e0
    # acpi_hotplug_work_fn+0x1e/0x30
    # process_one_work+0x16a/0x310
    # armed offline_migrate_max_passes=50
    # memory offlining [mem 0x108000000-0x10fffffff]: giving up after 50 passes
    ok 1 ACPI eject kworker hangs unbounded in offline_pages()
    ok 2 stuck offline rescued by writing offline_migrate_max_passes
    ok 3 memory block online and usable after the bailout
    # Totals: pass:3 fail:0 xfail:0 xpass:0 skip:0 error:0

The test SKIPs (never FAILs) without virtme-ng/qemu or on non-ACPI
arches.

This RFC is to discuss the approach and find the best way to address
the infinite retry loop in offline_pages().  Please let me know if you
have any comments. 

Thanks,
Aboorva

Aboorva Devarajan (2):
  mm/memory_hotplug: bound offline retry loops with a configurable limit
  selftests/mm: add pc-dimm ACPI eject selftest for offline_migrate_max_passes

 .../admin-guide/kernel-parameters.txt         |  14 +
 .../admin-guide/mm/memory-hotplug.rst         |  26 +-
 mm/memory_hotplug.c                           |  38 ++
 tools/testing/selftests/mm/Makefile           |   3 +
 tools/testing/selftests/mm/config             |   3 +
 .../selftests/mm/vmtest_memory_hotplug.sh     | 369 ++++++++++++++++++
 6 files changed, 452 insertions(+), 1 deletion(-)
 create mode 100755 tools/testing/selftests/mm/vmtest_memory_hotplug.sh

-- 
2.54.0