* Re: [RFC] PREEMPT_RT: livelock between lockless page cache lookup and the folio freeze->removal window
@ 2026-11-16 12:35 董辉
0 siblings, 0 replies; 3+ messages in thread
From: 董辉 @ 2026-11-16 12:35 UTC (permalink / raw)
To: Matthew Wilcox, linux-rt-devel@lists.linux.dev,
linux-fsdevel@vger.kernel.org
Cc: linux-mm@kvack.org, Andrew Morton, Sebastian Andrzej Siewior,
田孝斌, 李鹏程,
宁左斌, 宁博
Hello,
Thank you for the review. Nothing nicer surfaced on our side either,
so rather than let the thread stall we went ahead and implemented
direction B -- mostly to make the trade-off concrete and measurable.
The patch is inline at the end of this mail; the commit message
explains it. In short: every 64 failed attempts the reader waits for
the xarray lock once instead of spinning, so on PREEMPT_RT it sleeps
and priority inheritance boosts the holder. Non-RT is unaffected.
In a kprobe-driven reproducer that forces the race, with a second
SCHED_FIFO task boosting the holder as in our field case, the unfixed
kernel livelocks on every run, in each of the four lookup loops the
patch covers. With the patch, in 10-second runs of each loop, every
window hit took the wait path (87k-396k hits per loop), the longest
wait was 98 us, and nothing spun. Logs and reproducer code are
available on request.
On your comments:
- Direction A: agreed. The window also contains a sleeping lock:
__filemap_remove_folio() -> page_cache_delete() -> xas_store() ->
workingset_update_node() -> list_lru_add_obj() takes nlru->lock, a
spinlock_t that sleeps on PREEMPT_RT. So preempt_disable() around
the window would first need the shadow-node LRU update moved out of
xas_store(), an xarray/workingset change for every page cache
deletion.
On converting nlru->lock to a raw spinlock: that would remove the
sleeping lock from the window, but it is the generic list_lru lock,
and the isolate callbacks of every list_lru run under it.
shadow_lru_isolate() itself trylocks the xarray lock and i_lock
there, which are sleeping locks on PREEMPT_RT. So it is not a local
change. We have not pursued it, but are happy to look into it if
that is the direction you prefer.
- "spins on the folio refcount": you are right, that is the correct
wording; the patch uses it.
- Thanks for __folio_freeze_split_file(). It is not in 6.12, so we
will check it against the rule in the comment when rebasing onto
mainline.
- On "refcount == 0 is transient" being intentional but undocumented:
the patch adds a comment spelling out the rule the fallback depends
on -- whoever freezes a page cache folio holds the xarray lock until
the folio is either gone from i_pages or unfrozen. That is what
makes waiting for the lock enough, and any future freezer that
follows the rule is covered as well.
The patch is against mm/filemap.c of v6.12.73 (unmodified upstream
stable); we will rebase onto mainline before a proper submission.
Comments very welcome.
Best regards,
Hui Dong
-- >8 --
From: Hui Dong <donghui@xiaomi.com>
Subject: [RFC PATCH] mm/filemap: bound the lockless lookup retry loop for
PREEMPT_RT
On PREEMPT_RT, xa_lock_irq() becomes an rt_mutex that disables neither
interrupts nor preemption, and rt_spin_lock() additionally pins the
holder to its CPU with migrate_disable().
The reclaim and migration paths freeze a folio's refcount while the
folio is still in i_pages, and keep it reachable there until it is
deleted or replaced, all under the xarray lock. On a non-RT kernel
that window runs with interrupts disabled, so the lockless readers that
spin on the frozen refcount terminate after a few iterations.
On PREEMPT_RT the window is preemptible. When the holder is preempted
by a higher-priority task that then faults on the very page being
removed, that task retries in filemap_get_entry() without ever yielding
the CPU or joining the rt_mutex wait list, so priority inheritance
cannot see it. A SCHED_NORMAL holder then only runs on the fair
server's budget, and the reader stalls for up to a second. Once
another task blocks on the xarray lock and boosts the holder to the
spinning task's priority, as in our case, the holder is out of the fair
server's reach, and SCHED_FIFO does not round-robin equal priorities,
so the holder never runs again. The reader also keeps its RCU read
side open, which stalls grace periods.
Bound the spinning instead: every 64 attempts, wait for the xarray
lock once before retrying. On PREEMPT_RT that wait queues the
reader on the rt_mutex: it sleeps, gives the CPU back, and priority
inheritance boosts whoever is holding the window open. By the time the
lock has been acquired the holder has left its critical section, so the
retry that follows sees the final state of the slot. This is the same
fallback that commit 0f8821da4845 ("fs/namespace: Boost the
mount_lock.lock owner instead of spinning on PREEMPT_RT.") added to
mnt_get_write_access().
The wait is rare, so each lookup tells lockdep about the xarray lock up
front with might_lock(), as mnt_get_write_access() does for mount_lock.
Lockdep then checks every caller that may end up waiting, not only the
ones that actually did.
The bound is compiled out on non-RT kernels, so their behaviour is
unchanged, and callers that cannot sleep keep retrying as before.
Covered loops: filemap_get_entry(), find_get_entry(),
filemap_get_folios_contig() and filemap_get_read_batch();
ksm_get_folio() has no mapping at hand and is not covered.
Tested with a deterministic in-kernel reproducer that forces the race:
a SCHED_FIFO 99 reader is woken inside the freeze -> removal window and
looks up the frozen folio through each of the four loops, while a
second SCHED_FIFO 99 task blocks on the xarray lock and boosts the
holder. Without the patch the reader spins forever in every loop; with
it, every window hit took the wait path, the longest wait was 98 us,
and no reader spun.
Signed-off-by: Hui Dong <donghui@xiaomi.com>
---
mm/filemap.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 90 insertions(+), 3 deletions(-)
diff --git a/mm/filemap.c b/mm/filemap.c
index d8d9c0f0beb6..19959ad60fef 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1820,6 +1820,68 @@ EXPORT_SYMBOL(page_cache_prev_miss);
* folio_put().
*/
+/*
+ * PREEMPT_RT: the lookup loops below retry for as long as they find a folio
+ * whose refcount was frozen (step A above) but that is still in i_pages.
+ * Whoever freezes a page cache folio holds the xarray lock until it is
+ * resolved: __remove_mapping() until the folio has been deleted, the
+ * migration paths until it has been replaced and unfrozen.
+ *
+ * Without PREEMPT_RT the lock is held with interrupts disabled, so the
+ * retries end quickly. With PREEMPT_RT xa_lock_irq() is an rt_mutex that
+ * leaves interrupts and preemption enabled, and rt_spin_lock() pins the
+ * holder to its CPU with migrate_disable(). If the holder is preempted by
+ * the task that then retries here, the holder may never run again: the
+ * retry loop does not sleep, and priority inheritance cannot see a task
+ * that is not waiting on the lock. The task also keeps its RCU read side
+ * open, so grace periods stall as well.
+ *
+ * So on PREEMPT_RT, every FILEMAP_LOOKUP_RETRIES attempts, wait for the
+ * xarray lock once instead. The reader sleeps on the rt_mutex, which gives
+ * the CPU back and boosts the holder through priority inheritance. Once the
+ * lock has been acquired the holder has left its critical section, and the
+ * retry that follows sees the final state of the slot. Retry entries are
+ * counted too: they are transient, so waiting is merely unnecessary there.
+ *
+ * The wait is rare, so the loops tell lockdep about it up front with
+ * filemap_lookup_might_wait(): every caller that may end up waiting is then
+ * checked against the xarray lock, not only the ones that actually did.
+ */
+#define FILEMAP_LOOKUP_RETRIES64
+
+static inline void filemap_lookup_might_wait(struct xa_state *xas)
+{
+if (IS_ENABLED(CONFIG_PREEMPT_RT) && preemptible())
+might_lock(&xas->xa->xa_lock);
+}
+
+static inline bool filemap_lookup_should_wait(unsigned int *retries)
+{
+return IS_ENABLED(CONFIG_PREEMPT_RT) &&
+ unlikely(!(++*retries % FILEMAP_LOOKUP_RETRIES));
+}
+
+/*
+ * Wait for the holder of the xarray lock, see above. Called with the RCU
+ * read lock held and without a reference on the entry @xas points to;
+ * returns the same way, with @xas reset so that the caller retries the same
+ * index. The caller's own state stays valid: RCU only protects the walk,
+ * and folios collected so far hold their own references.
+ *
+ * Callers that cannot sleep keep retrying, as they did before.
+ */
+static noinline void filemap_lookup_wait(struct xa_state *xas)
+{
+xas_reset(xas);
+if (!preemptible())
+return;
+
+rcu_read_unlock();
+xas_lock_irq(xas);
+xas_unlock_irq(xas);
+rcu_read_lock();
+}
+
/*
* filemap_get_entry - Get a page cache entry.
* @mapping: the address_space to search
@@ -1836,9 +1898,13 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index)
{
XA_STATE(xas, &mapping->i_pages, index);
struct folio *folio;
+unsigned int retries = 0;
rcu_read_lock();
+filemap_lookup_might_wait(&xas);
repeat:
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(&xas);
xas_reset(&xas);
folio = xas_load(&xas);
if (xas_retry(&xas, folio))
@@ -1998,15 +2064,20 @@ static inline struct folio *find_get_entry(struct xa_state *xas, pgoff_t max,
xa_mark_t mark)
{
struct folio *folio;
+unsigned int retries = 0;
+filemap_lookup_might_wait(xas);
retry:
if (mark == XA_PRESENT)
folio = xas_find(xas, max);
else
folio = xas_find_marked(xas, max, mark);
-if (xas_retry(xas, folio))
+if (xas_retry(xas, folio)) {
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(xas);
goto retry;
+}
/*
* A shadow entry of a recently evicted page, a swap
* entry from shmem/tmpfs or a DAX entry. Return it
@@ -2026,6 +2097,8 @@ static inline struct folio *find_get_entry(struct xa_state *xas, pgoff_t max,
return folio;
reset:
xas_reset(xas);
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(xas);
goto retry;
}
@@ -2194,13 +2267,18 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,
XA_STATE(xas, &mapping->i_pages, *start);
unsigned long nr;
struct folio *folio;
+unsigned int retries = 0;
rcu_read_lock();
+filemap_lookup_might_wait(&xas);
for (folio = xas_load(&xas); folio && xas.xa_index <= end;
folio = xas_next(&xas)) {
-if (xas_retry(&xas, folio))
+if (xas_retry(&xas, folio)) {
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(&xas);
continue;
+}
/*
* If the entry has been swapped out, we can stop looking.
* No current caller is looking for DAX entries.
@@ -2230,6 +2308,8 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,
retry:
xas_reset(&xas);
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(&xas);
}
update_start:
@@ -2336,11 +2416,16 @@ static void filemap_get_read_batch(struct address_space *mapping,
{
XA_STATE(xas, &mapping->i_pages, index);
struct folio *folio;
+unsigned int retries = 0;
rcu_read_lock();
+filemap_lookup_might_wait(&xas);
for (folio = xas_load(&xas); folio; folio = xas_next(&xas)) {
-if (xas_retry(&xas, folio))
+if (xas_retry(&xas, folio)) {
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(&xas);
continue;
+}
if (xas.xa_index > max || xa_is_value(folio))
break;
if (xa_is_sibling(folio))
@@ -2363,6 +2448,8 @@ static void filemap_get_read_batch(struct address_space *mapping,
folio_put(folio);
retry:
xas_reset(&xas);
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(&xas);
}
rcu_read_unlock();
}
#/******本邮件及其附件含有小米公司的保密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件! This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#
^ permalink raw reply related [flat|nested] 3+ messages in thread* [RFC] PREEMPT_RT: livelock between lockless page cache lookup and the folio freeze->removal window
@ 2026-11-03 12:46 董辉
2026-11-04 20:36 ` Matthew Wilcox
0 siblings, 1 reply; 3+ messages in thread
From: 董辉 @ 2026-11-03 12:46 UTC (permalink / raw)
To: linux-rt-devel@lists.kernel.org
Cc: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Andrew Morton,
"Matthew Wilcox (Oracle)", 田孝斌,
李鹏程, 宁左斌,
宁博
[-- Attachment #1: Type: text/plain, Size: 15686 bytes --]
Hello,
We hit a permanent livelock on a PREEMPT_RT system and traced it to an
interaction between the lockless page cache lookup retry loop in
filemap_get_entry() and the folio freeze -> xarray removal window in
__remove_mapping() (and the migrate equivalents). On non-RT that window
is atomic (IRQs disabled) and therefore bounded; on PREEMPT_RT it is
preemptible and can become unbounded, which turns the "refcount == 0 is
transient" assumption of the lookup path into a permanent spin.
The affected code is vanilla upstream (our mm/filemap.c, mm/vmscan.c
and mm/migrate.c match 6.12 exactly); nothing vendor-specific is
involved.
The obvious fix -- making the window non-preemptible on RT -- does not
work for the reclaim path, which is the one we hit (section 4.1). We are
deliberately not sending a patch yet: we would like to hear which
fix direction upstream prefers before going further.
1. Symptom
----------
- 6.12.73-rt15 (PREEMPT_RT), arm64, 20 CPUs.
- A SCHED_FIFO 99 userspace workload was page-faulting on file-backed
code pages of a shared library (libc.so.6 in our case) while the system
was under memory pressure.
- One CPU reported rcu_preempt self-detected stall every ~63 s from then
on: 670+ reports over the ~12 hours covered by our captured logs.
- The next day, and again about two days after the first stall, we
inspected the machine over JTAG. It was still stuck, with the same
register values as in the first reports. It never recovered on its
own; only a reboot brought it back.
- On that CPU the running task was spinning in filemap_get_entry() on a
folio whose _refcount was 0 (frozen) while the folio was still
reachable from the mapping's xarray.
- kswapd0 was found preempted inside __remove_mapping(), its stack
ending in __filemap_remove_folio() -> filemap_unaccount_folio() ->
... -> preempt_schedule(): the folio was already frozen (refcount 0,
off-LRU, unmapped) but not yet removed from the xarray, and the task
still held the xarray lock.
All 670 stall reports show the same task and the same folio. Their PC
samples fall inside this retry loop, mostly in xas_load(); the path
taken after a successful refcount increment was never sampled. A
second, non-RT process later hung in the same loop on the same folio,
showing that once such an entry exists, every reader of that page is a
victim.
2. Root cause
-------------
2.1 Reader: unbounded retry on refcount == 0
/* mm/filemap.c, filemap_get_entry() */
rcu_read_lock();
repeat:
xas_reset(&xas);
folio = xas_load(&xas);
if (xas_retry(&xas, folio))
goto repeat;
/*
* A shadow entry of a recently evicted page, or a swap entry from
* shmem/tmpfs. Return it without attempting to raise page count.
*/
if (!folio || xa_is_value(folio))
goto out;
if (!folio_try_get(folio))
goto repeat;
The loop treats refcount == 0 as a transient state and retries. It never
sleeps or yields while doing so, and there is no retry bound, no timeout
and no warning.
2.2 Writer: the window is bounded on non-RT
/* mm/vmscan.c, __remove_mapping() */
if (!folio_test_swapcache(folio))
spin_lock(&mapping->host->i_lock);
xa_lock_irq(&mapping->i_pages);
...
refcount = 1 + folio_nr_pages(folio);
if (!folio_ref_freeze(folio, refcount))
goto cannot_free;
/* note: atomic_cmpxchg in folio_ref_freeze provides the smp_rmb */
...
__filemap_remove_folio(folio, shadow);
xa_unlock_irq(&mapping->i_pages);
...
cannot_free:
xa_unlock_irq(&mapping->i_pages);
Readers on other CPUs can briefly observe the frozen folio even on
non-RT -- that is what the retry loop is for. On non-RT, however,
xa_lock_irq() disables IRQs and preemption, so the interval from
folio_ref_freeze() to __filemap_remove_folio() cannot be preempted
and cannot outlive the locked code path: the window is bounded, and
the retry terminates after a few iterations.
2.3 On PREEMPT_RT the window is unbounded -- this is the break
On PREEMPT_RT xa_lock_irq() maps to rt_spin_lock(): an rt_mutex
plus migrate_disable() and rcu_read_lock(). It does not disable
preemption. A task inside the freeze -> removal window can therefore be
preempted, and the frozen folio stays visible in the xarray with
refcount == 0 for as long as the holder is off-CPU -- the window is no
longer bounded by the locked code path, but by scheduling.
migrate_disable() aggravates this: the holder is pinned to the CPU it
was preempted on. The holder does not have to be kswapd: the same
shrink_folio_list() -> __remove_mapping() path is used by direct
reclaim, so any task allocating memory can be the victim.
2.4 How it becomes permanent: priority inversion
In our case a SCHED_FIFO 99 task preempted the holder on the same CPU,
then faulted on the very page being removed and entered the
filemap_get_entry() spin. Two further SCHED_FIFO 99 threads of the
same workload blocked on the same xarray lock; through priority
inheritance the holder (kswapd) was boosted into the RT scheduling class
at the same effective priority as the spinner. SCHED_FIFO does not
time-slice among equal priorities, so a spinner that never blocks never
gives the CPU back to the holder.
This is where self-healing is lost. While the holder remains CFS, the
RT bandwidth limit (sched_rt_runtime_us, 950 ms per 1 s period by
default) reserves CPU time for CFS tasks exactly so that a run-away RT
task cannot lock up the machine (see
Documentation/scheduler/sched-rt-group.rst); the holder can still run
and close the window. Once PI pulls the holder into the RT class it is
throttled together with the spinner and no longer benefits from that
reserve. (With sched_rt_runtime_us = -1 even the CFS escape does not
exist.)
The spinner is invisible to PI: it waits on the folio, not on the lock,
so nothing ever boosts the holder on its behalf.
The four necessary conditions:
(a) frozen-but-visible folio -> reader spins
where: filemap_get_entry() retry loop
(b) window unbounded (holder preemptible, pinned)
where: PREEMPT_RT xa_lock_irq() semantics
(c) unyielding spinner on the same CPU
where: SCHED_FIFO 99 in the retry loop
(d) PI removes the holder's CFS bandwidth reserve
where: RT waiters on the same xarray lock
3. Blast radius
---------------
We went through every folio_ref_freeze()/page_ref_freeze() user in
6.12. Two of them can freeze a regular file folio and leave it
reachable from the page cache xarray, and on PREEMPT_RT both can be
preempted there:
- mm/vmscan.c, __remove_mapping()
trigger: reclaim -- kswapd or direct reclaim
- mm/migrate.c, __folio_migrate_mapping()
trigger: compaction / CMA / hotplug
Both hold the xarray lock from before the freeze until the folio has
been removed, or replaced and unfrozen.
The unbounded refcount == 0 spin exists in several lookup loops, not
just filemap_get_entry():
- filemap_get_entry() -- goto repeat, no bound
- find_get_entry() -- goto reset/goto retry, no bound
- filemap_get_folios_contig(), filemap_get_read_batch() -- goto retry,
no bound
- ksm_get_folio() -- while (!folio_try_get(folio)) cpu_relax() with a
swapcache-flag escape
ksm.c documents the transient state explicitly ("however, it might
mean that the page is under page_ref_freeze()..."), which suggests the
frozen-but-visible window is a known state of the design -- the filemap
loops simply have no bound on it.
4. Fix directions we have considered
------------------------------------
4.1 Direction A (writer side): restore non-RT atomicity of the window
The idea: on PREEMPT_RT, wrap the freeze -> xarray-removal interval in
preempt_disable()/preempt_enable(), keeping the lock acquisition
itself RT-semantic (it may sleep while waiting). Readers stay untouched
and "refcount == 0 is transient" holds again, for every reader.
By inspection this works for __folio_migrate_mapping(). Its window,
from the freeze to the xas_store() of the new folio, contains nothing
that sleeps on RT: that xas_store() call is not wrapped by
mapping_set_update(), so no update hook fires.
It does not work for __remove_mapping(), the site we hit. The
removal itself takes a spinlock_t whenever shadow-node tracking changes:
__filemap_remove_folio()
page_cache_delete()
mapping_set_update() /* xa_update = workingset_update_node */
xas_store(&xas, shadow)
update_node()
xas_update()
workingset_update_node()
list_lru_add_obj() / list_lru_del_obj()
spin_lock(&nlru->lock) /* spinlock_t */
__delete_from_swap_cache() installs the same hook, so the swapcache
branch is affected as well. This happens whenever an xa_node becomes
shadow-only or stops being so, which is routine when reclaim evicts a
run of file pages. shadow_nodes is a single list_lru whose per-node
lock is shared by all mappings and by the shadow-node shrinker, so it
can be contended. Unlike the other sleeping call we found, it cannot be
moved out of the window: it is taken from inside the very xas_store()
the window exists to cover.
(The other sleeping call in this function, mem_cgroup_swapout() in the
swapcache branch, could be moved: with memsw accounting it takes
swap_cgroup's ctrl->lock in swap_cgroup_record(). It would have to
stay under the xarray lock though, because memcg1_swapout() relies on
interrupts being disabled by it on non-RT.)
Making direction A work for reclaim would need the shadow-node list_lru
update to be decoupled from xas_store(), for example deferred until
after xa_unlock_irq(). That is an xarray/workingset change affecting
every page cache and swap cache deletion, which we do not want to
attempt without guidance.
4.2 Direction B (reader side): bounded retry + blocking fallback
Every freezer listed in section 3 (Blast radius) holds the xarray lock until the folio
is no longer frozen-and-visible. So a reader that keeps failing can
simply wait for that lock: on PREEMPT_RT it sleeps on the rt_mutex,
gives the CPU back, and priority inheritance boosts the holder. Once the
lock has been acquired the window is closed, and the retry sees the
final state of the slot.
This is the same fallback commit 0f8821da4845 uses in
mnt_get_write_access() (fs/namespace.c): on PREEMPT_RT the spinner on
MNT_WRITE_HOLD stops spinning and takes the writer's lock instead,
specifically to avoid this livelock.
The change would be small (mm/filemap.c only, we estimate about 80 lines):
- Applies to the four filemap loops listed in section 3 (Blast radius).
- Counts failed attempts; every 64 of them the reader calls
rcu_read_unlock(), then xas_lock_irq() + xas_unlock_irq(), then
rcu_read_lock(), resets the xa_state and retries the same index.
Nothing is skipped and nothing is returned early, so semantics do not
change: callers such as truncation still see every entry.
- Only on PREEMPT_RT: the check compiles away otherwise, so non-RT
behavior is identical.
- Only when preemptible(): callers that cannot sleep, such as some
BPF/perf paths reaching build_id_parse_nofault(), keep retrying as
before. An outer rcu_read_lock() does not need special handling:
PREEMPT_RT uses preemptible RCU, which allows blocking in a
read-side section (at the cost of a longer grace period).
We have not prototyped or tested this yet; the sketch is here so the
trade-off can be judged before we invest in a patch series.
- Pros: covers the reclaim site that direction A cannot; covers every
freezer that follows the "hold the xarray lock" rule, including ones
added later; adds no non-preemptible window.
- Cons: the lookups can now sleep on PREEMPT_RT in preemptible context,
and we have not audited all of their callers. Two things need checking:
that no caller holds a lock which is taken inside the xarray lock
(we spot-checked list_lru walkers and found no such caller, but this
was not exhaustive), and that every caller keeps the mapping alive
across the rcu_read_unlock(), not just by RCU.
- Not covered: callers that cannot sleep can still spin, and so can
ksm_get_folio(), which has no mapping at hand to wait on. KSM is
disabled on our system, so we did not look further into it.
Our current view is that B addresses the path we hit, while A, at best,
narrows the problem to the migrate path -- but we may be missing a
reason why upstream would rather restructure the writer side.
5. Questions
------------
1. Is the "refcount == 0 is transient" assumption of the lockless
lookup path documented as depending on the freeze -> removal window
being non-preemptible? If so, PREEMPT_RT breaking it is arguably an
RT-side invariant break; if not, should the lookup loops grow a
bound?
2. Given that the reclaim site cannot be made non-preemptible as long as
workingset_update_node() takes the list_lru spinlock_t from inside
xas_store(): is a reader-side fallback like direction B acceptable,
or would upstream rather decouple the shadow-node LRU update from
xas_store() and then fix the writers?
3. For direction B: is it acceptable that these lookups may sleep on
the xarray lock on PREEMPT_RT, in preemptible context only (the
same fallback commit 0f8821da4845 uses in mnt_get_write_access())?
Is
"every freezer holds the xarray lock until the folio is no longer
frozen-and-visible" an invariant we may rely on, and should it be
documented next to the lockless page cache protocol comment?
4. Are there other freeze sites we have missed?
5. Would a WARN_ON_ONCE() on excessive retry be acceptable as a
diagnostic, independent of the chosen fix?
6. Reproducer sketch
--------------------
1. Pin a SCHED_FIFO 99 task to CPU X; have it repeatedly page-fault on
file-backed code pages of a shared library under reclaim pressure.
2. Drive reclaim (memory pressure is enough; direct reclaim works too)
so a task runs __remove_mapping() for one of those pages on CPU X.
3. The FIFO task preempts the holder inside the freeze -> removal window
and faults on the frozen page -> spin.
4. Two more FIFO 99 threads of the same process fault on other pages of
the same mapping to block on the xarray lock and trigger PI.
Anonymized evidence (register sampling, stack traces of the three RT
threads and kswapd, decoded structure dumps) is available on request.
If direction B is the preferred one, we will turn the sketch above into
a patch series and send it for review.
Best regards,
Donghui
#/******本邮件及其附件含有小米公司的保密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件! This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#
[-- Attachment #2: Type: text/html, Size: 39729 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [RFC] PREEMPT_RT: livelock between lockless page cache lookup and the folio freeze->removal window
2026-11-03 12:46 董辉
@ 2026-11-04 20:36 ` Matthew Wilcox
0 siblings, 0 replies; 3+ messages in thread
From: Matthew Wilcox @ 2026-11-04 20:36 UTC (permalink / raw)
To: 董辉
Cc: linux-rt-devel@lists.kernel.org, linux-mm@kvack.org,
linux-fsdevel@vger.kernel.org, Andrew Morton,
田孝斌, 李鹏程,
宁左斌, 宁博
On Tue, Nov 03, 2026 at 12:46:36PM +0000, 董辉 wrote:
> We hit a permanent livelock on a PREEMPT_RT system and traced it to an
> interaction between the lockless page cache lookup retry loop in
> filemap_get_entry() and the folio freeze -> xarray removal window in
> __remove_mapping() (and the migrate equivalents). On non-RT that window
> is atomic (IRQs disabled) and therefore bounded; on PREEMPT_RT it is
> preemptible and can become unbounded, which turns the "refcount == 0 is
> transient" assumption of the lookup path into a permanent spin.
>
> The affected code is vanilla upstream (our mm/filemap.c, mm/vmscan.c
> and mm/migrate.c match 6.12 exactly); nothing vendor-specific is
> involved.
>
> The obvious fix -- making the window non-preemptible on RT -- does not
> work for the reclaim path, which is the one we hit (section 4.1). We are
> deliberately not sending a patch yet: we would like to hear which
> fix direction upstream prefers before going further.
Thanks for bringing it to us early so we can discuss the right solution.
Much easier than trying to infer from a patch what the problem was
originally ;-)
> 1. Symptom
> ----------
>
> - 6.12.73-rt15 (PREEMPT_RT), arm64, 20 CPUs.
> - A SCHED_FIFO 99 userspace workload was page-faulting on file-backed
> code pages of a shared library (libc.so.6 in our case) while the system
> was under memory pressure.
> - One CPU reported rcu_preempt self-detected stall every ~63 s from then
> on: 670+ reports over the ~12 hours covered by our captured logs.
> - The next day, and again about two days after the first stall, we
> inspected the machine over JTAG. It was still stuck, with the same
> register values as in the first reports. It never recovered on its
> own; only a reboot brought it back.
> - On that CPU the running task was spinning in filemap_get_entry() on a
> folio whose _refcount was 0 (frozen) while the folio was still
> reachable from the mapping's xarray.
> - kswapd0 was found preempted inside __remove_mapping(), its stack
> ending in __filemap_remove_folio() -> filemap_unaccount_folio() ->
> ... -> preempt_schedule(): the folio was already frozen (refcount 0,
> off-LRU, unmapped) but not yet removed from the xarray, and the task
> still held the xarray lock.
>
> All 670 stall reports show the same task and the same folio. Their PC
> samples fall inside this retry loop, mostly in xas_load(); the path
> taken after a successful refcount increment was never sampled. A
> second, non-RT process later hung in the same loop on the same folio,
> showing that once such an entry exists, every reader of that page is a
> victim.
>
> 2. Root cause
> -------------
>
> 2.1 Reader: unbounded retry on refcount == 0
>
> /* mm/filemap.c, filemap_get_entry() */
> rcu_read_lock();
> repeat:
> xas_reset(&xas);
> folio = xas_load(&xas);
> if (xas_retry(&xas, folio))
> goto repeat;
> /*
> * A shadow entry of a recently evicted page, or a swap entry from
> * shmem/tmpfs. Return it without attempting to raise page count.
> */
> if (!folio || xa_is_value(folio))
> goto out;
>
> if (!folio_try_get(folio))
> goto repeat;
>
> The loop treats refcount == 0 as a transient state and retries. It never
> sleeps or yields while doing so, and there is no retry bound, no timeout
> and no warning.
Right. As you suggest later, this is intended to be a transient state
that des not need to be bounded, slept on or warned about; usually finding
a folio with a zero refcount isn't from a freeze but a stale folio pointer.
> 2.2 Writer: the window is bounded on non-RT
>
> /* mm/vmscan.c, __remove_mapping() */
> if (!folio_test_swapcache(folio))
> spin_lock(&mapping->host->i_lock);
> xa_lock_irq(&mapping->i_pages);
> ...
> refcount = 1 + folio_nr_pages(folio);
> if (!folio_ref_freeze(folio, refcount))
> goto cannot_free;
> /* note: atomic_cmpxchg in folio_ref_freeze provides the smp_rmb */
> ...
> __filemap_remove_folio(folio, shadow);
> xa_unlock_irq(&mapping->i_pages);
> ...
> cannot_free:
> xa_unlock_irq(&mapping->i_pages);
>
> Readers on other CPUs can briefly observe the frozen folio even on
> non-RT -- that is what the retry loop is for. On non-RT, however,
> xa_lock_irq() disables IRQs and preemption, so the interval from
> folio_ref_freeze() to __filemap_remove_folio() cannot be preempted
> and cannot outlive the locked code path: the window is bounded, and
> the retry terminates after a few iterations.
>
> 2.3 On PREEMPT_RT the window is unbounded -- this is the break
>
> On PREEMPT_RT xa_lock_irq() maps to rt_spin_lock(): an rt_mutex
> plus migrate_disable() and rcu_read_lock(). It does not disable
> preemption. A task inside the freeze -> removal window can therefore be
> preempted, and the frozen folio stays visible in the xarray with
> refcount == 0 for as long as the holder is off-CPU -- the window is no
> longer bounded by the locked code path, but by scheduling.
>
> migrate_disable() aggravates this: the holder is pinned to the CPU it
> was preempted on. The holder does not have to be kswapd: the same
> shrink_folio_list() -> __remove_mapping() path is used by direct
> reclaim, so any task allocating memory can be the victim.
>
> 2.4 How it becomes permanent: priority inversion
>
> In our case a SCHED_FIFO 99 task preempted the holder on the same CPU,
> then faulted on the very page being removed and entered the
> filemap_get_entry() spin. Two further SCHED_FIFO 99 threads of the
> same workload blocked on the same xarray lock; through priority
> inheritance the holder (kswapd) was boosted into the RT scheduling class
> at the same effective priority as the spinner. SCHED_FIFO does not
> time-slice among equal priorities, so a spinner that never blocks never
> gives the CPU back to the holder.
>
> This is where self-healing is lost. While the holder remains CFS, the
> RT bandwidth limit (sched_rt_runtime_us, 950 ms per 1 s period by
> default) reserves CPU time for CFS tasks exactly so that a run-away RT
> task cannot lock up the machine (see
> Documentation/scheduler/sched-rt-group.rst); the holder can still run
> and close the window. Once PI pulls the holder into the RT class it is
> throttled together with the spinner and no longer benefits from that
> reserve. (With sched_rt_runtime_us = -1 even the CFS escape does not
> exist.)
>
> The spinner is invisible to PI: it waits on the folio, not on the lock,
> so nothing ever boosts the holder on its behalf.
I think you mean that it spins on the folio refcount, rather than it
waiting on the folio (when people say "wait on the folio", usually
they're talking abut folio_lock()).
> The four necessary conditions:
>
> (a) frozen-but-visible folio -> reader spins
> where: filemap_get_entry() retry loop
> (b) window unbounded (holder preemptible, pinned)
> where: PREEMPT_RT xa_lock_irq() semantics
> (c) unyielding spinner on the same CPU
> where: SCHED_FIFO 99 in the retry loop
> (d) PI removes the holder's CFS bandwidth reserve
> where: RT waiters on the same xarray lock
>
> 3. Blast radius
> ---------------
>
> We went through every folio_ref_freeze()/page_ref_freeze() user in
> 6.12. Two of them can freeze a regular file folio and leave it
> reachable from the page cache xarray, and on PREEMPT_RT both can be
> preempted there:
>
> - mm/vmscan.c, __remove_mapping()
> trigger: reclaim -- kswapd or direct reclaim
> - mm/migrate.c, __folio_migrate_mapping()
> trigger: compaction / CMA / hotplug
There's also __folio_freeze_split_file(). I know PREEMPT_RT disables
TRANSPARENT_HUGEPAGE at the moment, but hopefully that restriction will
be lifted.
> Both hold the xarray lock from before the freeze until the folio has
> been removed, or replaced and unfrozen.
>
> The unbounded refcount == 0 spin exists in several lookup loops, not
> just filemap_get_entry():
>
> - filemap_get_entry() -- goto repeat, no bound
> - find_get_entry() -- goto reset/goto retry, no bound
> - filemap_get_folios_contig(), filemap_get_read_batch() -- goto retry,
> no bound
> - ksm_get_folio() -- while (!folio_try_get(folio)) cpu_relax() with a
> swapcache-flag escape
>
> ksm.c documents the transient state explicitly ("however, it might
> mean that the page is under page_ref_freeze()..."), which suggests the
> frozen-but-visible window is a known state of the design -- the filemap
> loops simply have no bound on it.
>
> 4. Fix directions we have considered
> ------------------------------------
>
> 4.1 Direction A (writer side): restore non-RT atomicity of the window
>
> The idea: on PREEMPT_RT, wrap the freeze -> xarray-removal interval in
> preempt_disable()/preempt_enable(), keeping the lock acquisition
> itself RT-semantic (it may sleep while waiting). Readers stay untouched
> and "refcount == 0 is transient" holds again, for every reader.
>
> By inspection this works for __folio_migrate_mapping(). Its window,
> from the freeze to the xas_store() of the new folio, contains nothing
> that sleeps on RT: that xas_store() call is not wrapped by
> mapping_set_update(), so no update hook fires.
>
> It does not work for __remove_mapping(), the site we hit. The
> removal itself takes a spinlock_t whenever shadow-node tracking changes:
>
> __filemap_remove_folio()
> page_cache_delete()
> mapping_set_update() /* xa_update = workingset_update_node */
> xas_store(&xas, shadow)
> update_node()
> xas_update()
> workingset_update_node()
> list_lru_add_obj() / list_lru_del_obj()
> spin_lock(&nlru->lock) /* spinlock_t */
>
> __delete_from_swap_cache() installs the same hook, so the swapcache
> branch is affected as well. This happens whenever an xa_node becomes
> shadow-only or stops being so, which is routine when reclaim evicts a
> run of file pages. shadow_nodes is a single list_lru whose per-node
> lock is shared by all mappings and by the shadow-node shrinker, so it
> can be contended. Unlike the other sleeping call we found, it cannot be
> moved out of the window: it is taken from inside the very xas_store()
> the window exists to cover.
Is it possible / wise to convert the nlru->lock to be a raw spinlock?
In the old days when the BKL was magic, we could take a mutex and choose
to sleep. I don't think the preempt_disable() works like that. It's
a bit annying because we'd be fine with being preempted after the folio
is replaced in the XArray. It feels unutterably ugly to insert calls to
preempt_disable() / preempt_enable() / preempt_disable() /
preempt_enable() around all of this though.
> (The other sleeping call in this function, mem_cgroup_swapout() in the
> swapcache branch, could be moved: with memsw accounting it takes
> swap_cgroup's ctrl->lock in swap_cgroup_record(). It would have to
> stay under the xarray lock though, because memcg1_swapout() relies on
> interrupts being disabled by it on non-RT.)
>
> Making direction A work for reclaim would need the shadow-node list_lru
> update to be decoupled from xas_store(), for example deferred until
> after xa_unlock_irq(). That is an xarray/workingset change affecting
> every page cache and swap cache deletion, which we do not want to
> attempt without guidance.
Yeah, I don't think this is a great approach either.
> 4.2 Direction B (reader side): bounded retry + blocking fallback
>
> Every freezer listed in section 3 (Blast radius) holds the xarray lock until the folio
> is no longer frozen-and-visible. So a reader that keeps failing can
> simply wait for that lock: on PREEMPT_RT it sleeps on the rt_mutex,
> gives the CPU back, and priority inheritance boosts the holder. Once the
> lock has been acquired the window is closed, and the retry sees the
> final state of the slot.
>
> This is the same fallback commit 0f8821da4845 uses in
> mnt_get_write_access() (fs/namespace.c): on PREEMPT_RT the spinner on
> MNT_WRITE_HOLD stops spinning and takes the writer's lock instead,
> specifically to avoid this livelock.
>
> The change would be small (mm/filemap.c only, we estimate about 80 lines):
>
> - Applies to the four filemap loops listed in section 3 (Blast radius).
> - Counts failed attempts; every 64 of them the reader calls
> rcu_read_unlock(), then xas_lock_irq() + xas_unlock_irq(), then
> rcu_read_lock(), resets the xa_state and retries the same index.
> Nothing is skipped and nothing is returned early, so semantics do not
> change: callers such as truncation still see every entry.
> - Only on PREEMPT_RT: the check compiles away otherwise, so non-RT
> behavior is identical.
> - Only when preemptible(): callers that cannot sleep, such as some
> BPF/perf paths reaching build_id_parse_nofault(), keep retrying as
> before. An outer rcu_read_lock() does not need special handling:
> PREEMPT_RT uses preemptible RCU, which allows blocking in a
> read-side section (at the cost of a longer grace period).
>
> We have not prototyped or tested this yet; the sketch is here so the
> trade-off can be judged before we invest in a patch series.
>
> - Pros: covers the reclaim site that direction A cannot; covers every
> freezer that follows the "hold the xarray lock" rule, including ones
> added later; adds no non-preemptible window.
> - Cons: the lookups can now sleep on PREEMPT_RT in preemptible context,
> and we have not audited all of their callers. Two things need checking:
> that no caller holds a lock which is taken inside the xarray lock
> (we spot-checked list_lru walkers and found no such caller, but this
> was not exhaustive), and that every caller keeps the mapping alive
> across the rcu_read_unlock(), not just by RCU.
> - Not covered: callers that cannot sleep can still spin, and so can
> ksm_get_folio(), which has no mapping at hand to wait on. KSM is
> disabled on our system, so we did not look further into it.
>
> Our current view is that B addresses the path we hit, while A, at best,
> narrows the problem to the migrate path -- but we may be missing a
> reason why upstream would rather restructure the writer side.
But I don't like this either ;-( Hope somebody comes up with a nicer
idea.
> 5. Questions
> ------------
>
> 1. Is the "refcount == 0 is transient" assumption of the lockless
> lookup path documented as depending on the freeze -> removal window
> being non-preemptible? If so, PREEMPT_RT breaking it is arguably an
> RT-side invariant break; if not, should the lookup loops grow a
> bound?
I don't think it's _documented_. But I do think it's intentional.
> 2. Given that the reclaim site cannot be made non-preemptible as long as
> workingset_update_node() takes the list_lru spinlock_t from inside
> xas_store(): is a reader-side fallback like direction B acceptable,
> or would upstream rather decouple the shadow-node LRU update from
> xas_store() and then fix the writers?
> 3. For direction B: is it acceptable that these lookups may sleep on
> the xarray lock on PREEMPT_RT, in preemptible context only (the
> same fallback commit 0f8821da4845 uses in mnt_get_write_access())?
> Is
> "every freezer holds the xarray lock until the folio is no longer
> frozen-and-visible" an invariant we may rely on, and should it be
> documented next to the lockless page cache protocol comment?
> 4. Are there other freeze sites we have missed?
> 5. Would a WARN_ON_ONCE() on excessive retry be acceptable as a
> diagnostic, independent of the chosen fix?
No, I don't think we should do that.
> 6. Reproducer sketch
> --------------------
>
> 1. Pin a SCHED_FIFO 99 task to CPU X; have it repeatedly page-fault on
> file-backed code pages of a shared library under reclaim pressure.
> 2. Drive reclaim (memory pressure is enough; direct reclaim works too)
> so a task runs __remove_mapping() for one of those pages on CPU X.
> 3. The FIFO task preempts the holder inside the freeze -> removal window
> and faults on the frozen page -> spin.
> 4. Two more FIFO 99 threads of the same process fault on other pages of
> the same mapping to block on the xarray lock and trigger PI.
>
> Anonymized evidence (register sampling, stack traces of the three RT
> threads and kswapd, decoded structure dumps) is available on request.
> If direction B is the preferred one, we will turn the sketch above into
> a patch series and send it for review.
>
> Best regards,
> Donghui
> #/******本邮件及其附件含有小米公司的保密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件! This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-16 12:35 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 12:35 [RFC] PREEMPT_RT: livelock between lockless page cache lookup and the folio freeze->removal window 董辉
-- strict thread matches above, loose matches on Subject: below --
2026-11-03 12:46 董辉
2026-11-04 20:36 ` Matthew Wilcox
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox