From: 董辉 <donghui@xiaomi.com>
To: Matthew Wilcox <willy@infradead.org>,
"linux-rt-devel@lists.linux.dev" <linux-rt-devel@lists.linux.dev>,
"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>
Cc: "linux-mm@kvack.org" <linux-mm@kvack.org>,
"Andrew Morton" <akpm@linux-foundation.org>,
"Sebastian Andrzej Siewior" <bigeasy@linutronix.de>,
田孝斌 <tianxiaobin@xiaomi.com>, 李鹏程 <xiaoyaoli@xiaomi.com>,
宁左斌 <ningzuobin@xiaomi.com>, 宁博 <ningbo1@xiaomi.com>
Subject: Re: [RFC] PREEMPT_RT: livelock between lockless page cache lookup and the folio freeze->removal window
Date: Mon, 16 Nov 2026 12:35:09 +0000 [thread overview]
Message-ID: <b4ebeca28cc7472b901a354888e6d3ce@xiaomi.com> (raw)
Hello,
Thank you for the review. Nothing nicer surfaced on our side either,
so rather than let the thread stall we went ahead and implemented
direction B -- mostly to make the trade-off concrete and measurable.
The patch is inline at the end of this mail; the commit message
explains it. In short: every 64 failed attempts the reader waits for
the xarray lock once instead of spinning, so on PREEMPT_RT it sleeps
and priority inheritance boosts the holder. Non-RT is unaffected.
In a kprobe-driven reproducer that forces the race, with a second
SCHED_FIFO task boosting the holder as in our field case, the unfixed
kernel livelocks on every run, in each of the four lookup loops the
patch covers. With the patch, in 10-second runs of each loop, every
window hit took the wait path (87k-396k hits per loop), the longest
wait was 98 us, and nothing spun. Logs and reproducer code are
available on request.
On your comments:
- Direction A: agreed. The window also contains a sleeping lock:
__filemap_remove_folio() -> page_cache_delete() -> xas_store() ->
workingset_update_node() -> list_lru_add_obj() takes nlru->lock, a
spinlock_t that sleeps on PREEMPT_RT. So preempt_disable() around
the window would first need the shadow-node LRU update moved out of
xas_store(), an xarray/workingset change for every page cache
deletion.
On converting nlru->lock to a raw spinlock: that would remove the
sleeping lock from the window, but it is the generic list_lru lock,
and the isolate callbacks of every list_lru run under it.
shadow_lru_isolate() itself trylocks the xarray lock and i_lock
there, which are sleeping locks on PREEMPT_RT. So it is not a local
change. We have not pursued it, but are happy to look into it if
that is the direction you prefer.
- "spins on the folio refcount": you are right, that is the correct
wording; the patch uses it.
- Thanks for __folio_freeze_split_file(). It is not in 6.12, so we
will check it against the rule in the comment when rebasing onto
mainline.
- On "refcount == 0 is transient" being intentional but undocumented:
the patch adds a comment spelling out the rule the fallback depends
on -- whoever freezes a page cache folio holds the xarray lock until
the folio is either gone from i_pages or unfrozen. That is what
makes waiting for the lock enough, and any future freezer that
follows the rule is covered as well.
The patch is against mm/filemap.c of v6.12.73 (unmodified upstream
stable); we will rebase onto mainline before a proper submission.
Comments very welcome.
Best regards,
Hui Dong
-- >8 --
From: Hui Dong <donghui@xiaomi.com>
Subject: [RFC PATCH] mm/filemap: bound the lockless lookup retry loop for
PREEMPT_RT
On PREEMPT_RT, xa_lock_irq() becomes an rt_mutex that disables neither
interrupts nor preemption, and rt_spin_lock() additionally pins the
holder to its CPU with migrate_disable().
The reclaim and migration paths freeze a folio's refcount while the
folio is still in i_pages, and keep it reachable there until it is
deleted or replaced, all under the xarray lock. On a non-RT kernel
that window runs with interrupts disabled, so the lockless readers that
spin on the frozen refcount terminate after a few iterations.
On PREEMPT_RT the window is preemptible. When the holder is preempted
by a higher-priority task that then faults on the very page being
removed, that task retries in filemap_get_entry() without ever yielding
the CPU or joining the rt_mutex wait list, so priority inheritance
cannot see it. A SCHED_NORMAL holder then only runs on the fair
server's budget, and the reader stalls for up to a second. Once
another task blocks on the xarray lock and boosts the holder to the
spinning task's priority, as in our case, the holder is out of the fair
server's reach, and SCHED_FIFO does not round-robin equal priorities,
so the holder never runs again. The reader also keeps its RCU read
side open, which stalls grace periods.
Bound the spinning instead: every 64 attempts, wait for the xarray
lock once before retrying. On PREEMPT_RT that wait queues the
reader on the rt_mutex: it sleeps, gives the CPU back, and priority
inheritance boosts whoever is holding the window open. By the time the
lock has been acquired the holder has left its critical section, so the
retry that follows sees the final state of the slot. This is the same
fallback that commit 0f8821da4845 ("fs/namespace: Boost the
mount_lock.lock owner instead of spinning on PREEMPT_RT.") added to
mnt_get_write_access().
The wait is rare, so each lookup tells lockdep about the xarray lock up
front with might_lock(), as mnt_get_write_access() does for mount_lock.
Lockdep then checks every caller that may end up waiting, not only the
ones that actually did.
The bound is compiled out on non-RT kernels, so their behaviour is
unchanged, and callers that cannot sleep keep retrying as before.
Covered loops: filemap_get_entry(), find_get_entry(),
filemap_get_folios_contig() and filemap_get_read_batch();
ksm_get_folio() has no mapping at hand and is not covered.
Tested with a deterministic in-kernel reproducer that forces the race:
a SCHED_FIFO 99 reader is woken inside the freeze -> removal window and
looks up the frozen folio through each of the four loops, while a
second SCHED_FIFO 99 task blocks on the xarray lock and boosts the
holder. Without the patch the reader spins forever in every loop; with
it, every window hit took the wait path, the longest wait was 98 us,
and no reader spun.
Signed-off-by: Hui Dong <donghui@xiaomi.com>
---
mm/filemap.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 90 insertions(+), 3 deletions(-)
diff --git a/mm/filemap.c b/mm/filemap.c
index d8d9c0f0beb6..19959ad60fef 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -1820,6 +1820,68 @@ EXPORT_SYMBOL(page_cache_prev_miss);
* folio_put().
*/
+/*
+ * PREEMPT_RT: the lookup loops below retry for as long as they find a folio
+ * whose refcount was frozen (step A above) but that is still in i_pages.
+ * Whoever freezes a page cache folio holds the xarray lock until it is
+ * resolved: __remove_mapping() until the folio has been deleted, the
+ * migration paths until it has been replaced and unfrozen.
+ *
+ * Without PREEMPT_RT the lock is held with interrupts disabled, so the
+ * retries end quickly. With PREEMPT_RT xa_lock_irq() is an rt_mutex that
+ * leaves interrupts and preemption enabled, and rt_spin_lock() pins the
+ * holder to its CPU with migrate_disable(). If the holder is preempted by
+ * the task that then retries here, the holder may never run again: the
+ * retry loop does not sleep, and priority inheritance cannot see a task
+ * that is not waiting on the lock. The task also keeps its RCU read side
+ * open, so grace periods stall as well.
+ *
+ * So on PREEMPT_RT, every FILEMAP_LOOKUP_RETRIES attempts, wait for the
+ * xarray lock once instead. The reader sleeps on the rt_mutex, which gives
+ * the CPU back and boosts the holder through priority inheritance. Once the
+ * lock has been acquired the holder has left its critical section, and the
+ * retry that follows sees the final state of the slot. Retry entries are
+ * counted too: they are transient, so waiting is merely unnecessary there.
+ *
+ * The wait is rare, so the loops tell lockdep about it up front with
+ * filemap_lookup_might_wait(): every caller that may end up waiting is then
+ * checked against the xarray lock, not only the ones that actually did.
+ */
+#define FILEMAP_LOOKUP_RETRIES64
+
+static inline void filemap_lookup_might_wait(struct xa_state *xas)
+{
+if (IS_ENABLED(CONFIG_PREEMPT_RT) && preemptible())
+might_lock(&xas->xa->xa_lock);
+}
+
+static inline bool filemap_lookup_should_wait(unsigned int *retries)
+{
+return IS_ENABLED(CONFIG_PREEMPT_RT) &&
+ unlikely(!(++*retries % FILEMAP_LOOKUP_RETRIES));
+}
+
+/*
+ * Wait for the holder of the xarray lock, see above. Called with the RCU
+ * read lock held and without a reference on the entry @xas points to;
+ * returns the same way, with @xas reset so that the caller retries the same
+ * index. The caller's own state stays valid: RCU only protects the walk,
+ * and folios collected so far hold their own references.
+ *
+ * Callers that cannot sleep keep retrying, as they did before.
+ */
+static noinline void filemap_lookup_wait(struct xa_state *xas)
+{
+xas_reset(xas);
+if (!preemptible())
+return;
+
+rcu_read_unlock();
+xas_lock_irq(xas);
+xas_unlock_irq(xas);
+rcu_read_lock();
+}
+
/*
* filemap_get_entry - Get a page cache entry.
* @mapping: the address_space to search
@@ -1836,9 +1898,13 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index)
{
XA_STATE(xas, &mapping->i_pages, index);
struct folio *folio;
+unsigned int retries = 0;
rcu_read_lock();
+filemap_lookup_might_wait(&xas);
repeat:
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(&xas);
xas_reset(&xas);
folio = xas_load(&xas);
if (xas_retry(&xas, folio))
@@ -1998,15 +2064,20 @@ static inline struct folio *find_get_entry(struct xa_state *xas, pgoff_t max,
xa_mark_t mark)
{
struct folio *folio;
+unsigned int retries = 0;
+filemap_lookup_might_wait(xas);
retry:
if (mark == XA_PRESENT)
folio = xas_find(xas, max);
else
folio = xas_find_marked(xas, max, mark);
-if (xas_retry(xas, folio))
+if (xas_retry(xas, folio)) {
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(xas);
goto retry;
+}
/*
* A shadow entry of a recently evicted page, a swap
* entry from shmem/tmpfs or a DAX entry. Return it
@@ -2026,6 +2097,8 @@ static inline struct folio *find_get_entry(struct xa_state *xas, pgoff_t max,
return folio;
reset:
xas_reset(xas);
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(xas);
goto retry;
}
@@ -2194,13 +2267,18 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,
XA_STATE(xas, &mapping->i_pages, *start);
unsigned long nr;
struct folio *folio;
+unsigned int retries = 0;
rcu_read_lock();
+filemap_lookup_might_wait(&xas);
for (folio = xas_load(&xas); folio && xas.xa_index <= end;
folio = xas_next(&xas)) {
-if (xas_retry(&xas, folio))
+if (xas_retry(&xas, folio)) {
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(&xas);
continue;
+}
/*
* If the entry has been swapped out, we can stop looking.
* No current caller is looking for DAX entries.
@@ -2230,6 +2308,8 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,
retry:
xas_reset(&xas);
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(&xas);
}
update_start:
@@ -2336,11 +2416,16 @@ static void filemap_get_read_batch(struct address_space *mapping,
{
XA_STATE(xas, &mapping->i_pages, index);
struct folio *folio;
+unsigned int retries = 0;
rcu_read_lock();
+filemap_lookup_might_wait(&xas);
for (folio = xas_load(&xas); folio; folio = xas_next(&xas)) {
-if (xas_retry(&xas, folio))
+if (xas_retry(&xas, folio)) {
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(&xas);
continue;
+}
if (xas.xa_index > max || xa_is_value(folio))
break;
if (xa_is_sibling(folio))
@@ -2363,6 +2448,8 @@ static void filemap_get_read_batch(struct address_space *mapping,
folio_put(folio);
retry:
xas_reset(&xas);
+if (filemap_lookup_should_wait(&retries))
+filemap_lookup_wait(&xas);
}
rcu_read_unlock();
}
#/******本邮件及其附件含有小米公司的保密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件! This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#
next reply other threads:[~2026-11-16 12:35 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 12:35 董辉 [this message]
[not found] <c2ff0c66422c4e2d869499a1186ff0ae@xiaomi.com>
2026-11-04 20:36 ` [RFC] PREEMPT_RT: livelock between lockless page cache lookup and the folio freeze->removal window Matthew Wilcox
-- strict thread matches above, loose matches on Subject: below --
2026-11-04 3:45 董辉
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b4ebeca28cc7472b901a354888e6d3ce@xiaomi.com \
--to=donghui@xiaomi.com \
--cc=akpm@linux-foundation.org \
--cc=bigeasy@linutronix.de \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-rt-devel@lists.linux.dev \
--cc=ningbo1@xiaomi.com \
--cc=ningzuobin@xiaomi.com \
--cc=tianxiaobin@xiaomi.com \
--cc=willy@infradead.org \
--cc=xiaoyaoli@xiaomi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox