[RFC PATCH v2 09/18] mm/memory-failure: handle folios with PG_rcu_free

["Kiryl Shutsemau (Meta)" <kas@kernel.org>]

A page whose free waits for an RCU grace period has refcount 0 and is
neither on the LRU nor in the buddy allocator.  memory_failure() and
soft_offline_page() cannot handle such a page: get_any_page() retries a
few times and gives up.  Soft offline of the page fails, and a hard
error is ignored.  If the page is part of a large folio, the folio is
later freed with the poisoned page in it, and that page can be handed
out again.

Free the waiting pages first when the page looks like one of them, and
again on each retry in get_any_page(): the page can be freed while it
retries.  memory_failure() has to do it before it sets PG_hwpoison.  The
free of a poisoned order-0 page drops the page instead of returning it
to the buddy allocator, so a later drain would still leave the error
ignored.

A folio with PG_rcu_free that is still in use may be copied by a reader
that holds no reference.  When memory_failure() succeeds, its caller can
unmap the page from the direct map under such a reader.  Wait for a
grace period before returning success.  By then the page has left the
page cache, or is an order-0 folio with PG_hwpoison, which a reader that
starts after the grace period has to check.

Wait at the end, not right after setting PG_hwpoison.  A large folio is
split later, and until the new folios are in the page cache, a reader
can still copy the poisoned page as part of the large one.

Both callers hold mf_mutex and can wait for the grace period.

Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 mm/memory-failure.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index 7dde7c3c9a0c..398463b2e82d 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -1419,6 +1419,19 @@ static int __get_hwpoison_page(struct page *page, unsigned long flags)
 	return 0;
 }
 
+/* A page waiting for a grace period is not in the buddy yet: free it */
+static void mf_free_rcu_pages(struct page *p)
+{
+	if (!zone_rcu_free_waiting(page_zone(p)))
+		return;
+	if (page_count(p))
+		return;
+	if (is_free_buddy_page(p))
+		return;
+
+	rcu_free_sync();
+}
+
 #define GET_PAGE_MAX_RETRY_NUM 3
 
 static int get_any_page(struct page *p, unsigned long flags)
@@ -1443,6 +1456,7 @@ static int get_any_page(struct page *p, unsigned long flags)
 	}
 
 try_again:
+	mf_free_rcu_pages(p);
 	if (!count_increased) {
 		ret = __get_hwpoison_page(p, flags);
 		if (!ret) {
@@ -2440,6 +2454,7 @@ int memory_failure(unsigned long pfn, int flags)
 	if (res != -ENOENT)
 		goto unlock_mutex;
 
+	mf_free_rcu_pages(p);
 	if (TestSetPageHWPoison(p)) {
 		res = -EHWPOISON;
 		if (flags & MF_ACTION_REQUIRED)
@@ -2447,6 +2462,13 @@ int memory_failure(unsigned long pfn, int flags)
 		if (flags & MF_COUNT_INCREASED)
 			put_page(p);
 		action_result(pfn, MF_MSG_ALREADY_POISONED, MF_FAILED);
+		/*
+		 * The memory_failure() that set PG_hwpoison may have failed
+		 * without waiting for lockless readers, and the caller may
+		 * unmap the page once this returns 0.
+		 */
+		if (!res && folio_test_rcu_free(page_folio(p)))
+			synchronize_rcu();
 		goto unlock_mutex;
 	}
 
@@ -2612,6 +2634,13 @@ int memory_failure(unsigned long pfn, int flags)
 
 identify_page_state:
 	res = identify_page_state(pfn, p, page_flags);
+	/*
+	 * The caller may unmap the page once this returns 0.  Lockless
+	 * readers that started before the page got PG_hwpoison, or before a
+	 * split left it in a folio of its own, may still be copying it.
+	 */
+	if (!res && folio_test_rcu_free(folio))
+		synchronize_rcu();
 	mutex_unlock(&mf_mutex);
 	return res;
 unlock_page:
-- 
2.54.0