[PATCH 0/3] NFS/localio: issue IO inline when not reclaiming memory

[PATCH 0/3] NFS/localio: issue IO inline when not reclaiming memory

[Mike Snitzer]

Hi,

Every NFS LOCALIO read, write and commit is currently issued indirectly: it
is queued onto the dedicated !WQ_MEM_RECLAIM nfslocaliod_workqueue rather
than being submitted to the underlying filesystem in the calling context.
That intermediate hop was introduced by commit b9f5dd57f4a5 ("nfs/localio:
use dedicated workqueues for filesystem read and write") because:
LOCALIO submits IO directly into a stacked local filesystem (e.g. XFS)
which may in turn flush its own !WQ_MEM_RECLAIM workqueue. Doing so from a
WQ_MEM_RECLAIM worker -- most importantly writeback's wb_workfn running on
bdi_wq -- or from an explicit PF_MEMALLOC reclaim task trips
check_flush_dependency() and risks a forward-progress deadlock.

However, that hazard only exists when the submitting context is itself a
memory-reclaim context.  For ordinary application/task submission -- e.g.
O_DIRECT, or an fsync-driven commit -- the workqueue hop buys nothing: it
just adds a context switch and scheduling latency per IO and throws away the
NFS client's inherent application-context parallelism.

This series makes the hop conditional.  It adds a small workqueue-core
helper, current_is_workqueue_mem_reclaim(), that reports whether %current is
a WQ_MEM_RECLAIM worker using exactly the predicate check_flush_dependency()
warns on.  LOCALIO uses it (together with the existing PF_MEMALLOC test) in a
new nfs_local_defer_io() helper to decide per-IO whether it must defer to
nfslocaliod_workqueue or may issue the IO inline.  Reclaim contexts still
defer and are unaffected; everything else runs inline.

  Patch 1 adds current_is_workqueue_mem_reclaim() and applies the gating to
          the read and write paths.
  Patch 2 removes never-taken FLUSH_SYNC handling from nfs_local_commit()
          (every caller supplies a FLUSH_SYNC-stripped "how"), which also
          drops the sole user of the ctx->done completion plumbing and the
          now-unused "how" argument.  No functional change.
  Patch 3 extends the same gating to the commit (fsync) path.  Note the
          writeback-triggered commit does reach here in reclaim context --
          nfs_write_inode() (the ->write_inode super_op) runs under wb_workfn
          on the WQ_MEM_RECLAIM bdi_wq -- so that case correctly keeps
          deferring; only app-context commits run inline.

Patch 1 touches the workqueue core (kernel/workqueue.c, include/linux/
workqueue.h), hence the Cc to Tejun Heo, Lai Jiangshan and LKML.  The new
export mirrors the existing current_is_workqueue_rescuer()/current_work()
context-introspection helpers.

Additional note for reviewers:
 - The inline path now stacks the NFS pgio path plus the underlying
   filesystem's ->write_iter/->read_iter (and vfs_fsync_range) in task
   context.  The reclaim-context stack-depth concern of b9f5dd57f4a5 (2) is
   avoided for the deferred paths; feedback on inline stack headroom under
   deeply-stacking filesystems (XFS especially) is welcome.

All review appreciated, thanks.
Mike

Mike Snitzer (3):
  NFS/localio: issue IO inline when not in a memory-reclaim context
  NFS/localio: remove dead FLUSH_SYNC handling from nfs_local_commit
  NFS/localio: issue commit inline when not in a memory-reclaim context

 fs/nfs/internal.h         |  4 +--
 fs/nfs/localio.c          | 56 ++++++++++++++++++++++++++++++---------
 fs/nfs/write.c            |  2 +-
 include/linux/workqueue.h |  1 +
 kernel/workqueue.c        | 24 +++++++++++++++++
 5 files changed, 71 insertions(+), 16 deletions(-)

-- 
2.44.0

[PATCH 1/3] NFS/localio: issue IO inline when not in a memory-reclaim context

[Mike Snitzer]

Every LOCALIO read and write is currently bounced through the dedicated
!WQ_MEM_RECLAIM nfslocaliod_workqueue.  That bounce is only actually
required when the submitting context is a memory-reclaim context: LOCALIO
issues IO directly into a stacked local filesystem (e.g. XFS) which may in
turn flush its own !WQ_MEM_RECLAIM workqueue.  Doing that from a
WQ_MEM_RECLAIM worker (most importantly writeback's wb_workfn on bdi_wq) or
an explicit PF_MEMALLOC reclaim task trips check_flush_dependency() and
risks a forward-progress deadlock, which is why commit b9f5dd57f4a5
("nfs/localio: use dedicated workqueues for filesystem read and write")
introduced the intermediate workqueue.

Outside of reclaim context -- ordinary application/task submission such as
O_DIRECT or fsync-driven writeback -- the workqueue hop buys nothing and
merely adds a context switch and scheduling latency per IO while discarding
the NFS client's inherent application-context parallelism.

Add current_is_workqueue_mem_reclaim(), which reports whether %current is a
WQ_MEM_RECLAIM worker using the same predicate check_flush_dependency()
warns on.  Use it, together with the PF_MEMALLOC check, in the new
nfs_local_defer_io() helper to decide per-IO whether nfs_local_do_read()
and nfs_local_do_write() must defer to nfslocaliod_workqueue or may issue
the IO inline.  Buffered writeback continues to bounce (wb_workfn is a
WQ_MEM_RECLAIM worker); O_DIRECT and app-context submission now run inline.

Running nfs_local_call_write() inline is safe: it already saves and
restores current->flags around the PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO it
sets and scopes the file opener's creds.  The async O_DIRECT completion
path is likewise unaffected: when the underlying filesystem returns
-EIOCBQUEUED, the kiocb ki_complete callback (nfs_local_read_aio_complete /
nfs_local_write_aio_complete) can run in bottom-half context and so must
still defer the pgio completion (nfs_local_pgio_release -> rpc_call_done) to
nfsiod_workqueue via nfs_local_pgio_aio_complete().  That completion hop is
independent of how the IO was submitted, and this change leaves it as-is;
only the submission side stops unconditionally hopping through
nfslocaliod_workqueue.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
 fs/nfs/localio.c          | 33 +++++++++++++++++++++++++++++++--
 include/linux/workqueue.h |  1 +
 kernel/workqueue.c        | 24 ++++++++++++++++++++++++
 3 files changed, 56 insertions(+), 2 deletions(-)

diff --git a/fs/nfs/localio.c b/fs/nfs/localio.c
index e55c5977fcc3..d3e480888eb1 100644
--- a/fs/nfs/localio.c
+++ b/fs/nfs/localio.c
@@ -699,6 +699,29 @@ static void nfs_local_call_read(struct work_struct *work)
 	}
 }
 
+/*
+ * Decide whether LOCALIO must defer submission to the dedicated
+ * !WQ_MEM_RECLAIM nfslocaliod_workqueue rather than issue the IO inline.
+ *
+ * LOCALIO issues IO directly into a stacked local filesystem (e.g. XFS),
+ * which may in turn flush its own !WQ_MEM_RECLAIM workqueue.  Doing so from a
+ * memory-reclaim context -- either a WQ_MEM_RECLAIM worker (most importantly
+ * writeback's wb_workfn running on bdi_wq) or an explicit reclaim task
+ * (PF_MEMALLOC) -- would trip check_flush_dependency() and risks a
+ * forward-progress deadlock; see commit b9f5dd57f4a5 ("nfs/localio: use
+ * dedicated workqueues for filesystem read and write").  In that case defer
+ * to nfslocaliod_workqueue.
+ *
+ * Otherwise (ordinary application/task context, e.g. O_DIRECT or fsync-driven
+ * submission) issue the IO inline: this preserves the NFS client's inherent
+ * application-context parallelism and avoids the per-IO workqueue hop.
+ */
+static inline bool nfs_local_defer_io(void)
+{
+	return (current->flags & PF_MEMALLOC) ||
+		current_is_workqueue_mem_reclaim();
+}
+
 static void nfs_local_do_read(struct nfs_local_kiocb *iocb,
 			      const struct rpc_call_ops *call_ops)
 {
@@ -711,7 +734,10 @@ static void nfs_local_do_read(struct nfs_local_kiocb *iocb,
 	hdr->res.eof = false;
 
 	INIT_WORK(&iocb->work, nfs_local_call_read);
-	queue_work(nfslocaliod_workqueue, &iocb->work);
+	if (nfs_local_defer_io())
+		queue_work(nfslocaliod_workqueue, &iocb->work);
+	else
+		nfs_local_call_read(&iocb->work);
 }
 
 static void
@@ -929,7 +955,10 @@ static void nfs_local_do_write(struct nfs_local_kiocb *iocb,
 	nfs_set_local_verifier(hdr->inode, hdr->res.verf, hdr->args.stable);
 
 	INIT_WORK(&iocb->work, nfs_local_call_write);
-	queue_work(nfslocaliod_workqueue, &iocb->work);
+	if (nfs_local_defer_io())
+		queue_work(nfslocaliod_workqueue, &iocb->work);
+	else
+		nfs_local_call_write(&iocb->work);
 }
 
 static struct nfs_local_kiocb *
diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h
index bc1ccdfbfb1d..3d2e426bcf27 100644
--- a/include/linux/workqueue.h
+++ b/include/linux/workqueue.h
@@ -659,6 +659,7 @@ extern void workqueue_set_min_active(struct workqueue_struct *wq,
 				     int min_active);
 extern struct work_struct *current_work(void);
 extern bool current_is_workqueue_rescuer(void);
+extern bool current_is_workqueue_mem_reclaim(void);
 extern bool workqueue_congested(int cpu, struct workqueue_struct *wq);
 extern unsigned int work_busy(struct work_struct *work);
 extern __printf(1, 2) void set_worker_desc(const char *fmt, ...);
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 03d9588e16d7..4add75c621da 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -6169,6 +6169,30 @@ bool current_is_workqueue_rescuer(void)
 	return worker && worker->rescue_wq;
 }
 
+/**
+ * current_is_workqueue_mem_reclaim - is %current a %WQ_MEM_RECLAIM worker?
+ *
+ * Determine whether %current is a workqueue worker executing on a workqueue
+ * created with %WQ_MEM_RECLAIM.  This mirrors the condition that
+ * check_flush_dependency() warns on: flushing (or otherwise waiting on) a
+ * !WQ_MEM_RECLAIM workqueue from such a context breaks the forward-progress
+ * guarantee and can deadlock.  Callers that may recurse into such a flush --
+ * e.g. NFS LOCALIO submitting into a stacked filesystem that flushes its own
+ * !WQ_MEM_RECLAIM workqueue -- can use this to decide whether they must defer
+ * the work to a !WQ_MEM_RECLAIM workqueue rather than run it inline.
+ *
+ * Return: %true if %current is a %WQ_MEM_RECLAIM worker.  %false otherwise.
+ */
+bool current_is_workqueue_mem_reclaim(void)
+{
+	struct worker *worker = current_wq_worker();
+
+	return worker &&
+		((worker->current_pwq->wq->flags &
+		  (WQ_MEM_RECLAIM | __WQ_LEGACY)) == WQ_MEM_RECLAIM);
+}
+EXPORT_SYMBOL_GPL(current_is_workqueue_mem_reclaim);
+
 /**
  * workqueue_congested - test whether a workqueue is congested
  * @cpu: CPU in question
-- 
2.44.0

[PATCH 2/3] NFS/localio: remove dead FLUSH_SYNC handling from nfs_local_commit

[Mike Snitzer]

nfs_local_commit() is reached only through nfs_initiate_commit(), and every
path that supplies its "how" argument has already cleared FLUSH_SYNC:
__nfs_commit_inode() strips it (how &= ~FLUSH_SYNC) before dispatch and does
its own waiting via wait_on_commit(), while the O_DIRECT path passes how=0.
filelayout issues its DS commit with a NULL localio, so it never enters
nfs_local_commit() at all.  The FLUSH_SYNC branch has therefore been dead
since it was introduced with commit 70ba381e1a43 ("nfs: add LOCALIO
support").

Remove the never-taken FLUSH_SYNC branch along with the completion plumbing
it was the sole user of: the struct nfs_local_fsync_ctx::done member, its
initialization, and the complete() call in nfs_local_fsync_work().  With the
branch gone the "how" parameter is unused, so drop it from nfs_local_commit()
and its callers.  No functional change.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
 fs/nfs/internal.h |  4 ++--
 fs/nfs/localio.c  | 15 ++-------------
 fs/nfs/write.c    |  2 +-
 3 files changed, 5 insertions(+), 16 deletions(-)

diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 0338603e9674..66dbc9befdbb 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -476,7 +476,7 @@ extern int nfs_local_doio(struct nfs_client *,
 			  const struct rpc_call_ops *);
 extern int nfs_local_commit(struct nfsd_file *,
 			    struct nfs_commit_data *,
-			    const struct rpc_call_ops *, int);
+			    const struct rpc_call_ops *);
 extern bool nfs_server_is_local(const struct nfs_client *clp);
 
 #else /* CONFIG_NFS_LOCALIO */
@@ -498,7 +498,7 @@ static inline int nfs_local_doio(struct nfs_client *clp,
 }
 static inline int nfs_local_commit(struct nfsd_file *localio,
 				struct nfs_commit_data *data,
-				const struct rpc_call_ops *call_ops, int how)
+				const struct rpc_call_ops *call_ops)
 {
 	return -EINVAL;
 }
diff --git a/fs/nfs/localio.c b/fs/nfs/localio.c
index d3e480888eb1..acbc2bddcf81 100644
--- a/fs/nfs/localio.c
+++ b/fs/nfs/localio.c
@@ -52,7 +52,6 @@ struct nfs_local_fsync_ctx {
 	struct nfsd_file	*localio;
 	struct nfs_commit_data	*data;
 	struct work_struct	work;
-	struct completion	*done;
 };
 
 static bool localio_enabled __read_mostly = true;
@@ -1100,8 +1099,6 @@ nfs_local_fsync_work(struct work_struct *work)
 	status = nfs_local_run_commit(nfs_to->nfsd_file_file(ctx->localio),
 				      ctx->data);
 	nfs_local_commit_done(ctx->data, status);
-	if (ctx->done != NULL)
-		complete(ctx->done);
 	nfs_local_fsync_ctx_free(ctx);
 
 	current->flags = old_flags;
@@ -1117,14 +1114,13 @@ nfs_local_fsync_ctx_alloc(struct nfs_commit_data *data,
 		ctx->localio = localio;
 		ctx->data = data;
 		INIT_WORK(&ctx->work, nfs_local_fsync_work);
-		ctx->done = NULL;
 	}
 	return ctx;
 }
 
 int nfs_local_commit(struct nfsd_file *localio,
 		     struct nfs_commit_data *data,
-		     const struct rpc_call_ops *call_ops, int how)
+		     const struct rpc_call_ops *call_ops)
 {
 	struct nfs_local_fsync_ctx *ctx;
 
@@ -1136,14 +1132,7 @@ int nfs_local_commit(struct nfsd_file *localio,
 	}
 
 	nfs_local_init_commit(data, call_ops);
-
-	if (how & FLUSH_SYNC) {
-		DECLARE_COMPLETION_ONSTACK(done);
-		ctx->done = &done;
-		queue_work(nfslocaliod_workqueue, &ctx->work);
-		wait_for_completion(&done);
-	} else
-		queue_work(nfslocaliod_workqueue, &ctx->work);
+	queue_work(nfslocaliod_workqueue, &ctx->work);
 
 	return 0;
 }
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 0d7f2c2e599c..3afe243597fb 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -1663,7 +1663,7 @@ int nfs_initiate_commit(struct rpc_clnt *clnt, struct nfs_commit_data *data,
 	dprintk("NFS: initiated commit call\n");
 
 	if (localio)
-		return nfs_local_commit(localio, data, call_ops, how);
+		return nfs_local_commit(localio, data, call_ops);
 
 	task = rpc_run_task(&task_setup_data);
 	if (IS_ERR(task))
-- 
2.44.0

[PATCH 3/3] NFS/localio: issue commit inline when not in a memory-reclaim context

[Mike Snitzer]

Extend the memory-reclaim-context test used for LOCALIO reads and writes to
the commit (fsync) path.  As with data IO, bouncing every commit through the
dedicated !WQ_MEM_RECLAIM nfslocaliod_workqueue is only required when the
submitting context is a memory-reclaim context: nfs_local_run_commit() calls
vfs_fsync_range(), which may flush the underlying filesystem's own
!WQ_MEM_RECLAIM workqueue, and doing so from a WQ_MEM_RECLAIM worker or a
PF_MEMALLOC task trips check_flush_dependency().

The writeback path does exercise this: nfs_write_inode() (the ->write_inode
super_op) runs under wb_workfn on the WQ_MEM_RECLAIM bdi_wq and reaches
nfs_local_commit() via __nfs_commit_inode(), so that case must keep
deferring.  Application-context commits -- fsync (nfs_file_fsync), O_DIRECT
(nfs_direct), and copy/clone (nfs42) -- are not in a reclaim context and now
run the fsync inline via nfs_local_defer_io(), avoiding the per-commit
workqueue hop.

Completion (nfs_commit_release_pages -> nfs_commit_end) then runs
synchronously in the submitting context; higher layers already cope with
this, as __nfs_commit_inode() dispatches the commit async and waits for it
separately via wait_on_commit().

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
---
 fs/nfs/localio.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/localio.c b/fs/nfs/localio.c
index acbc2bddcf81..f42b6112a613 100644
--- a/fs/nfs/localio.c
+++ b/fs/nfs/localio.c
@@ -1132,7 +1132,19 @@ int nfs_local_commit(struct nfsd_file *localio,
 	}
 
 	nfs_local_init_commit(data, call_ops);
-	queue_work(nfslocaliod_workqueue, &ctx->work);
+
+	/*
+	 * Run the commit (fsync) inline when not in a memory-reclaim context,
+	 * rather than bouncing through nfslocaliod_workqueue; see
+	 * nfs_local_defer_io().  Completion (nfs_commit_release_pages ->
+	 * nfs_commit_end) then runs synchronously, which higher layers cope
+	 * with: __nfs_commit_inode() dispatches async and waits via
+	 * wait_on_commit().
+	 */
+	if (nfs_local_defer_io())
+		queue_work(nfslocaliod_workqueue, &ctx->work);
+	else
+		nfs_local_fsync_work(&ctx->work);
 
 	return 0;
 }
-- 
2.44.0