From: Mike Snitzer <snitzer@kernel.org>
To: Anna Schumaker <anna@kernel.org>, Trond Myklebust <trondmy@kernel.org>
Cc: Mike Snitzer <snitzer@hammerspace.com>,
linux-nfs@vger.kernel.org,
Jonathan Flynn <jonathan.flynn@hammerspace.com>
Subject: [PATCH v3 09/12] NFSv4/flexfiles: drop NFS4_FF_MIRROR_STAT_AVAIL
Date: Fri, 6 Nov 2026 13:20:57 -0500 [thread overview]
Message-ID: <20261106182105.78611-10-snitzer@kernel.org> (raw)
In-Reply-To: <20261106182105.78611-1-snitzer@kernel.org>
From: Mike Snitzer <snitzer@hammerspace.com>
Each DS RPC sets NFS4_FF_MIRROR_STAT_AVAIL twice, at start and at
completion. These are two atomic operations per RPC on a cacheline that
all stripes of the mirror share. The flag is read once per report
interval.
The flag also hides most stripes. It is per mirror, but
ff_layout_mirror_prepare_stats() tests and clears it in the per-stripe
loop. The first stripe clears it, so a LAYOUTSTATS report contains only
the lowest usable stripe of each mirror.
Remove the flag. Record each stripe's completed ops when it is
reported, and skip a stripe whose count did not change. A report then
contains each stripe that completed I/O since its last report, up to the
dev_limit of the caller. The reported values do not change.
LAYOUTRETURN still reports every stripe that fits.
The new last_reported_ops is in an existing hole in the read-mostly part
of the stripe. The I/O paths do not touch it, and the stripe stays 320
bytes. Both callers of ff_layout_mirror_prepare_stats() hold the
inode's i_lock, which serializes it.
NFS4_FF_MIRROR_STAT_AVAIL was the only flag in
nfs4_ff_layout_mirror::flags, so remove that field too.
Signed-off-by: Mike Snitzer <snitzer@hammerspace.com>
---
fs/nfs/flexfilelayout/flexfilelayout.c | 38 ++++++++++++++------------
fs/nfs/flexfilelayout/flexfilelayout.h | 5 ++--
2 files changed, 22 insertions(+), 21 deletions(-)
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 45bb2b15b728b..4f27b1ea8b621 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -812,7 +812,6 @@ nfs4_ff_layout_stat_io_start_read(struct inode *inode,
report = nfs4_ff_layoutstat_start_io(dss_info, &dss_info->read_stat,
requested, now);
spin_unlock(&dss_info->read_stat.lock);
- set_bit(NFS4_FF_MIRROR_STAT_AVAIL, &mirror->flags);
if (report)
pnfs_report_layoutstat(inode, nfs_io_gfp_mask());
@@ -832,7 +831,6 @@ nfs4_ff_layout_stat_io_end_read(struct rpc_task *task,
requested, completed,
ktime_get(), task->tk_start);
spin_unlock(&dss_info->read_stat.lock);
- set_bit(NFS4_FF_MIRROR_STAT_AVAIL, &mirror->flags);
}
static void
@@ -848,7 +846,6 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode,
report = nfs4_ff_layoutstat_start_io(dss_info, &dss_info->write_stat,
requested, now);
spin_unlock(&dss_info->write_stat.lock);
- set_bit(NFS4_FF_MIRROR_STAT_AVAIL, &mirror->flags);
if (report)
pnfs_report_layoutstat(inode, nfs_io_gfp_mask());
@@ -872,7 +869,6 @@ nfs4_ff_layout_stat_io_end_write(struct rpc_task *task,
requested, completed,
ktime_get(), task->tk_start);
spin_unlock(&dss_info->write_stat.lock);
- set_bit(NFS4_FF_MIRROR_STAT_AVAIL, &mirror->flags);
}
static struct nfs4_ff_layout_ds *
@@ -3139,6 +3135,7 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,
struct nfs4_ff_layout_mirror *mirror;
struct nfs4_ff_layout_ds_stripe *dss_info;
struct nfs4_ff_layout_ds *mirror_ds;
+ __u64 read_count, read_bytes, write_count, write_bytes, ops;
int i = 0, dss_id;
rcu_read_lock();
@@ -3150,13 +3147,26 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,
mirror_ds = rcu_dereference(dss_info->mirror_ds);
if (IS_ERR_OR_NULL(mirror_ds))
continue;
- if (!test_and_clear_bit(NFS4_FF_MIRROR_STAT_AVAIL,
- &mirror->flags) &&
+
+ spin_lock(&dss_info->read_stat.lock);
+ read_count = dss_info->read_stat.io_stat.ops_completed;
+ read_bytes = dss_info->read_stat.io_stat.bytes_completed;
+ spin_unlock(&dss_info->read_stat.lock);
+ spin_lock(&dss_info->write_stat.lock);
+ write_count = dss_info->write_stat.io_stat.ops_completed;
+ write_bytes = dss_info->write_stat.io_stat.bytes_completed;
+ spin_unlock(&dss_info->write_stat.lock);
+
+ /* Skip a stripe with no completions since its last report */
+ ops = read_count + write_count;
+ if (ops == dss_info->last_reported_ops &&
type != NFS4_FF_OP_LAYOUTRETURN)
continue;
+
/* mirror refcount put in cleanup_layoutstats */
if (!refcount_inc_not_zero(&mirror->ref))
continue;
+ dss_info->last_reported_ops = ops;
/* The pin holds a reference; it is exchanged out only
* under i_lock. Put in ff_layout_free_layoutstats().
*/
@@ -3166,18 +3176,10 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,
NFS4_DEVICEID4_SIZE);
devinfo->offset = 0;
devinfo->length = NFS4_MAX_UINT64;
- spin_lock(&dss_info->read_stat.lock);
- devinfo->read_count =
- dss_info->read_stat.io_stat.ops_completed;
- devinfo->read_bytes =
- dss_info->read_stat.io_stat.bytes_completed;
- spin_unlock(&dss_info->read_stat.lock);
- spin_lock(&dss_info->write_stat.lock);
- devinfo->write_count =
- dss_info->write_stat.io_stat.ops_completed;
- devinfo->write_bytes =
- dss_info->write_stat.io_stat.bytes_completed;
- spin_unlock(&dss_info->write_stat.lock);
+ devinfo->read_count = read_count;
+ devinfo->read_bytes = read_bytes;
+ devinfo->write_count = write_count;
+ devinfo->write_bytes = write_bytes;
devinfo->layout_type = LAYOUT_FLEX_FILES;
devinfo->ld_private.ops = &layoutstat_ops;
priv->dss_info = dss_info;
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.h b/fs/nfs/flexfilelayout/flexfilelayout.h
index f8280fa991bb8..69dfdcfb9664d 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.h
+++ b/fs/nfs/flexfilelayout/flexfilelayout.h
@@ -95,6 +95,8 @@ struct nfs4_ff_layout_ds_stripe {
struct nfs_file_localio nfl;
/* Published once by the first I/O; see nfs4_ff_layoutstat_set_start_time() */
atomic64_t start_time;
+ /* Completed ops, both directions, as of this stripe's last report */
+ __u64 last_reported_ops;
struct nfs4_ff_layoutstat read_stat;
struct nfs4_ff_layoutstat write_stat;
};
@@ -105,12 +107,9 @@ struct nfs4_ff_layout_mirror {
u32 dss_count;
struct nfs4_ff_layout_ds_stripe *dss;
refcount_t ref;
- unsigned long flags;
u32 report_interval;
};
-#define NFS4_FF_MIRROR_STAT_AVAIL (0)
-
struct nfs4_ff_layout_segment {
struct pnfs_layout_segment generic_hdr;
u64 stripe_unit;
--
2.52.0
next prev parent reply other threads:[~2026-11-06 18:21 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-06 18:20 [PATCH v3 00/12] NFSv4/flexfiles: scale layoutstats for striped layouts Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 01/12] NFSv4/flexfiles: move layoutstats accounting to a per-stripe lock Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 02/12] NFSv4/flexfiles: drop the now-unused per-mirror lock Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 03/12] NFSv4/flexfiles: cacheline align the per-stripe layoutstats Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 04/12] NFSv4/flexfiles: make dss_info->start_time write-once Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 05/12] NFSv4/flexfiles: drop the redundant atomic from the busy timer Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 06/12] NFSv4/flexfiles: pair the in-flight count with the layoutstats updates Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 07/12] NFSv4/flexfiles: derive ffil_ops_requested rather than storing it Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 08/12] NFSv4/flexfiles: give each direction its own lock and cacheline Mike Snitzer
2026-11-06 18:20 ` Mike Snitzer [this message]
2026-11-06 18:20 ` [PATCH v3 10/12] NFSv4/flexfiles: rotate LAYOUTSTATS reporting across a mirror's stripes Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 11/12] NFSv4/flexfiles: take the report decision out of the critical section Mike Snitzer
2026-11-06 18:21 ` [PATCH v3 12/12] NFSv4/flexfiles: share one helper for the start of a read and a write Mike Snitzer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261106182105.78611-10-snitzer@kernel.org \
--to=snitzer@kernel.org \
--cc=anna@kernel.org \
--cc=jonathan.flynn@hammerspace.com \
--cc=linux-nfs@vger.kernel.org \
--cc=snitzer@hammerspace.com \
--cc=trondmy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.