From: Mike Snitzer <snitzer@kernel.org>
To: Anna Schumaker <anna@kernel.org>, Trond Myklebust <trondmy@kernel.org>
Cc: Mike Snitzer <snitzer@hammerspace.com>,
linux-nfs@vger.kernel.org,
Jonathan Flynn <jonathan.flynn@hammerspace.com>
Subject: [PATCH v3 09/12] NFSv4/flexfiles: drop NFS4_FF_MIRROR_STAT_AVAIL
Date: Fri, 6 Nov 2026 13:20:57 -0500 [thread overview]
Message-ID: <20261106182105.78611-10-snitzer@kernel.org> (raw)
In-Reply-To: <20261106182105.78611-1-snitzer@kernel.org>
From: Mike Snitzer <snitzer@hammerspace.com>
Each DS RPC sets NFS4_FF_MIRROR_STAT_AVAIL twice, at start and at
completion. These are two atomic operations per RPC on a cacheline that
all stripes of the mirror share. The flag is read once per report
interval.
The flag also hides most stripes. It is per mirror, but
ff_layout_mirror_prepare_stats() tests and clears it in the per-stripe
loop. The first stripe clears it, so a LAYOUTSTATS report contains only
the lowest usable stripe of each mirror.
Remove the flag. Record each stripe's completed ops when it is
reported, and skip a stripe whose count did not change. A report then
contains each stripe that completed I/O since its last report, up to the
dev_limit of the caller. The reported values do not change.
LAYOUTRETURN still reports every stripe that fits.
The new last_reported_ops is in an existing hole in the read-mostly part
of the stripe. The I/O paths do not touch it, and the stripe stays 320
bytes. Both callers of ff_layout_mirror_prepare_stats() hold the
inode's i_lock, which serializes it.
NFS4_FF_MIRROR_STAT_AVAIL was the only flag in
nfs4_ff_layout_mirror::flags, so remove that field too.
Signed-off-by: Mike Snitzer <snitzer@hammerspace.com>
---
fs/nfs/flexfilelayout/flexfilelayout.c | 38 ++++++++++++++------------
fs/nfs/flexfilelayout/flexfilelayout.h | 5 ++--
2 files changed, 22 insertions(+), 21 deletions(-)
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 45bb2b15b728b..4f27b1ea8b621 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -812,7 +812,6 @@ nfs4_ff_layout_stat_io_start_read(struct inode *inode,
report = nfs4_ff_layoutstat_start_io(dss_info, &dss_info->read_stat,
requested, now);
spin_unlock(&dss_info->read_stat.lock);
- set_bit(NFS4_FF_MIRROR_STAT_AVAIL, &mirror->flags);
if (report)
pnfs_report_layoutstat(inode, nfs_io_gfp_mask());
@@ -832,7 +831,6 @@ nfs4_ff_layout_stat_io_end_read(struct rpc_task *task,
requested, completed,
ktime_get(), task->tk_start);
spin_unlock(&dss_info->read_stat.lock);
- set_bit(NFS4_FF_MIRROR_STAT_AVAIL, &mirror->flags);
}
static void
@@ -848,7 +846,6 @@ nfs4_ff_layout_stat_io_start_write(struct inode *inode,
report = nfs4_ff_layoutstat_start_io(dss_info, &dss_info->write_stat,
requested, now);
spin_unlock(&dss_info->write_stat.lock);
- set_bit(NFS4_FF_MIRROR_STAT_AVAIL, &mirror->flags);
if (report)
pnfs_report_layoutstat(inode, nfs_io_gfp_mask());
@@ -872,7 +869,6 @@ nfs4_ff_layout_stat_io_end_write(struct rpc_task *task,
requested, completed,
ktime_get(), task->tk_start);
spin_unlock(&dss_info->write_stat.lock);
- set_bit(NFS4_FF_MIRROR_STAT_AVAIL, &mirror->flags);
}
static struct nfs4_ff_layout_ds *
@@ -3139,6 +3135,7 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,
struct nfs4_ff_layout_mirror *mirror;
struct nfs4_ff_layout_ds_stripe *dss_info;
struct nfs4_ff_layout_ds *mirror_ds;
+ __u64 read_count, read_bytes, write_count, write_bytes, ops;
int i = 0, dss_id;
rcu_read_lock();
@@ -3150,13 +3147,26 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,
mirror_ds = rcu_dereference(dss_info->mirror_ds);
if (IS_ERR_OR_NULL(mirror_ds))
continue;
- if (!test_and_clear_bit(NFS4_FF_MIRROR_STAT_AVAIL,
- &mirror->flags) &&
+
+ spin_lock(&dss_info->read_stat.lock);
+ read_count = dss_info->read_stat.io_stat.ops_completed;
+ read_bytes = dss_info->read_stat.io_stat.bytes_completed;
+ spin_unlock(&dss_info->read_stat.lock);
+ spin_lock(&dss_info->write_stat.lock);
+ write_count = dss_info->write_stat.io_stat.ops_completed;
+ write_bytes = dss_info->write_stat.io_stat.bytes_completed;
+ spin_unlock(&dss_info->write_stat.lock);
+
+ /* Skip a stripe with no completions since its last report */
+ ops = read_count + write_count;
+ if (ops == dss_info->last_reported_ops &&
type != NFS4_FF_OP_LAYOUTRETURN)
continue;
+
/* mirror refcount put in cleanup_layoutstats */
if (!refcount_inc_not_zero(&mirror->ref))
continue;
+ dss_info->last_reported_ops = ops;
/* The pin holds a reference; it is exchanged out only
* under i_lock. Put in ff_layout_free_layoutstats().
*/
@@ -3166,18 +3176,10 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,
NFS4_DEVICEID4_SIZE);
devinfo->offset = 0;
devinfo->length = NFS4_MAX_UINT64;
- spin_lock(&dss_info->read_stat.lock);
- devinfo->read_count =
- dss_info->read_stat.io_stat.ops_completed;
- devinfo->read_bytes =
- dss_info->read_stat.io_stat.bytes_completed;
- spin_unlock(&dss_info->read_stat.lock);
- spin_lock(&dss_info->write_stat.lock);
- devinfo->write_count =
- dss_info->write_stat.io_stat.ops_completed;
- devinfo->write_bytes =
- dss_info->write_stat.io_stat.bytes_completed;
- spin_unlock(&dss_info->write_stat.lock);
+ devinfo->read_count = read_count;
+ devinfo->read_bytes = read_bytes;
+ devinfo->write_count = write_count;
+ devinfo->write_bytes = write_bytes;
devinfo->layout_type = LAYOUT_FLEX_FILES;
devinfo->ld_private.ops = &layoutstat_ops;
priv->dss_info = dss_info;
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.h b/fs/nfs/flexfilelayout/flexfilelayout.h
index f8280fa991bb8..69dfdcfb9664d 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.h
+++ b/fs/nfs/flexfilelayout/flexfilelayout.h
@@ -95,6 +95,8 @@ struct nfs4_ff_layout_ds_stripe {
struct nfs_file_localio nfl;
/* Published once by the first I/O; see nfs4_ff_layoutstat_set_start_time() */
atomic64_t start_time;
+ /* Completed ops, both directions, as of this stripe's last report */
+ __u64 last_reported_ops;
struct nfs4_ff_layoutstat read_stat;
struct nfs4_ff_layoutstat write_stat;
};
@@ -105,12 +107,9 @@ struct nfs4_ff_layout_mirror {
u32 dss_count;
struct nfs4_ff_layout_ds_stripe *dss;
refcount_t ref;
- unsigned long flags;
u32 report_interval;
};
-#define NFS4_FF_MIRROR_STAT_AVAIL (0)
-
struct nfs4_ff_layout_segment {
struct pnfs_layout_segment generic_hdr;
u64 stripe_unit;
--
2.52.0
next prev parent reply other threads:[~2026-11-06 18:21 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-06 18:20 [PATCH v3 00/12] NFSv4/flexfiles: scale layoutstats for striped layouts Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 01/12] NFSv4/flexfiles: move layoutstats accounting to a per-stripe lock Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 02/12] NFSv4/flexfiles: drop the now-unused per-mirror lock Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 03/12] NFSv4/flexfiles: cacheline align the per-stripe layoutstats Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 04/12] NFSv4/flexfiles: make dss_info->start_time write-once Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 05/12] NFSv4/flexfiles: drop the redundant atomic from the busy timer Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 06/12] NFSv4/flexfiles: pair the in-flight count with the layoutstats updates Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 07/12] NFSv4/flexfiles: derive ffil_ops_requested rather than storing it Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 08/12] NFSv4/flexfiles: give each direction its own lock and cacheline Mike Snitzer
2026-11-06 18:20 ` Mike Snitzer [this message]
2026-11-06 18:20 ` [PATCH v3 10/12] NFSv4/flexfiles: rotate LAYOUTSTATS reporting across a mirror's stripes Mike Snitzer
2026-11-06 18:20 ` [PATCH v3 11/12] NFSv4/flexfiles: take the report decision out of the critical section Mike Snitzer
2026-11-06 18:21 ` [PATCH v3 12/12] NFSv4/flexfiles: share one helper for the start of a read and a write Mike Snitzer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261106182105.78611-10-snitzer@kernel.org \
--to=snitzer@kernel.org \
--cc=anna@kernel.org \
--cc=jonathan.flynn@hammerspace.com \
--cc=linux-nfs@vger.kernel.org \
--cc=snitzer@hammerspace.com \
--cc=trondmy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox