[PATCH 10/11] NFSv4/flexfiles: rotate LAYOUTSTATS reporting across a mirror's stripes

[Mike Snitzer <snitzer@kernel.org>]

From: Mike Snitzer <snitzer@hammerspace.com>

Both callers of ff_layout_mirror_prepare_stats() cap it at four
ff_layoutupdate4 entries -- PNFS_LAYOUTSTATS_MAXDEV for LAYOUTSTATS and
ARRAY_SIZE(ff_args->devinfo), which is FF_LAYOUTSTATS_MAXDEV, for
LAYOUTRETURN.  The scan walks mirror->dss[] from index 0 every time and
stops once the budget is spent.

  What the server sees without this patch: for a mirror with more than
  four busy stripes, the same low dss_ids every report, forever.  The
  previous patch widened that from one stripe to four -- it did not stop
  it being the same four.  On the 32-stripe RAID0 layout this series
  targets, stripes 4..31 are reported only when the layout is returned,
  so a data server that is only ever backing a high stripe index stays
  invisible to the server's placement decisions no matter how much I/O it
  does.

Remember where each mirror's scan stopped and resume there.  The cursor
advances past each stripe that actually fills a slot, so a report picks
up where its predecessor left off and idle stripes, which are skipped
without consuming a slot, do not stall the rotation.

  What the server sees with this patch: still up to four entries per
  report, but rotating, so every busy stripe of a mirror is reported
  within ceil(busy_stripes / 4) report intervals -- eight intervals, or
  about eight minutes at the default 60 second interval, for 32 busy
  stripes.  A given stripe's numbers therefore arrive less often than
  before but, unlike before, they arrive.  The values are unchanged:
  cumulative ops_completed and bytes_completed since the layout was
  granted, so a server differencing successive reports for a stripe still
  gets a correct delta, just over a longer window.  Nothing about the
  encoding, the deviceid or the ordering within a report changes.

The mirror loop gets the same dev_limit test as the stripe loop, so a
full report stops scanning instead of walking the remaining mirrors to
break immediately on each.

[jonathan.flynn: read dss_report_start once per mirror, before the stripe
 loop.  The loop advances it after each stripe it reports, so re-reading
 it moved the index twice: stripes were skipped, and on a mirror with
 stripes not yet written (and so skipped) the index wrapped back onto a
 stripe already reported.  That put the same deviceid in one ff_iostats4
 array twice with identical counters.]
Signed-off-by: Jonathan Flynn <jonathan.flynn@hammerspace.com>
Signed-off-by: Mike Snitzer <snitzer@hammerspace.com>
---
 fs/nfs/flexfilelayout/flexfilelayout.c | 22 +++++++++++++++++++---
 fs/nfs/flexfilelayout/flexfilelayout.h |  7 +++++++
 2 files changed, 26 insertions(+), 3 deletions(-)

diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 670780bb3d353..591363e2cf460 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -3153,14 +3153,28 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,
 	struct nfs4_ff_layout_ds_stripe *dss_info;
 	struct nfs4_ff_layout_ds *mirror_ds;
 	__u64 read_count, read_bytes, write_count, write_bytes, ops;
-	int i = 0, dss_id;
+	u32 dss_id, scanned, base;
+	int i = 0;
 
 	rcu_read_lock();
 	list_for_each_entry(mirror, &ff_layout->mirrors, mirrors) {
-		for (dss_id = 0; dss_id < mirror->dss_count; ++dss_id) {
-			dss_info = &mirror->dss[dss_id];
+		if (i >= dev_limit)
+			break;
+		base = mirror->dss_report_start;
+		for (scanned = 0; scanned < mirror->dss_count; ++scanned) {
 			if (i >= dev_limit)
 				break;
+			/* Resume where the last report of this mirror left
+			 * off; one subtraction suffices to wrap, as both
+			 * terms are below dss_count.  Use the start as it
+			 * was on entry: this loop advances it for the next
+			 * report, and re-reading it would skip stripes and
+			 * wrap back onto one already reported.
+			 */
+			dss_id = base + scanned;
+			if (dss_id >= mirror->dss_count)
+				dss_id -= mirror->dss_count;
+			dss_info = &mirror->dss[dss_id];
 			mirror_ds = rcu_dereference(dss_info->mirror_ds);
 			if (IS_ERR_OR_NULL(mirror_ds))
 				continue;
@@ -3189,6 +3203,8 @@ ff_layout_mirror_prepare_stats(struct pnfs_layout_hdr *lo,
 			if (!refcount_inc_not_zero(&mirror->ref))
 				continue;
 			dss_info->last_reported_ops = ops;
+			mirror->dss_report_start =
+				dss_id + 1 < mirror->dss_count ? dss_id + 1 : 0;
 			/* The pin holds a reference; it is exchanged out only
 			 * under i_lock.  Put in ff_layout_free_layoutstats().
 			 */
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.h b/fs/nfs/flexfilelayout/flexfilelayout.h
index 743957cd12300..33724f2db5d62 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.h
+++ b/fs/nfs/flexfilelayout/flexfilelayout.h
@@ -141,6 +141,13 @@ struct nfs4_ff_layout_mirror {
 	struct nfs4_ff_layout_ds_stripe *dss;
 	refcount_t			ref;
 	u32				report_interval;
+	/*
+	 * dss[] index to begin the next LAYOUTSTATS scan of this mirror at,
+	 * so that a mirror with more stripes than a report has room for
+	 * rotates through them rather than always reporting the lowest.
+	 * Serialised by the inode's i_lock, like last_reported_ops.
+	 */
+	u32				dss_report_start;
 };
 
 struct nfs4_ff_layout_segment {
-- 
2.52.0