[PATCH] xfs: add per-mount read/write I/O completion counters

[PATCH] xfs: add per-mount read/write I/O completion counters

[Eric Peterson]

From: Eric Peterson <eric.peterson@hpe.com>

Add two per-mount statistics counters, xs_read_completions and
xs_write_completions, to complement the existing xs_read_calls and
xs_write_calls counters. The existing counters count I/O submissions
(entries); the new counters count I/O completions. The pair (calls,
completions) lets a consumer compute outstanding I/O as a queue depth
(calls - completions) and, via Little's law, derive an approximate
response time in userspace without any hot-path timestamping.

The counters are plain monotonic increments (no clock reads), so they
add negligible cost to the read/write path. Per-op timestamping was
deliberately not used: a clock read on the hot path costs ~20-30 ns on
TSC but hundreds of ns to ~1 us on HPET, which would be a regression for
general users. Queue depth from completion counters is an approximation
(instantaneous depth); this is a deliberate design choice, not
a placeholder.

Completions are accounted at exactly the same sites where XFS already
accounts the xs_*_bytes counters, so their semantics match the existing
byte counters per path:

  - Reads are counted at the frame in xfs_file_read_iter and
    xfs_file_splice_read.
  - Buffered writes are counted at the frame, i.e. when data reaches
    the page cache, mirroring how xs_write_bytes is accounted for
    buffered writes -- not at physical writeback.
  - DAX writes are counted at the frame after the synchronous
    dax_iomap_rw copy returns, mirroring xs_write_bytes for DAX.
  - Direct I/O writes are counted at true completion in
    xfs_dio_write_end_io, which is async-safe and fires for both sync
    and async DIO, mirroring xs_write_bytes for DIO.

Caveat: async O_DIRECT reads are counted at submission, not completion,
because XFS has no read end_io today (iomap_dio_rw is called with NULL
ops for reads). This matches the existing read-byte semantics.

The counters are uint32_t and wrap like the existing xs_*_calls
counters; userspace diffs handle wrap.

The per-mount stats file gains a new appended "rwcmpl" line printing
write and read completions. The existing "rw" line is unchanged, so
positional parsers of "rw" are unaffected:

  rw     <write_calls>       <read_calls>
  rwcmpl <write_completions> <read_completions>

Signed-off-by: Eric Peterson <eric.peterson@hpe.com>
---

Notes for reviewers (not part of the commit log):

* Placement: the new "rwcmpl" group is inserted between "rw" and
  "attr" in the xstats[] table. The "rw" line itself is unchanged,
  and "rwcmpl" is appended after it, but lines below "rw" in
  /proc/fs/xfs/stat shift by one for strictly positional parsers. I
  can instead append the group at the END of the table if preferred.

* checkpatch --strict reports two CHECKs preferring u32 over uint32_t
  for the new fields. They are kept as uint32_t to match struct
  __xfsstats, whose every field is uint32_t; changing only these two
  would break local consistency.

* Testing: fstests -g auto on v6.12.74 shows baseline and patched
  fail the identical 6/1277 tests -- zero regressions. The rwcmpl
  interface was verified on hardware (rw >= rwcmpl, counters
  advance under load). This for-next port applies cleanly with no
  drift and compiles clean; a runtime -g quick smoke on for-next was
  omitted as the logic is identical to the tested v6.12.74 patch.
 fs/xfs/xfs_file.c  | 11 +++++++++--
 fs/xfs/xfs_stats.c |  3 ++-
 fs/xfs/xfs_stats.h |  2 ++
 3 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 426a67b813..3ecd4ed534 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -347,8 +347,10 @@ xfs_file_read_iter(
 	else
 		ret = xfs_file_buffered_read(iocb, to);
 
-	if (ret > 0)
+	if (ret > 0) {
 		XFS_STATS_ADD(mp, xs_read_bytes, ret);
+		XFS_STATS_INC(mp, xs_read_completions);
+	}
 	return ret;
 }
 
@@ -375,8 +377,10 @@ xfs_file_splice_read(
 	xfs_ilock(ip, XFS_IOLOCK_SHARED);
 	ret = filemap_splice_read(in, ppos, pipe, len, flags);
 	xfs_iunlock(ip, XFS_IOLOCK_SHARED);
-	if (ret > 0)
+	if (ret > 0) {
 		XFS_STATS_ADD(mp, xs_read_bytes, ret);
+		XFS_STATS_INC(mp, xs_read_completions);
+	}
 	return ret;
 }
 
@@ -663,6 +667,7 @@ xfs_dio_write_end_io(
 	 * for it on submission.
 	 */
 	XFS_STATS_ADD(ip->i_mount, xs_write_bytes, size);
+	XFS_STATS_INC(ip->i_mount, xs_write_completions);
 
 	/*
 	 * We can allocate memory here while doing writeback on behalf of
@@ -1032,6 +1037,7 @@ xfs_file_dax_write(
 
 	if (ret > 0) {
 		XFS_STATS_ADD(ip->i_mount, xs_write_bytes, ret);
+		XFS_STATS_INC(ip->i_mount, xs_write_completions);
 
 		/* Handle various SYNC-type writes */
 		ret = generic_write_sync(iocb, ret);
@@ -1098,6 +1104,7 @@ xfs_file_buffered_write(
 
 	if (ret > 0) {
 		XFS_STATS_ADD(ip->i_mount, xs_write_bytes, ret);
+		XFS_STATS_INC(ip->i_mount, xs_write_completions);
 		/* Handle various SYNC-type writes */
 		ret = generic_write_sync(iocb, ret);
 	}
diff --git a/fs/xfs/xfs_stats.c b/fs/xfs/xfs_stats.c
index c13d600732..5b276666b6 100644
--- a/fs/xfs/xfs_stats.c
+++ b/fs/xfs/xfs_stats.c
@@ -40,7 +40,8 @@ int xfs_stats_format(struct xfsstats __percpu *stats, char *buf)
 		{ "log",		xfsstats_offset(xs_try_logspace)},
 		{ "push_ail",		xfsstats_offset(xs_xstrat_quick)},
 		{ "xstrat",		xfsstats_offset(xs_write_calls)	},
-		{ "rw",			xfsstats_offset(xs_attr_get)	},
+		{ "rw",			xfsstats_offset(xs_write_completions)	},
+		{ "rwcmpl",		xfsstats_offset(xs_attr_get)	},
 		{ "attr",		xfsstats_offset(xs_iflush_count)},
 		{ "icluster",		xfsstats_offset(xs_inodes_active) },
 		{ "vnodes",		xfsstats_offset(xb_get)		},
diff --git a/fs/xfs/xfs_stats.h b/fs/xfs/xfs_stats.h
index 57c32b86c3..608d12d0c6 100644
--- a/fs/xfs/xfs_stats.h
+++ b/fs/xfs/xfs_stats.h
@@ -93,6 +93,8 @@ struct __xfsstats {
 	uint32_t		xs_xstrat_split;
 	uint32_t		xs_write_calls;
 	uint32_t		xs_read_calls;
+	uint32_t		xs_write_completions;
+	uint32_t		xs_read_completions;
 	uint32_t		xs_attr_get;
 	uint32_t		xs_attr_set;
 	uint32_t		xs_attr_remove;
-- 
2.39.5

Re: [PATCH] xfs: add per-mount read/write I/O completion counters

[Dave Chinner]

On Thu, Aug 27, 2026 at 09:34:29PM -0600, Eric Peterson wrote:
> From: Eric Peterson <eric.peterson@hpe.com>
> 
> Add two per-mount statistics counters, xs_read_completions and
> xs_write_completions, to complement the existing xs_read_calls and
> xs_write_calls counters. The existing counters count I/O submissions
> (entries); the new counters count I/O completions. The pair (calls,
> completions) lets a consumer compute outstanding I/O as a queue depth
> (calls - completions) and, via Little's law, derive an approximate
> response time in userspace without any hot-path timestamping.

I'm not sure the fs is the right place for this - the bdev has long
exposed enough information for filesystem-wide queue depths to be
monitored directly.  e.g:

$ man iostat |grep -A 1 aqu-sz
	aqu-sz  The average queue length of the requests that were
		issued to the device.
$  pminfo -t disk.dev.avg_qlen
disk.dev.avg_qlen [average read and write queue length]
$

and so on.

Hence this really doesn't seem like something we should be trying to
infer from indirect filesystem stats. Why can't you use the bdev
stats to get the actual filesystem wide queue depth information?

-Dave.
-- 
Dave Chinner
dgc@kernel.org

Re: [PATCH] xfs: add per-mount read/write I/O completion counters

[Eric Peterson]

On Mon, Aug 31, 2026 at 07:26:52AM +1000, Dave Chinner wrote:
> Hence this really doesn't seem like something we should be trying to
> infer from indirect filesystem stats. Why can't you use the bdev
> stats to get the actual filesystem wide queue depth information?

The block device measures the device queue, which is a different
quantity than filesystem outstanding I/O - not just a lower-layer view
of the same thing.

Below are three cases where filesystem queue depth is not what the block
layer sees:

1. Cache hits never reach the block layer. Under a heavy read workload
   with a warm cache, a large share of ops are serviced from the page
   cache and are never seen at the block level. Device queue depth can
   sit near zero while the filesystem is servicing a very high op rate.

2. Filesystem ops don't map 1:1 to block I/O. A single read or write can
   produce one block I/O, several (metadata, readahead, writeback
   coalescing), or none at all. So device queue depth isn't the
   filesystem's outstanding-operation count.

3. Work can be outstanding inside the filesystem before any block I/O is
   issued - waiting on locks, log space/reservation, delalloc, etc.
   Such I/O has entered the filesystem but is invisible at the bdev.

The block-device queue depth answers "how deep is the device queue,"
which is not the same as "how much work is outstanding in the
filesystem." When the filesystem is just one layer an I/O passes
through, the block stats fold the layers together and structurally
cannot isolate the filesystem's own contribution.

To be clear about scope: I'm not proposing a queue-depth feature in
the kernel. The change just adds read/write completion counters to pair
with the existing call (submission) counters, so userspace can compute
outstanding I/O and derive a response-time estimate itself. The kernel
side is only exposing the complementary raw signal that's currently
missing - calls are counted, completions are not.

Being upfront: what userspace derives from this is an instantaneous
approximation, not a precise time-weighted queue length. It's meant as
a cheap, always-on aggregate, not a replacement for accurate per-op
tooling.

Does exposing the completion side of the existing call counters seem
reasonable on that basis?

-Eric

Re: [PATCH] xfs: add per-mount read/write I/O completion counters

[Carlos Maiolino]

On Sun, Aug 30, 2026 at 06:47:00PM -0600, Eric Peterson wrote:
> On Mon, Aug 31, 2026 at 07:26:52AM +1000, Dave Chinner wrote:
> > Hence this really doesn't seem like something we should be trying to
> > infer from indirect filesystem stats. Why can't you use the bdev
> > stats to get the actual filesystem wide queue depth information?
> 
> The block device measures the device queue, which is a different
> quantity than filesystem outstanding I/O - not just a lower-layer view
> of the same thing.
> 
> Below are three cases where filesystem queue depth is not what the block
> layer sees:
> 
> 1. Cache hits never reach the block layer. Under a heavy read workload
>    with a warm cache, a large share of ops are serviced from the page
>    cache and are never seen at the block level. Device queue depth can
>    sit near zero while the filesystem is servicing a very high op rate.
> 
> 2. Filesystem ops don't map 1:1 to block I/O. A single read or write can
>    produce one block I/O, several (metadata, readahead, writeback
>    coalescing), or none at all. So device queue depth isn't the
>    filesystem's outstanding-operation count.
> 
> 3. Work can be outstanding inside the filesystem before any block I/O is
>    issued - waiting on locks, log space/reservation, delalloc, etc.
>    Such I/O has entered the filesystem but is invisible at the bdev.

Could you please put those in the commit description? For historic
purposes would be good to keep track why this has been added (or not).


> 
> The block-device queue depth answers "how deep is the device queue,"
> which is not the same as "how much work is outstanding in the
> filesystem." When the filesystem is just one layer an I/O passes
> through, the block stats fold the layers together and structurally
> cannot isolate the filesystem's own contribution.
> 
> To be clear about scope: I'm not proposing a queue-depth feature in
> the kernel. The change just adds read/write completion counters to pair
> with the existing call (submission) counters, so userspace can compute
> outstanding I/O and derive a response-time estimate itself. The kernel
> side is only exposing the complementary raw signal that's currently
> missing - calls are counted, completions are not.
> 
> Being upfront: what userspace derives from this is an instantaneous
> approximation, not a precise time-weighted queue length. It's meant as
> a cheap, always-on aggregate, not a replacement for accurate per-op
> tooling.
> 
> Does exposing the completion side of the existing call counters seem
> reasonable on that basis?
> 

Particularly I liked the idea and the justification seems fair although
I'd want to see the justification for the counter in the patch
description.

Carlos

> -Eric
> 

[PATCH v2] xfs: add per-mount read/write I/O completion counters

[Eric Peterson]

From: Eric Peterson <eric.peterson@hpe.com>

Add two per-mount statistics counters, xs_read_completions and
xs_write_completions, to complement the existing xs_read_calls and
xs_write_calls counters. The existing counters count I/O submissions
(entries); the new counters count I/O completions. The pair (calls,
completions) lets a consumer compute outstanding I/O as a queue depth
(calls - completions) and, via Little's law, derive an approximate
response time in userspace without any hot-path timestamping.

Block device stats expose device queue depth, but that is a different
quantity from filesystem outstanding I/O. There are cases where the
filesystem queue depth is not what the block layer sees:

1. Cache hits never reach the block layer. Under a heavy read workload
   with a warm cache, a large share of ops are serviced from the page
   cache and are never seen at the block level. Device queue depth can
   sit near zero while the filesystem is servicing a very high op rate.

2. Filesystem ops don't map 1:1 to block I/O. A single read or write can
   produce one block I/O, several (metadata, readahead, writeback
   coalescing), or none at all. So device queue depth isn't the
   filesystem's outstanding-operation count.

3. Work can be outstanding inside the filesystem before any block I/O is
   issued - waiting on locks, log space/reservation, delalloc, etc.
   Such I/O has entered the filesystem but is invisible at the bdev.

The counters are plain monotonic increments (no clock reads), so they
add negligible cost to the read/write path. Per-op timestamping was
deliberately not used: a clock read on the hot path costs ~20-30 ns on
TSC but hundreds of ns to ~1 us on HPET, which would be a regression for
general users. Queue depth from completion counters is an approximation
(instantaneous depth, not time-weighted); this is a deliberate design
choice, not a placeholder.

Completions are accounted at exactly the same sites where XFS already
accounts the xs_*_bytes counters, so their semantics match the existing
byte counters per path:

  - Reads are counted at the frame in xfs_file_read_iter and
    xfs_file_splice_read.
  - Buffered writes are counted at the frame, i.e. when data reaches
    the page cache, mirroring how xs_write_bytes is accounted for
    buffered writes -- not at physical writeback.
  - DAX writes are counted at the frame after the synchronous
    dax_iomap_rw copy returns, mirroring xs_write_bytes for DAX.
  - Direct I/O writes are counted at true completion in
    xfs_dio_write_end_io, which is async-safe and fires for both sync
    and async DIO, mirroring xs_write_bytes for DIO.

Caveat: async O_DIRECT reads are counted at submission, not completion,
because XFS has no read end_io today (iomap_dio_rw is called with NULL
ops for reads). This matches the existing read-byte semantics. Adding a
read end_io for async-DIO-read precision is a larger change, deliberately
deferred.

The counters are uint32_t and wrap like the existing xs_*_calls
counters; userspace diffs handle wrap.

The per-mount stats file gains a new appended "rwcmpl" line printing
write and read completions. The existing "rw" line is unchanged, so
positional parsers of "rw" are unaffected:

  rw     <write_calls>       <read_calls>
  rwcmpl <write_completions> <read_completions>

Signed-off-by: Eric Peterson <eric.peterson@hpe.com>
---

v2:
- Expand the commit message with the rationale for why filesystem
  outstanding I/O differs from block-device queue depth (cache
  hits, no 1:1 op-to-block mapping, and work outstanding inside the
  filesystem before any block I/O). No code change from v1.
  (Carlos Maiolino)

Notes for reviewers (not part of the commit log):

* Placement: the new "rwcmpl" group is inserted between "rw" and
  "attr" in the xstats[] table. The "rw" line itself is unchanged,
  and "rwcmpl" is appended after it, but lines below "rw" in
  /proc/fs/xfs/stat shift by one for strictly positional parsers. I
  can instead append the group at the END of the table if preferred.

* checkpatch --strict reports two CHECKs preferring u32 over
  uint32_t for the new fields. They are kept as uint32_t to match
  struct __xfsstats, whose every field is uint32_t; changing only
  these two would break local consistency.

* Testing: fstests -g auto shows baseline and patched fail the
  identical tests -- zero regressions. The rwcmpl interface was
  verified on hardware (rw >= rwcmpl, counters advance under load).
 fs/xfs/xfs_file.c  | 11 +++++++++--
 fs/xfs/xfs_stats.c |  3 ++-
 fs/xfs/xfs_stats.h |  2 ++
 3 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 426a67b813..3ecd4ed534 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -347,8 +347,10 @@ xfs_file_read_iter(
 	else
 		ret = xfs_file_buffered_read(iocb, to);

-	if (ret > 0)
+	if (ret > 0) {
 		XFS_STATS_ADD(mp, xs_read_bytes, ret);
+		XFS_STATS_INC(mp, xs_read_completions);
+	}
 	return ret;
 }

@@ -375,8 +377,10 @@ xfs_file_splice_read(
 	xfs_ilock(ip, XFS_IOLOCK_SHARED);
 	ret = filemap_splice_read(in, ppos, pipe, len, flags);
 	xfs_iunlock(ip, XFS_IOLOCK_SHARED);
-	if (ret > 0)
+	if (ret > 0) {
 		XFS_STATS_ADD(mp, xs_read_bytes, ret);
+		XFS_STATS_INC(mp, xs_read_completions);
+	}
 	return ret;
 }

@@ -663,6 +667,7 @@ xfs_dio_write_end_io(
 	 * for it on submission.
 	 */
 	XFS_STATS_ADD(ip->i_mount, xs_write_bytes, size);
+	XFS_STATS_INC(ip->i_mount, xs_write_completions);

 	/*
 	 * We can allocate memory here while doing writeback on behalf of
@@ -1032,6 +1037,7 @@ xfs_file_dax_write(

 	if (ret > 0) {
 		XFS_STATS_ADD(ip->i_mount, xs_write_bytes, ret);
+		XFS_STATS_INC(ip->i_mount, xs_write_completions);

 		/* Handle various SYNC-type writes */
 		ret = generic_write_sync(iocb, ret);
@@ -1098,6 +1104,7 @@ xfs_file_buffered_write(

 	if (ret > 0) {
 		XFS_STATS_ADD(ip->i_mount, xs_write_bytes, ret);
+		XFS_STATS_INC(ip->i_mount, xs_write_completions);
 		/* Handle various SYNC-type writes */
 		ret = generic_write_sync(iocb, ret);
 	}
diff --git a/fs/xfs/xfs_stats.c b/fs/xfs/xfs_stats.c
index c13d600732..5b276666b6 100644
--- a/fs/xfs/xfs_stats.c
+++ b/fs/xfs/xfs_stats.c
@@ -40,7 +40,8 @@ int xfs_stats_format(struct xfsstats __percpu *stats, char *buf)
 		{ "log",		xfsstats_offset(xs_try_logspace)},
 		{ "push_ail",		xfsstats_offset(xs_xstrat_quick)},
 		{ "xstrat",		xfsstats_offset(xs_write_calls)	},
-		{ "rw",			xfsstats_offset(xs_attr_get)	},
+		{ "rw",			xfsstats_offset(xs_write_completions)	},
+		{ "rwcmpl",		xfsstats_offset(xs_attr_get)	},
 		{ "attr",		xfsstats_offset(xs_iflush_count)},
 		{ "icluster",		xfsstats_offset(xs_inodes_active) },
 		{ "vnodes",		xfsstats_offset(xb_get)		},
diff --git a/fs/xfs/xfs_stats.h b/fs/xfs/xfs_stats.h
index 57c32b86c3..608d12d0c6 100644
--- a/fs/xfs/xfs_stats.h
+++ b/fs/xfs/xfs_stats.h
@@ -93,6 +93,8 @@ struct __xfsstats {
 	uint32_t		xs_xstrat_split;
 	uint32_t		xs_write_calls;
 	uint32_t		xs_read_calls;
+	uint32_t		xs_write_completions;
+	uint32_t		xs_read_completions;
 	uint32_t		xs_attr_get;
 	uint32_t		xs_attr_set;
 	uint32_t		xs_attr_remove;
-- 
2.39.5

Re: [PATCH] xfs: add per-mount read/write I/O completion counters

[Dave Chinner]

On Sun, Aug 30, 2026 at 06:47:00PM -0600, Eric Peterson wrote:
> On Mon, Aug 31, 2026 at 07:26:52AM +1000, Dave Chinner wrote:
> > Hence this really doesn't seem like something we should be trying to
> > infer from indirect filesystem stats. Why can't you use the bdev
> > stats to get the actual filesystem wide queue depth information?
> 
> The block device measures the device queue, which is a different
> quantity than filesystem outstanding I/O - not just a lower-layer view
> of the same thing.
> 
> Below are three cases where filesystem queue depth is not what the block
> layer sees:

I do know the difference. Assume I understand what you are saying,
and that you don't need to explain how the IO stack works to me...

> To be clear about scope: I'm not proposing a queue-depth feature in
> the kernel. The change just adds read/write completion counters to pair
> with the existing call (submission) counters, so userspace can compute
> outstanding I/O and derive a response-time estimate itself. The kernel
> side is only exposing the complementary raw signal that's currently
> missing - calls are counted, completions are not.

I know, I just don't see how it can be used for a response time
metric that any way useful for behavioural correlation because of
the sampling method.

> Being upfront: what userspace derives from this is an instantaneous
> approximation, not a precise time-weighted queue length. It's meant as
> a cheap, always-on aggregate, not a replacement for accurate per-op
> tooling.

And that's exactly why I'm having trouble understanding how this
new metric means anything useful. Ignoring temporal sampling jitter
of multiple per-cpu counters, if you sample read + completions
it at some instant, all it tells you is what is happening at that
instant.

What happens the other 999.9ms of that second is not captured by
this new "in-flight" metric?  For example, if I sample read
submissions at 10Hz (annotated manually with rough deltas between
samples):

$ pmval -r -t 0.1 xfs.read

metric:    xfs.read
host:      devoid
semantics: cumulative counter
units:     count
samples:   all
  294454912
  294454912		S (0 IO in flight)
  294454912
  294454912
  294454912
  294454912
  294454912
  294454912
  294454912
  294454912
  294455553	+650
  294455553		S (0 IO in flight)
  294455555	+2
  294455555
  294455555
  294455555
  294455555
  294455555
  294455555
  294455555
  294455559	+4
  294455559		S (0 IO in flight)
  294455559
  294455559
  294455559
  294455559
  294455559
  294455559
  294455559
  294455559
  294455559
  294455559		S (0 IO in flight)
  294455561	+2
  294455561
  294455561
  294455564
  294455564
  294456360	+800
  294457344	+1000
  294457344
  294457346	+2
  294457352	+6	S (at most 6 IO in flight)
  294457352
  294458065	+700
  294458285
  294458285
  294458285
  294458285
  294458285
  294458285
  294458285
  294458285		S (0 IO in flight)

You can see that there are some 100ms periods where nothing happens,
whilst others have 650-1000 buffered reads. In all the cases where
there are periods with no submission, the in-flight calculation will
be zero. In the busy periods, it will likely be some non-zero
number, but it won't give any indication of IO behaviour in that
entire period.

If we pick a 1s sample time (marked with "S" above), only one of
those sample points had any chance of there being IO in flight.
If I pick a sampling pattern that hits one of those high
IO periods, it gives an unrealisticly high in flight value for the
sampling period, given that for most of the rest of the second
around that burst there was almost no read activity.

Hence I don't see how sampling a point in time "in-flight" metric
slowly provides reliable insight into application behaviour. To
address that, one would need to sample and calculate the inflight
metric at high resolution to be able to catch the concurrency of IO
in those high IOPS bursts.

However, the faster you sample to catch bursts, the closer the read
submission rate approaches the in-flight IO rate. i.e. if I sample
at 1000Hz instead of 10Hz, it'll capture the fact that there are
bursts much faster bursts than 8-10 read IOs per millisecond, yet
the in-flight counter still won't reflect that - it might still not
register any IO being in flight at all because at the sample instant
there was no IO in flight....

Hence I'm asking how this new metric is supposed to be used and
correlated to observed/measured application behaviour. i.e. what
insight does it give you into application performance that can only
be derived from this point in time snapshot?

-Dave.
-- 
Dave Chinner
dgc@kernel.org

Re: [PATCH] xfs: add per-mount read/write I/O completion counters

[Eric Peterson]

On Mon, Aug 31, 2026 at 09:38 UTC, Dave Chinner wrote:
> Hence I'm asking how this new metric is supposed to be used and
> correlated to observed/measured application behaviour. i.e. what
> insight does it give you into application performance that can only
> be derived from this point in time snapshot?

My apologies - it wasn't my intention to come across as patronizing.
I was unsure what background was or wasn't common ground, so I erred on
the side of more detail.

You're right about the sampling limitation: a slowly-sampled
point-in-time queue depth value cannot characterize bursty,
sub-interval concurrency. If the goal is to resolve what happens inside
a 10ms burst, this is the wrong tool - per-op tooling (tracepoints,
histograms) is the right one, and this is not meant to replace it.

The important part is that this is a property of the sampling rate, not
of the counters. Nyquist-Shannon says that to observe a phenomenon at
timescale T you have to sample at >= 2/T; if you sample slower than the
behavior you care about, it will be missed. This is true of any sampled
counter, including the existing submission counter - in your 10Hz pmval
example, xfs.read has exactly the same property. The sampling rate is a
policy choice for the user to match to what they're trying to observe.

Answering your question, it lets userspace characterize filesystem
queue depth over time. The places where this is useful are the ones
where the desired signal persists across multiple sample periods,
leading to a representative measurement:

 - Sustained/steady-state load.
   Database, NFS server, VM image store, etc. Outstanding I/O is stable
   across many sample periods. Most capacity and health monitoring
   lives here.

 - Long-horizon trends.
   Can show if queue depth is creeping up over hours or days as load
   grows or cache becomes insufficient. Leaving per-op tracing running
   for this kind of timescale is the wrong tool for the job; persistent,
   low-cost sampling is the better choice.

 - Sustained-backlog alerting.
   Consistent elevated depth can indicate saturation, a stuck consumer,
   or cache thrash. Filtering out small transients avoids adding noise.

 - Coarse steady-state latency.
   When load is steady, sustained depth over sustained completion rate
   gives an average latency - enough precision to tell 0.5ms from 5ms,
   but not tail latency. Histograms would be the correct tool if higher
   resolution is required.

For higher precision you'd want a time-weighted queue depth, but that
requires two clock reads on every I/O in the hot path, and the cost
grows with I/O load. This trade-off is the core motivation: the counter
is a near-free, always-on aggregate for the common steady-state and
trend cases. It does not replace per-op tooling where higher precision
is required.

-Eric