* [RFC 0/1] block: export windowed IO P99 latency
@ 2026-01-09 8:31 Diangang Li
2026-01-09 8:31 ` [RFC 1/1] " Diangang Li
2026-01-30 2:59 ` [RFC 0/1] " Diangang Li
0 siblings, 2 replies; 6+ messages in thread
From: Diangang Li @ 2026-01-09 8:31 UTC (permalink / raw)
To: axboe; +Cc: linux-block, linux-kernel, changfengnan, Diangang Li
Production environments occasionally run into elevated tail latencies. The
source can be the underlying device, but it can also be higher in the
stack (filesystem contention/journaling, memory reclaim, writeback, etc.).
Existing block IO statistics only provide throughput and average latency,
which fail to capture the critical tail end of the latency distribution
that often causes user-visible performance problems.
This patch adds windowed P99 latency tracking for block IO operations,
exposing the 99th percentile latency in /proc/diskstats and
/sys/block/<dev>/stat. System administrators can now monitor tail latency
trends over time using tools like iostat, enabling quick validation or
elimination of disk hardware as the source of latency issues.
Implementation uses per-CPU sliced ring histograms (21 buckets, 8us..~8s
range) with minimal overhead. P99 values are computed by aggregating
recent 1-second slices when reading statistics, reported in microseconds
using bucket midpoints.
The added work on the IO path is intentionally small (bucket selection and
a per-CPU counter update, with occasional per-slice reset), and in our
testing it does not have a measurable impact on IO performance.
Diangang Li (1):
block: export windowed IO P99 latency
block/blk-core.c | 5 ++-
block/blk-flush.c | 6 ++-
block/blk-mq.c | 5 ++-
block/genhd.c | 50 ++++++++++++++++++++++++-
include/linux/part_stat.h | 79 +++++++++++++++++++++++++++++++++++++++
5 files changed, 139 insertions(+), 6 deletions(-)
--
2.39.5
^ permalink raw reply [flat|nested] 6+ messages in thread
* [RFC 1/1] block: export windowed IO P99 latency
2026-01-09 8:31 [RFC 0/1] block: export windowed IO P99 latency Diangang Li
@ 2026-01-09 8:31 ` Diangang Li
2026-01-15 17:38 ` kernel test robot
` (2 more replies)
2026-01-30 2:59 ` [RFC 0/1] " Diangang Li
1 sibling, 3 replies; 6+ messages in thread
From: Diangang Li @ 2026-01-09 8:31 UTC (permalink / raw)
To: axboe; +Cc: linux-block, linux-kernel, changfengnan, Diangang Li
Track per-IO completion latency in a power-of-two histogram
(NR_STAT_BUCKETS buckets, DISK_LAT_BASE_USEC .. DISK_LAT_MAX_USEC).
Maintain a per-cpu sliced ring histogram and compute P99 by aggregating the
recent slices at read time in /proc/diskstats and /sys/block/<dev>/stat.
Report P99 in usecs using the bucket midpoint, clamp overflows to
DISK_LAT_MAX_USEC, and append the P99 for read/write/discard/flush.
Suggested-by: Fengnan Chang <changfengnan@bytedance.com>
Signed-off-by: Diangang Li <lidiangang@bytedance.com>
---
block/blk-core.c | 5 ++-
block/blk-flush.c | 6 ++-
block/blk-mq.c | 5 ++-
block/genhd.c | 50 ++++++++++++++++++++++++-
include/linux/part_stat.h | 79 +++++++++++++++++++++++++++++++++++++++
5 files changed, 139 insertions(+), 6 deletions(-)
diff --git a/block/blk-core.c b/block/blk-core.c
index 8387fe50ea156..832ba4fc1b75a 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -1062,12 +1062,15 @@ void bdev_end_io_acct(struct block_device *bdev, enum req_op op,
const int sgrp = op_stat_group(op);
unsigned long now = READ_ONCE(jiffies);
unsigned long duration = now - start_time;
+ u64 latency_ns = jiffies_to_nsecs(duration);
+ unsigned int bucket = diskstat_latency_bucket(latency_ns);
part_stat_lock();
update_io_ticks(bdev, now, true);
part_stat_inc(bdev, ios[sgrp]);
part_stat_add(bdev, sectors[sgrp], sectors);
- part_stat_add(bdev, nsecs[sgrp], jiffies_to_nsecs(duration));
+ part_stat_add(bdev, nsecs[sgrp], latency_ns);
+ part_stat_latency_record(bdev, sgrp, now, bucket);
part_stat_local_dec(bdev, in_flight[op_is_write(op)]);
part_stat_unlock();
}
diff --git a/block/blk-flush.c b/block/blk-flush.c
index 43d6152897a42..b3ff78025968f 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -124,11 +124,13 @@ static void blk_flush_restore_request(struct request *rq)
static void blk_account_io_flush(struct request *rq)
{
struct block_device *part = rq->q->disk->part0;
+ u64 latency_ns = blk_time_get_ns() - rq->start_time_ns;
+ unsigned int bucket = diskstat_latency_bucket(latency_ns);
part_stat_lock();
part_stat_inc(part, ios[STAT_FLUSH]);
- part_stat_add(part, nsecs[STAT_FLUSH],
- blk_time_get_ns() - rq->start_time_ns);
+ part_stat_add(part, nsecs[STAT_FLUSH], latency_ns);
+ part_stat_latency_record(part, STAT_FLUSH, jiffies, bucket);
part_stat_unlock();
}
diff --git a/block/blk-mq.c b/block/blk-mq.c
index eff4f72ce83be..6a7fd6681902e 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1068,11 +1068,14 @@ static inline void blk_account_io_done(struct request *req, u64 now)
*/
if ((req->rq_flags & (RQF_IO_STAT|RQF_FLUSH_SEQ)) == RQF_IO_STAT) {
const int sgrp = op_stat_group(req_op(req));
+ u64 latency_ns = now - req->start_time_ns;
+ unsigned int bucket = diskstat_latency_bucket(latency_ns);
part_stat_lock();
update_io_ticks(req->part, jiffies, true);
part_stat_inc(req->part, ios[sgrp]);
- part_stat_add(req->part, nsecs[sgrp], now - req->start_time_ns);
+ part_stat_add(req->part, nsecs[sgrp], latency_ns);
+ part_stat_latency_record(req->part, sgrp, jiffies, bucket);
part_stat_local_dec(req->part,
in_flight[op_is_write(req_op(req))]);
part_stat_unlock();
diff --git a/block/genhd.c b/block/genhd.c
index 69c75117ba2c0..56151c7880651 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -108,23 +108,60 @@ static void part_stat_read_all(struct block_device *part,
struct disk_stats *stat)
{
int cpu;
+ u32 now_epoch = (u32)(jiffies / HZ);
memset(stat, 0, sizeof(struct disk_stats));
for_each_possible_cpu(cpu) {
struct disk_stats *ptr = per_cpu_ptr(part->bd_stats, cpu);
int group;
+ int slice;
+ int bucket;
for (group = 0; group < NR_STAT_GROUPS; group++) {
stat->nsecs[group] += ptr->nsecs[group];
stat->sectors[group] += ptr->sectors[group];
stat->ios[group] += ptr->ios[group];
stat->merges[group] += ptr->merges[group];
+
+ for (slice = 0; slice < NR_STAT_SLICES; slice++) {
+ u32 slice_epoch = READ_ONCE(ptr->latency_epoch[slice]);
+ s32 age = (s32)(now_epoch - slice_epoch);
+
+ if (age < 0 || age >= NR_STAT_SLICES)
+ continue;
+
+ for (bucket = 0; bucket < NR_STAT_BUCKETS; bucket++)
+ stat->latency[group][0][bucket] +=
+ ptr->latency[group][slice][bucket];
+ }
}
stat->io_ticks += ptr->io_ticks;
}
}
+static u32 diskstat_p99_us(u32 buckets[NR_STAT_BUCKETS])
+{
+ u32 total = 0;
+ u32 accum = 0;
+ u32 target;
+ int bucket;
+
+ for (bucket = 0; bucket < NR_STAT_BUCKETS; bucket++)
+ total += buckets[bucket];
+ if (!total)
+ return 0;
+
+ target = total - div_u64((u64)total, 100);
+ for (bucket = 0; bucket < NR_STAT_BUCKETS; bucket++) {
+ accum += buckets[bucket];
+ if (accum >= target)
+ return diskstat_latency_bucket_us(bucket);
+ }
+
+ return diskstat_latency_bucket_us(NR_STAT_BUCKETS - 1);
+}
+
static void bdev_count_inflight_rw(struct block_device *part,
unsigned int inflight[2], bool mq_driver)
{
@@ -1078,7 +1115,8 @@ ssize_t part_stat_show(struct device *dev,
"%8lu %8lu %8llu %8u "
"%8u %8u %8u "
"%8lu %8lu %8llu %8u "
- "%8lu %8u"
+ "%8lu %8u "
+ "%8u %8u %8u %8u"
"\n",
stat.ios[STAT_READ],
stat.merges[STAT_READ],
@@ -1100,7 +1138,11 @@ ssize_t part_stat_show(struct device *dev,
(unsigned long long)stat.sectors[STAT_DISCARD],
(unsigned int)div_u64(stat.nsecs[STAT_DISCARD], NSEC_PER_MSEC),
stat.ios[STAT_FLUSH],
- (unsigned int)div_u64(stat.nsecs[STAT_FLUSH], NSEC_PER_MSEC));
+ (unsigned int)div_u64(stat.nsecs[STAT_FLUSH], NSEC_PER_MSEC),
+ diskstat_p99_us(stat.latency[STAT_READ][0]),
+ diskstat_p99_us(stat.latency[STAT_WRITE][0]),
+ diskstat_p99_us(stat.latency[STAT_DISCARD][0]),
+ diskstat_p99_us(stat.latency[STAT_FLUSH][0]));
}
/*
@@ -1406,6 +1448,10 @@ static int diskstats_show(struct seq_file *seqf, void *v)
seq_put_decimal_ull(seqf, " ", stat.ios[STAT_FLUSH]);
seq_put_decimal_ull(seqf, " ", (unsigned int)div_u64(stat.nsecs[STAT_FLUSH],
NSEC_PER_MSEC));
+ seq_put_decimal_ull(seqf, " ", diskstat_p99_us(stat.latency[STAT_READ][0]));
+ seq_put_decimal_ull(seqf, " ", diskstat_p99_us(stat.latency[STAT_WRITE][0]));
+ seq_put_decimal_ull(seqf, " ", diskstat_p99_us(stat.latency[STAT_DISCARD][0]));
+ seq_put_decimal_ull(seqf, " ", diskstat_p99_us(stat.latency[STAT_FLUSH][0]));
seq_putc(seqf, '\n');
}
rcu_read_unlock();
diff --git a/include/linux/part_stat.h b/include/linux/part_stat.h
index 729415e91215d..cbcb24abac21e 100644
--- a/include/linux/part_stat.h
+++ b/include/linux/part_stat.h
@@ -5,6 +5,19 @@
#include <linux/blkdev.h>
#include <asm/local.h>
+/*
+ * Diskstats latency histogram:
+ * - Bucket upper bounds are power-of-two in usecs, starting at DISK_LAT_BASE_USEC.
+ * - The last bucket is a saturation bucket for latencies >= DISK_LAT_MAX_USEC.
+ *
+ * Latency is tracked in NR_STAT_SLICES 1-second slices and
+ * summed to compute a NR_STAT_SLICES-second P99 latency.
+ */
+#define NR_STAT_BUCKETS 21
+#define NR_STAT_SLICES 5
+#define DISK_LAT_BASE_USEC 8U
+#define DISK_LAT_MAX_USEC (DISK_LAT_BASE_USEC << (NR_STAT_BUCKETS - 1))
+
struct disk_stats {
u64 nsecs[NR_STAT_GROUPS];
unsigned long sectors[NR_STAT_GROUPS];
@@ -12,6 +25,8 @@ struct disk_stats {
unsigned long merges[NR_STAT_GROUPS];
unsigned long io_ticks;
local_t in_flight[2];
+ u32 latency_epoch[NR_STAT_SLICES];
+ u32 latency[NR_STAT_GROUPS][NR_STAT_SLICES][NR_STAT_BUCKETS];
};
/*
@@ -81,4 +96,68 @@ static inline void part_stat_set_all(struct block_device *part, int value)
unsigned int bdev_count_inflight(struct block_device *part);
+static inline unsigned int diskstat_latency_bucket(u64 latency_ns)
+{
+ u64 latency_us = latency_ns / 1000;
+ u64 scaled;
+
+ if (latency_us <= DISK_LAT_BASE_USEC)
+ return 0;
+
+ if (latency_us >= DISK_LAT_MAX_USEC)
+ return NR_STAT_BUCKETS - 1;
+
+ scaled = div_u64(latency_us - 1, DISK_LAT_BASE_USEC);
+ return min_t(unsigned int, (unsigned int)fls64(scaled),
+ NR_STAT_BUCKETS - 1);
+}
+
+static inline u32 diskstat_latency_bucket_upper_us(unsigned int bucket)
+{
+ if (bucket >= NR_STAT_BUCKETS - 1)
+ return DISK_LAT_MAX_USEC;
+ return DISK_LAT_BASE_USEC << bucket;
+}
+
+static inline u32 diskstat_latency_bucket_us(unsigned int bucket)
+{
+ u32 high;
+ u32 low;
+
+ if (bucket >= NR_STAT_BUCKETS - 1)
+ return DISK_LAT_MAX_USEC;
+
+ high = diskstat_latency_bucket_upper_us(bucket);
+ low = high >> 1;
+ return low + (low >> 1);
+}
+
+static inline void __part_stat_latency_prepare(struct block_device *part,
+ u32 epoch, unsigned int slice)
+{
+ struct disk_stats *stats = per_cpu_ptr(part->bd_stats, smp_processor_id());
+ int group;
+
+ if (likely(stats->latency_epoch[slice] == epoch))
+ return;
+
+ for (group = 0; group < NR_STAT_GROUPS; group++)
+ memset(stats->latency[group][slice], 0,
+ sizeof(stats->latency[group][slice]));
+ stats->latency_epoch[slice] = epoch;
+}
+
+static inline void part_stat_latency_record(struct block_device *part,
+ int sgrp, unsigned long now, unsigned int bucket)
+{
+ u32 epoch = now / HZ;
+ unsigned int slice = epoch % NR_STAT_SLICES;
+
+ __part_stat_latency_prepare(part, epoch, slice);
+ if (bdev_is_partition(part))
+ __part_stat_latency_prepare(bdev_whole(part), epoch, slice);
+
+ part_stat_inc(part, latency[sgrp][slice][bucket]);
+}
+
#endif /* _LINUX_PART_STAT_H */
--
2.39.5
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [RFC 1/1] block: export windowed IO P99 latency
2026-01-09 8:31 ` [RFC 1/1] " Diangang Li
@ 2026-01-15 17:38 ` kernel test robot
2026-01-15 18:20 ` kernel test robot
2026-01-15 18:20 ` kernel test robot
2 siblings, 0 replies; 6+ messages in thread
From: kernel test robot @ 2026-01-15 17:38 UTC (permalink / raw)
To: Diangang Li; +Cc: oe-kbuild-all
Hi Diangang,
[This is a private test report for your RFC patch.]
kernel test robot noticed the following build warnings:
[auto build test WARNING on axboe/for-next]
[also build test WARNING on linus/master v6.19-rc5 next-20260115]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Diangang-Li/block-export-windowed-IO-P99-latency/20260109-163449
base: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git for-next
patch link: https://lore.kernel.org/r/20260109083126.15052-2-lidiangang%40bytedance.com
patch subject: [RFC 1/1] block: export windowed IO P99 latency
config: nios2-allnoconfig (https://download.01.org/0day-ci/archive/20260116/202601160139.JUgX771I-lkp@intel.com/config)
compiler: nios2-linux-gcc (GCC) 11.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260116/202601160139.JUgX771I-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202601160139.JUgX771I-lkp@intel.com/
All warnings (new ones prefixed by >>):
block/genhd.c: In function 'diskstats_show':
>> block/genhd.c:1460:1: warning: the frame size of 1800 bytes is larger than 1280 bytes [-Wframe-larger-than=]
1460 | }
| ^
block/genhd.c: In function 'part_stat_show':
block/genhd.c:1146:1: warning: the frame size of 1892 bytes is larger than 1280 bytes [-Wframe-larger-than=]
1146 | }
| ^
vim +1460 block/genhd.c
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1386
a6e2ba88774bc5 block/genhd.c Randy Dunlap 2008-05-23 1387 #ifdef CONFIG_PROC_FS
cf771cb5a7b716 block/genhd.c Tejun Heo 2008-09-03 1388 /*
cf771cb5a7b716 block/genhd.c Tejun Heo 2008-09-03 1389 * aggregate disk stat collector. Uses the same stats that the sysfs
cf771cb5a7b716 block/genhd.c Tejun Heo 2008-09-03 1390 * entries do, above, but makes them available through one seq_file.
cf771cb5a7b716 block/genhd.c Tejun Heo 2008-09-03 1391 *
cf771cb5a7b716 block/genhd.c Tejun Heo 2008-09-03 1392 * The output looks suspiciously like /proc/partitions with a bunch of
cf771cb5a7b716 block/genhd.c Tejun Heo 2008-09-03 1393 * extra fields.
cf771cb5a7b716 block/genhd.c Tejun Heo 2008-09-03 1394 */
cf771cb5a7b716 block/genhd.c Tejun Heo 2008-09-03 1395 static int diskstats_show(struct seq_file *seqf, void *v)
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1396 {
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1397 struct gendisk *gp = v;
ad1eaa5344b293 block/genhd.c Christoph Hellwig 2020-11-24 1398 struct block_device *hd;
e016b78201a2d9 block/genhd.c Mikulas Patocka 2018-12-06 1399 unsigned int inflight;
ea18e0f0a63af9 block/genhd.c Konstantin Khlebnikov 2020-03-25 1400 struct disk_stats stat;
7fae67cc9c0e06 block/genhd.c Christoph Hellwig 2021-04-06 1401 unsigned long idx;
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1402
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1403 /*
ed9e1982347b36 block/genhd.c Tejun Heo 2008-08-25 1404 if (&disk_to_dev(gp)->kobj.entry == block_class.devices.next)
cf771cb5a7b716 block/genhd.c Tejun Heo 2008-09-03 1405 seq_puts(seqf, "major minor name"
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1406 " rio rmerge rsect ruse wio wmerge "
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1407 "wsect wuse running use aveq"
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1408 "\n\n");
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1409 */
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1410
7fae67cc9c0e06 block/genhd.c Christoph Hellwig 2021-04-06 1411 rcu_read_lock();
7fae67cc9c0e06 block/genhd.c Christoph Hellwig 2021-04-06 1412 xa_for_each(&gp->part_tbl, idx, hd) {
7fae67cc9c0e06 block/genhd.c Christoph Hellwig 2021-04-06 1413 if (bdev_is_partition(hd) && !bdev_nr_sectors(hd))
7fae67cc9c0e06 block/genhd.c Christoph Hellwig 2021-04-06 1414 continue;
ea18e0f0a63af9 block/genhd.c Konstantin Khlebnikov 2020-03-25 1415
f2987c5816bda0 block/genhd.c Yu Kuai 2025-05-06 1416 inflight = bdev_count_inflight(hd);
86d7331299fda7 block/genhd.c Zhang Wensheng 2022-02-17 1417 if (inflight) {
86d7331299fda7 block/genhd.c Zhang Wensheng 2022-02-17 1418 part_stat_lock();
86d7331299fda7 block/genhd.c Zhang Wensheng 2022-02-17 1419 update_io_ticks(hd, jiffies, true);
86d7331299fda7 block/genhd.c Zhang Wensheng 2022-02-17 1420 part_stat_unlock();
86d7331299fda7 block/genhd.c Zhang Wensheng 2022-02-17 1421 }
86d7331299fda7 block/genhd.c Zhang Wensheng 2022-02-17 1422 part_stat_read_all(hd, &stat);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1423 seq_put_decimal_ull_width(seqf, "", MAJOR(hd->bd_dev), 4);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1424 seq_put_decimal_ull_width(seqf, " ", MINOR(hd->bd_dev), 7);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1425 seq_printf(seqf, " %pg", hd);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1426 seq_put_decimal_ull(seqf, " ", stat.ios[STAT_READ]);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1427 seq_put_decimal_ull(seqf, " ", stat.merges[STAT_READ]);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1428 seq_put_decimal_ull(seqf, " ", stat.sectors[STAT_READ]);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1429 seq_put_decimal_ull(seqf, " ", (unsigned int)div_u64(stat.nsecs[STAT_READ],
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1430 NSEC_PER_MSEC));
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1431 seq_put_decimal_ull(seqf, " ", stat.ios[STAT_WRITE]);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1432 seq_put_decimal_ull(seqf, " ", stat.merges[STAT_WRITE]);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1433 seq_put_decimal_ull(seqf, " ", stat.sectors[STAT_WRITE]);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1434 seq_put_decimal_ull(seqf, " ", (unsigned int)div_u64(stat.nsecs[STAT_WRITE],
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1435 NSEC_PER_MSEC));
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1436 seq_put_decimal_ull(seqf, " ", inflight);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1437 seq_put_decimal_ull(seqf, " ", jiffies_to_msecs(stat.io_ticks));
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1438 seq_put_decimal_ull(seqf, " ", (unsigned int)div_u64(stat.nsecs[STAT_READ] +
8cd5b8fc00716f block/genhd.c Konstantin Khlebnikov 2020-03-25 1439 stat.nsecs[STAT_WRITE] +
8cd5b8fc00716f block/genhd.c Konstantin Khlebnikov 2020-03-25 1440 stat.nsecs[STAT_DISCARD] +
8cd5b8fc00716f block/genhd.c Konstantin Khlebnikov 2020-03-25 1441 stat.nsecs[STAT_FLUSH],
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1442 NSEC_PER_MSEC));
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1443 seq_put_decimal_ull(seqf, " ", stat.ios[STAT_DISCARD]);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1444 seq_put_decimal_ull(seqf, " ", stat.merges[STAT_DISCARD]);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1445 seq_put_decimal_ull(seqf, " ", stat.sectors[STAT_DISCARD]);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1446 seq_put_decimal_ull(seqf, " ", (unsigned int)div_u64(stat.nsecs[STAT_DISCARD],
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1447 NSEC_PER_MSEC));
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1448 seq_put_decimal_ull(seqf, " ", stat.ios[STAT_FLUSH]);
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1449 seq_put_decimal_ull(seqf, " ", (unsigned int)div_u64(stat.nsecs[STAT_FLUSH],
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1450 NSEC_PER_MSEC));
f8b4ca85f47647 block/genhd.c Diangang Li 2026-01-09 1451 seq_put_decimal_ull(seqf, " ", diskstat_p99_us(stat.latency[STAT_READ][0]));
f8b4ca85f47647 block/genhd.c Diangang Li 2026-01-09 1452 seq_put_decimal_ull(seqf, " ", diskstat_p99_us(stat.latency[STAT_WRITE][0]));
f8b4ca85f47647 block/genhd.c Diangang Li 2026-01-09 1453 seq_put_decimal_ull(seqf, " ", diskstat_p99_us(stat.latency[STAT_DISCARD][0]));
f8b4ca85f47647 block/genhd.c Diangang Li 2026-01-09 1454 seq_put_decimal_ull(seqf, " ", diskstat_p99_us(stat.latency[STAT_FLUSH][0]));
bda9c7d92f24b6 block/genhd.c David Wang 2024-11-08 1455 seq_putc(seqf, '\n');
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1456 }
7fae67cc9c0e06 block/genhd.c Christoph Hellwig 2021-04-06 1457 rcu_read_unlock();
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1458
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1459 return 0;
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 @1460 }
^1da177e4c3f41 drivers/block/genhd.c Linus Torvalds 2005-04-16 1461
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC 1/1] block: export windowed IO P99 latency
2026-01-09 8:31 ` [RFC 1/1] " Diangang Li
2026-01-15 17:38 ` kernel test robot
@ 2026-01-15 18:20 ` kernel test robot
2026-01-15 18:20 ` kernel test robot
2 siblings, 0 replies; 6+ messages in thread
From: kernel test robot @ 2026-01-15 18:20 UTC (permalink / raw)
To: Diangang Li; +Cc: oe-kbuild-all
Hi Diangang,
[This is a private test report for your RFC patch.]
kernel test robot noticed the following build errors:
[auto build test ERROR on axboe/for-next]
[also build test ERROR on linus/master v6.19-rc5 next-20260115]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Diangang-Li/block-export-windowed-IO-P99-latency/20260109-163449
base: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git for-next
patch link: https://lore.kernel.org/r/20260109083126.15052-2-lidiangang%40bytedance.com
patch subject: [RFC 1/1] block: export windowed IO P99 latency
config: m68k-allnoconfig (https://download.01.org/0day-ci/archive/20260116/202601160136.m0I1HNWx-lkp@intel.com/config)
compiler: m68k-linux-gcc (GCC) 15.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260116/202601160136.m0I1HNWx-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202601160136.m0I1HNWx-lkp@intel.com/
All errors (new ones prefixed by >>):
m68k-linux-ld: block/blk-mq.o: in function `diskstat_latency_bucket':
blk-mq.c:(.text+0xe76): undefined reference to `__udivdi3'
m68k-linux-ld: block/blk-core.o: in function `diskstat_latency_bucket':
>> blk-core.c:(.text+0x736): undefined reference to `__udivdi3'
m68k-linux-ld: block/blk-flush.o: in function `blk_account_io_flush':
blk-flush.c:(.text+0x602): undefined reference to `__udivdi3'
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC 1/1] block: export windowed IO P99 latency
2026-01-09 8:31 ` [RFC 1/1] " Diangang Li
2026-01-15 17:38 ` kernel test robot
2026-01-15 18:20 ` kernel test robot
@ 2026-01-15 18:20 ` kernel test robot
2 siblings, 0 replies; 6+ messages in thread
From: kernel test robot @ 2026-01-15 18:20 UTC (permalink / raw)
To: Diangang Li; +Cc: llvm, oe-kbuild-all
Hi Diangang,
[This is a private test report for your RFC patch.]
kernel test robot noticed the following build warnings:
[auto build test WARNING on axboe/for-next]
[also build test WARNING on linus/master v6.19-rc5 next-20260115]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Diangang-Li/block-export-windowed-IO-P99-latency/20260109-163449
base: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git for-next
patch link: https://lore.kernel.org/r/20260109083126.15052-2-lidiangang%40bytedance.com
patch subject: [RFC 1/1] block: export windowed IO P99 latency
config: s390-allnoconfig (https://download.01.org/0day-ci/archive/20260116/202601160133.EiiYdMZh-lkp@intel.com/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 9b8addffa70cee5b2acc5454712d9cf78ce45710)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260116/202601160133.EiiYdMZh-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202601160133.EiiYdMZh-lkp@intel.com/
All warnings (new ones prefixed by >>):
>> block/genhd.c:1099:9: warning: stack frame size (2232) exceeds limit (2048) in 'part_stat_show' [-Wframe-larger-than]
1099 | ssize_t part_stat_show(struct device *dev,
| ^
1 warning generated.
vim +/part_stat_show +1099 block/genhd.c
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1098
3ad5cee5cd000d Christoph Hellwig 2020-03-24 @1099 ssize_t part_stat_show(struct device *dev,
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1100 struct device_attribute *attr, char *buf)
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1101 {
0d02129e76edf9 Christoph Hellwig 2020-11-27 1102 struct block_device *bdev = dev_to_bdev(dev);
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1103 struct disk_stats stat;
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1104 unsigned int inflight;
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1105
f2987c5816bda0 Yu Kuai 2025-05-06 1106 inflight = bdev_count_inflight(bdev);
86d7331299fda7 Zhang Wensheng 2022-02-17 1107 if (inflight) {
86d7331299fda7 Zhang Wensheng 2022-02-17 1108 part_stat_lock();
86d7331299fda7 Zhang Wensheng 2022-02-17 1109 update_io_ticks(bdev, jiffies, true);
86d7331299fda7 Zhang Wensheng 2022-02-17 1110 part_stat_unlock();
86d7331299fda7 Zhang Wensheng 2022-02-17 1111 }
86d7331299fda7 Zhang Wensheng 2022-02-17 1112 part_stat_read_all(bdev, &stat);
8e71afb94d6ed5 zhangguopeng 2024-11-07 1113 return sysfs_emit(buf,
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1114 "%8lu %8lu %8llu %8u "
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1115 "%8lu %8lu %8llu %8u "
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1116 "%8u %8u %8u "
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1117 "%8lu %8lu %8llu %8u "
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1118 "%8lu %8u "
f8b4ca85f47647 Diangang Li 2026-01-09 1119 "%8u %8u %8u %8u"
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1120 "\n",
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1121 stat.ios[STAT_READ],
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1122 stat.merges[STAT_READ],
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1123 (unsigned long long)stat.sectors[STAT_READ],
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1124 (unsigned int)div_u64(stat.nsecs[STAT_READ], NSEC_PER_MSEC),
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1125 stat.ios[STAT_WRITE],
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1126 stat.merges[STAT_WRITE],
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1127 (unsigned long long)stat.sectors[STAT_WRITE],
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1128 (unsigned int)div_u64(stat.nsecs[STAT_WRITE], NSEC_PER_MSEC),
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1129 inflight,
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1130 jiffies_to_msecs(stat.io_ticks),
8cd5b8fc00716f Konstantin Khlebnikov 2020-03-25 1131 (unsigned int)div_u64(stat.nsecs[STAT_READ] +
8cd5b8fc00716f Konstantin Khlebnikov 2020-03-25 1132 stat.nsecs[STAT_WRITE] +
8cd5b8fc00716f Konstantin Khlebnikov 2020-03-25 1133 stat.nsecs[STAT_DISCARD] +
8cd5b8fc00716f Konstantin Khlebnikov 2020-03-25 1134 stat.nsecs[STAT_FLUSH],
8cd5b8fc00716f Konstantin Khlebnikov 2020-03-25 1135 NSEC_PER_MSEC),
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1136 stat.ios[STAT_DISCARD],
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1137 stat.merges[STAT_DISCARD],
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1138 (unsigned long long)stat.sectors[STAT_DISCARD],
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1139 (unsigned int)div_u64(stat.nsecs[STAT_DISCARD], NSEC_PER_MSEC),
ea18e0f0a63af9 Konstantin Khlebnikov 2020-03-25 1140 stat.ios[STAT_FLUSH],
f8b4ca85f47647 Diangang Li 2026-01-09 1141 (unsigned int)div_u64(stat.nsecs[STAT_FLUSH], NSEC_PER_MSEC),
f8b4ca85f47647 Diangang Li 2026-01-09 1142 diskstat_p99_us(stat.latency[STAT_READ][0]),
f8b4ca85f47647 Diangang Li 2026-01-09 1143 diskstat_p99_us(stat.latency[STAT_WRITE][0]),
f8b4ca85f47647 Diangang Li 2026-01-09 1144 diskstat_p99_us(stat.latency[STAT_DISCARD][0]),
f8b4ca85f47647 Diangang Li 2026-01-09 1145 diskstat_p99_us(stat.latency[STAT_FLUSH][0]));
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1146 }
3ad5cee5cd000d Christoph Hellwig 2020-03-24 1147
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC 0/1] block: export windowed IO P99 latency
2026-01-09 8:31 [RFC 0/1] block: export windowed IO P99 latency Diangang Li
2026-01-09 8:31 ` [RFC 1/1] " Diangang Li
@ 2026-01-30 2:59 ` Diangang Li
1 sibling, 0 replies; 6+ messages in thread
From: Diangang Li @ 2026-01-30 2:59 UTC (permalink / raw)
To: Diangang Li, axboe; +Cc: linux-block, linux-kernel, changfengnan
On 2026/1/9 16:31, Diangang Li wrote:
> Production environments occasionally run into elevated tail latencies. The
> source can be the underlying device, but it can also be higher in the
> stack (filesystem contention/journaling, memory reclaim, writeback, etc.).
> Existing block IO statistics only provide throughput and average latency,
> which fail to capture the critical tail end of the latency distribution
> that often causes user-visible performance problems.
>
> This patch adds windowed P99 latency tracking for block IO operations,
> exposing the 99th percentile latency in /proc/diskstats and
> /sys/block/<dev>/stat. System administrators can now monitor tail latency
> trends over time using tools like iostat, enabling quick validation or
> elimination of disk hardware as the source of latency issues.
>
> Implementation uses per-CPU sliced ring histograms (21 buckets, 8us..~8s
> range) with minimal overhead. P99 values are computed by aggregating
> recent 1-second slices when reading statistics, reported in microseconds
> using bucket midpoints.
>
> The added work on the IO path is intentionally small (bucket selection and
> a per-CPU counter update, with occasional per-slice reset), and in our
> testing it does not have a measurable impact on IO performance.
>
> Diangang Li (1):
> block: export windowed IO P99 latency
>
> block/blk-core.c | 5 ++-
> block/blk-flush.c | 6 ++-
> block/blk-mq.c | 5 ++-
> block/genhd.c | 50 ++++++++++++++++++++++++-
> include/linux/part_stat.h | 79 +++++++++++++++++++++++++++++++++++++++
> 5 files changed, 139 insertions(+), 6 deletions(-)
>
Hi Jens, hi all,
Quick sanity check on the motivation/design before I respin.
I want to expose a simple tail metric (P99) via diskstats/sysfs stat,
since avg latency/throughput often miss the spikes seen in prod.
I considered read-to-read deltas, but diskstats is polled frequently
(often sub-second, multiple agents), so the effective window becomes
reader-dependent and too short/noisy. Current approach uses a fixed
window (per-CPU 1s slices in a small ring histogram) and aggregates on read.
Does this direction make sense? Is diskstats/sysfs the right place for
it? Any better low-overhead, polling-independent approach (and
preferences on percentile/window/buckets)?
Best regards,
Diangang Li
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-01-30 2:59 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-01-09 8:31 [RFC 0/1] block: export windowed IO P99 latency Diangang Li
2026-01-09 8:31 ` [RFC 1/1] " Diangang Li
2026-01-15 17:38 ` kernel test robot
2026-01-15 18:20 ` kernel test robot
2026-01-15 18:20 ` kernel test robot
2026-01-30 2:59 ` [RFC 0/1] " Diangang Li
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.