* [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis
@ 2026-07-02 12:57 wang.yaxin
2026-07-02 12:58 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: wang.yaxin @ 2026-07-02 12:57 UTC (permalink / raw)
To: akpm, fan.yu9, yang.yang29
Cc: corbet, linux-kernel, linux-doc, xu.xin16, wang.yaxin
From: Wang Yaxin <wang.yaxin@zte.com.cn>
Previously delaytop only showed average delays. This patch adds:
1. delay_max fields to track the maximum delay value for each delay type
(cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy)
per task.
2. The -t/--type option displays only the specified delay type with avg/max
values side by side, allowing focused analysis:
delaytop -t cpu # Show only CPU delay with avg/max
delaytop -t wpcopy # Show Copy-on-Write delay with avg/max
3. Wall-clock timestamp when each maximum delay occurred, displayed in the
MAX_TIMESTAMP column when using -t/--type option. This enables:
- Identifying the time when a process experienced an abnormal delay max
- Correlating delay max across multiple processes at the same timestamp
- Cross-referencing with logs, traces, or other metrics at that time
4. When using -t/--type option, tasks are sorted by maximum delay value in
descending order (largest delay first), enabling quick identification of
top N processes with highest delay spikes.
Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn>
v2->v3:
[patch 1/3]
1. fix get_field_by_name() MODE_TYPE filter breaking --sort for non-type fields
2. zero-init stats struct and cap memcpy size in netlink response parsing
3. fix compare_tasks() count field read as unsigned long long for 32-bit safety
4. strcmp → offset-based access via field_desc.max_offset
[patch 2/3]
1. fix 32-bit time_t overflow check to use max representable value instead of (time_t)-1
2. fix gmtime() pointer aliasing by assigning timespec64 tv_sec to local time_t
3. dropped pre-2000 check; gmtime → localtime_r + stack struct tm
[patch 3/3]
1. fixed Y2038 overflow false negative in format_kernel_timespec()
2. switched to localtime_r() with stack `struct tm` for thread safety.
3. `field_delay_max_and_ts()` `max_ts` nullable; removed unused ts_scratch
4. `-t/--type` mode: hide `[o]sort` and `[M]memverbose`, show only `[q]quit`
Wang Yaxin (3):
delaytop: add delay max for delaytop
delaytop: add timestamp of delay max
delaytop: sort by max delay to highlight top latency processes
Documentation/accounting/delay-accounting.rst | 43 +++
tools/accounting/delaytop.c | 286 +++++++++++++++---
tools/accounting/getdelays.c | 5 +
3 files changed, 296 insertions(+), 38 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 1/3] delaytop: add delay max for delaytop 2026-07-02 12:57 [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin @ 2026-07-02 12:58 ` wang.yaxin 2026-07-02 22:58 ` Andrew Morton 2026-07-02 12:58 ` [PATCH 2/3] delaytop: add timestamp of delay max wang.yaxin ` (2 subsequent siblings) 3 siblings, 1 reply; 9+ messages in thread From: wang.yaxin @ 2026-07-02 12:58 UTC (permalink / raw) To: wang.yaxin Cc: akpm, fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 From: Wang Yaxin <wang.yaxin@zte.com.cn> Previously delaytop only showed average delays. Add delay_max fields to track the maximum delay value for each delay type (cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy) per task. This provides a global view of all tasks' delay spikes, which is essential for identifying processes that experienced brief but significant latency events that would be hidden by average-only metrics. The -t/--type option displays only the specified delay type with avg/max values side by side, allowing focused analysis: delaytop -t cpu # Show only CPU delay with avg/max delaytop -t wpcopy # Show Copy-on-Write delay with avg/max Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> --- tools/accounting/delaytop.c | 165 ++++++++++++++++++++++++++++-------- 1 file changed, 131 insertions(+), 34 deletions(-) diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c index 72cc500b44b1..4410f8add3e4 100644 --- a/tools/accounting/delaytop.c +++ b/tools/accounting/delaytop.c @@ -75,13 +75,21 @@ {#name, #cmd, \ offsetof(struct task_info, name##_delay_total), \ offsetof(struct task_info, name##_count), \ + offsetof(struct task_info, name##_delay_max), \ modes} -#define END_FIELD {NULL, 0, 0} +#define SORT_FIELD_NO_MAX(name, cmd, modes) \ + {#name, #cmd, \ + offsetof(struct task_info, name##_delay_total), \ + offsetof(struct task_info, name##_count), \ + 0, \ + modes} +#define END_FIELD {NULL, 0, 0, 0, 0, 0} /* Display mode types */ #define MODE_TYPE_ALL (0xFFFFFFFF) #define MODE_DEFAULT (1 << 0) #define MODE_MEMVERBOSE (1 << 1) +#define MODE_TYPE (1 << 2) /* Display specific type with avg/max */ /* PSI statistics structure */ struct psi_stats { @@ -108,20 +116,28 @@ struct task_info { char command[TASK_COMM_LEN]; unsigned long long cpu_count; unsigned long long cpu_delay_total; + unsigned long long cpu_delay_max; unsigned long long blkio_count; unsigned long long blkio_delay_total; + unsigned long long blkio_delay_max; unsigned long long swapin_count; unsigned long long swapin_delay_total; + unsigned long long swapin_delay_max; unsigned long long freepages_count; unsigned long long freepages_delay_total; + unsigned long long freepages_delay_max; unsigned long long thrashing_count; unsigned long long thrashing_delay_total; + unsigned long long thrashing_delay_max; unsigned long long compact_count; unsigned long long compact_delay_total; + unsigned long long compact_delay_max; unsigned long long wpcopy_count; unsigned long long wpcopy_delay_total; + unsigned long long wpcopy_delay_max; unsigned long long irq_count; unsigned long long irq_delay_total; + unsigned long long irq_delay_max; unsigned long long mem_count; unsigned long long mem_delay_total; }; @@ -141,6 +157,7 @@ struct field_desc { const char *cmd_char; /* Interactive command */ unsigned long total_offset; /* Offset of total delay in task_info */ unsigned long count_offset; /* Offset of count in task_info */ + unsigned long max_offset; /* Offset of max delay in task_info */ size_t supported_modes; /* Supported display modes */ }; @@ -153,6 +170,7 @@ struct config { int monitor_pid; /* Monitor specific PID */ char *container_path; /* Path to container cgroup */ const struct field_desc *sort_field; /* Current sort field */ + const struct field_desc *type_field; /* Type field for -t option */ size_t display_mode; /* Current display mode */ }; @@ -164,15 +182,15 @@ static int task_count; static int running = 1; static struct container_stats container_stats; static const struct field_desc sort_fields[] = { - SORT_FIELD(cpu, c, MODE_DEFAULT), - SORT_FIELD(blkio, i, MODE_DEFAULT), - SORT_FIELD(irq, q, MODE_DEFAULT), - SORT_FIELD(mem, m, MODE_DEFAULT | MODE_MEMVERBOSE), - SORT_FIELD(swapin, s, MODE_MEMVERBOSE), - SORT_FIELD(freepages, r, MODE_MEMVERBOSE), - SORT_FIELD(thrashing, t, MODE_MEMVERBOSE), - SORT_FIELD(compact, p, MODE_MEMVERBOSE), - SORT_FIELD(wpcopy, w, MODE_MEMVERBOSE), + SORT_FIELD(cpu, c, MODE_DEFAULT | MODE_TYPE), + SORT_FIELD(blkio, i, MODE_DEFAULT | MODE_TYPE), + SORT_FIELD(irq, q, MODE_DEFAULT | MODE_TYPE), + SORT_FIELD_NO_MAX(mem, m, MODE_DEFAULT | MODE_MEMVERBOSE), + SORT_FIELD(swapin, s, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(freepages, r, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(thrashing, t, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(compact, p, MODE_MEMVERBOSE | MODE_TYPE), + SORT_FIELD(wpcopy, w, MODE_MEMVERBOSE | MODE_TYPE), END_FIELD }; static int sort_selected; @@ -265,6 +283,7 @@ static void usage(void) " -p, --pid=PID Monitor only the specified PID\n" " -C, --container=PATH Monitor the container at specified cgroup path\n" " -s, --sort=FIELD Sort by delay field (default: cpu)\n" + " -t, --type=FIELD Display only specified delay type with avg/max\n" " -M, --memverbose Display memory detailed information\n"); exit(0); } @@ -283,6 +302,7 @@ static void parse_args(int argc, char **argv) {"processes", required_argument, 0, 'P'}, {"sort", required_argument, 0, 's'}, {"container", required_argument, 0, 'C'}, + {"type", required_argument, 0, 't'}, {"memverbose", no_argument, 0, 'M'}, {0, 0, 0, 0} }; @@ -292,6 +312,7 @@ static void parse_args(int argc, char **argv) cfg.iterations = 0; cfg.max_processes = 20; cfg.sort_field = &sort_fields[0]; /* Default sorted by CPU delay */ + cfg.type_field = NULL; /* No type field by default */ cfg.output_one_time = 0; cfg.monitor_pid = 0; /* 0 means monitor all PIDs */ cfg.container_path = NULL; @@ -300,7 +321,7 @@ static void parse_args(int argc, char **argv) while (1) { int option_index = 0; - c = getopt_long(argc, argv, "hd:n:p:oP:C:s:M", long_options, &option_index); + c = getopt_long(argc, argv, "hd:n:p:oP:C:s:t:M", long_options, &option_index); if (c == -1) break; @@ -363,9 +384,32 @@ static void parse_args(int argc, char **argv) cfg.sort_field = field; break; + case 't': + if (strlen(optarg) == 0) { + fprintf(stderr, "Error: empty type field\n"); + exit(1); + } + + field = get_field_by_name(optarg); + /* Show available fields if invalid option provided */ + if (!field || !(field->supported_modes & MODE_TYPE)) { + fprintf(stderr, "Error: invalid type field '%s'\n", optarg); + display_available_fields(MODE_TYPE); + exit(1); + } + + cfg.type_field = field; + cfg.display_mode = MODE_TYPE; + break; case 'M': cfg.display_mode = MODE_MEMVERBOSE; - cfg.sort_field = get_field_by_name("mem"); + /* Find first field supporting MODE_MEMVERBOSE for sorting */ + for (field = sort_fields; field->name != NULL; field++) { + if (field->supported_modes & MODE_MEMVERBOSE) { + cfg.sort_field = field; + break; + } + } break; default: fprintf(stderr, "Try 'delaytop --help' for more information.\n"); @@ -690,7 +734,12 @@ static void fetch_and_fill_task_info(int pid, const char *comm) nested_len = NLA_PAYLOAD(na->nla_len); while (nested_len > 0) { if (nested->nla_type == TASKSTATS_TYPE_STATS) { - memcpy(&stats, NLA_DATA(nested), sizeof(stats)); + size_t payload_len = NLA_PAYLOAD(nested->nla_len); + + memset(&stats, 0, sizeof(stats)); + if (payload_len > sizeof(stats)) + payload_len = sizeof(stats); + memcpy(&stats, NLA_DATA(nested), payload_len); if (task_count < MAX_TASKS) { tasks[task_count].pid = pid; tasks[task_count].tgid = pid; @@ -699,20 +748,28 @@ static void fetch_and_fill_task_info(int pid, const char *comm) tasks[task_count].command[TASK_COMM_LEN - 1] = '\0'; SET_TASK_STAT(task_count, cpu_count); SET_TASK_STAT(task_count, cpu_delay_total); + SET_TASK_STAT(task_count, cpu_delay_max); SET_TASK_STAT(task_count, blkio_count); SET_TASK_STAT(task_count, blkio_delay_total); + SET_TASK_STAT(task_count, blkio_delay_max); SET_TASK_STAT(task_count, swapin_count); SET_TASK_STAT(task_count, swapin_delay_total); + SET_TASK_STAT(task_count, swapin_delay_max); SET_TASK_STAT(task_count, freepages_count); SET_TASK_STAT(task_count, freepages_delay_total); + SET_TASK_STAT(task_count, freepages_delay_max); SET_TASK_STAT(task_count, thrashing_count); SET_TASK_STAT(task_count, thrashing_delay_total); + SET_TASK_STAT(task_count, thrashing_delay_max); SET_TASK_STAT(task_count, compact_count); SET_TASK_STAT(task_count, compact_delay_total); + SET_TASK_STAT(task_count, compact_delay_max); SET_TASK_STAT(task_count, wpcopy_count); SET_TASK_STAT(task_count, wpcopy_delay_total); + SET_TASK_STAT(task_count, wpcopy_delay_max); SET_TASK_STAT(task_count, irq_count); SET_TASK_STAT(task_count, irq_delay_total); + SET_TASK_STAT(task_count, irq_delay_max); set_mem_count(&tasks[task_count]); set_mem_delay_total(&tasks[task_count]); task_count++; @@ -777,14 +834,14 @@ static int compare_tasks(const void *a, const void *b) const struct task_info *t2 = (const struct task_info *)b; unsigned long long total1; unsigned long long total2; - unsigned long count1; - unsigned long count2; + unsigned long long count1; + unsigned long long count2; double avg1, avg2; total1 = *(unsigned long long *)((char *)t1 + cfg.sort_field->total_offset); total2 = *(unsigned long long *)((char *)t2 + cfg.sort_field->total_offset); - count1 = *(unsigned long *)((char *)t1 + cfg.sort_field->count_offset); - count2 = *(unsigned long *)((char *)t2 + cfg.sort_field->count_offset); + count1 = *(unsigned long long *)((char *)t1 + cfg.sort_field->count_offset); + count2 = *(unsigned long long *)((char *)t2 + cfg.sort_field->count_offset); avg1 = average_ms(total1, count1); avg2 = average_ms(total2, count2); @@ -794,6 +851,26 @@ static int compare_tasks(const void *a, const void *b) return 0; } +/* Get delay values for a specific field */ +static void get_field_delay_values(const struct task_info *task, const struct field_desc *field, + double *avg_ms, double *max_ms) +{ + unsigned long long total, count, max; + + if (!field || !field->max_offset) { + *avg_ms = 0; + *max_ms = 0; + return; + } + + total = *(unsigned long long *)((char *)task + field->total_offset); + count = *(unsigned long long *)((char *)task + field->count_offset); + *avg_ms = average_ms(total, count); + + max = *(unsigned long long *)((char *)task + field->max_offset); + *max_ms = (double)max / 1000000.0; /* Convert nanoseconds to milliseconds */ +} + /* Sort tasks by selected field */ static void sort_tasks(void) { @@ -847,7 +924,12 @@ static void get_container_stats(void) while (nl_len > 0) { if (na->nla_type == CGROUPSTATS_TYPE_CGROUP_STATS) { /* Get the cgroupstats structure */ - memcpy(&stats, NLA_DATA(na), sizeof(stats)); + size_t payload_len = NLA_PAYLOAD(na->nla_len); + + memset(&stats, 0, sizeof(stats)); + if (payload_len > sizeof(stats)) + payload_len = sizeof(stats); + memcpy(&stats, NLA_DATA(na), payload_len); /* Fill container stats */ container_stats.nr_sleeping = stats.nr_sleeping; @@ -950,29 +1032,44 @@ static void display_results(int psi_ret) suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n", cfg.max_processes, get_name_by_field(cfg.sort_field)); - suc &= BOOL_FPRINT(out, "%8s %8s %-17s", "PID", "TGID", "COMMAND"); - if (cfg.display_mode == MODE_MEMVERBOSE) { - suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s %8s %8s\n", - "MEM(ms)", "SWAP(ms)", "RCL(ms)", - "THR(ms)", "CMP(ms)", "WP(ms)"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "---------------------\n"); + if (cfg.display_mode == MODE_TYPE && cfg.type_field) { + /* Display mode for -t option: show only specified type with avg/max */ + suc &= BOOL_FPRINT(out, "%8s %8s %-17s %12s %12s\n", + "PID", "TGID", "COMMAND", + "AVG(ms)", "MAX(ms)"); + suc &= BOOL_FPRINT(out, "----------------------------------------------------\n"); } else { - suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s\n", - "CPU(ms)", "IO(ms)", "IRQ(ms)", "MEM(ms)"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "-----------------------"); - suc &= BOOL_FPRINT(out, "--------------------------\n"); + suc &= BOOL_FPRINT(out, "%8s %8s %-17s", "PID", "TGID", "COMMAND"); + if (cfg.display_mode == MODE_MEMVERBOSE) { + suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s %8s %8s\n", + "MEM(ms)", "SWAP(ms)", "RCL(ms)", + "THR(ms)", "CMP(ms)", "WP(ms)"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "---------------------\n"); + } else { + suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s\n", + "CPU(ms)", "IO(ms)", "IRQ(ms)", "MEM(ms)"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "-----------------------"); + suc &= BOOL_FPRINT(out, "--------------------------\n"); + } } count = task_count < cfg.max_processes ? task_count : cfg.max_processes; for (i = 0; i < count; i++) { - suc &= BOOL_FPRINT(out, "%8d %8d %-15s", + suc &= BOOL_FPRINT(out, "%8d %8d %-17s", tasks[i].pid, tasks[i].tgid, tasks[i].command); - if (cfg.display_mode == MODE_MEMVERBOSE) { + if (cfg.display_mode == MODE_TYPE && cfg.type_field) { + double avg_ms, max_ms; + + get_field_delay_values(&tasks[i], cfg.type_field, &avg_ms, &max_ms); + + suc &= BOOL_FPRINT(out, "%12.2f %12.2f\n", + avg_ms, max_ms); + } else if (cfg.display_mode == MODE_MEMVERBOSE) { suc &= BOOL_FPRINT(out, DELAY_FMT_MEMVERBOSE, TASK_AVG(tasks[i], mem), TASK_AVG(tasks[i], swapin), -- 2.25.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 1/3] delaytop: add delay max for delaytop 2026-07-02 12:58 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin @ 2026-07-02 22:58 ` Andrew Morton 0 siblings, 0 replies; 9+ messages in thread From: Andrew Morton @ 2026-07-02 22:58 UTC (permalink / raw) To: wang.yaxin Cc: fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 On Thu, 2 Jul 2026 20:58:15 +0800 (CST) <wang.yaxin@zte.com.cn> wrote: > From: Wang Yaxin <wang.yaxin@zte.com.cn> > > Previously delaytop only showed average delays. Add delay_max fields to > track the maximum delay value for each delay type (cpu, blkio, irq, swapin, > freepages, thrashing, compact, wpcopy) per task. > > This provides a global view of all tasks' delay spikes, which is essential > for identifying processes that experienced brief but significant latency > events that would be hidden by average-only metrics. > > The -t/--type option displays only the specified delay type with avg/max > values side by side, allowing focused analysis: > delaytop -t cpu # Show only CPU delay with avg/max > delaytop -t wpcopy # Show Copy-on-Write delay with avg/max > > ... > > total1 = *(unsigned long long *)((char *)t1 + cfg.sort_field->total_offset); The code does this pointer operation in many places. Is there a better way? Should all those `unsigned long long' fields in `struct task_info' be in an array, something like that? ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 2/3] delaytop: add timestamp of delay max 2026-07-02 12:57 [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin 2026-07-02 12:58 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin @ 2026-07-02 12:58 ` wang.yaxin 2026-07-02 23:01 ` Andrew Morton 2026-07-02 13:00 ` [PATCH 3/3] delaytop: sort by max delay to highlight top latency processes wang.yaxin 2026-07-02 23:01 ` [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis Andrew Morton 3 siblings, 1 reply; 9+ messages in thread From: wang.yaxin @ 2026-07-02 12:58 UTC (permalink / raw) To: wang.yaxin Cc: akpm, fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 From: Wang Yaxin <wang.yaxin@zte.com.cn> Record the wall-clock timestamp when each maximum delay occurred for all delay types. The timestamp is displayed in the MAX_TIMESTAMP column when using -t/--type option. This enables: - Identifying the time when a process experienced an abnormal delay spike - Correlating delay peaks across multiple processes at the same timestamp - Cross-referencing with system logs, traces, or other metrics at that time - Pinpointing the root cause of latency issues by finding concurrent events Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> --- tools/accounting/delaytop.c | 84 ++++++++++++++++++++++++++++++++----- 1 file changed, 74 insertions(+), 10 deletions(-) diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c index 4410f8add3e4..be28a4b70f70 100644 --- a/tools/accounting/delaytop.c +++ b/tools/accounting/delaytop.c @@ -76,14 +76,16 @@ offsetof(struct task_info, name##_delay_total), \ offsetof(struct task_info, name##_count), \ offsetof(struct task_info, name##_delay_max), \ + offsetof(struct task_info, name##_delay_max_ts), \ modes} #define SORT_FIELD_NO_MAX(name, cmd, modes) \ {#name, #cmd, \ offsetof(struct task_info, name##_delay_total), \ offsetof(struct task_info, name##_count), \ 0, \ + 0, \ modes} -#define END_FIELD {NULL, 0, 0, 0, 0, 0} +#define END_FIELD {NULL, 0, 0, 0, 0, 0, 0} /* Display mode types */ #define MODE_TYPE_ALL (0xFFFFFFFF) @@ -117,27 +119,35 @@ struct task_info { unsigned long long cpu_count; unsigned long long cpu_delay_total; unsigned long long cpu_delay_max; + struct __kernel_timespec cpu_delay_max_ts; unsigned long long blkio_count; unsigned long long blkio_delay_total; unsigned long long blkio_delay_max; + struct __kernel_timespec blkio_delay_max_ts; unsigned long long swapin_count; unsigned long long swapin_delay_total; unsigned long long swapin_delay_max; + struct __kernel_timespec swapin_delay_max_ts; unsigned long long freepages_count; unsigned long long freepages_delay_total; unsigned long long freepages_delay_max; + struct __kernel_timespec freepages_delay_max_ts; unsigned long long thrashing_count; unsigned long long thrashing_delay_total; unsigned long long thrashing_delay_max; + struct __kernel_timespec thrashing_delay_max_ts; unsigned long long compact_count; unsigned long long compact_delay_total; unsigned long long compact_delay_max; + struct __kernel_timespec compact_delay_max_ts; unsigned long long wpcopy_count; unsigned long long wpcopy_delay_total; unsigned long long wpcopy_delay_max; + struct __kernel_timespec wpcopy_delay_max_ts; unsigned long long irq_count; unsigned long long irq_delay_total; unsigned long long irq_delay_max; + struct __kernel_timespec irq_delay_max_ts; unsigned long long mem_count; unsigned long long mem_delay_total; }; @@ -158,6 +168,7 @@ struct field_desc { unsigned long total_offset; /* Offset of total delay in task_info */ unsigned long count_offset; /* Offset of count in task_info */ unsigned long max_offset; /* Offset of max delay in task_info */ + unsigned long max_ts_offset; /* Offset of max delay timestamp in task_info */ size_t supported_modes; /* Supported display modes */ }; @@ -283,7 +294,7 @@ static void usage(void) " -p, --pid=PID Monitor only the specified PID\n" " -C, --container=PATH Monitor the container at specified cgroup path\n" " -s, --sort=FIELD Sort by delay field (default: cpu)\n" - " -t, --type=FIELD Display only specified delay type with avg/max\n" + " -t, --type=FIELD Display only specified delay type with avg/max/timestamp\n" " -M, --memverbose Display memory detailed information\n"); exit(0); } @@ -749,27 +760,35 @@ static void fetch_and_fill_task_info(int pid, const char *comm) SET_TASK_STAT(task_count, cpu_count); SET_TASK_STAT(task_count, cpu_delay_total); SET_TASK_STAT(task_count, cpu_delay_max); + SET_TASK_STAT(task_count, cpu_delay_max_ts); SET_TASK_STAT(task_count, blkio_count); SET_TASK_STAT(task_count, blkio_delay_total); SET_TASK_STAT(task_count, blkio_delay_max); + SET_TASK_STAT(task_count, blkio_delay_max_ts); SET_TASK_STAT(task_count, swapin_count); SET_TASK_STAT(task_count, swapin_delay_total); SET_TASK_STAT(task_count, swapin_delay_max); + SET_TASK_STAT(task_count, swapin_delay_max_ts); SET_TASK_STAT(task_count, freepages_count); SET_TASK_STAT(task_count, freepages_delay_total); SET_TASK_STAT(task_count, freepages_delay_max); + SET_TASK_STAT(task_count, freepages_delay_max_ts); SET_TASK_STAT(task_count, thrashing_count); SET_TASK_STAT(task_count, thrashing_delay_total); SET_TASK_STAT(task_count, thrashing_delay_max); + SET_TASK_STAT(task_count, thrashing_delay_max_ts); SET_TASK_STAT(task_count, compact_count); SET_TASK_STAT(task_count, compact_delay_total); SET_TASK_STAT(task_count, compact_delay_max); + SET_TASK_STAT(task_count, compact_delay_max_ts); SET_TASK_STAT(task_count, wpcopy_count); SET_TASK_STAT(task_count, wpcopy_delay_total); SET_TASK_STAT(task_count, wpcopy_delay_max); + SET_TASK_STAT(task_count, wpcopy_delay_max_ts); SET_TASK_STAT(task_count, irq_count); SET_TASK_STAT(task_count, irq_delay_total); SET_TASK_STAT(task_count, irq_delay_max); + SET_TASK_STAT(task_count, irq_delay_max_ts); set_mem_count(&tasks[task_count]); set_mem_delay_total(&tasks[task_count]); task_count++; @@ -827,6 +846,41 @@ static double average_ms(unsigned long long total, unsigned long long count) return (double)total / 1000000.0 / count; } +/* + * Format __kernel_timespec to human readable string (YYYY-MM-DDTHH:MM:SS) + * Returns formatted string or "N/A" if timestamp is zero + */ +static const char *format_timespec64(struct __kernel_timespec *ts) +{ + static char buffer[32]; + time_t time_sec; + struct tm tm_info; + + /* Check if timestamp is zero (not set) */ + if (ts->tv_sec == 0 && ts->tv_nsec == 0) + return "N/A"; + + /* Avoid Y2038 truncation: check if timestamp fits in time_t on 32-bit platforms */ + if (sizeof(time_t) < sizeof(ts->tv_sec) && + ts->tv_sec > (__u64)((1ULL << (sizeof(time_t) * 8 - 1)) - 1)) + return "N/A"; + + time_sec = (time_t)ts->tv_sec; + + if (localtime_r(&time_sec, &tm_info) == NULL) + return "N/A"; + + snprintf(buffer, sizeof(buffer), "%04d-%02d-%02dT%02d:%02d:%02d", + tm_info.tm_year + 1900, + tm_info.tm_mon + 1, + tm_info.tm_mday, + tm_info.tm_hour, + tm_info.tm_min, + tm_info.tm_sec); + + return buffer; +} + /* Comparison function for sorting tasks */ static int compare_tasks(const void *a, const void *b) { @@ -853,13 +907,15 @@ static int compare_tasks(const void *a, const void *b) /* Get delay values for a specific field */ static void get_field_delay_values(const struct task_info *task, const struct field_desc *field, - double *avg_ms, double *max_ms) + double *avg_ms, double *max_ms, + struct __kernel_timespec *max_ts) { unsigned long long total, count, max; if (!field || !field->max_offset) { *avg_ms = 0; *max_ms = 0; + memset(max_ts, 0, sizeof(*max_ts)); return; } @@ -869,6 +925,11 @@ static void get_field_delay_values(const struct task_info *task, const struct fi max = *(unsigned long long *)((char *)task + field->max_offset); *max_ms = (double)max / 1000000.0; /* Convert nanoseconds to milliseconds */ + + if (field->max_ts_offset) + *max_ts = *(struct __kernel_timespec *)((char *)task + field->max_ts_offset); + else + memset(max_ts, 0, sizeof(*max_ts)); } /* Sort tasks by selected field */ @@ -1033,11 +1094,12 @@ static void display_results(int psi_ret) cfg.max_processes, get_name_by_field(cfg.sort_field)); if (cfg.display_mode == MODE_TYPE && cfg.type_field) { - /* Display mode for -t option: show only specified type with avg/max */ - suc &= BOOL_FPRINT(out, "%8s %8s %-17s %12s %12s\n", + /* Display mode for -t option: show only specified type with avg/max/timestamp */ + suc &= BOOL_FPRINT(out, "%8s %8s %-17s %12s %12s %20s\n", "PID", "TGID", "COMMAND", - "AVG(ms)", "MAX(ms)"); - suc &= BOOL_FPRINT(out, "----------------------------------------------------\n"); + "AVG(ms)", "MAX(ms)", "MAX_TIMESTAMP"); + suc &= BOOL_FPRINT(out, "--------------------------------------------------------"); + suc &= BOOL_FPRINT(out, "----------------------------------------\n"); } else { suc &= BOOL_FPRINT(out, "%8s %8s %-17s", "PID", "TGID", "COMMAND"); if (cfg.display_mode == MODE_MEMVERBOSE) { @@ -1064,11 +1126,13 @@ static void display_results(int psi_ret) tasks[i].pid, tasks[i].tgid, tasks[i].command); if (cfg.display_mode == MODE_TYPE && cfg.type_field) { double avg_ms, max_ms; + struct __kernel_timespec max_ts; - get_field_delay_values(&tasks[i], cfg.type_field, &avg_ms, &max_ms); + get_field_delay_values(&tasks[i], cfg.type_field, &avg_ms, + &max_ms, &max_ts); - suc &= BOOL_FPRINT(out, "%12.2f %12.2f\n", - avg_ms, max_ms); + suc &= BOOL_FPRINT(out, "%12.2f %12.2f %20s\n", + avg_ms, max_ms, format_timespec64(&max_ts)); } else if (cfg.display_mode == MODE_MEMVERBOSE) { suc &= BOOL_FPRINT(out, DELAY_FMT_MEMVERBOSE, TASK_AVG(tasks[i], mem), -- 2.25.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 2/3] delaytop: add timestamp of delay max 2026-07-02 12:58 ` [PATCH 2/3] delaytop: add timestamp of delay max wang.yaxin @ 2026-07-02 23:01 ` Andrew Morton 0 siblings, 0 replies; 9+ messages in thread From: Andrew Morton @ 2026-07-02 23:01 UTC (permalink / raw) To: wang.yaxin Cc: fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 On Thu, 2 Jul 2026 20:58:54 +0800 (CST) <wang.yaxin@zte.com.cn> wrote: > From: Wang Yaxin <wang.yaxin@zte.com.cn> > > Record the wall-clock timestamp when each maximum delay occurred for > all delay types. The timestamp is displayed in the MAX_TIMESTAMP column > when using -t/--type option. > > This enables: > - Identifying the time when a process experienced an abnormal delay spike > - Correlating delay peaks across multiple processes at the same timestamp > - Cross-referencing with system logs, traces, or other metrics at that time > - Pinpointing the root cause of latency issues by finding concurrent events > > ... > > +/* > + * Format __kernel_timespec to human readable string (YYYY-MM-DDTHH:MM:SS) > + * Returns formatted string or "N/A" if timestamp is zero > + */ > +static const char *format_timespec64(struct __kernel_timespec *ts) > +{ > + static char buffer[32]; > + time_t time_sec; > + struct tm tm_info; > + > + /* Check if timestamp is zero (not set) */ > + if (ts->tv_sec == 0 && ts->tv_nsec == 0) > + return "N/A"; > + > + /* Avoid Y2038 truncation: check if timestamp fits in time_t on 32-bit platforms */ > + if (sizeof(time_t) < sizeof(ts->tv_sec) && > + ts->tv_sec > (__u64)((1ULL << (sizeof(time_t) * 8 - 1)) - 1)) > + return "N/A"; > + > + time_sec = (time_t)ts->tv_sec; > + > + if (localtime_r(&time_sec, &tm_info) == NULL) > + return "N/A"; > + > + snprintf(buffer, sizeof(buffer), "%04d-%02d-%02dT%02d:%02d:%02d", > + tm_info.tm_year + 1900, > + tm_info.tm_mon + 1, > + tm_info.tm_mday, > + tm_info.tm_hour, > + tm_info.tm_min, > + tm_info.tm_sec); > + > + return buffer; > +} This appears to be a copy-paste-edit from format_timespec() in tools/accounting/getdelays.c. Why do the two differ? Is it possible to use a common implementation? tools/accounting/format_tiomespec.o? Does the getdelays.c version have the possible issue which AI review identified? ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 3/3] delaytop: sort by max delay to highlight top latency processes 2026-07-02 12:57 [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin 2026-07-02 12:58 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin 2026-07-02 12:58 ` [PATCH 2/3] delaytop: add timestamp of delay max wang.yaxin @ 2026-07-02 13:00 ` wang.yaxin 2026-07-02 23:01 ` [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis Andrew Morton 3 siblings, 0 replies; 9+ messages in thread From: wang.yaxin @ 2026-07-02 13:00 UTC (permalink / raw) To: wang.yaxin Cc: akpm, fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 From: Wang Yaxin <wang.yaxin@zte.com.cn> When using -t/--type option, sort tasks by the maximum delay value of the selected type in descending order (largest delay first). This enables quickly identifying the top N processes with the highest delay spikes, which is essential for diagnosing latency problems by pinpointing which processes contributed most to system delays. Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> --- Documentation/accounting/delay-accounting.rst | 43 +++++++++++ tools/accounting/delaytop.c | 71 ++++++++++++++++--- tools/accounting/getdelays.c | 5 ++ 3 files changed, 108 insertions(+), 11 deletions(-) diff --git a/Documentation/accounting/delay-accounting.rst b/Documentation/accounting/delay-accounting.rst index e209c46241b0..4bb2e72a6043 100644 --- a/Documentation/accounting/delay-accounting.rst +++ b/Documentation/accounting/delay-accounting.rst @@ -212,3 +212,46 @@ Advanced usage examples:: # ./delaytop -d secs Specify refresh interval as secs + + # ./delaytop -t type + Display only specified delay type with avg/max/timestamp + (rows sorted by MAX for that type, largest first) + + + +delaytop add delay_max fields to track the maximum delay value for each delay type +(cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy) per task. + + bash# ./delaytop -t cpu + System Pressure Information: (avg10/avg60/avg300/total) + CPU some: 0.4%/ 0.2%/ 0.1%/ 220(ms) + CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms) + Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms) + Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms) + IO full: 0.0%/ 0.0%/ 0.0%/ 12(ms) + IO some: 0.0%/ 0.0%/ 0.0%/ 13(ms) + IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms) + [q]quit + Top 20 processes (sorted by cpu MAX delay, largest first): + PID TGID COMMAND AVG(ms) MAX(ms) MAX_TIMESTAMP + ------------------------------------------------------------------------ + 9 9 kworker/0:0-eve 0.59 16.87 2026-05-27T13:32:39 + 30 30 kworker/2:0H-kb 2.87 11.36 2026-05-27T13:32:36 + 27 27 migration/2 1.05 9.51 2026-05-27T13:32:37 + 50 50 kworker/2:1-eve 0.50 9.13 2026-05-27T13:32:37 + 15 15 rcu_preempt 0.11 8.98 2026-05-27T13:32:37 + 1 1 init 0.17 7.12 2026-05-27T13:32:38 + 67 67 scsi_eh_0 1.20 4.23 2026-05-27T13:32:37 + 23 23 ksoftirqd/1 1.12 3.77 2026-05-27T13:32:36 + 3 3 pool_workqueue_ 0.72 3.55 2026-05-27T13:32:38 + 62 62 kworker/u20:2-a 0.49 3.03 2026-05-27T13:32:37 + 2 2 kthreadd 0.18 2.82 2026-05-27T13:32:37 + 11 11 kworker/0:1 1.42 2.76 2026-05-27T13:32:36 + 39 39 kworker/u20:0-a 0.10 2.71 2026-05-27T13:32:38 + 17 17 rcu_exp_gp_kthr 0.25 2.65 2026-05-27T13:32:37 + 66 66 kworker/u20:3-e 0.38 2.55 2026-05-27T13:32:37 + 20 20 cpuhp/0 0.53 2.51 2026-05-27T13:32:37 + 28 28 ksoftirqd/2 0.59 2.48 2026-05-27T13:32:37 + 55 55 kworker/u19:1 0.88 2.42 2026-05-27T13:32:37 + 13 13 kworker/R-mm_pe 1.18 2.35 2026-05-27T13:32:36 + 54 54 kworker/3:1-eve 0.14 2.20 2026-05-27T13:32:38 diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c index be28a4b70f70..1c40bb477320 100644 --- a/tools/accounting/delaytop.c +++ b/tools/accounting/delaytop.c @@ -295,6 +295,7 @@ static void usage(void) " -C, --container=PATH Monitor the container at specified cgroup path\n" " -s, --sort=FIELD Sort by delay field (default: cpu)\n" " -t, --type=FIELD Display only specified delay type with avg/max/timestamp\n" + " (rows sorted by MAX for that type, largest first)\n" " -M, --memverbose Display memory detailed information\n"); exit(0); } @@ -838,6 +839,15 @@ static void get_task_delays(void) closedir(dir); } +static void field_delay_max_and_ts(const struct task_info *task, + const struct field_desc *field, + unsigned long long *max_ns, + struct __kernel_timespec *max_ts); +static void get_field_delay_values(const struct task_info *task, + const struct field_desc *field, + double *avg_ms, double *max_ms, + struct __kernel_timespec *max_ts); + /* Calculate average delay in milliseconds */ static double average_ms(unsigned long long total, unsigned long long count) { @@ -850,7 +860,7 @@ static double average_ms(unsigned long long total, unsigned long long count) * Format __kernel_timespec to human readable string (YYYY-MM-DDTHH:MM:SS) * Returns formatted string or "N/A" if timestamp is zero */ -static const char *format_timespec64(struct __kernel_timespec *ts) +static const char *format_kernel_timespec(struct __kernel_timespec *ts) { static char buffer[32]; time_t time_sec; @@ -891,6 +901,16 @@ static int compare_tasks(const void *a, const void *b) unsigned long long count1; unsigned long long count2; double avg1, avg2; + unsigned long long max1, max2; + + /* -t/--type: default sort by MAX column for the selected type (descending) */ + if (cfg.display_mode == MODE_TYPE && cfg.type_field) { + field_delay_max_and_ts(t1, cfg.type_field, &max1, NULL); + field_delay_max_and_ts(t2, cfg.type_field, &max2, NULL); + if (max1 != max2) + return max2 > max1 ? 1 : -1; + return 0; + } total1 = *(unsigned long long *)((char *)t1 + cfg.sort_field->total_offset); total2 = *(unsigned long long *)((char *)t2 + cfg.sort_field->total_offset); @@ -905,6 +925,28 @@ static int compare_tasks(const void *a, const void *b) return 0; } +/* Max delay (ns) and timestamp for field (shared by display and sort) */ +static void field_delay_max_and_ts(const struct task_info *task, const struct field_desc *field, + unsigned long long *max_ns, struct __kernel_timespec *max_ts) +{ + if (!field || !field->max_offset) { + *max_ns = 0; + if (max_ts) + memset(max_ts, 0, sizeof(*max_ts)); + return; + } + + *max_ns = *(unsigned long long *)((char *)task + field->max_offset); + + if (max_ts) { + if (field->max_ts_offset) + *max_ts = *(struct __kernel_timespec *)((char *)task + + field->max_ts_offset); + else + memset(max_ts, 0, sizeof(*max_ts)); + } +} + /* Get delay values for a specific field */ static void get_field_delay_values(const struct task_info *task, const struct field_desc *field, double *avg_ms, double *max_ms, @@ -923,13 +965,8 @@ static void get_field_delay_values(const struct task_info *task, const struct fi count = *(unsigned long long *)((char *)task + field->count_offset); *avg_ms = average_ms(total, count); - max = *(unsigned long long *)((char *)task + field->max_offset); + field_delay_max_and_ts(task, field, &max, max_ts); *max_ms = (double)max / 1000000.0; /* Convert nanoseconds to milliseconds */ - - if (field->max_ts_offset) - *max_ts = *(struct __kernel_timespec *)((char *)task + field->max_ts_offset); - else - memset(max_ts, 0, sizeof(*max_ts)); } /* Sort tasks by selected field */ @@ -1079,7 +1116,10 @@ static void display_results(int psi_ret) } /* Interacive command */ - suc &= BOOL_FPRINT(out, "[o]sort [M]memverbose [q]quit\n"); + if (cfg.display_mode == MODE_TYPE && cfg.type_field) + suc &= BOOL_FPRINT(out, "[q]quit\n"); + else + suc &= BOOL_FPRINT(out, "[o]sort [M]memverbose [q]quit\n"); if (sort_selected) { if (cfg.display_mode == MODE_MEMVERBOSE) suc &= BOOL_FPRINT(out, @@ -1090,8 +1130,13 @@ static void display_results(int psi_ret) } /* Task delay output */ - suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n", - cfg.max_processes, get_name_by_field(cfg.sort_field)); + if (cfg.display_mode == MODE_TYPE && cfg.type_field) + suc &= BOOL_FPRINT(out, + "Top %d processes (sorted by %s MAX delay, largest first):\n", + cfg.max_processes, get_name_by_field(cfg.type_field)); + else + suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n", + cfg.max_processes, get_name_by_field(cfg.sort_field)); if (cfg.display_mode == MODE_TYPE && cfg.type_field) { /* Display mode for -t option: show only specified type with avg/max/timestamp */ @@ -1132,7 +1177,7 @@ static void display_results(int psi_ret) &max_ms, &max_ts); suc &= BOOL_FPRINT(out, "%12.2f %12.2f %20s\n", - avg_ms, max_ms, format_timespec64(&max_ts)); + avg_ms, max_ms, format_kernel_timespec(&max_ts)); } else if (cfg.display_mode == MODE_MEMVERBOSE) { suc &= BOOL_FPRINT(out, DELAY_FMT_MEMVERBOSE, TASK_AVG(tasks[i], mem), @@ -1201,9 +1246,13 @@ static void handle_keypress(char ch, int *running) } else { switch (ch) { case 'o': + if (cfg.display_mode == MODE_TYPE) + break; sort_selected = 1; break; case 'M': + if (cfg.display_mode == MODE_TYPE) + break; toggle_display_mode(); for (field = sort_fields; field->name != NULL; field++) { if (field->supported_modes & cfg.display_mode) { diff --git a/tools/accounting/getdelays.c b/tools/accounting/getdelays.c index caa5fe9dd573..6ac30d4f96f7 100644 --- a/tools/accounting/getdelays.c +++ b/tools/accounting/getdelays.c @@ -235,6 +235,11 @@ static const char *format_timespec(struct __kernel_timespec *ts) if (ts->tv_sec == 0 && ts->tv_nsec == 0) return "N/A"; + /* Avoid Y2038 truncation on 32-bit platforms */ + if (sizeof(time_sec) < sizeof(ts->tv_sec) && + ts->tv_sec > (__u64)((1ULL << (sizeof(time_sec) * 8 - 1)) - 1)) + return "N/A"; + time_sec = ts->tv_sec; /* Use thread-safe localtime_r */ -- 2.25.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis 2026-07-02 12:57 [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin ` (2 preceding siblings ...) 2026-07-02 13:00 ` [PATCH 3/3] delaytop: sort by max delay to highlight top latency processes wang.yaxin @ 2026-07-02 23:01 ` Andrew Morton 3 siblings, 0 replies; 9+ messages in thread From: Andrew Morton @ 2026-07-02 23:01 UTC (permalink / raw) To: wang.yaxin Cc: fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 On Thu, 2 Jul 2026 20:57:04 +0800 (CST) <wang.yaxin@zte.com.cn> wrote: > Previously delaytop only showed average delays. This patch adds: > > 1. delay_max fields to track the maximum delay value for each delay type > (cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy) > per task. > > 2. The -t/--type option displays only the specified delay type with avg/max > values side by side, allowing focused analysis: > delaytop -t cpu # Show only CPU delay with avg/max > delaytop -t wpcopy # Show Copy-on-Write delay with avg/max > > 3. Wall-clock timestamp when each maximum delay occurred, displayed in the > MAX_TIMESTAMP column when using -t/--type option. This enables: > - Identifying the time when a process experienced an abnormal delay max > - Correlating delay max across multiple processes at the same timestamp > - Cross-referencing with logs, traces, or other metrics at that time > > 4. When using -t/--type option, tasks are sorted by maximum delay value in > descending order (largest delay first), enabling quick identification of > top N processes with highest delay spikes. Seems useful, thanks - I'll queue this for testing. AI review might have found a couple of things: https://sashiko.dev/#/patchset/20260702205704180NZ3cu_QF04KfBIL6vjTHL@zte.com.cn ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 0/3 v2] delaytop: add delay max, timestamp and sorting for top latency analysis
@ 2026-05-30 3:16 wang.yaxin
2026-05-30 3:18 ` [PATCH 2/3] delaytop: add timestamp of delay max wang.yaxin
0 siblings, 1 reply; 9+ messages in thread
From: wang.yaxin @ 2026-05-30 3:16 UTC (permalink / raw)
To: akpm, fan.yu9, yang.yang29
Cc: corbet, linux-kernel, linux-doc, xu.xin16, wang.yaxin
From: Wang Yaxin <wang.yaxin@zte.com.cn>
Previously delaytop only showed average delays. This patch adds:
1. delay_max fields to track the maximum delay value for each delay type
(cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy)
per task.
2. The -t/--type option displays only the specified delay type with avg/max
values side by side, allowing focused analysis:
delaytop -t cpu # Show only CPU delay with avg/max
delaytop -t wpcopy # Show Copy-on-Write delay with avg/max
3. Wall-clock timestamp when each maximum delay occurred, displayed in the
MAX_TIMESTAMP column when using -t/--type option. This enables:
- Identifying the time when a process experienced an abnormal delay max
- Correlating delay max across multiple processes at the same timestamp
- Cross-referencing with logs, traces, or other metrics at that time
4. When using -t/--type option, tasks are sorted by maximum delay value in
descending order (largest delay first), enabling quick identification of
top N processes with highest delay spikes.
Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn>
v1->v2:
https://lore.kernel.org/all/20260527124210.19726c1c89a94b89310e5a47@linux-foundation.org/
[patch 1/3]
1. Fix potential NULL pointer dereference in get_field_by_name().
2. Fix data truncation by changing count pointer type from unsigned long to
unsigned long long.
3. Fix column misalignment by changing COMMAND format from %-15s to %-17s.
[patch 2/3]
1. Add missing freepages_delay_max_ts initialization to avoid stale
timestamps.
2. Fix Year 2038 regression by removing explicit time_t cast.
[patch 3/3]
1. Fix Y2038 truncation on 32-bit platforms by checking timestamp
overflow before cast.
2. Mark field_delay_max_and_ts and get_field_delay_values as static
to avoid missing-prototypes warnings.
3. Fix unexpected unindent in the Documentation.
Wang Yaxin (3):
delaytop: add delay max for delaytop
delaytop: add timestamp of delay max
delaytop: sort by max delay to highlight top latency processes
Documentation/accounting/delay-accounting.rst | 43 +++
tools/accounting/delaytop.c | 263 ++++++++++++++++--
2 files changed, 276 insertions(+), 30 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 2/3] delaytop: add timestamp of delay max 2026-05-30 3:16 [PATCH 0/3 v2] " wang.yaxin @ 2026-05-30 3:18 ` wang.yaxin 0 siblings, 0 replies; 9+ messages in thread From: wang.yaxin @ 2026-05-30 3:18 UTC (permalink / raw) To: wang.yaxin Cc: akpm, fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 From: Wang Yaxin <wang.yaxin@zte.com.cn> Record the wall-clock timestamp when each maximum delay occurred for all delay types. The timestamp is displayed in the MAX_TIMESTAMP column when using -t/--type option. This enables: - Identifying the time when a process experienced an abnormal delay spike - Correlating delay peaks across multiple processes at the same timestamp - Cross-referencing with system logs, traces, or other metrics at that time - Pinpointing the root cause of latency issues by finding concurrent events Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> --- tools/accounting/delaytop.c | 103 +++++++++++++++++++++++++++++------- 1 file changed, 83 insertions(+), 20 deletions(-) diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c index 015ce8b6917f..2be975cbc093 100644 --- a/tools/accounting/delaytop.c +++ b/tools/accounting/delaytop.c @@ -110,27 +110,35 @@ struct task_info { unsigned long long cpu_count; unsigned long long cpu_delay_total; unsigned long long cpu_delay_max; + struct __kernel_timespec cpu_delay_max_ts; unsigned long long blkio_count; unsigned long long blkio_delay_total; unsigned long long blkio_delay_max; + struct __kernel_timespec blkio_delay_max_ts; unsigned long long swapin_count; unsigned long long swapin_delay_total; unsigned long long swapin_delay_max; + struct __kernel_timespec swapin_delay_max_ts; unsigned long long freepages_count; unsigned long long freepages_delay_total; unsigned long long freepages_delay_max; + struct __kernel_timespec freepages_delay_max_ts; unsigned long long thrashing_count; unsigned long long thrashing_delay_total; unsigned long long thrashing_delay_max; + struct __kernel_timespec thrashing_delay_max_ts; unsigned long long compact_count; unsigned long long compact_delay_total; unsigned long long compact_delay_max; + struct __kernel_timespec compact_delay_max_ts; unsigned long long wpcopy_count; unsigned long long wpcopy_delay_total; unsigned long long wpcopy_delay_max; + struct __kernel_timespec wpcopy_delay_max_ts; unsigned long long irq_count; unsigned long long irq_delay_total; unsigned long long irq_delay_max; + struct __kernel_timespec irq_delay_max_ts; unsigned long long mem_count; unsigned long long mem_delay_total; }; @@ -275,7 +283,7 @@ static void usage(void) " -p, --pid=PID Monitor only the specified PID\n" " -C, --container=PATH Monitor the container at specified cgroup path\n" " -s, --sort=FIELD Sort by delay field (default: cpu)\n" - " -t, --type=FIELD Display only specified delay type with avg/max\n" + " -t, --type=FIELD Display only specified delay type with avg/max/timestamp\n" " -M, --memverbose Display memory detailed information\n"); exit(0); } @@ -736,27 +744,35 @@ static void fetch_and_fill_task_info(int pid, const char *comm) SET_TASK_STAT(task_count, cpu_count); SET_TASK_STAT(task_count, cpu_delay_total); SET_TASK_STAT(task_count, cpu_delay_max); + SET_TASK_STAT(task_count, cpu_delay_max_ts); SET_TASK_STAT(task_count, blkio_count); SET_TASK_STAT(task_count, blkio_delay_total); SET_TASK_STAT(task_count, blkio_delay_max); + SET_TASK_STAT(task_count, blkio_delay_max_ts); SET_TASK_STAT(task_count, swapin_count); SET_TASK_STAT(task_count, swapin_delay_total); SET_TASK_STAT(task_count, swapin_delay_max); + SET_TASK_STAT(task_count, swapin_delay_max_ts); SET_TASK_STAT(task_count, freepages_count); SET_TASK_STAT(task_count, freepages_delay_total); SET_TASK_STAT(task_count, freepages_delay_max); + SET_TASK_STAT(task_count, freepages_delay_max_ts); SET_TASK_STAT(task_count, thrashing_count); SET_TASK_STAT(task_count, thrashing_delay_total); SET_TASK_STAT(task_count, thrashing_delay_max); + SET_TASK_STAT(task_count, thrashing_delay_max_ts); SET_TASK_STAT(task_count, compact_count); SET_TASK_STAT(task_count, compact_delay_total); SET_TASK_STAT(task_count, compact_delay_max); + SET_TASK_STAT(task_count, compact_delay_max_ts); SET_TASK_STAT(task_count, wpcopy_count); SET_TASK_STAT(task_count, wpcopy_delay_total); SET_TASK_STAT(task_count, wpcopy_delay_max); + SET_TASK_STAT(task_count, wpcopy_delay_max_ts); SET_TASK_STAT(task_count, irq_count); SET_TASK_STAT(task_count, irq_delay_total); SET_TASK_STAT(task_count, irq_delay_max); + SET_TASK_STAT(task_count, irq_delay_max_ts); set_mem_count(&tasks[task_count]); set_mem_delay_total(&tasks[task_count]); task_count++; @@ -814,6 +830,40 @@ static double average_ms(unsigned long long total, unsigned long long count) return (double)total / 1000000.0 / count; } +/* + * Format __kernel_timespec to human readable string (YYYY-MM-DD HH:MM:SS) + * Returns formatted string or "N/A" if timestamp is zero + */ +static const char *format_timespec64(struct __kernel_timespec *ts) +{ + static char buffer[32]; + struct tm *tm_info; + + /* Check if timestamp is zero (not set) or invalid (before year 2000) */ + if ((ts->tv_sec == 0 && ts->tv_nsec == 0) || ts->tv_sec < 946684800) { + /* 946684800 is timestamp for 2000-01-01 00:00:00 UTC */ + return "N/A"; + } + + /* Check if timestamp is too large for time_t on 32-bit platforms */ + if (sizeof(time_t) < sizeof(ts->tv_sec) && ts->tv_sec > (time_t)-1) + return "N/A"; + + tm_info = gmtime((const time_t *)&ts->tv_sec); + if (!tm_info) + return "N/A"; + + snprintf(buffer, sizeof(buffer), "%04d-%02d-%02dT%02d:%02d:%02d", + tm_info->tm_year + 1900, + tm_info->tm_mon + 1, + tm_info->tm_mday, + tm_info->tm_hour, + tm_info->tm_min, + tm_info->tm_sec); + + return buffer; +} + /* Comparison function for sorting tasks */ static int compare_tasks(const void *a, const void *b) { @@ -840,13 +890,14 @@ static int compare_tasks(const void *a, const void *b) /* Get delay values for a specific field */ static void get_field_delay_values(const struct task_info *task, const struct field_desc *field, - double *avg_ms, double *max_ms) + double *avg_ms, double *max_ms, struct __kernel_timespec *max_ts) { unsigned long long total, count, max; if (!field) { *avg_ms = 0; *max_ms = 0; + memset(max_ts, 0, sizeof(*max_ts)); return; } @@ -854,26 +905,35 @@ static void get_field_delay_values(const struct task_info *task, const struct fi count = *(unsigned long long *)((char *)task + field->count_offset); *avg_ms = average_ms(total, count); - /* Get max delay based on field name */ - if (strcmp(field->name, "cpu") == 0) + /* Get max delay and timestamp based on field name */ + if (strcmp(field->name, "cpu") == 0) { max = task->cpu_delay_max; - else if (strcmp(field->name, "blkio") == 0) + *max_ts = task->cpu_delay_max_ts; + } else if (strcmp(field->name, "blkio") == 0) { max = task->blkio_delay_max; - else if (strcmp(field->name, "irq") == 0) + *max_ts = task->blkio_delay_max_ts; + } else if (strcmp(field->name, "irq") == 0) { max = task->irq_delay_max; - else if (strcmp(field->name, "swapin") == 0) + *max_ts = task->irq_delay_max_ts; + } else if (strcmp(field->name, "swapin") == 0) { max = task->swapin_delay_max; - else if (strcmp(field->name, "freepages") == 0) + *max_ts = task->swapin_delay_max_ts; + } else if (strcmp(field->name, "freepages") == 0) { max = task->freepages_delay_max; - else if (strcmp(field->name, "thrashing") == 0) + *max_ts = task->freepages_delay_max_ts; + } else if (strcmp(field->name, "thrashing") == 0) { max = task->thrashing_delay_max; - else if (strcmp(field->name, "compact") == 0) + *max_ts = task->thrashing_delay_max_ts; + } else if (strcmp(field->name, "compact") == 0) { max = task->compact_delay_max; - else if (strcmp(field->name, "wpcopy") == 0) + *max_ts = task->compact_delay_max_ts; + } else if (strcmp(field->name, "wpcopy") == 0) { max = task->wpcopy_delay_max; - else + *max_ts = task->wpcopy_delay_max_ts; + } else { max = 0; - + memset(max_ts, 0, sizeof(*max_ts)); + } *max_ms = (double)max / 1000000.0; /* Convert nanoseconds to milliseconds */ } @@ -1034,11 +1094,12 @@ static void display_results(int psi_ret) cfg.max_processes, get_name_by_field(cfg.sort_field)); if (cfg.display_mode == MODE_TYPE && cfg.type_field) { - /* Display mode for -t option: show only specified type with avg/max */ - suc &= BOOL_FPRINT(out, "%8s %8s %-17s %12s %12s\n", + /* Display mode for -t option: show only specified type with avg/max/timestamp */ + suc &= BOOL_FPRINT(out, "%8s %8s %-17s %12s %12s %20s\n", "PID", "TGID", "COMMAND", - "AVG(ms)", "MAX(ms)"); - suc &= BOOL_FPRINT(out, "----------------------------------------------------\n"); + "AVG(ms)", "MAX(ms)", "MAX_TIMESTAMP"); + suc &= BOOL_FPRINT(out, "--------------------------------------------------------"); + suc &= BOOL_FPRINT(out, "----------------------------------------\n"); } else { suc &= BOOL_FPRINT(out, "%8s %8s %-17s", "PID", "TGID", "COMMAND"); if (cfg.display_mode == MODE_MEMVERBOSE) { @@ -1065,11 +1126,13 @@ static void display_results(int psi_ret) tasks[i].pid, tasks[i].tgid, tasks[i].command); if (cfg.display_mode == MODE_TYPE && cfg.type_field) { double avg_ms, max_ms; + struct __kernel_timespec max_ts; - get_field_delay_values(&tasks[i], cfg.type_field, &avg_ms, &max_ms); + get_field_delay_values(&tasks[i], cfg.type_field, &avg_ms, + &max_ms, &max_ts); - suc &= BOOL_FPRINT(out, "%12.2f %12.2f\n", - avg_ms, max_ms); + suc &= BOOL_FPRINT(out, "%12.2f %12.2f %20s\n", + avg_ms, max_ms, format_timespec64(&max_ts)); } else if (cfg.display_mode == MODE_MEMVERBOSE) { suc &= BOOL_FPRINT(out, DELAY_FMT_MEMVERBOSE, TASK_AVG(tasks[i], mem), -- 2.25.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis
@ 2026-05-27 13:55 wang.yaxin
2026-05-27 13:58 ` [PATCH 2/3] delaytop: add timestamp of delay max wang.yaxin
0 siblings, 1 reply; 9+ messages in thread
From: wang.yaxin @ 2026-05-27 13:55 UTC (permalink / raw)
To: akpm, fan.yu9, yang.yang29; +Cc: corbet, linux-kernel, linux-doc, xu.xin16
From: Wang Yaxin <wang.yaxin@zte.com.cn>
Previously delaytop only showed average delays. This patch adds:
1. delay_max fields to track the maximum delay value for each delay type
(cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy)
per task.
2. The -t/--type option displays only the specified delay type with avg/max
values side by side, allowing focused analysis:
delaytop -t cpu # Show only CPU delay with avg/max
delaytop -t wpcopy # Show Copy-on-Write delay with avg/max
3. Wall-clock timestamp when each maximum delay occurred, displayed in the
MAX_TIMESTAMP column when using -t/--type option. This enables:
- Identifying the time when a process experienced an abnormal delay max
- Correlating delay max across multiple processes at the same timestamp
- Cross-referencing with logs, traces, or other metrics at that time
4. When using -t/--type option, tasks are sorted by maximum delay value in
descending order (largest delay first), enabling quick identification of
top N processes with highest delay spikes.
Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn>
Wang Yaxin (3):
delaytop: add delay max for delaytop
delaytop: add timestamp of delay max
delaytop: sort by max delay to highlight top latency processes
Documentation/accounting/delay-accounting.rst | 43 ++++
tools/accounting/delaytop.c | 232 +++++++++++++++---
2 files changed, 247 insertions(+), 28 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 2/3] delaytop: add timestamp of delay max 2026-05-27 13:55 [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin @ 2026-05-27 13:58 ` wang.yaxin 0 siblings, 0 replies; 9+ messages in thread From: wang.yaxin @ 2026-05-27 13:58 UTC (permalink / raw) To: wang.yaxin Cc: akpm, fan.yu9, yang.yang29, corbet, linux-kernel, linux-doc, xu.xin16 From: Wang Yaxin <wang.yaxin@zte.com.cn> Record the wall-clock timestamp when each maximum delay occurred for all delay types. The timestamp is displayed in the MAX_TIMESTAMP column when using -t/--type option. This enables: - Identifying the time when a process experienced an abnormal delay spike - Correlating delay peaks across multiple processes at the same timestamp - Cross-referencing with system logs, traces, or other metrics at that time - Pinpointing the root cause of latency issues by finding concurrent events Signed-off-by: Wang Yaxin <wang.yaxin@zte.com.cn> --- tools/accounting/delaytop.c | 100 ++++++++++++++++++++++++++++-------- 1 file changed, 80 insertions(+), 20 deletions(-) diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c index 586294b29834..8d22c43dceed 100644 --- a/tools/accounting/delaytop.c +++ b/tools/accounting/delaytop.c @@ -110,27 +110,35 @@ struct task_info { unsigned long long cpu_count; unsigned long long cpu_delay_total; unsigned long long cpu_delay_max; + struct __kernel_timespec cpu_delay_max_ts; unsigned long long blkio_count; unsigned long long blkio_delay_total; unsigned long long blkio_delay_max; + struct __kernel_timespec blkio_delay_max_ts; unsigned long long swapin_count; unsigned long long swapin_delay_total; unsigned long long swapin_delay_max; + struct __kernel_timespec swapin_delay_max_ts; unsigned long long freepages_count; unsigned long long freepages_delay_total; unsigned long long freepages_delay_max; + struct __kernel_timespec freepages_delay_max_ts; unsigned long long thrashing_count; unsigned long long thrashing_delay_total; unsigned long long thrashing_delay_max; + struct __kernel_timespec thrashing_delay_max_ts; unsigned long long compact_count; unsigned long long compact_delay_total; unsigned long long compact_delay_max; + struct __kernel_timespec compact_delay_max_ts; unsigned long long wpcopy_count; unsigned long long wpcopy_delay_total; unsigned long long wpcopy_delay_max; + struct __kernel_timespec wpcopy_delay_max_ts; unsigned long long irq_count; unsigned long long irq_delay_total; unsigned long long irq_delay_max; + struct __kernel_timespec irq_delay_max_ts; unsigned long long mem_count; unsigned long long mem_delay_total; }; @@ -275,7 +283,7 @@ static void usage(void) " -p, --pid=PID Monitor only the specified PID\n" " -C, --container=PATH Monitor the container at specified cgroup path\n" " -s, --sort=FIELD Sort by delay field (default: cpu)\n" - " -t, --type=FIELD Display only specified delay type with avg/max\n" + " -t, --type=FIELD Display only specified delay type with avg/max/timestamp\n" " -M, --memverbose Display memory detailed information\n"); exit(0); } @@ -730,27 +738,34 @@ static void fetch_and_fill_task_info(int pid, const char *comm) SET_TASK_STAT(task_count, cpu_count); SET_TASK_STAT(task_count, cpu_delay_total); SET_TASK_STAT(task_count, cpu_delay_max); + SET_TASK_STAT(task_count, cpu_delay_max_ts); SET_TASK_STAT(task_count, blkio_count); SET_TASK_STAT(task_count, blkio_delay_total); SET_TASK_STAT(task_count, blkio_delay_max); + SET_TASK_STAT(task_count, blkio_delay_max_ts); SET_TASK_STAT(task_count, swapin_count); SET_TASK_STAT(task_count, swapin_delay_total); SET_TASK_STAT(task_count, swapin_delay_max); + SET_TASK_STAT(task_count, swapin_delay_max_ts); SET_TASK_STAT(task_count, freepages_count); SET_TASK_STAT(task_count, freepages_delay_total); SET_TASK_STAT(task_count, freepages_delay_max); SET_TASK_STAT(task_count, thrashing_count); SET_TASK_STAT(task_count, thrashing_delay_total); SET_TASK_STAT(task_count, thrashing_delay_max); + SET_TASK_STAT(task_count, thrashing_delay_max_ts); SET_TASK_STAT(task_count, compact_count); SET_TASK_STAT(task_count, compact_delay_total); SET_TASK_STAT(task_count, compact_delay_max); + SET_TASK_STAT(task_count, compact_delay_max_ts); SET_TASK_STAT(task_count, wpcopy_count); SET_TASK_STAT(task_count, wpcopy_delay_total); SET_TASK_STAT(task_count, wpcopy_delay_max); + SET_TASK_STAT(task_count, wpcopy_delay_max_ts); SET_TASK_STAT(task_count, irq_count); SET_TASK_STAT(task_count, irq_delay_total); SET_TASK_STAT(task_count, irq_delay_max); + SET_TASK_STAT(task_count, irq_delay_max_ts); set_mem_count(&tasks[task_count]); set_mem_delay_total(&tasks[task_count]); task_count++; @@ -808,6 +823,38 @@ static double average_ms(unsigned long long total, unsigned long long count) return (double)total / 1000000.0 / count; } +/* + * Format __kernel_timespec to human readable string (YYYY-MM-DD HH:MM:SS) + * Returns formatted string or "N/A" if timestamp is zero + */ +static const char *format_timespec64(struct __kernel_timespec *ts) +{ + static char buffer[32]; + struct tm *tm_info; + time_t time_sec; + + /* Check if timestamp is zero (not set) or invalid (before year 2000) */ + if ((ts->tv_sec == 0 && ts->tv_nsec == 0) || ts->tv_sec < 946684800) { + /* 946684800 is timestamp for 2000-01-01 00:00:00 UTC */ + return "N/A"; + } + + time_sec = (time_t)ts->tv_sec; + tm_info = localtime(&time_sec); + if (!tm_info) + return "N/A"; + + snprintf(buffer, sizeof(buffer), "%04d-%02d-%02d %02d:%02d:%02d", + tm_info->tm_year + 1900, + tm_info->tm_mon + 1, + tm_info->tm_mday, + tm_info->tm_hour, + tm_info->tm_min, + tm_info->tm_sec); + + return buffer; +} + /* Comparison function for sorting tasks */ static int compare_tasks(const void *a, const void *b) { @@ -834,13 +881,14 @@ static int compare_tasks(const void *a, const void *b) /* Get delay values for a specific field */ static void get_field_delay_values(const struct task_info *task, const struct field_desc *field, - double *avg_ms, double *max_ms) + double *avg_ms, double *max_ms, struct __kernel_timespec *max_ts) { unsigned long long total, count, max; if (!field) { *avg_ms = 0; *max_ms = 0; + memset(max_ts, 0, sizeof(*max_ts)); return; } @@ -848,26 +896,35 @@ static void get_field_delay_values(const struct task_info *task, const struct fi count = *(unsigned long *)((char *)task + field->count_offset); *avg_ms = average_ms(total, count); - /* Get max delay based on field name */ - if (strcmp(field->name, "cpu") == 0) + /* Get max delay and timestamp based on field name */ + if (strcmp(field->name, "cpu") == 0) { max = task->cpu_delay_max; - else if (strcmp(field->name, "blkio") == 0) + *max_ts = task->cpu_delay_max_ts; + } else if (strcmp(field->name, "blkio") == 0) { max = task->blkio_delay_max; - else if (strcmp(field->name, "irq") == 0) + *max_ts = task->blkio_delay_max_ts; + } else if (strcmp(field->name, "irq") == 0) { max = task->irq_delay_max; - else if (strcmp(field->name, "swapin") == 0) + *max_ts = task->irq_delay_max_ts; + } else if (strcmp(field->name, "swapin") == 0) { max = task->swapin_delay_max; - else if (strcmp(field->name, "freepages") == 0) + *max_ts = task->swapin_delay_max_ts; + } else if (strcmp(field->name, "freepages") == 0) { max = task->freepages_delay_max; - else if (strcmp(field->name, "thrashing") == 0) + *max_ts = task->freepages_delay_max_ts; + } else if (strcmp(field->name, "thrashing") == 0) { max = task->thrashing_delay_max; - else if (strcmp(field->name, "compact") == 0) + *max_ts = task->thrashing_delay_max_ts; + } else if (strcmp(field->name, "compact") == 0) { max = task->compact_delay_max; - else if (strcmp(field->name, "wpcopy") == 0) + *max_ts = task->compact_delay_max_ts; + } else if (strcmp(field->name, "wpcopy") == 0) { max = task->wpcopy_delay_max; - else + *max_ts = task->wpcopy_delay_max_ts; + } else { max = 0; - + memset(max_ts, 0, sizeof(*max_ts)); + } *max_ms = (double)max / 1000000.0; /* Convert nanoseconds to milliseconds */ } @@ -1028,11 +1085,12 @@ static void display_results(int psi_ret) cfg.max_processes, get_name_by_field(cfg.sort_field)); if (cfg.display_mode == MODE_TYPE && cfg.type_field) { - /* Display mode for -t option: show only specified type with avg/max */ - suc &= BOOL_FPRINT(out, "%8s %8s %-17s %12s %12s\n", + /* Display mode for -t option: show only specified type with avg/max/timestamp */ + suc &= BOOL_FPRINT(out, "%8s %8s %-17s %12s %12s %20s\n", "PID", "TGID", "COMMAND", - "AVG(ms)", "MAX(ms)"); - suc &= BOOL_FPRINT(out, "----------------------------------------------------\n"); + "AVG(ms)", "MAX(ms)", "MAX_TIMESTAMP"); + suc &= BOOL_FPRINT(out, "--------------------------------------------------------"); + suc &= BOOL_FPRINT(out, "----------------------------------------\n"); } else { suc &= BOOL_FPRINT(out, "%8s %8s %-17s", "PID", "TGID", "COMMAND"); if (cfg.display_mode == MODE_MEMVERBOSE) { @@ -1059,11 +1117,13 @@ static void display_results(int psi_ret) tasks[i].pid, tasks[i].tgid, tasks[i].command); if (cfg.display_mode == MODE_TYPE && cfg.type_field) { double avg_ms, max_ms; + struct __kernel_timespec max_ts; - get_field_delay_values(&tasks[i], cfg.type_field, &avg_ms, &max_ms); + get_field_delay_values(&tasks[i], cfg.type_field, &avg_ms, + &max_ms, &max_ts); - suc &= BOOL_FPRINT(out, "%12.2f %12.2f\n", - avg_ms, max_ms); + suc &= BOOL_FPRINT(out, "%12.2f %12.2f %20s\n", + avg_ms, max_ms, format_timespec64(&max_ts)); } else if (cfg.display_mode == MODE_MEMVERBOSE) { suc &= BOOL_FPRINT(out, DELAY_FMT_MEMVERBOSE, TASK_AVG(tasks[i], mem), -- 2.25.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-07-02 23:01 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-07-02 12:57 [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin 2026-07-02 12:58 ` [PATCH 1/3] delaytop: add delay max for delaytop wang.yaxin 2026-07-02 22:58 ` Andrew Morton 2026-07-02 12:58 ` [PATCH 2/3] delaytop: add timestamp of delay max wang.yaxin 2026-07-02 23:01 ` Andrew Morton 2026-07-02 13:00 ` [PATCH 3/3] delaytop: sort by max delay to highlight top latency processes wang.yaxin 2026-07-02 23:01 ` [PATCH 0/3 v3] delaytop: add delay max, timestamp and sorting for top latency analysis Andrew Morton -- strict thread matches above, loose matches on Subject: below -- 2026-05-30 3:16 [PATCH 0/3 v2] " wang.yaxin 2026-05-30 3:18 ` [PATCH 2/3] delaytop: add timestamp of delay max wang.yaxin 2026-05-27 13:55 [PATCH 0/3] delaytop: add delay max, timestamp and sorting for top latency analysis wang.yaxin 2026-05-27 13:58 ` [PATCH 2/3] delaytop: add timestamp of delay max wang.yaxin
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox