From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793408777; b=tuQj+Vm8jf5+z1iRpDUF47S+6MX9eGedfvjk31WL5LDgWczkjbJxO47fCI0oijqNDGXB KVmLq6zl5AU74a0K3dnOrdAnfcvROXpZeTuyHKDsuvnLp5DYaT4d6pKz4Gt7+3IC6ZiDW +gUulExIhCe0jm99tZxmMPvx2J5poZ75/jCE26RDtmBKQ5gV+OUiEMWygHcJEYhGkx3ZH Y1CP7ObD3mKZuK6GX28+SmpqNu9IMu0NBOrXRiWcaenIs73D1ZMn4bVgN46+MzHMv14WJ MbqkrBYSYXabJtg9d75x2L1rVdanOOWqqZxxemBdOXi8xmKXLKBxWF4fhRre3Es7wBQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793408777; h=Received:DKIM-Signature:Date:To:From:Subject:Message-Id; bh=PBH/Lhkc65X9rFjrOpqJlubdkgjF1lt41OZDDmdh2m0=; b=FRXINJU4POd82Oi/RzYHGjzbh1O6XsknqC5rGjarDM38TQVL+GvhNiRJNOuP+gAE1xC1 OyYvH3j+yjUpw85XlBfrhlqvobo6Qr5JZtmoMMtc6fzD3PLiOel7dxenKxi+DxuhKOxrB yTtEAghIG9vgPF0fWZboht1831w2EzfXZzyrpECjlBMO0ydA8VwDEv1UgO16lLkQoWwnj 3Lw/IRwrYQnxTCuzXxFlzCrO7eSqv4b3nZ7qJYxHi0jlTkZFDZPs62abyqGWCbkq2qyYQ aiUFoF9FSz6WS/h2oFJJnFdsK2KiKt6aNh08Fhj+GbDGPCEueGY1Rn3geSpZY3HVIbQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=linux-foundation.org header.i=@linux-foundation.org header.a=rsa-sha256 header.s=korg header.b=D3E7SVlU; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46107606A31E for ; Sat, 31 Oct 2026 01:06:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key, unprotected) header.d=linux-foundation.org header.i=@linux-foundation.org header.a=rsa-sha256 header.s=korg header.b=D3E7SVlU Received: by smtp.kernel.org (Postfix) with ESMTPSA id 013521F000C1; Sat, 31 Oct 2026 01:06:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=korg; t=1793408777; bh=PBH/Lhkc65X9rFjrOpqJlubdkgjF1lt41OZDDmdh2m0=; h=Date:To:From:Subject; b=D3E7SVlU/gAojDkaHzOmCMprdKdXMzrqrgW5FWLDBRTGnhddmLF4Ocg2MY+Og71ok gDTjkqh4h384ayyiu8yLiZ2fsvFGpbYl2ogMT+Me89Th99NX2O/zZ0xAxoI2b/5RsQ 8gN/rwEMzMbxQXSEdE3rL20ItgX0/p/3f+P3WAmg= Date: Fri, 30 Oct 2026 18:06:16 -0700 To: mm-commits@vger.kernel.org,yang.yang29@zte.com.cn,xu.xin@linux.dev, thomas.weissschuh@linutronix.de,fan.yu9@zte.com.cn,corbet@lwn.net, wang.yaxin@zte.com.cn,akpm@linux-foundation.org From: Andrew Morton Subject: + delaytop-sort-by-max-delay-to-highlight-top-latency-processes.patch added to mm-nonmm-unstable branch Message-Id: <20261031010617.013521F000C1@smtp.kernel.org> X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list The patch titled Subject: delaytop: sort by max delay to highlight top latency processes has been added to the -mm mm-nonmm-unstable branch. Its filename is delaytop-sort-by-max-delay-to-highlight-top-latency-processes.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/delaytop-sort-by-max-delay-to-highlight-top-latency-processes.patch This patch will later appear in the mm-nonmm-unstable branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via various branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there most days ------------------------------------------------------ From: Wang Yaxin Subject: delaytop: sort by max delay to highlight top latency processes Date: Wed, 23 Sep 2026 21:05:42 +0800 When using -t/--type option, sort tasks by the maximum delay value of the selected type in descending order (largest delay first). This enables quickly identifying the top N processes with the highest delay spikes, which is essential for diagnosing latency problems by pinpointing which processes contributed most to system delays. Link: https://lore.kernel.org/20260923210542517bP5gb_zEY1jsriLQHagUT@zte.com.cn Signed-off-by: Wang Yaxin Signed-off-by: Andrew Morton Cc: Fan Yu Cc: Yang Yang Cc: Thomas Weißschuh Cc: Jonathan Corbet Cc: Xu Xin --- Documentation/accounting/delay-accounting.rst | 87 +++++++++++----- tools/accounting/delaytop.c | 71 +++++++++++-- tools/accounting/getdelays.c | 8 + 3 files changed, 133 insertions(+), 33 deletions(-) --- a/Documentation/accounting/delay-accounting.rst~delaytop-sort-by-max-delay-to-highlight-top-latency-processes +++ a/Documentation/accounting/delay-accounting.rst @@ -152,28 +152,28 @@ Basic usage with default settings (sorts IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms) [o]sort [M]memverbose [q]quit Top 20 processes (sorted by cpu delay): - PID TGID COMMAND CPU(ms) IO(ms) IRQ(ms) MEM(ms) + PID TGID COMMAND CPU(ms) IO(ms) IRQ(ms) MEM(ms) ------------------------------------------------------------------------ - 110 110 kworker/15:0H-s 27.91 0.00 0.00 0.00 - 57 57 cpuhp/7 3.18 0.00 0.00 0.00 - 99 99 cpuhp/14 2.97 0.00 0.00 0.00 - 51 51 cpuhp/6 0.90 0.00 0.00 0.00 - 44 44 kworker/4:0H-sy 0.80 0.00 0.00 0.00 - 60 60 ksoftirqd/7 0.74 0.00 0.00 0.00 - 76 76 idle_inject/10 0.31 0.00 0.00 0.00 - 100 100 idle_inject/14 0.30 0.00 0.00 0.00 - 1309 1309 systemsettings 0.29 0.00 0.00 0.00 - 45 45 cpuhp/5 0.22 0.00 0.00 0.00 - 63 63 cpuhp/8 0.20 0.00 0.00 0.00 - 87 87 cpuhp/12 0.18 0.00 0.00 0.00 - 93 93 cpuhp/13 0.17 0.00 0.00 0.00 - 1265 1265 acpid 0.17 0.00 0.00 0.00 - 1552 1552 sshd 0.17 0.00 0.00 0.00 - 2584 2584 sddm-helper 0.16 0.00 0.00 0.00 - 1284 1284 rtkit-daemon 0.15 0.00 0.00 0.00 - 1326 1326 nde-netfilter 0.14 0.00 0.00 0.00 - 27 27 cpuhp/2 0.13 0.00 0.00 0.00 - 631 631 kworker/11:2-rc 0.11 0.00 0.00 0.00 + 110 110 kworker/15:0H-s 27.91 0.00 0.00 0.00 + 57 57 cpuhp/7 3.18 0.00 0.00 0.00 + 99 99 cpuhp/14 2.97 0.00 0.00 0.00 + 51 51 cpuhp/6 0.90 0.00 0.00 0.00 + 44 44 kworker/4:0H-sy 0.80 0.00 0.00 0.00 + 60 60 ksoftirqd/7 0.74 0.00 0.00 0.00 + 76 76 idle_inject/10 0.31 0.00 0.00 0.00 + 100 100 idle_inject/14 0.30 0.00 0.00 0.00 + 1309 1309 systemsettings 0.29 0.00 0.00 0.00 + 45 45 cpuhp/5 0.22 0.00 0.00 0.00 + 63 63 cpuhp/8 0.20 0.00 0.00 0.00 + 87 87 cpuhp/12 0.18 0.00 0.00 0.00 + 93 93 cpuhp/13 0.17 0.00 0.00 0.00 + 1265 1265 acpid 0.17 0.00 0.00 0.00 + 1552 1552 sshd 0.17 0.00 0.00 0.00 + 2584 2584 sddm-helper 0.16 0.00 0.00 0.00 + 1284 1284 rtkit-daemon 0.15 0.00 0.00 0.00 + 1326 1326 nde-netfilter 0.14 0.00 0.00 0.00 + 27 27 cpuhp/2 0.13 0.00 0.00 0.00 + 631 631 kworker/11:2-rc 0.11 0.00 0.00 0.00 Interactive keyboard controls during runtime:: @@ -181,7 +181,7 @@ Interactive keyboard controls during run M - Toggle display mode (Default/Memory Verbose) q - Quit -Available sort fields(use -s/--sort or interactive command):: +Available sort fields (use -s/--sort or interactive command):: cpu(c) - CPU delay blkio(i) - I/O delay @@ -212,3 +212,46 @@ Advanced usage examples:: # ./delaytop -d secs Specify refresh interval as secs + + # ./delaytop -t type + Display only specified delay type with avg/max/timestamp + (rows sorted by MAX for that type, largest first) + + + +delaytop add delay_max fields to track the maximum delay value for each delay +type (cpu, blkio, irq, swapin, freepages, thrashing, compact, wpcopy) per task:: + + bash# ./delaytop -t cpu + System Pressure Information: (avg10/avg60/avg300/total) + CPU some: 0.4%/ 0.2%/ 0.1%/ 220(ms) + CPU full: 0.0%/ 0.0%/ 0.0%/ 0(ms) + Memory full: 0.0%/ 0.0%/ 0.0%/ 0(ms) + Memory some: 0.0%/ 0.0%/ 0.0%/ 0(ms) + IO full: 0.0%/ 0.0%/ 0.0%/ 12(ms) + IO some: 0.0%/ 0.0%/ 0.0%/ 13(ms) + IRQ full: 0.0%/ 0.0%/ 0.0%/ 0(ms) + [q]quit + Top 20 processes (sorted by cpu MAX delay, largest first): + PID TGID COMMAND AVG(ms) MAX(ms) MAX_TIMESTAMP + ------------------------------------------------------------------------ + 9 9 kworker/0:0-eve 0.59 16.87 2026-05-27T13:32:39 + 30 30 kworker/2:0H-kb 2.87 11.36 2026-05-27T13:32:36 + 27 27 migration/2 1.05 9.51 2026-05-27T13:32:37 + 50 50 kworker/2:1-eve 0.50 9.13 2026-05-27T13:32:37 + 15 15 rcu_preempt 0.11 8.98 2026-05-27T13:32:37 + 1 1 init 0.17 7.12 2026-05-27T13:32:38 + 67 67 scsi_eh_0 1.20 4.23 2026-05-27T13:32:37 + 23 23 ksoftirqd/1 1.12 3.77 2026-05-27T13:32:36 + 3 3 pool_workqueue 0.72 3.55 2026-05-27T13:32:38 + 62 62 kworker/u20:2-a 0.49 3.03 2026-05-27T13:32:37 + 2 2 kthreadd 0.18 2.82 2026-05-27T13:32:37 + 11 11 kworker/0:1 1.42 2.76 2026-05-27T13:32:36 + 39 39 kworker/u20:0-a 0.10 2.71 2026-05-27T13:32:38 + 17 17 rcu_exp_gp_kthr 0.25 2.65 2026-05-27T13:32:36 + 66 66 kworker/u20:3-e 0.38 2.55 2026-05-27T13:32:37 + 20 20 cpuhp/0 0.53 2.51 2026-05-27T13:32:38 + 28 28 ksoftirqd/2 0.59 2.48 2026-05-27T13:32:36 + 55 55 kworker/u19:1 0.88 2.42 2026-05-27T13:32:36 + 13 13 kworker/R-mm_pe 1.18 2.35 2026-05-27T13:32:36 + 54 54 kworker/3:1-eve 0.14 2.20 2026-05-27T13:32:38 --- a/tools/accounting/delaytop.c~delaytop-sort-by-max-delay-to-highlight-top-latency-processes +++ a/tools/accounting/delaytop.c @@ -295,6 +295,7 @@ static void usage(void) " -C, --container=PATH Monitor the container at specified cgroup path\n" " -s, --sort=FIELD Sort by delay field (default: cpu)\n" " -t, --type=FIELD Display only specified delay type with avg/max/timestamp\n" + " (rows sorted by MAX for that type, largest first)\n" " -M, --memverbose Display memory detailed information\n"); exit(0); } @@ -838,6 +839,15 @@ static void get_task_delays(void) closedir(dir); } +static void field_delay_max_and_ts(const struct task_info *task, + const struct field_desc *field, + unsigned long long *max_ns, + struct __kernel_timespec *max_ts); +static void get_field_delay_values(const struct task_info *task, + const struct field_desc *field, + double *avg_ms, double *max_ms, + struct __kernel_timespec *max_ts); + /* Calculate average delay in milliseconds */ static double average_ms(unsigned long long total, unsigned long long count) { @@ -850,7 +860,7 @@ static double average_ms(unsigned long l * Format __kernel_timespec to human readable string (YYYY-MM-DDTHH:MM:SS) * Returns formatted string or "N/A" if timestamp is zero */ -static const char *format_timespec64(struct __kernel_timespec *ts) +static const char *format_kernel_timespec(struct __kernel_timespec *ts) { static char buffer[32]; time_t time_sec; @@ -894,6 +904,16 @@ static int compare_tasks(const void *a, unsigned long long count1; unsigned long long count2; double avg1, avg2; + unsigned long long max1, max2; + + /* -t/--type: default sort by MAX column for the selected type (descending) */ + if (cfg.display_mode == MODE_TYPE && cfg.type_field) { + field_delay_max_and_ts(t1, cfg.type_field, &max1, NULL); + field_delay_max_and_ts(t2, cfg.type_field, &max2, NULL); + if (max1 != max2) + return max2 > max1 ? 1 : -1; + return 0; + } total1 = *(unsigned long long *)((char *)t1 + cfg.sort_field->total_offset); total2 = *(unsigned long long *)((char *)t2 + cfg.sort_field->total_offset); @@ -908,6 +928,28 @@ static int compare_tasks(const void *a, return 0; } +/* Max delay (ns) and timestamp for field (shared by display and sort) */ +static void field_delay_max_and_ts(const struct task_info *task, const struct field_desc *field, + unsigned long long *max_ns, struct __kernel_timespec *max_ts) +{ + if (!field || !field->max_offset) { + *max_ns = 0; + if (max_ts) + memset(max_ts, 0, sizeof(*max_ts)); + return; + } + + *max_ns = *(unsigned long long *)((char *)task + field->max_offset); + + if (max_ts) { + if (field->max_ts_offset) + *max_ts = *(struct __kernel_timespec *)((char *)task + + field->max_ts_offset); + else + memset(max_ts, 0, sizeof(*max_ts)); + } +} + /* Get delay values for a specific field */ static void get_field_delay_values(const struct task_info *task, const struct field_desc *field, double *avg_ms, double *max_ms, @@ -926,13 +968,8 @@ static void get_field_delay_values(const count = *(unsigned long long *)((char *)task + field->count_offset); *avg_ms = average_ms(total, count); - max = *(unsigned long long *)((char *)task + field->max_offset); + field_delay_max_and_ts(task, field, &max, max_ts); *max_ms = (double)max / 1000000.0; /* Convert nanoseconds to milliseconds */ - - if (field->max_ts_offset) - *max_ts = *(struct __kernel_timespec *)((char *)task + field->max_ts_offset); - else - memset(max_ts, 0, sizeof(*max_ts)); } /* Sort tasks by selected field */ @@ -1082,7 +1119,10 @@ static void display_results(int psi_ret) } /* Interacive command */ - suc &= BOOL_FPRINT(out, "[o]sort [M]memverbose [q]quit\n"); + if (cfg.display_mode == MODE_TYPE && cfg.type_field) + suc &= BOOL_FPRINT(out, "[q]quit\n"); + else + suc &= BOOL_FPRINT(out, "[o]sort [M]memverbose [q]quit\n"); if (sort_selected) { if (cfg.display_mode == MODE_MEMVERBOSE) suc &= BOOL_FPRINT(out, @@ -1093,8 +1133,13 @@ static void display_results(int psi_ret) } /* Task delay output */ - suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n", - cfg.max_processes, get_name_by_field(cfg.sort_field)); + if (cfg.display_mode == MODE_TYPE && cfg.type_field) + suc &= BOOL_FPRINT(out, + "Top %d processes (sorted by %s MAX delay, largest first):\n", + cfg.max_processes, get_name_by_field(cfg.type_field)); + else + suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n", + cfg.max_processes, get_name_by_field(cfg.sort_field)); if (cfg.display_mode == MODE_TYPE && cfg.type_field) { /* Display mode for -t option: show only specified type with avg/max/timestamp */ @@ -1135,7 +1180,7 @@ static void display_results(int psi_ret) &max_ms, &max_ts); suc &= BOOL_FPRINT(out, " %12.2f %12.2f %20s\n", - avg_ms, max_ms, format_timespec64(&max_ts)); + avg_ms, max_ms, format_kernel_timespec(&max_ts)); } else if (cfg.display_mode == MODE_MEMVERBOSE) { suc &= BOOL_FPRINT(out, DELAY_FMT_MEMVERBOSE, TASK_AVG(tasks[i], mem), @@ -1204,9 +1249,13 @@ static void handle_keypress(char ch, int } else { switch (ch) { case 'o': + if (cfg.display_mode == MODE_TYPE) + break; sort_selected = 1; break; case 'M': + if (cfg.display_mode == MODE_TYPE) + break; toggle_display_mode(); for (field = sort_fields; field->name != NULL; field++) { if (field->supported_modes & cfg.display_mode) { --- a/tools/accounting/getdelays.c~delaytop-sort-by-max-delay-to-highlight-top-latency-processes +++ a/tools/accounting/getdelays.c @@ -235,6 +235,14 @@ static const char *format_timespec(struc if (ts->tv_sec == 0 && ts->tv_nsec == 0) return "N/A"; + /* Avoid Y2038 truncation on 32-bit platforms */ + if (sizeof(time_sec) < sizeof(ts->tv_sec)) { + __s64 max = (__s64)((1ULL << (sizeof(time_sec) * 8 - 1)) - 1); + + if (ts->tv_sec > max || ts->tv_sec < -max - 1) + return "N/A"; + } + time_sec = ts->tv_sec; /* Use thread-safe localtime_r */ _ Patches currently in -mm which might be from wang.yaxin@zte.com.cn are delaytop-add-delay-max-for-delaytop.patch delaytop-add-timestamp-of-delay-max.patch delaytop-sort-by-max-delay-to-highlight-top-latency-processes.patch