[PATCH linux-next 0/3] tools/delaytop: implement real-time keyboard interaction support

[PATCH linux-next 0/3] tools/delaytop: implement real-time keyboard interaction support

[fan.yu9]

From: Fan Yu <fan.yu9@zte.com.cn>

Current Limitations
===================
The current delaytop implementation has two main limitations:
1) Static sorting only by CPU delay
Forcing users to restart with different parameters to analyze
other resource bottlenecks.
2) Memory delay information is always expanded
Causing information overload when only high-level memory pressure
monitoring is needed.

Improvements
============
1) Implemented dynamic sorting capability
- Interactive key 'o' triggers sort mode.
- Supports sorting by CPU/IO/Memory/IRQ delays.
- Memory subcategories available in verbose mode.
 * c - CPU delay (default)
 * i - IO delay
 * m - Total memory delay
 * q - IRQ delay
 * s/r/t/p/w - Memory subcategories (in verbose mode)
2) Added memory display modes
- Compact view (default): shows aggregated memory delays.
- Verbose view ('M' key): breaks down into memory sub-delays.
 * SWAP - swapin delays
 * RCL - freepages reclaim delays
 * THR - thrashing delays
 * CMP - compaction delays
 * WP - write-protect copy delays

Practical benefits
==================
1) Dynamic Sorting for Real-Time Bottleneck Detection
System administrators can now dynamically change sorting to identify
different types of resource bottlenecks without restarting.

2) Enhanced Usability with On-Screen Keybindings
More intuitive interactive usage with on-screen keybindings help.
Reduced screen clutter when only memory overview is needed.

Fan Yu (3):
  tools/delaytop: add memory verbose mode support
  tools/delaytop: add flexible sorting by delay field
  tools/delaytop: add interactive mode with keyboard controls

 tools/accounting/delaytop.c | 347 +++++++++++++++++++++++++++++-------
 1 file changed, 287 insertions(+), 60 deletions(-)

-- 
2.25.1

[PATCH linux-next 1/3] tools/delaytop: add memory verbose mode support

[fan.yu9]

From: Fan Yu <fan.yu9@zte.com.cn>

The original delaytop tool always displayed detailed memory
subsystem breakdown, which could be overwhelming for users
who only need high-level overview.

Add flexible display control allowing users to choose their
preferred information granularity.

The new flexibility provides:
1) For quick monitoring: use normal mode to reduce visual clutter
2) For deep analysis: use verbose mode to see all memory subsystem details

Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
---
 tools/accounting/delaytop.c | 111 ++++++++++++++++++++++++++++--------
 1 file changed, 87 insertions(+), 24 deletions(-)

diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c
index 9afb1ffc00ba..f1e2e1cca4b8 100644
--- a/tools/accounting/delaytop.c
+++ b/tools/accounting/delaytop.c
@@ -68,6 +68,8 @@
 	ret >= 0; \
 })
 #define PSI_LINE_FORMAT "%-12s %6.1f%%/%6.1f%%/%6.1f%%/%8llu(ms)\n"
+#define FMT_NORMAL "%8.2f %8.2f %8.2f %8.2f\n"
+#define FMT_MEMVERBOSE "%8.2f %8.2f %8.2f %8.2f %8.2f %8.2f %8.2f %8.2f %8.2f\n"

 /* Program settings structure */
 struct config {
@@ -78,6 +80,7 @@ struct config {
 	int output_one_time;	/* Output once and exit */
 	int monitor_pid;		/* Monitor specific PID */
 	char *container_path;	/* Path to container cgroup */
+	int mem_verbose_mode;	/* Memory detailed display mode */
 };

 /* PSI statistics structure */
@@ -163,13 +166,14 @@ static void usage(void)
 {
 	printf("Usage: delaytop [Options]\n"
 	"Options:\n"
-	"  -h, --help				Show this help message and exit\n"
-	"  -d, --delay=SECONDS	  Set refresh interval (default: 2 seconds, min: 1)\n"
-	"  -n, --iterations=COUNT	Set number of updates (default: 0 = infinite)\n"
-	"  -P, --processes=NUMBER	Set maximum number of processes to show (default: 20, max: 1000)\n"
-	"  -o, --once				Display once and exit\n"
-	"  -p, --pid=PID			Monitor only the specified PID\n"
-	"  -C, --container=PATH	 Monitor the container at specified cgroup path\n");
+	"  -h, --help               Show this help message and exit\n"
+	"  -d, --delay=SECONDS      Set refresh interval (default: 2 seconds, min: 1)\n"
+	"  -n, --iterations=COUNT   Set number of updates (default: 0 = infinite)\n"
+	"  -P, --processes=NUMBER   Set maximum number of processes to show (default: 20, max: 1000)\n"
+	"  -o, --once               Display once and exit\n"
+	"  -p, --pid=PID            Monitor only the specified PID\n"
+	"  -C, --container=PATH     Monitor the container at specified cgroup path\n"
+	"  -M, --memverbose         Display memory detailed information\n");
 	exit(0);
 }

@@ -185,6 +189,7 @@ static void parse_args(int argc, char **argv)
 		{"once", no_argument, 0, 'o'},
 		{"processes", required_argument, 0, 'P'},
 		{"container", required_argument, 0, 'C'},
+		{"memverbose", no_argument, 0, 'M'},
 		{0, 0, 0, 0}
 	};

@@ -196,11 +201,12 @@ static void parse_args(int argc, char **argv)
 	cfg.output_one_time = 0;
 	cfg.monitor_pid = 0;	/* 0 means monitor all PIDs */
 	cfg.container_path = NULL;
+	cfg.mem_verbose_mode = 0;

 	while (1) {
 		int option_index = 0;

-		c = getopt_long(argc, argv, "hd:n:p:oP:C:", long_options, &option_index);
+		c = getopt_long(argc, argv, "hd:n:p:oP:C:M", long_options, &option_index);
 		if (c == -1)
 			break;

@@ -247,6 +253,9 @@ static void parse_args(int argc, char **argv)
 		case 'C':
 			cfg.container_path = strdup(optarg);
 			break;
+		case 'M':
+			cfg.mem_verbose_mode = 1;
+			break;
 		default:
 			fprintf(stderr, "Try 'delaytop --help' for more information.\n");
 			exit(1);
@@ -582,6 +591,25 @@ static double average_ms(unsigned long long total, unsigned long long count)
 	return (double)total / 1000000.0 / count;
 }

+/* Calculate average delay in milliseconds for memory */
+static unsigned long long task_total_mem_delay(const struct task_info *t)
+{
+	return t->swapin_delay_total +
+		t->freepages_delay_total +
+		t->thrashing_delay_total +
+		t->compact_delay_total +
+		t->wpcopy_delay_total;
+}
+
+static unsigned long long task_total_mem_count(const struct task_info *t)
+{
+	return t->swapin_count +
+		t->freepages_count +
+		t->thrashing_count +
+		t->compact_count +
+		t->wpcopy_count;
+}
+
 /* Comparison function for sorting tasks */
 static int compare_tasks(const void *a, const void *b)
 {
@@ -740,27 +768,62 @@ static void display_results(void)
 	}
 	suc &= BOOL_FPRINT(out, "Top %d processes (sorted by CPU delay):\n",
 			cfg.max_processes);
-	suc &= BOOL_FPRINT(out, "%5s  %5s  %-17s", "PID", "TGID", "COMMAND");
-	suc &= BOOL_FPRINT(out, "%7s %7s %7s %7s %7s %7s %7s %7s\n",
-		"CPU(ms)", "IO(ms)", "SWAP(ms)", "RCL(ms)",
-		"THR(ms)", "CMP(ms)", "WP(ms)", "IRQ(ms)");
+	suc &= BOOL_FPRINT(out, "%8s  %8s  %-17s", "PID", "TGID", "COMMAND");
+
+	if (!cfg.mem_verbose_mode) {
+		suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s\n",
+			"CPU(ms)", "IO(ms)", "IRQ(ms)", "MEM(ms)");
+		suc &= BOOL_FPRINT(out, "-----------------------");
+		suc &= BOOL_FPRINT(out, "-----------------------");
+		suc &= BOOL_FPRINT(out, "--------------------------\n");
+	} else {
+		suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s %8s %8s %8s %8s %8s\n",
+			"CPU(ms)", "IO(ms)", "IRQ(ms)", "MEM(ms)",
+			"SWAP(ms)", "RCL(ms)", "THR(ms)", "CMP(ms)", "WP(ms)");
+		suc &= BOOL_FPRINT(out, "-----------------------");
+		suc &= BOOL_FPRINT(out, "-----------------------");
+		suc &= BOOL_FPRINT(out, "-----------------------");
+		suc &= BOOL_FPRINT(out, "-----------------------");
+		suc &= BOOL_FPRINT(out, "-------------------------\n");
+	}
+

-	suc &= BOOL_FPRINT(out, "-----------------------------------------------");
-	suc &= BOOL_FPRINT(out, "----------------------------------------------\n");
 	count = task_count < cfg.max_processes ? task_count : cfg.max_processes;

 	for (i = 0; i < count; i++) {
-		suc &= BOOL_FPRINT(out, "%5d  %5d  %-15s",
+		suc &= BOOL_FPRINT(out, "%8d  %8d  %-15s",
 			tasks[i].pid, tasks[i].tgid, tasks[i].command);
-		suc &= BOOL_FPRINT(out, "%7.2f %7.2f %7.2f %7.2f %7.2f %7.2f %7.2f %7.2f\n",
-			average_ms(tasks[i].cpu_delay_total, tasks[i].cpu_count),
-			average_ms(tasks[i].blkio_delay_total, tasks[i].blkio_count),
-			average_ms(tasks[i].swapin_delay_total, tasks[i].swapin_count),
-			average_ms(tasks[i].freepages_delay_total, tasks[i].freepages_count),
-			average_ms(tasks[i].thrashing_delay_total, tasks[i].thrashing_count),
-			average_ms(tasks[i].compact_delay_total, tasks[i].compact_count),
-			average_ms(tasks[i].wpcopy_delay_total, tasks[i].wpcopy_count),
-			average_ms(tasks[i].irq_delay_total, tasks[i].irq_count));
+		if (!cfg.mem_verbose_mode) {
+			suc &= BOOL_FPRINT(out, FMT_NORMAL,
+				average_ms(tasks[i].cpu_delay_total,
+						   tasks[i].cpu_count),
+				average_ms(tasks[i].blkio_delay_total,
+						   tasks[i].blkio_count),
+				average_ms(tasks[i].irq_delay_total,
+						   tasks[i].irq_count),
+				average_ms(task_total_mem_delay(&tasks[i]),
+						   task_total_mem_count(&tasks[i])));
+		} else {
+			suc &= BOOL_FPRINT(out, FMT_MEMVERBOSE,
+				average_ms(tasks[i].cpu_delay_total,
+						   tasks[i].cpu_count),
+				average_ms(tasks[i].blkio_delay_total,
+						   tasks[i].blkio_count),
+				average_ms(tasks[i].irq_delay_total,
+						   tasks[i].irq_count),
+				average_ms(task_total_mem_delay(&tasks[i]),
+						   task_total_mem_count(&tasks[i])),
+				average_ms(tasks[i].swapin_delay_total,
+						   tasks[i].swapin_count),
+				average_ms(tasks[i].freepages_delay_total,
+						   tasks[i].freepages_count),
+				average_ms(tasks[i].thrashing_delay_total,
+						   tasks[i].thrashing_count),
+				average_ms(tasks[i].compact_delay_total,
+						   tasks[i].compact_count),
+				average_ms(tasks[i].wpcopy_delay_total,
+						   tasks[i].wpcopy_count));
+		}
 	}

 	suc &= BOOL_FPRINT(out, "\n");
-- 
2.25.1

[PATCH linux-next 2/3] tools/delaytop: add flexible sorting by delay field

[fan.yu9]

From: Fan Yu <fan.yu9@zte.com.cn>

The delaytop tool only supported sorting by CPU delay, which limited
its usefulness when users needed to identify bottlenecks in other
subsystems. Users had no way to sort processes by IO, memory, or
other delay types to quickly pinpoint specific performance issues.

Add -s/--sort option to allow sorting by different delay types:
1) Basic modes: cpu, io, irq, mem  
2) Detailed modes (-M required): swap, reclaim, thrashing, compact, wpcopy

Users can now quickly identify bottlenecks in specific subsystems
by sorting processes by the relevant delay metric.

Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
---
 tools/accounting/delaytop.c | 130 +++++++++++++++++++++++++++++++++---
 1 file changed, 121 insertions(+), 9 deletions(-)

diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c
index f1e2e1cca4b8..39852cd70bdf 100644
--- a/tools/accounting/delaytop.c
+++ b/tools/accounting/delaytop.c
@@ -173,7 +173,9 @@ static void usage(void)
 	"  -o, --once               Display once and exit\n"
 	"  -p, --pid=PID            Monitor only the specified PID\n"
 	"  -C, --container=PATH     Monitor the container at specified cgroup path\n"
-	"  -M, --memverbose         Display memory detailed information\n");
+	"  -M, --memverbose         Display memory detailed information\n"
+	"  -s, --sort=FIELD         Sort by delay field (default: cpu)\n"
+	"                           Types: cpu|io|irq|mem|swap|reclaim|thrashing|compact|wpcopy\n");
 	exit(0);
 }

@@ -188,6 +190,7 @@ static void parse_args(int argc, char **argv)
 		{"pid", required_argument, 0, 'p'},
 		{"once", no_argument, 0, 'o'},
 		{"processes", required_argument, 0, 'P'},
+		{"sort", required_argument, 0, 's'},
 		{"container", required_argument, 0, 'C'},
 		{"memverbose", no_argument, 0, 'M'},
 		{0, 0, 0, 0}
@@ -206,7 +209,7 @@ static void parse_args(int argc, char **argv)
 	while (1) {
 		int option_index = 0;

-		c = getopt_long(argc, argv, "hd:n:p:oP:C:M", long_options, &option_index);
+		c = getopt_long(argc, argv, "hd:n:p:oP:C:Ms:", long_options, &option_index);
 		if (c == -1)
 			break;

@@ -256,11 +259,53 @@ static void parse_args(int argc, char **argv)
 		case 'M':
 			cfg.mem_verbose_mode = 1;
 			break;
+		case 's':
+			if (strlen(optarg) == 0) {
+				fprintf(stderr, "Error: empty sort field\n");
+				exit(1);
+			}
+
+			if (strncmp(optarg, "cpu", 3) == 0)
+				cfg.sort_field = 'c';
+			else if (strncmp(optarg, "io", 2) == 0)
+				cfg.sort_field = 'i';
+			else if (strncmp(optarg, "irq", 3) == 0)
+				cfg.sort_field = 'q';
+			else if (strncmp(optarg, "mem", 3) == 0)
+				cfg.sort_field = 'm';
+			else if (strncmp(optarg, "swap", 4) == 0)
+				cfg.sort_field = 's';
+			else if (strncmp(optarg, "reclaim", 7) == 0)
+				cfg.sort_field = 'r';
+			else if (strncmp(optarg, "thrashing", 9) == 0)
+				cfg.sort_field = 't';
+			else if (strncmp(optarg, "compact", 7) == 0)
+				cfg.sort_field = 'p';
+			else if (strncmp(optarg, "wpcopy", 7) == 0)
+				cfg.sort_field = 'w';
+			else {
+				fprintf(stderr, "Error: invalid sort field\n");
+				fprintf(stderr, "Try to use cpu|io|irq|mem|");
+				fprintf(stderr, "swap|reclaim|thrashing|compact|wpcopy\n");
+				exit(1);
+			}
+			break;
 		default:
 			fprintf(stderr, "Try 'delaytop --help' for more information.\n");
 			exit(1);
 		}
 	}
+
+	/* Validate sorting field compatibility with memory verbose mode */
+	if (cfg.mem_verbose_mode == 0 &&
+		cfg.sort_field == 's' ||
+		cfg.sort_field == 'r' ||
+		cfg.sort_field == 't' ||
+		cfg.sort_field == 'p' ||
+		cfg.sort_field == 'w') {
+		fprintf(stderr, "Error: mem verbose mode is off, try to use -M\n");
+		exit(1);
+	}
 }

 /* Create a raw netlink socket and bind */
@@ -621,12 +666,77 @@ static int compare_tasks(const void *a, const void *b)
 	case 'c': /* CPU */
 		avg1 = average_ms(t1->cpu_delay_total, t1->cpu_count);
 		avg2 = average_ms(t2->cpu_delay_total, t2->cpu_count);
-		if (avg1 != avg2)
-			return avg2 > avg1 ? 1 : -1;
-		return t2->cpu_delay_total > t1->cpu_delay_total ? 1 : -1;
+		break;
+	case 'i': /* IO */
+		avg1 = average_ms(t1->blkio_delay_total, t1->blkio_count);
+		avg2 = average_ms(t2->blkio_delay_total, t2->blkio_count);
+		break;
+	case 'q': /* IRQ */
+		avg1 = average_ms(t1->irq_delay_total, t1->irq_count);
+		avg2 = average_ms(t2->irq_delay_total, t2->irq_count);
+		break;
+	case 'm': /* MEM(total) */
+		avg1 = average_ms(task_total_mem_delay(t1), task_total_mem_count(t1));
+		avg2 = average_ms(task_total_mem_delay(t2), task_total_mem_count(t2));
+		break;
+	/* Memory detailed display mode */
+	case 's': /* swapin (SWAP) */
+		avg1 = average_ms(t1->swapin_delay_total, t1->swapin_count);
+		avg2 = average_ms(t2->swapin_delay_total, t2->swapin_count);
+		break;
+	case 'r': /* freepages (RCL) */
+		avg1 = average_ms(t1->freepages_delay_total, t1->freepages_count);
+		avg2 = average_ms(t2->freepages_delay_total, t2->freepages_count);
+		break;
+	case 't': /* thrashing (THR) */
+		avg1 = average_ms(t1->thrashing_delay_total, t1->thrashing_count);
+		avg2 = average_ms(t2->thrashing_delay_total, t2->thrashing_count);
+		break;
+	case 'p': /* compact (CMP) */
+		avg1 = average_ms(t1->compact_delay_total, t1->compact_count);
+		avg2 = average_ms(t2->compact_delay_total, t2->compact_count);
+		break;
+	case 'w': /* wpcopy (WP) */
+		avg1 = average_ms(t1->wpcopy_delay_total, t1->wpcopy_count);
+		avg2 = average_ms(t2->wpcopy_delay_total, t2->wpcopy_count);
+		break;
+	default:
+		avg1 = average_ms(t1->cpu_delay_total, t1->cpu_count);
+		avg2 = average_ms(t2->cpu_delay_total, t2->cpu_count);
+		break;
+	}
+
+	if (avg1 != avg2)
+		return avg2 > avg1 ? 1 : -1;
+
+	return 0;
+}

+static const char *get_sort_field(char sort_field)
+{
+	switch (sort_field) {
+	case 'c':
+		return "CPU";
+	case 'i':
+		return "IO";
+	case 'q':
+		return "IRQ";
+	/* MEM(total) */
+	case 'm':
+		return "MEM";
+	/* Memory detailed display mode */
+	case 's':
+		return "SWAP";
+	case 'r':
+		return "RCL";
+	case 't':
+		return "THR";
+	case 'p':
+		return "CMP";
+	case 'w':
+		return "WP";
 	default:
-		return t2->cpu_delay_total > t1->cpu_delay_total ? 1 : -1;
+		return "UNKNOWN"; /* handle error */
 	}
 }

@@ -705,6 +815,7 @@ static void display_results(void)
 {
 	time_t now = time(NULL);
 	struct tm *tm_now = localtime(&now);
+	const char *sort_field;
 	FILE *out = stdout;
 	char timestamp[32];
 	bool suc = true;
@@ -766,8 +877,10 @@ static void display_results(void)
 			container_stats.nr_stopped, container_stats.nr_uninterruptible,
 			container_stats.nr_io_wait);
 	}
-	suc &= BOOL_FPRINT(out, "Top %d processes (sorted by CPU delay):\n",
-			cfg.max_processes);
+
+	/* Task delay output */
+	suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n",
+			cfg.max_processes, get_sort_field(cfg.sort_field));
 	suc &= BOOL_FPRINT(out, "%8s  %8s  %-17s", "PID", "TGID", "COMMAND");

 	if (!cfg.mem_verbose_mode) {
@@ -787,7 +900,6 @@ static void display_results(void)
 		suc &= BOOL_FPRINT(out, "-------------------------\n");
 	}

-
 	count = task_count < cfg.max_processes ? task_count : cfg.max_processes;

 	for (i = 0; i < count; i++) {
-- 
2.25.1

[PATCH linux-next 3/3] tools/delaytop: add interactive mode with keyboard controls

[fan.yu9]

From: Fan Yu <fan.yu9@zte.com.cn>

The original delaytop only supported static output with limited
interaction. Users had to restart the tool with different command-line
options to change sorting or display modes, which disrupted continuous
monitoring and reduced productivity during performance investigations.

Adds real-time interactive controls through keyboard input:
1) Add interactive menu system with visual prompts
2) Support dynamic sorting changes without restarting
3) Enable toggle of memory verbose mode with 'M' key

The interactive mode transforms delaytop from a static monitoring tool
into a dynamic investigation platform, allowing users to adapt the
view in real-time based on observed performance patterns.

Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
---
 tools/accounting/delaytop.c | 112 ++++++++++++++++++++++++++----------
 1 file changed, 82 insertions(+), 30 deletions(-)

diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c
index 39852cd70bdf..21fb215d2928 100644
--- a/tools/accounting/delaytop.c
+++ b/tools/accounting/delaytop.c
@@ -140,6 +140,7 @@ static struct task_info tasks[MAX_TASKS];
 static int task_count;
 static int running = 1;
 static struct container_stats container_stats;
+static int sort_selected;

 /* Netlink socket variables */
 static int nl_sd = -1;
@@ -878,6 +879,17 @@ static void display_results(void)
 			container_stats.nr_io_wait);
 	}

+	/* Interacive command */
+	suc &= BOOL_FPRINT(out, "[o]sort [M]memverbose [q]quit\n");
+	if (sort_selected) {
+		if (cfg.mem_verbose_mode)
+			suc &= BOOL_FPRINT(out,
+			"sort selection: [c]CPU [i]IO [s]SWAP [r]RCL [t]THR [p]CMP [w]WP [q]IRQ\n");
+		else
+			suc &= BOOL_FPRINT(out,
+			"sort selection: [c]CPU [i]IO [m]MEM [q]IRQ\n");
+	}
+
 	/* Task delay output */
 	suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n",
 			cfg.max_processes, get_sort_field(cfg.sort_field));
@@ -944,11 +956,73 @@ static void display_results(void)
 		perror("Error writing to output");
 }

+/* Check for keyboard input with timeout based on cfg.delay */
+static char check_for_keypress(void)
+{
+	struct timeval tv = {cfg.delay, 0};
+	fd_set readfds;
+	char ch = 0;
+
+	FD_ZERO(&readfds);
+	FD_SET(STDIN_FILENO, &readfds);
+	int r = select(STDIN_FILENO + 1, &readfds, NULL, NULL, &tv);
+
+	if (r > 0 && FD_ISSET(STDIN_FILENO, &readfds)) {
+		read(STDIN_FILENO, &ch, 1);
+		return ch;
+	}
+
+	return 0;
+}
+
+/* Handle keyboard input: sorting selection, mode toggle, or quit */
+static void handle_keypress(char ch, int *running)
+{
+	if (sort_selected) {
+		switch (ch) {
+		case 'c':
+		case 'i':
+		case 'q':
+		case 'm':
+			cfg.sort_field = ch;
+			break;
+		/* Only for memory verbose mode */
+		case 's':
+		case 'r':
+		case 't':
+		case 'p':
+		case 'w':
+			if (cfg.mem_verbose_mode)
+				cfg.sort_field = ch;
+			break;
+		default:
+			break;
+		}
+		sort_selected = 0;
+	} else {
+		switch (ch) {
+		case 'o':
+			sort_selected = 1;
+			break;
+		case 'M':
+			cfg.mem_verbose_mode = !cfg.mem_verbose_mode;
+			cfg.sort_field = 'c'; /* Change to default sort mode */
+			break;
+		case 'q':
+		case 'Q':
+			*running = 0;
+			break;
+		default:
+			break;
+		}
+	}
+}
+
 /* Main function */
 int main(int argc, char **argv)
 {
 	int iterations = 0;
-	int use_q_quit = 0;
+	char keypress;

 	/* Parse command line arguments */
 	parse_args(argc, argv);
@@ -968,12 +1042,8 @@ int main(int argc, char **argv)
 		exit(1);
 	}

-	if (!cfg.output_one_time) {
-		use_q_quit = 1;
-		enable_raw_mode();
-		printf("Press 'q' to quit.\n");
-		fflush(stdout);
-	}
+	/* Set terminal to non-canonical mode for interaction */
+	enable_raw_mode();

 	/* Main loop */
 	while (running) {
@@ -1001,32 +1071,14 @@ int main(int argc, char **argv)
 		if (cfg.output_one_time)
 			break;

-		/* Check for 'q' key to quit */
-		if (use_q_quit) {
-			struct timeval tv = {cfg.delay, 0};
-			fd_set readfds;
-
-			FD_ZERO(&readfds);
-			FD_SET(STDIN_FILENO, &readfds);
-			int r = select(STDIN_FILENO+1, &readfds, NULL, NULL, &tv);
-
-			if (r > 0 && FD_ISSET(STDIN_FILENO, &readfds)) {
-				char ch = 0;
-
-				read(STDIN_FILENO, &ch, 1);
-				if (ch == 'q' || ch == 'Q') {
-					running = 0;
-					break;
-				}
-			}
-		} else {
-			sleep(cfg.delay);
-		}
+		/* Keypress for interactive usage */
+		keypress = check_for_keypress();
+		if (keypress)
+			handle_keypress(keypress, &running);
 	}

 	/* Restore terminal mode */
-	if (use_q_quit)
-		disable_raw_mode();
+	disable_raw_mode();

 	/* Cleanup */
 	close(nl_sd);
-- 
2.25.1

Re: [PATCH linux-next 1/3] tools/delaytop: add memory verbose mode support

[xu.xin16]

> From: Fan Yu <fan.yu9@zte.com.cn>
> 
> The original delaytop tool always displayed detailed memory
> subsystem breakdown, which could be overwhelming for users
> who only need high-level overview.
> 
> Add flexible display control allowing users to choose their
> preferred information granularity.
> 
> The new flexibility provides:
> 1) For quick monitoring: use normal mode to reduce visual clutter
> 2) For deep analysis: use verbose mode to see all memory subsystem details
> 
> Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>

This feature is very useful to analyze specific delay sources due to memory operations.

But these shown datas are basically average values rather than min/max, it's not enough

for debugging delay jitters. Will you add showing min/max values to delaytop in future? 

> ---
>  tools/accounting/delaytop.c | 111 ++++++++++++++++++++++++++++--------
>  1 file changed, 87 insertions(+), 24 deletions(-)
> 
>  /* PSI statistics structure */
> @@ -163,13 +166,14 @@ static void usage(void)
>  {
>  	printf("Usage: delaytop [Options]\n"
>  	"Options:\n"
> -	"  -h, --help				Show this help message and exit\n"
> -	"  -d, --delay=SECONDS	  Set refresh interval (default: 2 seconds, min: 1)\n"
> -	"  -n, --iterations=COUNT	Set number of updates (default: 0 = infinite)\n"
> -	"  -P, --processes=NUMBER	Set maximum number of processes to show (default: 20, max: 1000)\n"
> -	"  -o, --once				Display once and exit\n"
> -	"  -p, --pid=PID			Monitor only the specified PID\n"
> -	"  -C, --container=PATH	 Monitor the container at specified cgroup path\n");
> +	"  -h, --help               Show this help message and exit\n"
> +	"  -d, --delay=SECONDS      Set refresh interval (default: 2 seconds, min: 1)\n"
> +	"  -n, --iterations=COUNT   Set number of updates (default: 0 = infinite)\n"
> +	"  -P, --processes=NUMBER   Set maximum number of processes to show (default: 20, max: 1000)\n"
> +	"  -o, --once               Display once and exit\n"
> +	"  -p, --pid=PID            Monitor only the specified PID\n"
> +	"  -C, --container=PATH     Monitor the container at specified cgroup path\n"
> +	"  -M, --memverbose         Display memory detailed information\n");
>  	exit(0);
>  }
> 

..

> 
> +	suc &= BOOL_FPRINT(out, "%8s  %8s  %-17s", "PID", "TGID", "COMMAND");
> +
> +	if (!cfg.mem_verbose_mode) {
> +		suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s\n",
> +			"CPU(ms)", "IO(ms)", "IRQ(ms)", "MEM(ms)");
> +		suc &= BOOL_FPRINT(out, "-----------------------");
> +		suc &= BOOL_FPRINT(out, "-----------------------");
> +		suc &= BOOL_FPRINT(out, "--------------------------\n");
> +	} else {
> +		suc &= BOOL_FPRINT(out, "%8s %8s %8s %8s %8s %8s %8s %8s %8s\n",
> +			"CPU(ms)", "IO(ms)", "IRQ(ms)", "MEM(ms)",
> +			"SWAP(ms)", "RCL(ms)", "THR(ms)", "CMP(ms)", "WP(ms)");
> +		suc &= BOOL_FPRINT(out, "-----------------------");
> +		suc &= BOOL_FPRINT(out, "-----------------------");
> +		suc &= BOOL_FPRINT(out, "-----------------------");
> +		suc &= BOOL_FPRINT(out, "-----------------------");
> +		suc &= BOOL_FPRINT(out, "-------------------------\n");
> +	}
> +
> 
> -	suc &= BOOL_FPRINT(out, "-----------------------------------------------");
> -	suc &= BOOL_FPRINT(out, "----------------------------------------------\n");
>  	count = task_count < cfg.max_processes ? task_count : cfg.max_processes;
> 
>  	for (i = 0; i < count; i++) {
> -		suc &= BOOL_FPRINT(out, "%5d  %5d  %-15s",
> +		suc &= BOOL_FPRINT(out, "%8d  %8d  %-15s",
>  			tasks[i].pid, tasks[i].tgid, tasks[i].command);
> -		suc &= BOOL_FPRINT(out, "%7.2f %7.2f %7.2f %7.2f %7.2f %7.2f %7.2f %7.2f\n",
> -			average_ms(tasks[i].cpu_delay_total, tasks[i].cpu_count),
> -			average_ms(tasks[i].blkio_delay_total, tasks[i].blkio_count),
> -			average_ms(tasks[i].swapin_delay_total, tasks[i].swapin_count),
> -			average_ms(tasks[i].thrashing_delay_total, tasks[i].thrashing_count),
> -			average_ms(tasks[i].compact_delay_total, tasks[i].compact_count),
> -			average_ms(tasks[i].wpcopy_delay_total, tasks[i].wpcopy_count),
> -			average_ms(tasks[i].irq_delay_total, tasks[i].irq_count));
> +		if (!cfg.mem_verbose_mode) {
> +			suc &= BOOL_FPRINT(out, FMT_NORMAL,
> +				average_ms(tasks[i].cpu_delay_total,
> +						   tasks[i].cpu_count),
> +				average_ms(tasks[i].blkio_delay_total,
> +						   tasks[i].blkio_count),
> +				average_ms(tasks[i].irq_delay_total,
> +						   tasks[i].irq_count),
> +				average_ms(task_total_mem_delay(&tasks[i]),
> +						   task_total_mem_count(&tasks[i])));
> +		} else {
> +			suc &= BOOL_FPRINT(out, FMT_MEMVERBOSE,
> +				average_ms(tasks[i].cpu_delay_total,
> +						   tasks[i].cpu_count),
> +				average_ms(tasks[i].blkio_delay_total,
> +						   tasks[i].blkio_count),
> +				average_ms(tasks[i].irq_delay_total,
> +						   tasks[i].irq_count),
> +				average_ms(task_total_mem_delay(&tasks[i]),
> +						   task_total_mem_count(&tasks[i])),
> +				average_ms(tasks[i].swapin_delay_total,
> +						   tasks[i].swapin_count),
> +				average_ms(tasks[i].freepages_delay_total,
> +						   tasks[i].freepages_count),
> +				average_ms(tasks[i].thrashing_delay_total,
> +						   tasks[i].thrashing_count),
> +				average_ms(tasks[i].compact_delay_total,
> +						   tasks[i].compact_count),
> +				average_ms(tasks[i].wpcopy_delay_total,
> +						   tasks[i].wpcopy_count));
> +		}
>  	}

I think if users press 'M', delaytop doesn't need to shown CPU/IRQ/IO delays, just show the detailed memory delays.


> 
>  	suc &= BOOL_FPRINT(out, "\n");
> -- 
> 2.25.1

Re: [PATCH linux-next 2/3] tools/delaytop: add flexible sorting by delay field

[xu.xin16]

> From: Fan Yu <fan.yu9@zte.com.cn>
> 
> The delaytop tool only supported sorting by CPU delay, which limited
> its usefulness when users needed to identify bottlenecks in other
> subsystems. Users had no way to sort processes by IO, memory, or
> other delay types to quickly pinpoint specific performance issues.
> 
> Add -s/--sort option to allow sorting by different delay types:
> 1) Basic modes: cpu, io, irq, mem  
> 2) Detailed modes (-M required): swap, reclaim, thrashing, compact, wpcopy
> 
> Users can now quickly identify bottlenecks in specific subsystems
> by sorting processes by the relevant delay metric.
> 
> Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
> ---
>  tools/accounting/delaytop.c | 130 +++++++++++++++++++++++++++++++++---
>  1 file changed, 121 insertions(+), 9 deletions(-)

I have tried this function. Nice to use.

Reviewed-by: xu xin <xu.xin16@zte.com.cn>

> 
> diff --git a/tools/accounting/delaytop.c b/tools/accounting/delaytop.c
> index f1e2e1cca4b8..39852cd70bdf 100644
> --- a/tools/accounting/delaytop.c
> +++ b/tools/accounting/delaytop.c
> @@ -173,7 +173,9 @@ static void usage(void)
>  	"  -o, --once               Display once and exit\n"
>  	"  -p, --pid=PID            Monitor only the specified PID\n"
>  	"  -C, --container=PATH     Monitor the container at specified cgroup path\n"
> -	"  -M, --memverbose         Display memory detailed information\n");
> +	"  -M, --memverbose         Display memory detailed information\n"
> +	"  -s, --sort=FIELD         Sort by delay field (default: cpu)\n"
> +	"                           Types: cpu|io|irq|mem|swap|reclaim|thrashing|compact|wpcopy\n");
>  	exit(0);
>  }
> 
> @@ -188,6 +190,7 @@ static void parse_args(int argc, char **argv)
>  		{"pid", required_argument, 0, 'p'},
>  		{"once", no_argument, 0, 'o'},
>  		{"processes", required_argument, 0, 'P'},
> +		{"sort", required_argument, 0, 's'},
>  		{"container", required_argument, 0, 'C'},
>  		{"memverbose", no_argument, 0, 'M'},
>  		{0, 0, 0, 0}
> @@ -206,7 +209,7 @@ static void parse_args(int argc, char **argv)
>  	while (1) {
>  		int option_index = 0;
> 
> -		c = getopt_long(argc, argv, "hd:n:p:oP:C:M", long_options, &option_index);
> +		c = getopt_long(argc, argv, "hd:n:p:oP:C:Ms:", long_options, &option_index);
>  		if (c == -1)
>  			break;
> 
> @@ -256,11 +259,53 @@ static void parse_args(int argc, char **argv)
>  		case 'M':
>  			cfg.mem_verbose_mode = 1;
>  			break;
> +		case 's':
> +			if (strlen(optarg) == 0) {
> +				fprintf(stderr, "Error: empty sort field\n");
> +				exit(1);
> +			}
> +
> +			if (strncmp(optarg, "cpu", 3) == 0)
> +				cfg.sort_field = 'c';
> +			else if (strncmp(optarg, "io", 2) == 0)
> +				cfg.sort_field = 'i';
> +			else if (strncmp(optarg, "irq", 3) == 0)
> +				cfg.sort_field = 'q';
> +			else if (strncmp(optarg, "mem", 3) == 0)
> +				cfg.sort_field = 'm';
> +			else if (strncmp(optarg, "swap", 4) == 0)
> +				cfg.sort_field = 's';
> +			else if (strncmp(optarg, "reclaim", 7) == 0)
> +				cfg.sort_field = 'r';
> +			else if (strncmp(optarg, "thrashing", 9) == 0)
> +				cfg.sort_field = 't';
> +			else if (strncmp(optarg, "compact", 7) == 0)
> +				cfg.sort_field = 'p';
> +			else if (strncmp(optarg, "wpcopy", 7) == 0)
> +				cfg.sort_field = 'w';
> +			else {
> +				fprintf(stderr, "Error: invalid sort field\n");
> +				fprintf(stderr, "Try to use cpu|io|irq|mem|");
> +				fprintf(stderr, "swap|reclaim|thrashing|compact|wpcopy\n");
> +				exit(1);
> +			}
> +			break;
>  		default:
>  			fprintf(stderr, "Try 'delaytop --help' for more information.\n");
>  			exit(1);
>  		}
>  	}
> +
> +	/* Validate sorting field compatibility with memory verbose mode */
> +	if (cfg.mem_verbose_mode == 0 &&
> +		cfg.sort_field == 's' ||
> +		cfg.sort_field == 'r' ||
> +		cfg.sort_field == 't' ||
> +		cfg.sort_field == 'p' ||
> +		cfg.sort_field == 'w') {
> +		fprintf(stderr, "Error: mem verbose mode is off, try to use -M\n");
> +		exit(1);
> +	}
>  }
> 
>  /* Create a raw netlink socket and bind */
> @@ -621,12 +666,77 @@ static int compare_tasks(const void *a, const void *b)
>  	case 'c': /* CPU */
>  		avg1 = average_ms(t1->cpu_delay_total, t1->cpu_count);
>  		avg2 = average_ms(t2->cpu_delay_total, t2->cpu_count);
> -		if (avg1 != avg2)
> -			return avg2 > avg1 ? 1 : -1;
> -		return t2->cpu_delay_total > t1->cpu_delay_total ? 1 : -1;
> +		break;
> +	case 'i': /* IO */
> +		avg1 = average_ms(t1->blkio_delay_total, t1->blkio_count);
> +		avg2 = average_ms(t2->blkio_delay_total, t2->blkio_count);
> +		break;
> +	case 'q': /* IRQ */
> +		avg1 = average_ms(t1->irq_delay_total, t1->irq_count);
> +		avg2 = average_ms(t2->irq_delay_total, t2->irq_count);
> +		break;
> +	case 'm': /* MEM(total) */
> +		avg1 = average_ms(task_total_mem_delay(t1), task_total_mem_count(t1));
> +		avg2 = average_ms(task_total_mem_delay(t2), task_total_mem_count(t2));
> +		break;
> +	/* Memory detailed display mode */
> +	case 's': /* swapin (SWAP) */
> +		avg1 = average_ms(t1->swapin_delay_total, t1->swapin_count);
> +		avg2 = average_ms(t2->swapin_delay_total, t2->swapin_count);
> +		break;
> +	case 'r': /* freepages (RCL) */
> +		avg1 = average_ms(t1->freepages_delay_total, t1->freepages_count);
> +		avg2 = average_ms(t2->freepages_delay_total, t2->freepages_count);
> +		break;
> +	case 't': /* thrashing (THR) */
> +		avg1 = average_ms(t1->thrashing_delay_total, t1->thrashing_count);
> +		avg2 = average_ms(t2->thrashing_delay_total, t2->thrashing_count);
> +		break;
> +	case 'p': /* compact (CMP) */
> +		avg1 = average_ms(t1->compact_delay_total, t1->compact_count);
> +		avg2 = average_ms(t2->compact_delay_total, t2->compact_count);
> +		break;
> +	case 'w': /* wpcopy (WP) */
> +		avg1 = average_ms(t1->wpcopy_delay_total, t1->wpcopy_count);
> +		avg2 = average_ms(t2->wpcopy_delay_total, t2->wpcopy_count);
> +		break;
> +	default:
> +		avg1 = average_ms(t1->cpu_delay_total, t1->cpu_count);
> +		avg2 = average_ms(t2->cpu_delay_total, t2->cpu_count);
> +		break;
> +	}
> +
> +	if (avg1 != avg2)
> +		return avg2 > avg1 ? 1 : -1;
> +
> +	return 0;
> +}
> 
> +static const char *get_sort_field(char sort_field)
> +{
> +	switch (sort_field) {
> +	case 'c':
> +		return "CPU";
> +	case 'i':
> +		return "IO";
> +	case 'q':
> +		return "IRQ";
> +	/* MEM(total) */
> +	case 'm':
> +		return "MEM";
> +	/* Memory detailed display mode */
> +	case 's':
> +		return "SWAP";
> +	case 'r':
> +		return "RCL";
> +	case 't':
> +		return "THR";
> +	case 'p':
> +		return "CMP";
> +	case 'w':
> +		return "WP";
>  	default:
> -		return t2->cpu_delay_total > t1->cpu_delay_total ? 1 : -1;
> +		return "UNKNOWN"; /* handle error */
>  	}
>  }
> 
> @@ -705,6 +815,7 @@ static void display_results(void)
>  {
>  	time_t now = time(NULL);
>  	struct tm *tm_now = localtime(&now);
> +	const char *sort_field;
>  	FILE *out = stdout;
>  	char timestamp[32];
>  	bool suc = true;
> @@ -766,8 +877,10 @@ static void display_results(void)
>  			container_stats.nr_stopped, container_stats.nr_uninterruptible,
>  			container_stats.nr_io_wait);
>  	}
> -	suc &= BOOL_FPRINT(out, "Top %d processes (sorted by CPU delay):\n",
> -			cfg.max_processes);
> +
> +	/* Task delay output */
> +	suc &= BOOL_FPRINT(out, "Top %d processes (sorted by %s delay):\n",
> +			cfg.max_processes, get_sort_field(cfg.sort_field));
>  	suc &= BOOL_FPRINT(out, "%8s  %8s  %-17s", "PID", "TGID", "COMMAND");
> 
>  	if (!cfg.mem_verbose_mode) {
> @@ -787,7 +900,6 @@ static void display_results(void)
>  		suc &= BOOL_FPRINT(out, "-------------------------\n");
>  	}
> 
> -
>  	count = task_count < cfg.max_processes ? task_count : cfg.max_processes;
> 
>  	for (i = 0; i < count; i++) {
> -- 
> 2.25.1

Re: [PATCH linux-next 2/3] tools/delaytop: add flexible sorting by delay field

[xu.xin16]

> From: Fan Yu <fan.yu9@zte.com.cn>
> 
> The original delaytop only supported static output with limited
> interaction. Users had to restart the tool with different command-line
> options to change sorting or display modes, which disrupted continuous
> monitoring and reduced productivity during performance investigations.
> 
> Adds real-time interactive controls through keyboard input:
> 1) Add interactive menu system with visual prompts
> 2) Support dynamic sorting changes without restarting
> 3) Enable toggle of memory verbose mode with 'M' key
> 
> The interactive mode transforms delaytop from a static monitoring tool
> into a dynamic investigation platform, allowing users to adapt the
> view in real-time based on observed performance patterns.
> 
> Signed-off-by: Fan Yu <fan.yu9@zte.com.cn>
> ---
>  tools/accounting/delaytop.c | 112 ++++++++++++++++++++++++++----------
>  1 file changed, 82 insertions(+), 30 deletions(-)

Reviewed-by: xu xin <xu.xin16@zte.com.cn>

Re: [PATCH linux-next 0/3] tools/delaytop: implement real-time keyboard interaction support

[xu.xin16]

From: Fan Yu <fan.yu9@zte.com.cn>
> Current Limitations
> ===================
> The current delaytop implementation has two main limitations:
> 1) Static sorting only by CPU delay
> Forcing users to restart with different parameters to analyze
> other resource bottlenecks.
> 2) Memory delay information is always expanded
> Causing information overload when only high-level memory pressure
> monitoring is needed.
> 
> Improvements
> ============
> 1) Implemented dynamic sorting capability
> - Interactive key 'o' triggers sort mode.
> - Supports sorting by CPU/IO/Memory/IRQ delays.
> - Memory subcategories available in verbose mode.
>  * c - CPU delay (default)
>  * i - IO delay
>  * m - Total memory delay
>  * q - IRQ delay
>  * s/r/t/p/w - Memory subcategories (in verbose mode)
> 2) Added memory display modes
> - Compact view (default): shows aggregated memory delays.
> - Verbose view ('M' key): breaks down into memory sub-delays.
>  * SWAP - swapin delays
>  * RCL - freepages reclaim delays
>  * THR - thrashing delays
>  * CMP - compaction delays
>  * WP - write-protect copy delays
>

What about updating usages into Documentation/accounting/delay-accounting.rst ?

> 
> 
> Fan Yu (3):
>   tools/delaytop: add memory verbose mode support
>   tools/delaytop: add flexible sorting by delay field
>   tools/delaytop: add interactive mode with keyboard controls
> 
>  tools/accounting/delaytop.c | 347 +++++++++++++++++++++++++++++-------
>  1 file changed, 287 insertions(+), 60 deletions(-)

Other parts are ok except [PATCH 1/3]

Re: [PATCH linux-next 0/3] tools/delaytop: implement real-time keyboard interaction support

[yang.yang29]

> 2) Enhanced Usability with On-Screen Keybindings
> More intuitive interactive usage with on-screen keybindings help.
> Reduced screen clutter when only memory overview is needed.

It's more intuitive and easier to understand the patches if show the
output of new features in git log. And we notice that if PSI or
delayaccouting is off, delaytop will output zero data, I think we
should tell user why it's zero, since not all user knows kernel config
will. Else: 
Acked-by: Yang Yang <yang.yang29@zte.com.cn>