Linux Device Performance Analyses - Where are my cylces?

[Lucas Tanure <tanure@linux.com>]

Hey there,

I'm a kernel developer working on an Android device with a 5.10 kernel. 
I'm trying to improve performance, and to do that, I need to figure out 
exactly where the CPU is spending its time.

Our vendor provides a generic Android BSP, so the system is likely 
running many unnecessary processes for things like GPS and phone calls 
that my device doesn't even need. Since I don't have a good handle on 
all these processes, I want to create a detailed breakdown of all the 
tasks, workqueues, IRQs, kernel threads, softIRQs, and tasklets running 
and measure the CPU time each one is using.

To do this, I've been using kernel traces with the following events:

task:task_newtask sched:sched_process_fork
sched:sched_process_exec sched:sched_process_exit sched:sched_switch 
irq:irq_handler_entry irq:irq_handler_exit irq:softirq_entry 
irq:softirq_exit workqueue:workqueue_execute_start 
workqueue:workqueue_execute_end syscalls:sys_enter_execve 
syscalls:sys_exit_execve syscalls:sys_enter_execveat 
syscalls:sys_exit_execveat

However, the trace logs don't provide the full picture. For example, 
when a new process is executed, the logs don't specify the script being 
run. The sched_switch event tells me which PID is being executed, but 
multiple processes can originate from the same binary, so it's hard to 
distinguish them.

To solve this, I've developed a patch for the do_execveat_common 
function to log all new processes along with their full command lines. I 
plan to use this patch with a Python script to parse the logs and 
generate a report on CPU usage.

My main questions are:

- Is my patch correct? When I log the new process, do current->pid and 
argv refer to the same new process?
- It feels like logging new processes with their command lines is a 
fundamental requirement for this kind of analysis. Am I missing a better 
or more standard way of doing this?
- I don't want to reinvent the wheel, so if there's a better way to 
analyze kernel performance on a device, I'd love to hear about it.

After looking into other methods like eBPF and kprobes, I found that 
this patch is the most straightforward way to get the information I need.

Any insights would be greatly appreciated!

Thanks
Lucas Tanure

diff --git a/fs/exec.c b/fs/exec.c
index 6e1f8628ab9c..ab797ba0571c 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -513,6 +513,49 @@ static int bprm_stack_limits(struct linux_binprm *bprm)
  	return 0;
  }

+static char* str_cmdline(int argc, struct user_arg_ptr argv, struct 
linux_binprm *bprm)
+{
+	char *buf = (char *)get_zeroed_page(GFP_KERNEL);
+	ssize_t pos = 0;
+	int arg;
+
+	for (arg = 0; arg < argc; arg++) {
+		const char __user *str;
+		int len;
+
+		str = get_user_arg_ptr(argv, arg);
+		if (IS_ERR(str))
+			goto free_page;
+
+		// this includes a final null.
+		len = strnlen_user(str, MAX_ARG_STRLEN);
+		if (!len)
+			goto free_page;
+
+		if (!valid_arg_len(bprm, len))
+			goto free_page;
+
+		if (pos + len >= PAGE_SIZE)
+			break; // Return the command line up to a page
+
+		len -= 1; // Don't copy the final null.
+		if (copy_from_user(buf+pos, str, len))
+			goto free_page;
+
+		pos += len;
+
+		if (arg < argc - 1)
+			buf[pos++] = ' ';
+	}
+
+	return buf;
+
+free_page:
+	free_page((unsigned long)buf);
+
+	return NULL;
+}
+
  /*
   * 'copy_strings()' copies argument/environment strings from the old
   * processes's memory to the new process's stack.  The call to 
get_user_pages()
@@ -1874,6 +1917,7 @@ static int do_execveat_common(int fd, struct 
filename *filename,
  {
  	struct linux_binprm *bprm;
  	int retval;
+	char *cmdline;

  	if (IS_ERR(filename))
  		return PTR_ERR(filename);
@@ -1943,6 +1987,15 @@ static int do_execveat_common(int fd, struct 
filename *filename,
  		bprm->argc = 1;
  	}

+	cmdline = str_cmdline(bprm->argc, argv, bprm);
+	pr_info("%s pid: %d, comm: %s # filename: %s # cmdline: %s\n", __func__,
+								       current->pid,
+								       current->comm,
+								       bprm->filename,
+								       cmdline ? cmdline : "NULL");
+	if (cmdline)
+		free_page((unsigned long)cmdline);
+
  	retval = bprm_execve(bprm, fd, filename, flags);
  out_free:
  	free_bprm(bprm);

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies