From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A3EE1F890F for ; Thu, 19 Dec 2024 06:06:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734588416; cv=none; b=XC9SRnM/98MQ9N+aVoJWmw/S/UWsOlU7KhRgUHKbPbzhV9hnP/JwYgLDoB/oiXda/k6OUSJve2VN7Z8ixZ/4WjDht4r058zAQtshwBLjqDpvlG24PtimD4sySa8e2Vo/yCYe8FKK5/+yo61DAsYat834fHCwQE0zrjszEwx1IXc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734588416; c=relaxed/simple; bh=/+1C0IvxQ8P1Z7TSSe4D99SGTydPNYnJTM0KhFC+iEA=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=fesBXft4l/OWY+KDjUhN2/kWMRqb3ChrWyNnmPXo4H6lxrKnpzESpRY4/8jpiDH/D8Bf6dVi+kmf2AItHfbSI08I+kNa5tBXiR3+AMMf7bmwRohfHDiIvrW/btj+1TYGZUa72qdT39ikAlzZK0ELnqxmI6MgjLgu5Ybd2ymbi/s= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=NwoAaMVa; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NwoAaMVa" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-801c27e67a6so459847a12.2 for ; Wed, 18 Dec 2024 22:06:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734588414; x=1735193214; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=sjzEPhk1aMRT4dKtdtSYRJDQpJYMwDhdB6quer1fCoQ=; b=NwoAaMVa/yR8O3FCUjF/Qm/4bg4wWBOZu1BD3DEDZqlHorFF6a7nH5a+KrlC8qGK1z TPdN6IAyAxPZkN5WI4D+3beA/pPUN+Ej94nIvvKDmXMcvlyneWGS8ohW02ParD5DP5hp rjVNHj0Rn92t6GHf/U/co+lir+tHsZFmOMXUGrtaFiJMhjMDWXm0XwZq8shQ5Apl87Ts a7NbI0fbwMXTqCxlhw1oPAyHYJga2km7/VrH2AX/UNg8loVsV0o/X4ybQ/oTcXoanzkM fxTy3lPeRyqNIX0GlQmyePrb6Gq4ztx+cSDrl0pU3ReOE7IQtjtWbbz7xdacmOkCQXcD j5NQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734588414; x=1735193214; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=sjzEPhk1aMRT4dKtdtSYRJDQpJYMwDhdB6quer1fCoQ=; b=vKFhW2VNO8ohLOYysy9gT0qUlIGYwfjYcANLRFKFeb5+i7y+A1Ft4/AXNwzXj+hAyO HNDmAqMJdtEwyej5Q+1NmG6Dlwqpeec6HDH1jnWcojo3+6EHcdnbZV+CQ0w3yX4dSY/E YDUe9Vwx2jbi3Y+PjbN2dev+R2rGhVBBnxw/YEXQGrSaiX4s3FQ9ji4CF+MMDrWboyZ1 CbbTS9bKbWMwj97VB1L2kIX8J2duT0XBRNPqGQFjAPj2UdY60/hecsXBpWxfsE+6JzCN mR3bvCTSkFJGR4voxsk1pI7tqSWR39pktX6slCmpnsiEclq9Uy+IQV8lO/wUkOCFf3tG 3ApA== X-Gm-Message-State: AOJu0YwYiZAYOFluDbFfKTvir+miymNxE1t2rMe4UawgFHioQsBElkzK TPUR0BTyWrgsLhUDUX2lLD7pP7TZosm1nPcuKLRQ533qXeSTCSVP3BIIZzAar6dLz2VgsMHfDAd ZLTlWz1zCcv41rmI7vSTcptCnMWzlDws2L5HMXvdZCV4PaYFoZHzDNSz05C1MvraswN2XqmvnoG 90RF5y6HIdm4/drVViH0XcSzTNl88KWviB/DIXtRsPZv05 X-Google-Smtp-Source: AGHT+IH6NrWl0pPGE3a2BSHhK08KTtahCksXFUKdBML7eg0YhDBfvUekODv5FuTqqphfy6s6yI5HZIm0tfsR X-Received: from pguo10.prod.google.com ([2002:a65:6a4a:0:b0:7fd:4075:406d]) (user=jstultz job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:320d:b0:215:50fb:ae4a with SMTP id d9443c01a7336-219d95513b2mr41389315ad.0.1734588413736; Wed, 18 Dec 2024 22:06:53 -0800 (PST) Date: Wed, 18 Dec 2024 22:06:05 -0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241219060643.2360926-1-jstultz@google.com> Subject: [PATCH v3 1/2] lib: stacklog_debug: Introduce helper tool for collecting and displaying stacktraces From: John Stultz To: LKML Cc: John Stultz , Steven Rostedt , Alexander Potapenko , Andrey Konovalov , Andrew Morton , kernel-team@android.com Content-Type: text/plain; charset="UTF-8" When debugging, its often useful to understand how a function is called and the different paths taken to get there. Usually dump_stack() can be used for this purpose. However there are a number of cases where a function is called very frequently, making dump_stack far too noisy to be useful. This is a little debug tool that utilizes stackdepot to capture unique stack traces and store them in a circular buffer. In the code, the developer adds: stacklog_debug_save() calls at points of interest (as they might with stack_dump()). Then after running the kernel, the developer can dump the unique stack traces from the buffer via: cat /sys/kernel/debug/stacklog_debug This is pretty trivial, but I've had this hanging around for awhile and recently hit another case where it was helpful, so I figured it would be worth sending it out for feedback as to if others thought it would be useful enough to merge upstream or to possibly rework into stackdepot itself? Cc: Steven Rostedt Cc: Alexander Potapenko Cc: Andrey Konovalov Cc: Andrew Morton Cc: kernel-team@android.com Signed-off-by: John Stultz --- v2: * Add dependency on STACKTRACE_SUPPORT as reported by kernel test robot * Expand the Kconfig help to include basic usage v3: * Make globals static as Reported-by: kernel test robot --- include/linux/stacklog_debug.h | 13 ++++ lib/Kconfig | 16 +++++ lib/Makefile | 1 + lib/stacklog_debug.c | 110 +++++++++++++++++++++++++++++++++ 4 files changed, 140 insertions(+) create mode 100644 include/linux/stacklog_debug.h create mode 100644 lib/stacklog_debug.c diff --git a/include/linux/stacklog_debug.h b/include/linux/stacklog_debug.h new file mode 100644 index 0000000000000..d88f05d7000a5 --- /dev/null +++ b/include/linux/stacklog_debug.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +#ifndef _LINUX_STACKLOG_DEBUG_H +#define _LINUX_STACKLOG_DEBUG_H + +#ifdef CONFIG_STACKLOG_DEBUG +void stacklog_debug_save(void); +#else +static inline void stacklog_debug_save(void) +{ +} +#endif +#endif diff --git a/lib/Kconfig b/lib/Kconfig index 5a318f753b2f4..d0320329260b2 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -737,6 +737,22 @@ config REF_TRACKER depends on STACKTRACE_SUPPORT select STACKDEPOT +config STACKLOG_DEBUG + bool "Debug tool for logging and later displaying stacktraces" + depends on STACKTRACE_SUPPORT + select STACKDEPOT + select STACKDEPOT_ALWAYS_INIT + help + Enables debug infrastructure for logging unique stack traces at + a specific point, which can be later displayed from userland. + + The developer adds: stacklog_debug_save() calls at points of + interest (as they might with stack_dump()). + + Then after running the kernel, the developer can dump the unique + stack traces from the buffer via: + cat /sys/kernel/debug/stacklog_debug + config SBITMAP bool diff --git a/lib/Makefile b/lib/Makefile index a8155c972f028..931347245334e 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -268,6 +268,7 @@ obj-$(CONFIG_IRQ_POLL) += irq_poll.o obj-$(CONFIG_POLYNOMIAL) += polynomial.o +obj-$(CONFIG_STACKLOG_DEBUG) += stacklog_debug.o # stackdepot.c should not be instrumented or call instrumented functions. # Prevent the compiler from calling builtins like memcmp() or bcmp() from this # file. diff --git a/lib/stacklog_debug.c b/lib/stacklog_debug.c new file mode 100644 index 0000000000000..70d2e203b36c6 --- /dev/null +++ b/lib/stacklog_debug.c @@ -0,0 +1,110 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2014, The Linux Foundation. All rights reserved. + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define STACKDEPTH 32 +#define BUFSZ 4096 + +#define LIST_ENTRIES 512 +static DEFINE_SPINLOCK(stack_lock); +static depot_stack_handle_t stack_list[LIST_ENTRIES]; +static int head, tail; + +void stacklog_debug_save(void) +{ + unsigned long entries[STACKDEPTH]; + depot_stack_handle_t stack_hash; + unsigned long flags; + unsigned int n; + int i; + + n = stack_trace_save(entries, ARRAY_SIZE(entries), 1); + stack_hash = stack_depot_save(entries, n, GFP_NOWAIT); + if (!stack_hash) + return; + + spin_lock_irqsave(&stack_lock, flags); + for (i = head; i < tail; i++) + if (stack_list[i % LIST_ENTRIES] == stack_hash) + goto out; + + stack_list[(tail++ % LIST_ENTRIES)] = stack_hash; + + if (tail % LIST_ENTRIES == head % LIST_ENTRIES) + head++; + + if (tail >= 2 * LIST_ENTRIES) { + head %= LIST_ENTRIES; + tail %= LIST_ENTRIES; + if (tail < head) + tail += LIST_ENTRIES; + } +out: + spin_unlock_irqrestore(&stack_lock, flags); +} + +#ifdef CONFIG_DEBUG_FS +static int stacklog_stats_show(struct seq_file *s, void *unused) +{ + char *buf = kmalloc(BUFSZ, GFP_NOWAIT); + unsigned int nr_entries; + unsigned long flags; + int i, start, stop; + + if (!buf) + return -ENOMEM; + + spin_lock_irqsave(&stack_lock, flags); + start = head; + stop = tail; + spin_unlock_irqrestore(&stack_lock, flags); + + if (start == stop) + goto out; + + for (i = start; i < stop; i++) { + unsigned long *ent; + u32 hash; + + /* + * We avoid holdings the lock over the entire loop + * just to be careful as we don't want to trip a + * call path that calls back into stacklog_debug_save + * which would deadlock, so hold the lock minimally + * (and be ok with the data changing between loop + * iterations). + */ + spin_lock_irqsave(&stack_lock, flags); + hash = stack_list[i % LIST_ENTRIES]; + spin_unlock_irqrestore(&stack_lock, flags); + + nr_entries = stack_depot_fetch(hash, &ent); + stack_trace_snprint(buf, BUFSZ, ent, nr_entries, 0); + seq_printf(s, "[idx: %i hash: %ld]====================\n%s\n\n", + i - start, (long)hash, buf); + } +out: + kfree(buf); + return 0; +} +DEFINE_SHOW_ATTRIBUTE(stacklog_stats); + +static int __init stacklog_debug_init(void) +{ + debugfs_create_file("stacklog_debug", 0400, NULL, NULL, + &stacklog_stats_fops); + return 0; +} + +late_initcall(stacklog_debug_init); +#endif -- 2.47.1.613.gc27f4b7a9f-goog