From mboxrd@z Thu Jan 1 00:00:00 1970 From: Valentin Schneider Subject: [RFC PATCH 0/5] Generic IPI sending tracepoint Date: Fri, 7 Oct 2022 16:41:40 +0100 Message-ID: <20221007154145.1877054-1-vschneid@redhat.com> Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665157364; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=1lKCnBjxRPjgpNNPlEylbLIMLTQ3r/0aTk3SgXwAPQE=; b=N7sQrgdSZPwOY3ZJUgISKw+SC3hBO/RHz6umRRQW4ie1INCaUWCjb3vy0v5wah0psYYkvz fQtvgtdRDTaigxwjQkRn1uZP3hrluP3fb7iiWMY8rSIeMGz5G5YT/6Bd6GNXLwQ+gPIakr kCmn7vL2fab3I6qWl+AKt1arkPoLJRk= List-ID: Content-Type: text/plain; charset="us-ascii" To: linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, x86@kernel.org Cc: "Paul E. McKenney" , Steven Rostedt , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Frederic Weisbecker , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Marc Zyngier , Mark Rutland , Russell King , Nicholas Piggin , Guo Ren , "David S. Miller" Background ========== Detecting IPI *reception* is relatively easy, e.g. using trace_irq_handler_{entry,exit} or even just function-trace flush_smp_call_function_queue() for SMP calls. Figuring out their *origin*, is trickier as there is no generic tracepoint tied to e.g. smp_call_function(): o AFAIA x86 has no tracepoint tied to sending IPIs, only receiving them (cf. trace_call_function{_single}_entry()). o arm/arm64 do have trace_ipi_raise(), which gives us the target cpus but also a mostly useless string (smp_calls will all be "Function call interrupts"). o Other architectures don't seem to have any IPI-sending related tracepoint. I believe one reason those tracepoints used by arm/arm64 ended up as they were is because these archs used to handle IPIs differently from regular interrupts (the IRQ driver would directly invoke an IPI-handling routine), which meant they never showed up in trace_irq_handler_{entry, exit}. The trace_ipi_{entry,exit} tracepoints gave a way to trace IPI reception but those have become redundant as of: 56afcd3dbd19 ("ARM: Allow IPIs to be handled as normal interrupts") d3afc7f12987 ("arm64: Allow IPIs to be handled as normal interrupts") which gave IPIs a "proper" handler function used through generic_handle_domain_irq(), which makes them show up via trace_irq_handler_{entry, exit}. Changing stuff up ================= Per the above, it would make sense to reshuffle trace_ipi_raise() and move it into generic code. This also came up during Daniel's talk on Osnoise at the CPU isolation MC of LPC 2022 [1]. Now, to be useful, such a tracepoint needs to export: o targeted CPU(s) o calling context The only way to get the calling context with trace_ipi_raise() is to trigger a stack dump, e.g. $(trace-cmd -e ipi* -T echo 42). As for the targeted CPUs, the existing tracepoint does export them, albeit in cpumask form, which is quite inconvenient from a tooling perspective. For instance, as far as I'm aware, it's not possible to do event filtering on a cpumask via trace-cmd. Because of the above points, this is introducing a new tracepoint. Patches ======= This results in having trace events for: o smp_call_function*() o smp_send_reschedule() o irq_work_queue*() This is incomplete, just looking at arm64 there's more IPI types that aren't covered: IPI_CPU_STOP, IPI_CPU_CRASH_STOP, IPI_TIMER, IPI_WAKEUP, ... But it feels like a good starting point. Another thing worth mentioning is that depending on the callsite, the _RET_IP_ fed to the tracepoint is not always useful - generic_exec_single() doesn't tell you much about the actual callback being sent via IPI, so there might be value in exploding the single tracepoint into at least one variant for smp_calls. Links ===== [1]: https://youtu.be/5gT57y4OzBM?t=14234 Valentin Schneider (5): trace: Add trace_ipi_send_{cpu, cpumask} sched, smp: Trace send_call_function_single_ipi() smp: Add a multi-CPU variant to send_call_function_single_ipi() irq_work: Trace calls to arch_irq_work_raise() treewide: Rename and trace arch-definitions of smp_send_reschedule() arch/alpha/kernel/smp.c | 2 +- arch/arc/kernel/smp.c | 2 +- arch/arm/kernel/smp.c | 5 +---- arch/arm64/kernel/smp.c | 3 +-- arch/csky/kernel/smp.c | 2 +- arch/hexagon/kernel/smp.c | 2 +- arch/ia64/kernel/smp.c | 4 ++-- arch/loongarch/include/asm/smp.h | 2 +- arch/mips/include/asm/smp.h | 2 +- arch/openrisc/kernel/smp.c | 2 +- arch/parisc/kernel/smp.c | 4 ++-- arch/powerpc/kernel/smp.c | 4 ++-- arch/riscv/kernel/smp.c | 4 ++-- arch/s390/kernel/smp.c | 2 +- arch/sh/kernel/smp.c | 2 +- arch/sparc/kernel/smp_32.c | 2 +- arch/sparc/kernel/smp_64.c | 2 +- arch/x86/include/asm/smp.h | 2 +- arch/xtensa/kernel/smp.c | 2 +- include/linux/smp.h | 1 + include/trace/events/ipi.h | 27 +++++++++++++++++++++++++++ kernel/irq_work.c | 12 +++++++++++- kernel/sched/core.c | 7 +++++-- kernel/smp.c | 18 +++++++++++++++++- 24 files changed, 84 insertions(+), 31 deletions(-) -- 2.31.1