From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marcelo Tosatti Subject: Re: [RFC PATCH 0/5] Generic IPI sending tracepoint Date: Fri, 7 Oct 2022 17:01:33 -0300 Message-ID: References: <20221007154145.1877054-1-vschneid@redhat.com> Mime-Version: 1.0 Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665173019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=jaDcObpjcx0whPL96jk38beBrOuRKo+eqX0wqNpM3Cc=; b=P+lIyk+e0CPKAEcXbbQsDGytM3pSE2HiXWeW7NP38w5yUSHLYGCuSuucP33CsCkagHA0Xl 151o2pP9yjDQ1PeDz0xhPVbLkmo8lCO3ZH7OMrITtTWxWfWY5mkXbEVSe9xPFuqCCFlhIl 3Pk9fHEL2zLDpOzOjIrrd39Z4gNTtAk= Content-Disposition: inline In-Reply-To: <20221007154145.1877054-1-vschneid@redhat.com> List-ID: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Valentin Schneider Cc: linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, x86@kernel.org, "Paul E. McKenney" , Steven Rostedt , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , Juri Lelli , Dani Hi Valentin, On Fri, Oct 07, 2022 at 04:41:40PM +0100, Valentin Schneider wrote: > Background > ========== > > Detecting IPI *reception* is relatively easy, e.g. using > trace_irq_handler_{entry,exit} or even just function-trace > flush_smp_call_function_queue() for SMP calls. > > Figuring out their *origin*, is trickier as there is no generic tracepoint tied > to e.g. smp_call_function(): > > o AFAIA x86 has no tracepoint tied to sending IPIs, only receiving them > (cf. trace_call_function{_single}_entry()). > o arm/arm64 do have trace_ipi_raise(), which gives us the target cpus but also a > mostly useless string (smp_calls will all be "Function call interrupts"). > o Other architectures don't seem to have any IPI-sending related tracepoint. > > I believe one reason those tracepoints used by arm/arm64 ended up as they were > is because these archs used to handle IPIs differently from regular interrupts > (the IRQ driver would directly invoke an IPI-handling routine), which meant they > never showed up in trace_irq_handler_{entry, exit}. The trace_ipi_{entry,exit} > tracepoints gave a way to trace IPI reception but those have become redundant as > of: > > 56afcd3dbd19 ("ARM: Allow IPIs to be handled as normal interrupts") > d3afc7f12987 ("arm64: Allow IPIs to be handled as normal interrupts") > > which gave IPIs a "proper" handler function used through > generic_handle_domain_irq(), which makes them show up via > trace_irq_handler_{entry, exit}. > > Changing stuff up > ================= > > Per the above, it would make sense to reshuffle trace_ipi_raise() and move it > into generic code. This also came up during Daniel's talk on Osnoise at the CPU > isolation MC of LPC 2022 [1]. > > Now, to be useful, such a tracepoint needs to export: > o targeted CPU(s) > o calling context > > The only way to get the calling context with trace_ipi_raise() is to trigger a > stack dump, e.g. $(trace-cmd -e ipi* -T echo 42). > > As for the targeted CPUs, the existing tracepoint does export them, albeit in > cpumask form, which is quite inconvenient from a tooling perspective. For > instance, as far as I'm aware, it's not possible to do event filtering on a > cpumask via trace-cmd. https://man7.org/linux/man-pages/man1/trace-cmd-set.1.html -f filter Specify a filter for the previous event. This must come after a -e. This will filter what events get recorded based on the content of the event. Filtering is passed to the kernel directly so what filtering is allowed may depend on what version of the kernel you have. Basically, it will let you use C notation to check if an event should be processed or not. ==, >=, <=, >, <, &, |, && and || The above are usually safe to use to compare fields. This looks overkill to me (consider large number of bits set in mask). +#define trace_ipi_send_cpumask(callsite, mask) do { \ + if (static_key_false(&__tracepoint_ipi_send_cpu.key)) { \ + int cpu; \ + for_each_cpu(cpu, mask) \ + trace_ipi_send_cpu(callsite, cpu); \ + } \ +} while (0) > > Because of the above points, this is introducing a new tracepoint. > > Patches > ======= > > This results in having trace events for: > > o smp_call_function*() > o smp_send_reschedule() > o irq_work_queue*() > > This is incomplete, just looking at arm64 there's more IPI types that aren't covered: > > IPI_CPU_STOP, > IPI_CPU_CRASH_STOP, > IPI_TIMER, > IPI_WAKEUP, > > ... But it feels like a good starting point. Can't you have a single tracepoint (or variant with cpumask) that would cover such cases as well? Maybe (as parameters for tracepoint): * type (reschedule, smp_call_function, timer, wakeup, ...). * function address: valid for smp_call_function, irq_work_queue types. > Another thing worth mentioning is that depending on the callsite, the _RET_IP_ > fed to the tracepoint is not always useful - generic_exec_single() doesn't tell > you much about the actual callback being sent via IPI, so there might be value > in exploding the single tracepoint into at least one variant for smp_calls. Not sure i grasp what you mean by "exploding the single tracepoint...", but yes knowing the function or irq work function is very useful. > > Links > ===== > > [1]: https://youtu.be/5gT57y4OzBM?t=14234 > > Valentin Schneider (5): > trace: Add trace_ipi_send_{cpu, cpumask} > sched, smp: Trace send_call_function_single_ipi() > smp: Add a multi-CPU variant to send_call_function_single_ipi() > irq_work: Trace calls to arch_irq_work_raise() > treewide: Rename and trace arch-definitions of smp_send_reschedule() > > arch/alpha/kernel/smp.c | 2 +- > arch/arc/kernel/smp.c | 2 +- > arch/arm/kernel/smp.c | 5 +---- > arch/arm64/kernel/smp.c | 3 +-- > arch/csky/kernel/smp.c | 2 +- > arch/hexagon/kernel/smp.c | 2 +- > arch/ia64/kernel/smp.c | 4 ++-- > arch/loongarch/include/asm/smp.h | 2 +- > arch/mips/include/asm/smp.h | 2 +- > arch/openrisc/kernel/smp.c | 2 +- > arch/parisc/kernel/smp.c | 4 ++-- > arch/powerpc/kernel/smp.c | 4 ++-- > arch/riscv/kernel/smp.c | 4 ++-- > arch/s390/kernel/smp.c | 2 +- > arch/sh/kernel/smp.c | 2 +- > arch/sparc/kernel/smp_32.c | 2 +- > arch/sparc/kernel/smp_64.c | 2 +- > arch/x86/include/asm/smp.h | 2 +- > arch/xtensa/kernel/smp.c | 2 +- > include/linux/smp.h | 1 + > include/trace/events/ipi.h | 27 +++++++++++++++++++++++++++ > kernel/irq_work.c | 12 +++++++++++- > kernel/sched/core.c | 7 +++++-- > kernel/smp.c | 18 +++++++++++++++++- > 24 files changed, 84 insertions(+), 31 deletions(-) > > -- > 2.31.1 > > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EC1CC4332F for ; Fri, 7 Oct 2022 20:03:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230058AbiJGUDr (ORCPT ); Fri, 7 Oct 2022 16:03:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230059AbiJGUDo (ORCPT ); Fri, 7 Oct 2022 16:03:44 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AB9A12C8B3 for ; Fri, 7 Oct 2022 13:03:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665173019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=jaDcObpjcx0whPL96jk38beBrOuRKo+eqX0wqNpM3Cc=; b=P+lIyk+e0CPKAEcXbbQsDGytM3pSE2HiXWeW7NP38w5yUSHLYGCuSuucP33CsCkagHA0Xl 151o2pP9yjDQ1PeDz0xhPVbLkmo8lCO3ZH7OMrITtTWxWfWY5mkXbEVSe9xPFuqCCFlhIl 3Pk9fHEL2zLDpOzOjIrrd39Z4gNTtAk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-264-kStNREYhN0OL2iSIbHz80Q-1; Fri, 07 Oct 2022 16:03:36 -0400 X-MC-Unique: kStNREYhN0OL2iSIbHz80Q-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 67B08882826; Fri, 7 Oct 2022 20:03:34 +0000 (UTC) Received: from fuller.cnet (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 29A1A2166B4D; Fri, 7 Oct 2022 20:03:32 +0000 (UTC) Received: by fuller.cnet (Postfix, from userid 1000) id 258DA416CE48; Fri, 7 Oct 2022 17:01:33 -0300 (-03) Date: Fri, 7 Oct 2022 17:01:33 -0300 From: Marcelo Tosatti To: Valentin Schneider Cc: linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, x86@kernel.org, "Paul E. McKenney" , Steven Rostedt , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , Juri Lelli , Daniel Bristot de Oliveira , Frederic Weisbecker , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Marc Zyngier , Mark Rutland , Russell King , Nicholas Piggin , Guo Ren , "David S. Miller" Subject: Re: [RFC PATCH 0/5] Generic IPI sending tracepoint Message-ID: References: <20221007154145.1877054-1-vschneid@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20221007154145.1877054-1-vschneid@redhat.com> X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Precedence: bulk List-ID: X-Mailing-List: linux-csky@vger.kernel.org Hi Valentin, On Fri, Oct 07, 2022 at 04:41:40PM +0100, Valentin Schneider wrote: > Background > ========== > > Detecting IPI *reception* is relatively easy, e.g. using > trace_irq_handler_{entry,exit} or even just function-trace > flush_smp_call_function_queue() for SMP calls. > > Figuring out their *origin*, is trickier as there is no generic tracepoint tied > to e.g. smp_call_function(): > > o AFAIA x86 has no tracepoint tied to sending IPIs, only receiving them > (cf. trace_call_function{_single}_entry()). > o arm/arm64 do have trace_ipi_raise(), which gives us the target cpus but also a > mostly useless string (smp_calls will all be "Function call interrupts"). > o Other architectures don't seem to have any IPI-sending related tracepoint. > > I believe one reason those tracepoints used by arm/arm64 ended up as they were > is because these archs used to handle IPIs differently from regular interrupts > (the IRQ driver would directly invoke an IPI-handling routine), which meant they > never showed up in trace_irq_handler_{entry, exit}. The trace_ipi_{entry,exit} > tracepoints gave a way to trace IPI reception but those have become redundant as > of: > > 56afcd3dbd19 ("ARM: Allow IPIs to be handled as normal interrupts") > d3afc7f12987 ("arm64: Allow IPIs to be handled as normal interrupts") > > which gave IPIs a "proper" handler function used through > generic_handle_domain_irq(), which makes them show up via > trace_irq_handler_{entry, exit}. > > Changing stuff up > ================= > > Per the above, it would make sense to reshuffle trace_ipi_raise() and move it > into generic code. This also came up during Daniel's talk on Osnoise at the CPU > isolation MC of LPC 2022 [1]. > > Now, to be useful, such a tracepoint needs to export: > o targeted CPU(s) > o calling context > > The only way to get the calling context with trace_ipi_raise() is to trigger a > stack dump, e.g. $(trace-cmd -e ipi* -T echo 42). > > As for the targeted CPUs, the existing tracepoint does export them, albeit in > cpumask form, which is quite inconvenient from a tooling perspective. For > instance, as far as I'm aware, it's not possible to do event filtering on a > cpumask via trace-cmd. https://man7.org/linux/man-pages/man1/trace-cmd-set.1.html -f filter Specify a filter for the previous event. This must come after a -e. This will filter what events get recorded based on the content of the event. Filtering is passed to the kernel directly so what filtering is allowed may depend on what version of the kernel you have. Basically, it will let you use C notation to check if an event should be processed or not. ==, >=, <=, >, <, &, |, && and || The above are usually safe to use to compare fields. This looks overkill to me (consider large number of bits set in mask). +#define trace_ipi_send_cpumask(callsite, mask) do { \ + if (static_key_false(&__tracepoint_ipi_send_cpu.key)) { \ + int cpu; \ + for_each_cpu(cpu, mask) \ + trace_ipi_send_cpu(callsite, cpu); \ + } \ +} while (0) > > Because of the above points, this is introducing a new tracepoint. > > Patches > ======= > > This results in having trace events for: > > o smp_call_function*() > o smp_send_reschedule() > o irq_work_queue*() > > This is incomplete, just looking at arm64 there's more IPI types that aren't covered: > > IPI_CPU_STOP, > IPI_CPU_CRASH_STOP, > IPI_TIMER, > IPI_WAKEUP, > > ... But it feels like a good starting point. Can't you have a single tracepoint (or variant with cpumask) that would cover such cases as well? Maybe (as parameters for tracepoint): * type (reschedule, smp_call_function, timer, wakeup, ...). * function address: valid for smp_call_function, irq_work_queue types. > Another thing worth mentioning is that depending on the callsite, the _RET_IP_ > fed to the tracepoint is not always useful - generic_exec_single() doesn't tell > you much about the actual callback being sent via IPI, so there might be value > in exploding the single tracepoint into at least one variant for smp_calls. Not sure i grasp what you mean by "exploding the single tracepoint...", but yes knowing the function or irq work function is very useful. > > Links > ===== > > [1]: https://youtu.be/5gT57y4OzBM?t=14234 > > Valentin Schneider (5): > trace: Add trace_ipi_send_{cpu, cpumask} > sched, smp: Trace send_call_function_single_ipi() > smp: Add a multi-CPU variant to send_call_function_single_ipi() > irq_work: Trace calls to arch_irq_work_raise() > treewide: Rename and trace arch-definitions of smp_send_reschedule() > > arch/alpha/kernel/smp.c | 2 +- > arch/arc/kernel/smp.c | 2 +- > arch/arm/kernel/smp.c | 5 +---- > arch/arm64/kernel/smp.c | 3 +-- > arch/csky/kernel/smp.c | 2 +- > arch/hexagon/kernel/smp.c | 2 +- > arch/ia64/kernel/smp.c | 4 ++-- > arch/loongarch/include/asm/smp.h | 2 +- > arch/mips/include/asm/smp.h | 2 +- > arch/openrisc/kernel/smp.c | 2 +- > arch/parisc/kernel/smp.c | 4 ++-- > arch/powerpc/kernel/smp.c | 4 ++-- > arch/riscv/kernel/smp.c | 4 ++-- > arch/s390/kernel/smp.c | 2 +- > arch/sh/kernel/smp.c | 2 +- > arch/sparc/kernel/smp_32.c | 2 +- > arch/sparc/kernel/smp_64.c | 2 +- > arch/x86/include/asm/smp.h | 2 +- > arch/xtensa/kernel/smp.c | 2 +- > include/linux/smp.h | 1 + > include/trace/events/ipi.h | 27 +++++++++++++++++++++++++++ > kernel/irq_work.c | 12 +++++++++++- > kernel/sched/core.c | 7 +++++-- > kernel/smp.c | 18 +++++++++++++++++- > 24 files changed, 84 insertions(+), 31 deletions(-) > > -- > 2.31.1 > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marcelo Tosatti Date: Fri, 07 Oct 2022 19:58:19 +0000 Subject: Re: [RFC PATCH 0/5] Generic IPI sending tracepoint Message-Id: List-Id: References: <20221007154145.1877054-1-vschneid@redhat.com> In-Reply-To: <20221007154145.1877054-1-vschneid@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Valentin Schneider Cc: linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, x86@kernel.org, "Paul E. McKenney" , Steven Rostedt , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , Juri Lelli , Daniel Bristot de Oliveira , Frederic Weisbecker , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Marc Zyngier , Mark Rutland , Russell King , Nicholas Piggin , Guo Ren , "David S. Miller" Hi Valentin, On Fri, Oct 07, 2022 at 04:41:40PM +0100, Valentin Schneider wrote: > Background > ===== > > Detecting IPI *reception* is relatively easy, e.g. using > trace_irq_handler_{entry,exit} or even just function-trace > flush_smp_call_function_queue() for SMP calls. > > Figuring out their *origin*, is trickier as there is no generic tracepoint tied > to e.g. smp_call_function(): > > o AFAIA x86 has no tracepoint tied to sending IPIs, only receiving them > (cf. trace_call_function{_single}_entry()). > o arm/arm64 do have trace_ipi_raise(), which gives us the target cpus but also a > mostly useless string (smp_calls will all be "Function call interrupts"). > o Other architectures don't seem to have any IPI-sending related tracepoint. > > I believe one reason those tracepoints used by arm/arm64 ended up as they were > is because these archs used to handle IPIs differently from regular interrupts > (the IRQ driver would directly invoke an IPI-handling routine), which meant they > never showed up in trace_irq_handler_{entry, exit}. The trace_ipi_{entry,exit} > tracepoints gave a way to trace IPI reception but those have become redundant as > of: > > 56afcd3dbd19 ("ARM: Allow IPIs to be handled as normal interrupts") > d3afc7f12987 ("arm64: Allow IPIs to be handled as normal interrupts") > > which gave IPIs a "proper" handler function used through > generic_handle_domain_irq(), which makes them show up via > trace_irq_handler_{entry, exit}. > > Changing stuff up > ========> > Per the above, it would make sense to reshuffle trace_ipi_raise() and move it > into generic code. This also came up during Daniel's talk on Osnoise at the CPU > isolation MC of LPC 2022 [1]. > > Now, to be useful, such a tracepoint needs to export: > o targeted CPU(s) > o calling context > > The only way to get the calling context with trace_ipi_raise() is to trigger a > stack dump, e.g. $(trace-cmd -e ipi* -T echo 42). > > As for the targeted CPUs, the existing tracepoint does export them, albeit in > cpumask form, which is quite inconvenient from a tooling perspective. For > instance, as far as I'm aware, it's not possible to do event filtering on a > cpumask via trace-cmd. https://man7.org/linux/man-pages/man1/trace-cmd-set.1.html -f filter Specify a filter for the previous event. This must come after a -e. This will filter what events get recorded based on the content of the event. Filtering is passed to the kernel directly so what filtering is allowed may depend on what version of the kernel you have. Basically, it will let you use C notation to check if an event should be processed or not. =, >=, <=, >, <, &, |, && and || The above are usually safe to use to compare fields. This looks overkill to me (consider large number of bits set in mask). +#define trace_ipi_send_cpumask(callsite, mask) do { \ + if (static_key_false(&__tracepoint_ipi_send_cpu.key)) { \ + int cpu; \ + for_each_cpu(cpu, mask) \ + trace_ipi_send_cpu(callsite, cpu); \ + } \ +} while (0) > > Because of the above points, this is introducing a new tracepoint. > > Patches > ===> > This results in having trace events for: > > o smp_call_function*() > o smp_send_reschedule() > o irq_work_queue*() > > This is incomplete, just looking at arm64 there's more IPI types that aren't covered: > > IPI_CPU_STOP, > IPI_CPU_CRASH_STOP, > IPI_TIMER, > IPI_WAKEUP, > > ... But it feels like a good starting point. Can't you have a single tracepoint (or variant with cpumask) that would cover such cases as well? Maybe (as parameters for tracepoint): * type (reschedule, smp_call_function, timer, wakeup, ...). * function address: valid for smp_call_function, irq_work_queue types. > Another thing worth mentioning is that depending on the callsite, the _RET_IP_ > fed to the tracepoint is not always useful - generic_exec_single() doesn't tell > you much about the actual callback being sent via IPI, so there might be value > in exploding the single tracepoint into at least one variant for smp_calls. Not sure i grasp what you mean by "exploding the single tracepoint...", but yes knowing the function or irq work function is very useful. > > Links > ==> > [1]: https://youtu.be/5gT57y4OzBM?t234 > > Valentin Schneider (5): > trace: Add trace_ipi_send_{cpu, cpumask} > sched, smp: Trace send_call_function_single_ipi() > smp: Add a multi-CPU variant to send_call_function_single_ipi() > irq_work: Trace calls to arch_irq_work_raise() > treewide: Rename and trace arch-definitions of smp_send_reschedule() > > arch/alpha/kernel/smp.c | 2 +- > arch/arc/kernel/smp.c | 2 +- > arch/arm/kernel/smp.c | 5 +---- > arch/arm64/kernel/smp.c | 3 +-- > arch/csky/kernel/smp.c | 2 +- > arch/hexagon/kernel/smp.c | 2 +- > arch/ia64/kernel/smp.c | 4 ++-- > arch/loongarch/include/asm/smp.h | 2 +- > arch/mips/include/asm/smp.h | 2 +- > arch/openrisc/kernel/smp.c | 2 +- > arch/parisc/kernel/smp.c | 4 ++-- > arch/powerpc/kernel/smp.c | 4 ++-- > arch/riscv/kernel/smp.c | 4 ++-- > arch/s390/kernel/smp.c | 2 +- > arch/sh/kernel/smp.c | 2 +- > arch/sparc/kernel/smp_32.c | 2 +- > arch/sparc/kernel/smp_64.c | 2 +- > arch/x86/include/asm/smp.h | 2 +- > arch/xtensa/kernel/smp.c | 2 +- > include/linux/smp.h | 1 + > include/trace/events/ipi.h | 27 +++++++++++++++++++++++++++ > kernel/irq_work.c | 12 +++++++++++- > kernel/sched/core.c | 7 +++++-- > kernel/smp.c | 18 +++++++++++++++++- > 24 files changed, 84 insertions(+), 31 deletions(-) > > -- > 2.31.1 > > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 92B9FC433F5 for ; Fri, 7 Oct 2022 20:03:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Uo89VfbDMVKRx2OTcdj+X4qdh7JnUx3x6BU3uZYQQCk=; b=PpKd2X/2i5BCZs cfgY2Mx07sZTPiJpLOSGLmFyQAMgsaPkWqFSRBxNX1/j7sG4XI2xhx4SPd4rBY87g6/kCmN5vSbcG iAG+1skSpAct6FpBEuUtroZSxTTKf90FkDjrZvd2gEB0J6j6WqBDtJbYhF5BobM5TBTa0e2xzdtyn ukb9KGJPExJB9yl5txRfkea+pCOW+hOKwFFuwpc2sN0bIDfmRUoEJaa4exw6a4wulaRvt+/l1o62F YFrIwmKcFkDd2Ah9vnzo00R0V++rX7+JcLv1QnKACGo9QamR/jl3AbcQgKT5y8/KGUcY+yBJ3vVnf a9V5goRNJeut5sGfgAYg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ogtZP-00AJEF-DV; Fri, 07 Oct 2022 20:03:43 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ogtZM-00AJBL-Td for linux-riscv@lists.infradead.org; Fri, 07 Oct 2022 20:03:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665173017; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=jaDcObpjcx0whPL96jk38beBrOuRKo+eqX0wqNpM3Cc=; b=NGLzyLDo0+mQ8eWEGVtjMEihZBPP5mK6ZxBG/r5WCtTkjAkppJF2fWFqgyxIzRBsQ9j1kD LuDA9auAKdRo9xTXyv3/8eX9ZUt6E1FmJQquoyEjuUBJxdKxps/pL92Wr9UYxGI1GkOfku qVc4N5FIMrgVF7/V9lv31pSIg3+y99Y= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-264-kStNREYhN0OL2iSIbHz80Q-1; Fri, 07 Oct 2022 16:03:36 -0400 X-MC-Unique: kStNREYhN0OL2iSIbHz80Q-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 67B08882826; Fri, 7 Oct 2022 20:03:34 +0000 (UTC) Received: from fuller.cnet (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 29A1A2166B4D; Fri, 7 Oct 2022 20:03:32 +0000 (UTC) Received: by fuller.cnet (Postfix, from userid 1000) id 258DA416CE48; Fri, 7 Oct 2022 17:01:33 -0300 (-03) Date: Fri, 7 Oct 2022 17:01:33 -0300 From: Marcelo Tosatti To: Valentin Schneider Cc: linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, x86@kernel.org, "Paul E. McKenney" , Steven Rostedt , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , Juri Lelli , Daniel Bristot de Oliveira , Frederic Weisbecker , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Marc Zyngier , Mark Rutland , Russell King , Nicholas Piggin , Guo Ren , "David S. Miller" Subject: Re: [RFC PATCH 0/5] Generic IPI sending tracepoint Message-ID: References: <20221007154145.1877054-1-vschneid@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20221007154145.1877054-1-vschneid@redhat.com> X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221007_130341_069457_1C6D2DDF X-CRM114-Status: GOOD ( 35.40 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Hi Valentin, On Fri, Oct 07, 2022 at 04:41:40PM +0100, Valentin Schneider wrote: > Background > ========== > > Detecting IPI *reception* is relatively easy, e.g. using > trace_irq_handler_{entry,exit} or even just function-trace > flush_smp_call_function_queue() for SMP calls. > > Figuring out their *origin*, is trickier as there is no generic tracepoint tied > to e.g. smp_call_function(): > > o AFAIA x86 has no tracepoint tied to sending IPIs, only receiving them > (cf. trace_call_function{_single}_entry()). > o arm/arm64 do have trace_ipi_raise(), which gives us the target cpus but also a > mostly useless string (smp_calls will all be "Function call interrupts"). > o Other architectures don't seem to have any IPI-sending related tracepoint. > > I believe one reason those tracepoints used by arm/arm64 ended up as they were > is because these archs used to handle IPIs differently from regular interrupts > (the IRQ driver would directly invoke an IPI-handling routine), which meant they > never showed up in trace_irq_handler_{entry, exit}. The trace_ipi_{entry,exit} > tracepoints gave a way to trace IPI reception but those have become redundant as > of: > > 56afcd3dbd19 ("ARM: Allow IPIs to be handled as normal interrupts") > d3afc7f12987 ("arm64: Allow IPIs to be handled as normal interrupts") > > which gave IPIs a "proper" handler function used through > generic_handle_domain_irq(), which makes them show up via > trace_irq_handler_{entry, exit}. > > Changing stuff up > ================= > > Per the above, it would make sense to reshuffle trace_ipi_raise() and move it > into generic code. This also came up during Daniel's talk on Osnoise at the CPU > isolation MC of LPC 2022 [1]. > > Now, to be useful, such a tracepoint needs to export: > o targeted CPU(s) > o calling context > > The only way to get the calling context with trace_ipi_raise() is to trigger a > stack dump, e.g. $(trace-cmd -e ipi* -T echo 42). > > As for the targeted CPUs, the existing tracepoint does export them, albeit in > cpumask form, which is quite inconvenient from a tooling perspective. For > instance, as far as I'm aware, it's not possible to do event filtering on a > cpumask via trace-cmd. https://man7.org/linux/man-pages/man1/trace-cmd-set.1.html -f filter Specify a filter for the previous event. This must come after a -e. This will filter what events get recorded based on the content of the event. Filtering is passed to the kernel directly so what filtering is allowed may depend on what version of the kernel you have. Basically, it will let you use C notation to check if an event should be processed or not. ==, >=, <=, >, <, &, |, && and || The above are usually safe to use to compare fields. This looks overkill to me (consider large number of bits set in mask). +#define trace_ipi_send_cpumask(callsite, mask) do { \ + if (static_key_false(&__tracepoint_ipi_send_cpu.key)) { \ + int cpu; \ + for_each_cpu(cpu, mask) \ + trace_ipi_send_cpu(callsite, cpu); \ + } \ +} while (0) > > Because of the above points, this is introducing a new tracepoint. > > Patches > ======= > > This results in having trace events for: > > o smp_call_function*() > o smp_send_reschedule() > o irq_work_queue*() > > This is incomplete, just looking at arm64 there's more IPI types that aren't covered: > > IPI_CPU_STOP, > IPI_CPU_CRASH_STOP, > IPI_TIMER, > IPI_WAKEUP, > > ... But it feels like a good starting point. Can't you have a single tracepoint (or variant with cpumask) that would cover such cases as well? Maybe (as parameters for tracepoint): * type (reschedule, smp_call_function, timer, wakeup, ...). * function address: valid for smp_call_function, irq_work_queue types. > Another thing worth mentioning is that depending on the callsite, the _RET_IP_ > fed to the tracepoint is not always useful - generic_exec_single() doesn't tell > you much about the actual callback being sent via IPI, so there might be value > in exploding the single tracepoint into at least one variant for smp_calls. Not sure i grasp what you mean by "exploding the single tracepoint...", but yes knowing the function or irq work function is very useful. > > Links > ===== > > [1]: https://youtu.be/5gT57y4OzBM?t=14234 > > Valentin Schneider (5): > trace: Add trace_ipi_send_{cpu, cpumask} > sched, smp: Trace send_call_function_single_ipi() > smp: Add a multi-CPU variant to send_call_function_single_ipi() > irq_work: Trace calls to arch_irq_work_raise() > treewide: Rename and trace arch-definitions of smp_send_reschedule() > > arch/alpha/kernel/smp.c | 2 +- > arch/arc/kernel/smp.c | 2 +- > arch/arm/kernel/smp.c | 5 +---- > arch/arm64/kernel/smp.c | 3 +-- > arch/csky/kernel/smp.c | 2 +- > arch/hexagon/kernel/smp.c | 2 +- > arch/ia64/kernel/smp.c | 4 ++-- > arch/loongarch/include/asm/smp.h | 2 +- > arch/mips/include/asm/smp.h | 2 +- > arch/openrisc/kernel/smp.c | 2 +- > arch/parisc/kernel/smp.c | 4 ++-- > arch/powerpc/kernel/smp.c | 4 ++-- > arch/riscv/kernel/smp.c | 4 ++-- > arch/s390/kernel/smp.c | 2 +- > arch/sh/kernel/smp.c | 2 +- > arch/sparc/kernel/smp_32.c | 2 +- > arch/sparc/kernel/smp_64.c | 2 +- > arch/x86/include/asm/smp.h | 2 +- > arch/xtensa/kernel/smp.c | 2 +- > include/linux/smp.h | 1 + > include/trace/events/ipi.h | 27 +++++++++++++++++++++++++++ > kernel/irq_work.c | 12 +++++++++++- > kernel/sched/core.c | 7 +++++-- > kernel/smp.c | 18 +++++++++++++++++- > 24 files changed, 84 insertions(+), 31 deletions(-) > > -- > 2.31.1 > > _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3CADDC433FE for ; Fri, 7 Oct 2022 20:03:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QPhPr2nmG2SQul15hGvNMqY7P4ysQI7Nsn76vPdNyUs=; b=Z+iHe2N2Jr4hGy 4UKkoMWBWgMSmTpR/aF+tLmo+kSrTrA2EnnSV6/vvyT7tSAXauY+dIl0Yck2fIOaUZhOKogiNf0ov lKXkjRAOGlM1rFbTOMZeDq+xphW658OmZ+7kUAaNdTXa7S6HCY2rdw2Y3Or3epOERYQbkS3xCsqU8 yvCP+HPuPp3+lHOmoAANYJny+/KesDUKoqqAD+77uza31/wULt6MT/byyPhVTYr4ZgqBCTrGNIZUC ZRd2gri7t8gyNWWDY0qwexjAZ70D8prRpsyRS4PdCkaiu3gOxH940/FrsUcNuKMLVajfNmh59G10z RjtlrZqk8aR8Y53JpZMg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ogtZR-00AJF3-On; Fri, 07 Oct 2022 20:03:45 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ogtZO-00AJBp-I2 for linux-snps-arc@lists.infradead.org; Fri, 07 Oct 2022 20:03:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665173019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=jaDcObpjcx0whPL96jk38beBrOuRKo+eqX0wqNpM3Cc=; b=P+lIyk+e0CPKAEcXbbQsDGytM3pSE2HiXWeW7NP38w5yUSHLYGCuSuucP33CsCkagHA0Xl 151o2pP9yjDQ1PeDz0xhPVbLkmo8lCO3ZH7OMrITtTWxWfWY5mkXbEVSe9xPFuqCCFlhIl 3Pk9fHEL2zLDpOzOjIrrd39Z4gNTtAk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-264-kStNREYhN0OL2iSIbHz80Q-1; Fri, 07 Oct 2022 16:03:36 -0400 X-MC-Unique: kStNREYhN0OL2iSIbHz80Q-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 67B08882826; Fri, 7 Oct 2022 20:03:34 +0000 (UTC) Received: from fuller.cnet (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 29A1A2166B4D; Fri, 7 Oct 2022 20:03:32 +0000 (UTC) Received: by fuller.cnet (Postfix, from userid 1000) id 258DA416CE48; Fri, 7 Oct 2022 17:01:33 -0300 (-03) Date: Fri, 7 Oct 2022 17:01:33 -0300 From: Marcelo Tosatti To: Valentin Schneider Cc: linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, x86@kernel.org, "Paul E. McKenney" , Steven Rostedt , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , Juri Lelli , Daniel Bristot de Oliveira , Frederic Weisbecker , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Marc Zyngier , Mark Rutland , Russell King , Nicholas Piggin , Guo Ren , "David S. Miller" Subject: Re: [RFC PATCH 0/5] Generic IPI sending tracepoint Message-ID: References: <20221007154145.1877054-1-vschneid@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20221007154145.1877054-1-vschneid@redhat.com> X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221007_130342_698290_3CB6FE39 X-CRM114-Status: GOOD ( 35.30 ) X-BeenThere: linux-snps-arc@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Linux on Synopsys ARC Processors List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-snps-arc" Errors-To: linux-snps-arc-bounces+linux-snps-arc=archiver.kernel.org@lists.infradead.org Hi Valentin, On Fri, Oct 07, 2022 at 04:41:40PM +0100, Valentin Schneider wrote: > Background > ========== > > Detecting IPI *reception* is relatively easy, e.g. using > trace_irq_handler_{entry,exit} or even just function-trace > flush_smp_call_function_queue() for SMP calls. > > Figuring out their *origin*, is trickier as there is no generic tracepoint tied > to e.g. smp_call_function(): > > o AFAIA x86 has no tracepoint tied to sending IPIs, only receiving them > (cf. trace_call_function{_single}_entry()). > o arm/arm64 do have trace_ipi_raise(), which gives us the target cpus but also a > mostly useless string (smp_calls will all be "Function call interrupts"). > o Other architectures don't seem to have any IPI-sending related tracepoint. > > I believe one reason those tracepoints used by arm/arm64 ended up as they were > is because these archs used to handle IPIs differently from regular interrupts > (the IRQ driver would directly invoke an IPI-handling routine), which meant they > never showed up in trace_irq_handler_{entry, exit}. The trace_ipi_{entry,exit} > tracepoints gave a way to trace IPI reception but those have become redundant as > of: > > 56afcd3dbd19 ("ARM: Allow IPIs to be handled as normal interrupts") > d3afc7f12987 ("arm64: Allow IPIs to be handled as normal interrupts") > > which gave IPIs a "proper" handler function used through > generic_handle_domain_irq(), which makes them show up via > trace_irq_handler_{entry, exit}. > > Changing stuff up > ================= > > Per the above, it would make sense to reshuffle trace_ipi_raise() and move it > into generic code. This also came up during Daniel's talk on Osnoise at the CPU > isolation MC of LPC 2022 [1]. > > Now, to be useful, such a tracepoint needs to export: > o targeted CPU(s) > o calling context > > The only way to get the calling context with trace_ipi_raise() is to trigger a > stack dump, e.g. $(trace-cmd -e ipi* -T echo 42). > > As for the targeted CPUs, the existing tracepoint does export them, albeit in > cpumask form, which is quite inconvenient from a tooling perspective. For > instance, as far as I'm aware, it's not possible to do event filtering on a > cpumask via trace-cmd. https://man7.org/linux/man-pages/man1/trace-cmd-set.1.html -f filter Specify a filter for the previous event. This must come after a -e. This will filter what events get recorded based on the content of the event. Filtering is passed to the kernel directly so what filtering is allowed may depend on what version of the kernel you have. Basically, it will let you use C notation to check if an event should be processed or not. ==, >=, <=, >, <, &, |, && and || The above are usually safe to use to compare fields. This looks overkill to me (consider large number of bits set in mask). +#define trace_ipi_send_cpumask(callsite, mask) do { \ + if (static_key_false(&__tracepoint_ipi_send_cpu.key)) { \ + int cpu; \ + for_each_cpu(cpu, mask) \ + trace_ipi_send_cpu(callsite, cpu); \ + } \ +} while (0) > > Because of the above points, this is introducing a new tracepoint. > > Patches > ======= > > This results in having trace events for: > > o smp_call_function*() > o smp_send_reschedule() > o irq_work_queue*() > > This is incomplete, just looking at arm64 there's more IPI types that aren't covered: > > IPI_CPU_STOP, > IPI_CPU_CRASH_STOP, > IPI_TIMER, > IPI_WAKEUP, > > ... But it feels like a good starting point. Can't you have a single tracepoint (or variant with cpumask) that would cover such cases as well? Maybe (as parameters for tracepoint): * type (reschedule, smp_call_function, timer, wakeup, ...). * function address: valid for smp_call_function, irq_work_queue types. > Another thing worth mentioning is that depending on the callsite, the _RET_IP_ > fed to the tracepoint is not always useful - generic_exec_single() doesn't tell > you much about the actual callback being sent via IPI, so there might be value > in exploding the single tracepoint into at least one variant for smp_calls. Not sure i grasp what you mean by "exploding the single tracepoint...", but yes knowing the function or irq work function is very useful. > > Links > ===== > > [1]: https://youtu.be/5gT57y4OzBM?t=14234 > > Valentin Schneider (5): > trace: Add trace_ipi_send_{cpu, cpumask} > sched, smp: Trace send_call_function_single_ipi() > smp: Add a multi-CPU variant to send_call_function_single_ipi() > irq_work: Trace calls to arch_irq_work_raise() > treewide: Rename and trace arch-definitions of smp_send_reschedule() > > arch/alpha/kernel/smp.c | 2 +- > arch/arc/kernel/smp.c | 2 +- > arch/arm/kernel/smp.c | 5 +---- > arch/arm64/kernel/smp.c | 3 +-- > arch/csky/kernel/smp.c | 2 +- > arch/hexagon/kernel/smp.c | 2 +- > arch/ia64/kernel/smp.c | 4 ++-- > arch/loongarch/include/asm/smp.h | 2 +- > arch/mips/include/asm/smp.h | 2 +- > arch/openrisc/kernel/smp.c | 2 +- > arch/parisc/kernel/smp.c | 4 ++-- > arch/powerpc/kernel/smp.c | 4 ++-- > arch/riscv/kernel/smp.c | 4 ++-- > arch/s390/kernel/smp.c | 2 +- > arch/sh/kernel/smp.c | 2 +- > arch/sparc/kernel/smp_32.c | 2 +- > arch/sparc/kernel/smp_64.c | 2 +- > arch/x86/include/asm/smp.h | 2 +- > arch/xtensa/kernel/smp.c | 2 +- > include/linux/smp.h | 1 + > include/trace/events/ipi.h | 27 +++++++++++++++++++++++++++ > kernel/irq_work.c | 12 +++++++++++- > kernel/sched/core.c | 7 +++++-- > kernel/smp.c | 18 +++++++++++++++++- > 24 files changed, 84 insertions(+), 31 deletions(-) > > -- > 2.31.1 > > _______________________________________________ linux-snps-arc mailing list linux-snps-arc@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-snps-arc From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.librecores.org (lists.librecores.org [88.198.125.70]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC6C0C4167B for ; Sat, 8 Oct 2022 05:01:10 +0000 (UTC) Received: from [172.31.1.100] (localhost.localdomain [127.0.0.1]) by mail.librecores.org (Postfix) with ESMTP id 41A0B2499B; Sat, 8 Oct 2022 07:01:09 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mail.librecores.org (Postfix) with ESMTP id 6051B24827 for ; Fri, 7 Oct 2022 22:03:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665173019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=jaDcObpjcx0whPL96jk38beBrOuRKo+eqX0wqNpM3Cc=; b=P+lIyk+e0CPKAEcXbbQsDGytM3pSE2HiXWeW7NP38w5yUSHLYGCuSuucP33CsCkagHA0Xl 151o2pP9yjDQ1PeDz0xhPVbLkmo8lCO3ZH7OMrITtTWxWfWY5mkXbEVSe9xPFuqCCFlhIl 3Pk9fHEL2zLDpOzOjIrrd39Z4gNTtAk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-264-kStNREYhN0OL2iSIbHz80Q-1; Fri, 07 Oct 2022 16:03:36 -0400 X-MC-Unique: kStNREYhN0OL2iSIbHz80Q-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 67B08882826; Fri, 7 Oct 2022 20:03:34 +0000 (UTC) Received: from fuller.cnet (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 29A1A2166B4D; Fri, 7 Oct 2022 20:03:32 +0000 (UTC) Received: by fuller.cnet (Postfix, from userid 1000) id 258DA416CE48; Fri, 7 Oct 2022 17:01:33 -0300 (-03) Date: Fri, 7 Oct 2022 17:01:33 -0300 From: Marcelo Tosatti To: Valentin Schneider Subject: Re: [RFC PATCH 0/5] Generic IPI sending tracepoint Message-ID: References: <20221007154145.1877054-1-vschneid@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20221007154145.1877054-1-vschneid@redhat.com> X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Mailman-Approved-At: Sat, 08 Oct 2022 07:01:05 +0200 X-BeenThere: openrisc@lists.librecores.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Discussion around the OpenRISC processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Juri Lelli , Mark Rutland , linux-ia64@vger.kernel.org, linux-sh@vger.kernel.org, Peter Zijlstra , Sebastian Andrzej Siewior , Dave Hansen , linux-mips@vger.kernel.org, Guo Ren , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, Marc Zyngier , linux-hexagon@vger.kernel.org, x86@kernel.org, Russell King , linux-csky@vger.kernel.org, Ingo Molnar , linux-snps-arc@lists.infradead.org, linux-xtensa@linux-xtensa.org, "Paul E. McKenney" , Frederic Weisbecker , Steven Rostedt , openrisc@lists.librecores.org, Borislav Petkov , Nicholas Piggin , loongarch@lists.linux.dev, Thomas Gleixner , linux-arm-kernel@lists.infradead.org, linux-parisc@vger.kernel.org, Daniel Bristot de Oliveira , linux-kernel@vger.kernel.org, linux-alpha@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, "David S. Miller" Errors-To: openrisc-bounces@lists.librecores.org Sender: "OpenRISC" Hi Valentin, On Fri, Oct 07, 2022 at 04:41:40PM +0100, Valentin Schneider wrote: > Background > ========== > > Detecting IPI *reception* is relatively easy, e.g. using > trace_irq_handler_{entry,exit} or even just function-trace > flush_smp_call_function_queue() for SMP calls. > > Figuring out their *origin*, is trickier as there is no generic tracepoint tied > to e.g. smp_call_function(): > > o AFAIA x86 has no tracepoint tied to sending IPIs, only receiving them > (cf. trace_call_function{_single}_entry()). > o arm/arm64 do have trace_ipi_raise(), which gives us the target cpus but also a > mostly useless string (smp_calls will all be "Function call interrupts"). > o Other architectures don't seem to have any IPI-sending related tracepoint. > > I believe one reason those tracepoints used by arm/arm64 ended up as they were > is because these archs used to handle IPIs differently from regular interrupts > (the IRQ driver would directly invoke an IPI-handling routine), which meant they > never showed up in trace_irq_handler_{entry, exit}. The trace_ipi_{entry,exit} > tracepoints gave a way to trace IPI reception but those have become redundant as > of: > > 56afcd3dbd19 ("ARM: Allow IPIs to be handled as normal interrupts") > d3afc7f12987 ("arm64: Allow IPIs to be handled as normal interrupts") > > which gave IPIs a "proper" handler function used through > generic_handle_domain_irq(), which makes them show up via > trace_irq_handler_{entry, exit}. > > Changing stuff up > ================= > > Per the above, it would make sense to reshuffle trace_ipi_raise() and move it > into generic code. This also came up during Daniel's talk on Osnoise at the CPU > isolation MC of LPC 2022 [1]. > > Now, to be useful, such a tracepoint needs to export: > o targeted CPU(s) > o calling context > > The only way to get the calling context with trace_ipi_raise() is to trigger a > stack dump, e.g. $(trace-cmd -e ipi* -T echo 42). > > As for the targeted CPUs, the existing tracepoint does export them, albeit in > cpumask form, which is quite inconvenient from a tooling perspective. For > instance, as far as I'm aware, it's not possible to do event filtering on a > cpumask via trace-cmd. https://man7.org/linux/man-pages/man1/trace-cmd-set.1.html -f filter Specify a filter for the previous event. This must come after a -e. This will filter what events get recorded based on the content of the event. Filtering is passed to the kernel directly so what filtering is allowed may depend on what version of the kernel you have. Basically, it will let you use C notation to check if an event should be processed or not. ==, >=, <=, >, <, &, |, && and || The above are usually safe to use to compare fields. This looks overkill to me (consider large number of bits set in mask). +#define trace_ipi_send_cpumask(callsite, mask) do { \ + if (static_key_false(&__tracepoint_ipi_send_cpu.key)) { \ + int cpu; \ + for_each_cpu(cpu, mask) \ + trace_ipi_send_cpu(callsite, cpu); \ + } \ +} while (0) > > Because of the above points, this is introducing a new tracepoint. > > Patches > ======= > > This results in having trace events for: > > o smp_call_function*() > o smp_send_reschedule() > o irq_work_queue*() > > This is incomplete, just looking at arm64 there's more IPI types that aren't covered: > > IPI_CPU_STOP, > IPI_CPU_CRASH_STOP, > IPI_TIMER, > IPI_WAKEUP, > > ... But it feels like a good starting point. Can't you have a single tracepoint (or variant with cpumask) that would cover such cases as well? Maybe (as parameters for tracepoint): * type (reschedule, smp_call_function, timer, wakeup, ...). * function address: valid for smp_call_function, irq_work_queue types. > Another thing worth mentioning is that depending on the callsite, the _RET_IP_ > fed to the tracepoint is not always useful - generic_exec_single() doesn't tell > you much about the actual callback being sent via IPI, so there might be value > in exploding the single tracepoint into at least one variant for smp_calls. Not sure i grasp what you mean by "exploding the single tracepoint...", but yes knowing the function or irq work function is very useful. > > Links > ===== > > [1]: https://youtu.be/5gT57y4OzBM?t=14234 > > Valentin Schneider (5): > trace: Add trace_ipi_send_{cpu, cpumask} > sched, smp: Trace send_call_function_single_ipi() > smp: Add a multi-CPU variant to send_call_function_single_ipi() > irq_work: Trace calls to arch_irq_work_raise() > treewide: Rename and trace arch-definitions of smp_send_reschedule() > > arch/alpha/kernel/smp.c | 2 +- > arch/arc/kernel/smp.c | 2 +- > arch/arm/kernel/smp.c | 5 +---- > arch/arm64/kernel/smp.c | 3 +-- > arch/csky/kernel/smp.c | 2 +- > arch/hexagon/kernel/smp.c | 2 +- > arch/ia64/kernel/smp.c | 4 ++-- > arch/loongarch/include/asm/smp.h | 2 +- > arch/mips/include/asm/smp.h | 2 +- > arch/openrisc/kernel/smp.c | 2 +- > arch/parisc/kernel/smp.c | 4 ++-- > arch/powerpc/kernel/smp.c | 4 ++-- > arch/riscv/kernel/smp.c | 4 ++-- > arch/s390/kernel/smp.c | 2 +- > arch/sh/kernel/smp.c | 2 +- > arch/sparc/kernel/smp_32.c | 2 +- > arch/sparc/kernel/smp_64.c | 2 +- > arch/x86/include/asm/smp.h | 2 +- > arch/xtensa/kernel/smp.c | 2 +- > include/linux/smp.h | 1 + > include/trace/events/ipi.h | 27 +++++++++++++++++++++++++++ > kernel/irq_work.c | 12 +++++++++++- > kernel/sched/core.c | 7 +++++-- > kernel/smp.c | 18 +++++++++++++++++- > 24 files changed, 84 insertions(+), 31 deletions(-) > > -- > 2.31.1 > > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 31AF6C433FE for ; Fri, 7 Oct 2022 20:04:43 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MkfRF2Cnyz3dsW for ; Sat, 8 Oct 2022 07:04:41 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=P+lIyk+e; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=aBoAisTh; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=170.10.133.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=mtosatti@redhat.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=P+lIyk+e; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=aBoAisTh; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MkfQ86Lwwz3c69 for ; Sat, 8 Oct 2022 07:03:42 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665173019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=jaDcObpjcx0whPL96jk38beBrOuRKo+eqX0wqNpM3Cc=; b=P+lIyk+e0CPKAEcXbbQsDGytM3pSE2HiXWeW7NP38w5yUSHLYGCuSuucP33CsCkagHA0Xl 151o2pP9yjDQ1PeDz0xhPVbLkmo8lCO3ZH7OMrITtTWxWfWY5mkXbEVSe9xPFuqCCFlhIl 3Pk9fHEL2zLDpOzOjIrrd39Z4gNTtAk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665173020; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=jaDcObpjcx0whPL96jk38beBrOuRKo+eqX0wqNpM3Cc=; b=aBoAisThuPu/ZF7+U3dSnEbg7djv/f4KmkoIwlBP6WxAT8t2I4r5q0iE7iDSi+J4ffWjQZ SJF2ptiBgFnSBAKSmIZbzbUHrbmOKH2qrAyvw9HlJdNaw71tZanLU4bK6Lhh+3PNx89l3m 4VQZN2aJcBrEFjV9nLXZH2NJirvXJzQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-264-kStNREYhN0OL2iSIbHz80Q-1; Fri, 07 Oct 2022 16:03:36 -0400 X-MC-Unique: kStNREYhN0OL2iSIbHz80Q-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 67B08882826; Fri, 7 Oct 2022 20:03:34 +0000 (UTC) Received: from fuller.cnet (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 29A1A2166B4D; Fri, 7 Oct 2022 20:03:32 +0000 (UTC) Received: by fuller.cnet (Postfix, from userid 1000) id 258DA416CE48; Fri, 7 Oct 2022 17:01:33 -0300 (-03) Date: Fri, 7 Oct 2022 17:01:33 -0300 From: Marcelo Tosatti To: Valentin Schneider Subject: Re: [RFC PATCH 0/5] Generic IPI sending tracepoint Message-ID: References: <20221007154145.1877054-1-vschneid@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20221007154145.1877054-1-vschneid@redhat.com> X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Juri Lelli , Mark Rutland , linux-ia64@vger.kernel.org, linux-sh@vger.kernel.org, Peter Zijlstra , Sebastian Andrzej Siewior , Dave Hansen , linux-mips@vger.kernel.org, Guo Ren , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, Marc Zyngier , linux-hexagon@vger.kernel.org, x86@kernel.org, Russell King , linux-csky@vger.kernel.org, Ingo Molnar , linux-snps-arc@lists.infradead.org, linux-xtensa@linux-xtensa.org, "Paul E. McKenney" , Frederic Weisbecker , Steven Rostedt , openrisc@lists.librecores.org, Borislav Petkov , Nicholas Piggin , loongarch@lists.linux.dev, Thomas Gleixner , linux-arm-kernel@lists.i nfradead.org, linux-parisc@vger.kernel.org, Daniel Bristot de Oliveira , linux-kernel@vger.kernel.org, linux-alpha@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, "David S. Miller" Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" Hi Valentin, On Fri, Oct 07, 2022 at 04:41:40PM +0100, Valentin Schneider wrote: > Background > ========== > > Detecting IPI *reception* is relatively easy, e.g. using > trace_irq_handler_{entry,exit} or even just function-trace > flush_smp_call_function_queue() for SMP calls. > > Figuring out their *origin*, is trickier as there is no generic tracepoint tied > to e.g. smp_call_function(): > > o AFAIA x86 has no tracepoint tied to sending IPIs, only receiving them > (cf. trace_call_function{_single}_entry()). > o arm/arm64 do have trace_ipi_raise(), which gives us the target cpus but also a > mostly useless string (smp_calls will all be "Function call interrupts"). > o Other architectures don't seem to have any IPI-sending related tracepoint. > > I believe one reason those tracepoints used by arm/arm64 ended up as they were > is because these archs used to handle IPIs differently from regular interrupts > (the IRQ driver would directly invoke an IPI-handling routine), which meant they > never showed up in trace_irq_handler_{entry, exit}. The trace_ipi_{entry,exit} > tracepoints gave a way to trace IPI reception but those have become redundant as > of: > > 56afcd3dbd19 ("ARM: Allow IPIs to be handled as normal interrupts") > d3afc7f12987 ("arm64: Allow IPIs to be handled as normal interrupts") > > which gave IPIs a "proper" handler function used through > generic_handle_domain_irq(), which makes them show up via > trace_irq_handler_{entry, exit}. > > Changing stuff up > ================= > > Per the above, it would make sense to reshuffle trace_ipi_raise() and move it > into generic code. This also came up during Daniel's talk on Osnoise at the CPU > isolation MC of LPC 2022 [1]. > > Now, to be useful, such a tracepoint needs to export: > o targeted CPU(s) > o calling context > > The only way to get the calling context with trace_ipi_raise() is to trigger a > stack dump, e.g. $(trace-cmd -e ipi* -T echo 42). > > As for the targeted CPUs, the existing tracepoint does export them, albeit in > cpumask form, which is quite inconvenient from a tooling perspective. For > instance, as far as I'm aware, it's not possible to do event filtering on a > cpumask via trace-cmd. https://man7.org/linux/man-pages/man1/trace-cmd-set.1.html -f filter Specify a filter for the previous event. This must come after a -e. This will filter what events get recorded based on the content of the event. Filtering is passed to the kernel directly so what filtering is allowed may depend on what version of the kernel you have. Basically, it will let you use C notation to check if an event should be processed or not. ==, >=, <=, >, <, &, |, && and || The above are usually safe to use to compare fields. This looks overkill to me (consider large number of bits set in mask). +#define trace_ipi_send_cpumask(callsite, mask) do { \ + if (static_key_false(&__tracepoint_ipi_send_cpu.key)) { \ + int cpu; \ + for_each_cpu(cpu, mask) \ + trace_ipi_send_cpu(callsite, cpu); \ + } \ +} while (0) > > Because of the above points, this is introducing a new tracepoint. > > Patches > ======= > > This results in having trace events for: > > o smp_call_function*() > o smp_send_reschedule() > o irq_work_queue*() > > This is incomplete, just looking at arm64 there's more IPI types that aren't covered: > > IPI_CPU_STOP, > IPI_CPU_CRASH_STOP, > IPI_TIMER, > IPI_WAKEUP, > > ... But it feels like a good starting point. Can't you have a single tracepoint (or variant with cpumask) that would cover such cases as well? Maybe (as parameters for tracepoint): * type (reschedule, smp_call_function, timer, wakeup, ...). * function address: valid for smp_call_function, irq_work_queue types. > Another thing worth mentioning is that depending on the callsite, the _RET_IP_ > fed to the tracepoint is not always useful - generic_exec_single() doesn't tell > you much about the actual callback being sent via IPI, so there might be value > in exploding the single tracepoint into at least one variant for smp_calls. Not sure i grasp what you mean by "exploding the single tracepoint...", but yes knowing the function or irq work function is very useful. > > Links > ===== > > [1]: https://youtu.be/5gT57y4OzBM?t=14234 > > Valentin Schneider (5): > trace: Add trace_ipi_send_{cpu, cpumask} > sched, smp: Trace send_call_function_single_ipi() > smp: Add a multi-CPU variant to send_call_function_single_ipi() > irq_work: Trace calls to arch_irq_work_raise() > treewide: Rename and trace arch-definitions of smp_send_reschedule() > > arch/alpha/kernel/smp.c | 2 +- > arch/arc/kernel/smp.c | 2 +- > arch/arm/kernel/smp.c | 5 +---- > arch/arm64/kernel/smp.c | 3 +-- > arch/csky/kernel/smp.c | 2 +- > arch/hexagon/kernel/smp.c | 2 +- > arch/ia64/kernel/smp.c | 4 ++-- > arch/loongarch/include/asm/smp.h | 2 +- > arch/mips/include/asm/smp.h | 2 +- > arch/openrisc/kernel/smp.c | 2 +- > arch/parisc/kernel/smp.c | 4 ++-- > arch/powerpc/kernel/smp.c | 4 ++-- > arch/riscv/kernel/smp.c | 4 ++-- > arch/s390/kernel/smp.c | 2 +- > arch/sh/kernel/smp.c | 2 +- > arch/sparc/kernel/smp_32.c | 2 +- > arch/sparc/kernel/smp_64.c | 2 +- > arch/x86/include/asm/smp.h | 2 +- > arch/xtensa/kernel/smp.c | 2 +- > include/linux/smp.h | 1 + > include/trace/events/ipi.h | 27 +++++++++++++++++++++++++++ > kernel/irq_work.c | 12 +++++++++++- > kernel/sched/core.c | 7 +++++-- > kernel/smp.c | 18 +++++++++++++++++- > 24 files changed, 84 insertions(+), 31 deletions(-) > > -- > 2.31.1 > > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A63B0C433F5 for ; Fri, 7 Oct 2022 20:04:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4T9mHkWiXmQBnirt8jvRPv4ZKR1DsK4ETw/SKqgQfn8=; b=KeGTqUwbM5rXbD UKDKHlezr0QrIPLq282xOqXj36Gfrn4554x8we7kbW9+XQnJmoWf3pAZE+0JCG6SKhFITCojo6KN0 pdO1dPBrfo/bMpp833N6PQmBAQgVy2KFbuGQjoZ+iTX0pAjmrM2vnD0BtyFpm1+2i9QelKfjNqOgR cVLZJLaEyhWZcTghIZ+iwFk2sXVJLgZCd8pRb843ZCMF7L+XuSpVSv7wB98nY0ignkotp6iAbL8KW l3C4HF6dTq/XDEKaD93MfmhUdGmnI92r+BamsZp1VIvW5QecfMdE0rMIKeqY7qt4+fGkp9bcCS5ZG BjWPcMmjA2NOSpdeRhcg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ogtZS-00AJF9-7f; Fri, 07 Oct 2022 20:03:46 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ogtZO-00AJBo-La for linux-arm-kernel@lists.infradead.org; Fri, 07 Oct 2022 20:03:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665173019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=jaDcObpjcx0whPL96jk38beBrOuRKo+eqX0wqNpM3Cc=; b=P+lIyk+e0CPKAEcXbbQsDGytM3pSE2HiXWeW7NP38w5yUSHLYGCuSuucP33CsCkagHA0Xl 151o2pP9yjDQ1PeDz0xhPVbLkmo8lCO3ZH7OMrITtTWxWfWY5mkXbEVSe9xPFuqCCFlhIl 3Pk9fHEL2zLDpOzOjIrrd39Z4gNTtAk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-264-kStNREYhN0OL2iSIbHz80Q-1; Fri, 07 Oct 2022 16:03:36 -0400 X-MC-Unique: kStNREYhN0OL2iSIbHz80Q-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 67B08882826; Fri, 7 Oct 2022 20:03:34 +0000 (UTC) Received: from fuller.cnet (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 29A1A2166B4D; Fri, 7 Oct 2022 20:03:32 +0000 (UTC) Received: by fuller.cnet (Postfix, from userid 1000) id 258DA416CE48; Fri, 7 Oct 2022 17:01:33 -0300 (-03) Date: Fri, 7 Oct 2022 17:01:33 -0300 From: Marcelo Tosatti To: Valentin Schneider Cc: linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, x86@kernel.org, "Paul E. McKenney" , Steven Rostedt , Peter Zijlstra , Thomas Gleixner , Sebastian Andrzej Siewior , Juri Lelli , Daniel Bristot de Oliveira , Frederic Weisbecker , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Marc Zyngier , Mark Rutland , Russell King , Nicholas Piggin , Guo Ren , "David S. Miller" Subject: Re: [RFC PATCH 0/5] Generic IPI sending tracepoint Message-ID: References: <20221007154145.1877054-1-vschneid@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20221007154145.1877054-1-vschneid@redhat.com> X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221007_130342_812359_837C7A68 X-CRM114-Status: GOOD ( 36.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Valentin, On Fri, Oct 07, 2022 at 04:41:40PM +0100, Valentin Schneider wrote: > Background > ========== > > Detecting IPI *reception* is relatively easy, e.g. using > trace_irq_handler_{entry,exit} or even just function-trace > flush_smp_call_function_queue() for SMP calls. > > Figuring out their *origin*, is trickier as there is no generic tracepoint tied > to e.g. smp_call_function(): > > o AFAIA x86 has no tracepoint tied to sending IPIs, only receiving them > (cf. trace_call_function{_single}_entry()). > o arm/arm64 do have trace_ipi_raise(), which gives us the target cpus but also a > mostly useless string (smp_calls will all be "Function call interrupts"). > o Other architectures don't seem to have any IPI-sending related tracepoint. > > I believe one reason those tracepoints used by arm/arm64 ended up as they were > is because these archs used to handle IPIs differently from regular interrupts > (the IRQ driver would directly invoke an IPI-handling routine), which meant they > never showed up in trace_irq_handler_{entry, exit}. The trace_ipi_{entry,exit} > tracepoints gave a way to trace IPI reception but those have become redundant as > of: > > 56afcd3dbd19 ("ARM: Allow IPIs to be handled as normal interrupts") > d3afc7f12987 ("arm64: Allow IPIs to be handled as normal interrupts") > > which gave IPIs a "proper" handler function used through > generic_handle_domain_irq(), which makes them show up via > trace_irq_handler_{entry, exit}. > > Changing stuff up > ================= > > Per the above, it would make sense to reshuffle trace_ipi_raise() and move it > into generic code. This also came up during Daniel's talk on Osnoise at the CPU > isolation MC of LPC 2022 [1]. > > Now, to be useful, such a tracepoint needs to export: > o targeted CPU(s) > o calling context > > The only way to get the calling context with trace_ipi_raise() is to trigger a > stack dump, e.g. $(trace-cmd -e ipi* -T echo 42). > > As for the targeted CPUs, the existing tracepoint does export them, albeit in > cpumask form, which is quite inconvenient from a tooling perspective. For > instance, as far as I'm aware, it's not possible to do event filtering on a > cpumask via trace-cmd. https://man7.org/linux/man-pages/man1/trace-cmd-set.1.html -f filter Specify a filter for the previous event. This must come after a -e. This will filter what events get recorded based on the content of the event. Filtering is passed to the kernel directly so what filtering is allowed may depend on what version of the kernel you have. Basically, it will let you use C notation to check if an event should be processed or not. ==, >=, <=, >, <, &, |, && and || The above are usually safe to use to compare fields. This looks overkill to me (consider large number of bits set in mask). +#define trace_ipi_send_cpumask(callsite, mask) do { \ + if (static_key_false(&__tracepoint_ipi_send_cpu.key)) { \ + int cpu; \ + for_each_cpu(cpu, mask) \ + trace_ipi_send_cpu(callsite, cpu); \ + } \ +} while (0) > > Because of the above points, this is introducing a new tracepoint. > > Patches > ======= > > This results in having trace events for: > > o smp_call_function*() > o smp_send_reschedule() > o irq_work_queue*() > > This is incomplete, just looking at arm64 there's more IPI types that aren't covered: > > IPI_CPU_STOP, > IPI_CPU_CRASH_STOP, > IPI_TIMER, > IPI_WAKEUP, > > ... But it feels like a good starting point. Can't you have a single tracepoint (or variant with cpumask) that would cover such cases as well? Maybe (as parameters for tracepoint): * type (reschedule, smp_call_function, timer, wakeup, ...). * function address: valid for smp_call_function, irq_work_queue types. > Another thing worth mentioning is that depending on the callsite, the _RET_IP_ > fed to the tracepoint is not always useful - generic_exec_single() doesn't tell > you much about the actual callback being sent via IPI, so there might be value > in exploding the single tracepoint into at least one variant for smp_calls. Not sure i grasp what you mean by "exploding the single tracepoint...", but yes knowing the function or irq work function is very useful. > > Links > ===== > > [1]: https://youtu.be/5gT57y4OzBM?t=14234 > > Valentin Schneider (5): > trace: Add trace_ipi_send_{cpu, cpumask} > sched, smp: Trace send_call_function_single_ipi() > smp: Add a multi-CPU variant to send_call_function_single_ipi() > irq_work: Trace calls to arch_irq_work_raise() > treewide: Rename and trace arch-definitions of smp_send_reschedule() > > arch/alpha/kernel/smp.c | 2 +- > arch/arc/kernel/smp.c | 2 +- > arch/arm/kernel/smp.c | 5 +---- > arch/arm64/kernel/smp.c | 3 +-- > arch/csky/kernel/smp.c | 2 +- > arch/hexagon/kernel/smp.c | 2 +- > arch/ia64/kernel/smp.c | 4 ++-- > arch/loongarch/include/asm/smp.h | 2 +- > arch/mips/include/asm/smp.h | 2 +- > arch/openrisc/kernel/smp.c | 2 +- > arch/parisc/kernel/smp.c | 4 ++-- > arch/powerpc/kernel/smp.c | 4 ++-- > arch/riscv/kernel/smp.c | 4 ++-- > arch/s390/kernel/smp.c | 2 +- > arch/sh/kernel/smp.c | 2 +- > arch/sparc/kernel/smp_32.c | 2 +- > arch/sparc/kernel/smp_64.c | 2 +- > arch/x86/include/asm/smp.h | 2 +- > arch/xtensa/kernel/smp.c | 2 +- > include/linux/smp.h | 1 + > include/trace/events/ipi.h | 27 +++++++++++++++++++++++++++ > kernel/irq_work.c | 12 +++++++++++- > kernel/sched/core.c | 7 +++++-- > kernel/smp.c | 18 +++++++++++++++++- > 24 files changed, 84 insertions(+), 31 deletions(-) > > -- > 2.31.1 > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel