From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751437AbcHQJT5 (ORCPT ); Wed, 17 Aug 2016 05:19:57 -0400 Received: from bombadil.infradead.org ([198.137.202.9]:39857 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750896AbcHQJTz (ORCPT ); Wed, 17 Aug 2016 05:19:55 -0400 Date: Wed, 17 Aug 2016 11:19:54 +0200 From: Peter Zijlstra To: Steven Rostedt , Thomas Gleixner , Ingo Molnar , Alexander Shishkin Cc: linux-kernel@vger.kernel.org Subject: [RFC] ftrace / perf 'recursion' Message-ID: <20160817091953.GH7141@twins.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23.1 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org blergh, now with LKML added... --- Much like: d525211f9d1b ("perf: Fix irq_work 'tail' recursion") I found another infinite recursion problem with irq_work: [] ? perf_output_begin_forward+0x5/0x1e0 [] ? arch_irq_work_raise+0x5/0x40 [] ? perf_event_output_forward+0x30/0x60 [] arch_irq_work_raise+0x5/0x40 [] irq_work_queue+0x97/0xa0 [] ? arch_irq_work_raise+0x5/0x40 [] ? irq_work_queue+0x97/0xa0 [] __perf_event_overflow+0xcf/0x1b0 [] perf_swevent_overflow+0x9a/0xc0 [] perf_swevent_event+0x5d/0x80 [] perf_tp_event+0x1a2/0x1b0 [] ? _raw_spin_trylock+0x30/0x30 [] ? perf_ftrace_function_call+0x83/0xd0 [] ? ftrace_ops_assist_func+0xb5/0x110 [] ? ftrace_ops_assist_func+0xb5/0x110 [] ? do_send_sig_info+0x5d/0x80 [] ? _raw_spin_trylock+0x30/0x30 [] ? perf_trace_buf_alloc+0x1f/0xa0 [] ? kill_fasync+0x6b/0x90 [] ? _raw_spin_trylock+0x30/0x30 [] ? perf_ftrace_function_call+0x83/0xd0 [] ? smp_irq_work_interrupt+0x33/0x40 [] ? irq_enter+0x70/0x70 [] perf_ftrace_function_call+0xbf/0xd0 [] ? ftrace_ops_assist_func+0xb5/0x110 [] ftrace_ops_assist_func+0xb5/0x110 [] ? smp_irq_work_interrupt+0x33/0x40 [] ? irq_enter+0x70/0x70 [] 0xffffffffa157e077 [] ? kill_fasync+0x6b/0x90 [] ? irq_exit+0x5/0xb0 [] irq_exit+0x5/0xb0 [] smp_irq_work_interrupt+0x33/0x40 [] ? irq_exit+0x5/0xb0 [] ? smp_irq_work_interrupt+0x33/0x40 [] irq_work_interrupt+0x89/0x90 Here every irq_work execution triggers another irq_work queue, which gets us stuck in an IRQ loop ad infinitum. This is through function tracing of irq_exit(), but the same can be done through function tracing of pretty much anything else around there and through the explicit IRQ_WORK_VECTOR tracepoints. The only 'solution' is something like the below, which I absolutely detest because it makes the irq_work code slower for everyone. Also, this doesn't fix the problem for any other arch :/ I would much rather tag the whole irq_work thing notrace and remove the tracepoints, but I'm sure that'll not be a popular solution either :/ --- arch/x86/kernel/irq_work.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/irq_work.c b/arch/x86/kernel/irq_work.c index 3512ba607361..a8a7999f1147 100644 --- a/arch/x86/kernel/irq_work.c +++ b/arch/x86/kernel/irq_work.c @@ -10,26 +10,41 @@ #include #include +/* + * I'm sure header recursion will bite my head off + */ +#ifdef CONFIG_PERF_EVENTS +extern int perf_swevent_get_recursion_context(void); +extern void perf_swevent_put_recursion_context(int rctx); +#else +static inline int perf_swevent_get_recursion_context(void) { return -1; } +static inline void perf_swevent_put_recursion_context(int rctx) { } +#endif + static inline void __smp_irq_work_interrupt(void) { inc_irq_stat(apic_irq_work_irqs); irq_work_run(); } -__visible void smp_irq_work_interrupt(struct pt_regs *regs) +__visible notrace void smp_irq_work_interrupt(struct pt_regs *regs) { + int rctx = perf_swevent_get_recursion_context(); ipi_entering_ack_irq(); __smp_irq_work_interrupt(); exiting_irq(); + perf_swevent_put_recursionn_context(rctx); } -__visible void smp_trace_irq_work_interrupt(struct pt_regs *regs) +__visible notrace void smp_trace_irq_work_interrupt(struct pt_regs *regs) { + int rctx = perf_swevent_get_recursion_context(); ipi_entering_ack_irq(); trace_irq_work_entry(IRQ_WORK_VECTOR); __smp_irq_work_interrupt(); trace_irq_work_exit(IRQ_WORK_VECTOR); exiting_irq(); + perf_swevent_put_recursionn_context(rctx); } void arch_irq_work_raise(void)