From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Metcalf Subject: Re: [PATCH v13 00/12] support "task_isolation" mode Date: Thu, 21 Jul 2016 10:06:56 -0400 Message-ID: References: <1468529299-27929-1-git-send-email-cmetcalf@mellanox.com> Mime-Version: 1.0 Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Christoph Lameter Cc: Gilad Ben Yossef , Steven Rostedt , Ingo Molnar , Peter Zijlstra , Andrew Morton , Rik van Riel , Tejun Heo , Frederic Weisbecker , Thomas Gleixner , "Paul E. McKenney" , Viresh Kumar , Catalin Marinas , Will Deacon , Andy Lutomirski , Daniel Lezcano , linux-doc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-api-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: linux-api@vger.kernel.org On 7/20/2016 10:04 PM, Christoph Lameter wrote: > We are trying to test the patchset on x86 and are getting strange > backtraces and aborts. It seems that the cpu before the cpu we are running > on creates an irq_work event that causes a latency event on the next cpu. > > This is weird. Is there a new round robin IPI feature in the kernel that I > am not aware of? This seems to be from your clocksource declaring itself to be unstable, and then scheduling work to safely remove that timer. I haven't looked at this code before (in kernel/time/clocksource.c under CONFIG_CLOCKSOURCE_WATCHDOG) since the timers on arm64 and tile aren't unstable. Is it possible to boot your machine with a stable clocksource? > Backtraces from dmesg: > > [ 956.603223] latencytest/7928: task_isolation mode lost due to irq_work > [ 956.610817] cpu 12: irq_work violating task isolation for latencytest/7928 on cpu 13 > [ 956.619985] CPU: 12 PID: 0 Comm: swapper/12 Not tainted 4.7.0-rc7-stream1 #1 > [ 956.628765] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.0.2 03/15/2016 > [ 956.637642] 0000000000000086 ce6735c7b39e7b81 ffff88103e783d00 ffffffff8134f6ff > [ 956.646739] ffff88102c50d700 000000000000000d ffff88103e783d28 ffffffff811986f4 > [ 956.655828] ffff88102c50d700 ffff88203cf97f80 000000000000000d ffff88103e783d68 > [ 956.664924] Call Trace: > [ 956.667945] [] dump_stack+0x63/0x84 > [ 956.674740] [] task_isolation_debug_task+0xb4/0xd0 > [ 956.682229] [] _task_isolation_debug+0x83/0xc0 > [ 956.689331] [] irq_work_queue_on+0x9c/0x120 > [ 956.696142] [] tick_nohz_full_kick_cpu+0x44/0x50 > [ 956.703438] [] wake_up_nohz_cpu+0x99/0x110 > [ 956.710150] [] internal_add_timer+0x71/0xb0 > [ 956.716959] [] add_timer_on+0xbb/0x140 > [ 956.723283] [] clocksource_watchdog+0x230/0x300 > [ 956.730480] [] ? __clocksource_unstable.isra.2+0x40/0x40 > [ 956.738555] [] call_timer_fn+0x35/0x120 > [ 956.744973] [] ? __clocksource_unstable.isra.2+0x40/0x40 > [ 956.753046] [] run_timer_softirq+0x23c/0x2f0 > [ 956.759952] [] __do_softirq+0xd7/0x2c5 > [ 956.766272] [] irq_exit+0xf5/0x100 > [ 956.772209] [] smp_apic_timer_interrupt+0x42/0x50 > [ 956.779600] [] apic_timer_interrupt+0x8c/0xa0 > [ 956.786602] [] ? poll_idle+0x40/0x80 > [ 956.793490] [] cpuidle_enter_state+0x9c/0x260 > [ 956.800498] [] cpuidle_enter+0x17/0x20 > [ 956.806810] [] cpu_startup_entry+0x2b7/0x3a0 > [ 956.813717] [] start_secondary+0x15c/0x1a0 > [ 1036.601758] cpu 12: irq_work violating task isolation for latencytest/8447 on cpu 13 > [ 1036.610922] CPU: 12 PID: 0 Comm: swapper/12 Not tainted 4.7.0-rc7-stream1 #1 > [ 1036.619692] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.0.2 03/15/2016 > [ 1036.628551] 0000000000000086 ce6735c7b39e7b81 ffff88103e783d00 ffffffff8134f6ff > [ 1036.637648] ffff88102dca0000 000000000000000d ffff88103e783d28 ffffffff811986f4 > [ 1036.646741] ffff88102dca0000 ffff88203cf97f80 000000000000000d ffff88103e783d68 > [ 1036.655833] Call Trace: > [ 1036.658852] [] dump_stack+0x63/0x84 > [ 1036.665649] [] task_isolation_debug_task+0xb4/0xd0 > [ 1036.673136] [] _task_isolation_debug+0x83/0xc0 > [ 1036.680237] [] irq_work_queue_on+0x9c/0x120 > [ 1036.687091] [] tick_nohz_full_kick_cpu+0x44/0x50 > [ 1036.694388] [] wake_up_nohz_cpu+0x99/0x110 > [ 1036.701089] [] internal_add_timer+0x71/0xb0 > [ 1036.707896] [] add_timer_on+0xbb/0x140 > [ 1036.714210] [] clocksource_watchdog+0x230/0x300 > [ 1036.721411] [] ? __clocksource_unstable.isra.2+0x40/0x40 > [ 1036.729478] [] call_timer_fn+0x35/0x120 > [ 1036.735899] [] ? __clocksource_unstable.isra.2+0x40/0x40 > [ 1036.743970] [] run_timer_softirq+0x23c/0x2f0 > [ 1036.750878] [] __do_softirq+0xd7/0x2c5 > [ 1036.757199] [] irq_exit+0xf5/0x100 > [ 1036.763132] [] smp_apic_timer_interrupt+0x42/0x50 > [ 1036.770520] [] apic_timer_interrupt+0x8c/0xa0 > [ 1036.777520] [] ? poll_idle+0x40/0x80 > [ 1036.784410] [] cpuidle_enter_state+0x9c/0x260 > [ 1036.791413] [] cpuidle_enter+0x17/0x20 > [ 1036.797734] [] cpu_startup_entry+0x2b7/0x3a0 > [ 1036.804641] [] start_secondary+0x15c/0x1a0 > > -- Chris Metcalf, Mellanox Technologies http://www.mellanox.com