From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756918Ab0ELRso (ORCPT ); Wed, 12 May 2010 13:48:44 -0400 Received: from mail.openrapids.net ([64.15.138.104]:49654 "EHLO blackscsi.openrapids.net" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1756663Ab0ELRsm (ORCPT ); Wed, 12 May 2010 13:48:42 -0400 Date: Wed, 12 May 2010 13:48:40 -0400 From: Mathieu Desnoyers To: Masami Hiramatsu Cc: Ingo Molnar , lkml , systemtap , DLE , Ananth N Mavinakayanahalli , Jim Keniston , Jason Baron Subject: Re: [PATCH -tip 4/5] kprobes/x86: Use text_poke_smp_batch Message-ID: <20100512174840.GA32496@Krystal> References: <20100510175313.27396.34605.stgit@localhost6.localdomain6> <20100510175340.27396.7222.stgit@localhost6.localdomain6> <20100511144013.GA17656@Krystal> <4BE9F952.3060505@redhat.com> <20100512152747.GA12326@Krystal> <4BEAE8B8.8080809@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4BEAE8B8.8080809@redhat.com> X-Editor: vi X-Info: http://www.efficios.com X-Operating-System: Linux/2.6.26-2-686 (i686) X-Uptime: 13:46:39 up 109 days, 20:23, 10 users, load average: 0.19, 0.18, 0.17 User-Agent: Mutt/1.5.18 (2008-05-17) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org * Masami Hiramatsu (mhiramat@redhat.com) wrote: > Mathieu Desnoyers wrote: > > * Masami Hiramatsu (mhiramat@redhat.com) wrote: > >> Mathieu Desnoyers wrote: > >>> * Masami Hiramatsu (mhiramat@redhat.com) wrote: > >>>> Use text_poke_smp_batch() in optimization path for reducing > >>>> the number of stop_machine() issues. > >>>> > >>>> Signed-off-by: Masami Hiramatsu > >>>> Cc: Ananth N Mavinakayanahalli > >>>> Cc: Ingo Molnar > >>>> Cc: Jim Keniston > >>>> Cc: Jason Baron > >>>> Cc: Mathieu Desnoyers > >>>> --- > >>>> > >>>> arch/x86/kernel/kprobes.c | 37 ++++++++++++++++++++++++++++++------- > >>>> include/linux/kprobes.h | 2 +- > >>>> kernel/kprobes.c | 13 +------------ > >>>> 3 files changed, 32 insertions(+), 20 deletions(-) > >>>> > >>>> diff --git a/arch/x86/kernel/kprobes.c b/arch/x86/kernel/kprobes.c > >>>> index 345a4b1..63a5c24 100644 > >>>> --- a/arch/x86/kernel/kprobes.c > >>>> +++ b/arch/x86/kernel/kprobes.c > >>>> @@ -1385,10 +1385,14 @@ int __kprobes arch_prepare_optimized_kprobe(struct optimized_kprobe *op) > >>>> return 0; > >>>> } > >>>> > >>>> -/* Replace a breakpoint (int3) with a relative jump. */ > >>>> -int __kprobes arch_optimize_kprobe(struct optimized_kprobe *op) > >>>> +#define MAX_OPTIMIZE_PROBES 256 > >>> > >>> So what kind of interrupt latency does a 256-probes batch generate on the > >>> system ? Are we talking about a few milliseconds, a few seconds ? > >> > >> From my experiment on kvm/4cpu, it took about 3 seconds in average. > > > > That's 3 seconds for multiple calls to stop_machine(). So we can expect > > latencies in the area of few microseconds for each call, right ? > > Theoretically yes. > But if we register more than 1000 probes at once, it's hard to do > anything except optimizing a while(more than 10 sec), because > it stops machine so frequently. > > >> With this patch, it went down to 30ms. (x100 faster :)) > > > > This is beefing up the latency from few microseconds to 30ms. It sounds like a > > regression rather than a gain to me. > > If it is not acceptable, I can add a knob for control how many probes > optimize/unoptimize at once. Anyway, it is expectable latency (after > registering/unregistering probes) and it will be small if we put a few probes. > (30ms is the worst case) > And if you want, it can be disabled by sysctl. I think we are starting to see the stop_machine() approach is really limiting our ability to do even relatively small amount of work without hurting responsiveness significantly. What's the current showstopper with the breakpoint-bypass-ipi approach that solves this issue properly and makes this batching approach unnecessary ? Thanks, Mathieu > > Thank you, > > -- > Masami Hiramatsu > e-mail: mhiramat@redhat.com -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com