[PATCH 0/3][RFC] ftrace: track dynamic ftrace update failures

[PATCH 0/3][RFC] ftrace: track dynamic ftrace update failures

[Abhishek Sagar]

Hi Steven/Ingo,

In dynamic ftrace any record which fails gets freed to be recycled. This
happens normally during (although not limited to) tracing of init functions.
In general, failures can happen due to multiple reason, such as external
patching of kernel functions, instrumentation of the mcount calls-sites,
hardware error etc. As an example, in the case of kprobes if a probe is
installed on the mcount call site (before being converted to NOP or after) then
eventually ftrace will detect it and free the corresponding record. Kprobes
however will keep on single-sepping the instruction it installed the probe on.

These patches prevent freeing of records which have failed (except for init
functions). This prevents re-recording of failed functions and allows tracking
them so that they can be listed using a new debugfs file -> 'failures'. The
main change here is preventing removal of ftrace records from ftace_hash. This
way records can be looked up by their 'ip' at any time.

--
Regards,
Abhishek Sagar

Re: [PATCH 0/3][RFC] ftrace: track dynamic ftrace update failures

[Steven Rostedt]

On Sun, 1 Jun 2008, Abhishek Sagar wrote:

>
> Hi Steven/Ingo,
>
> In dynamic ftrace any record which fails gets freed to be recycled. This
> happens normally during (although not limited to) tracing of init functions.
> In general, failures can happen due to multiple reason, such as external
> patching of kernel functions, instrumentation of the mcount calls-sites,
> hardware error etc. As an example, in the case of kprobes if a probe is
> installed on the mcount call site (before being converted to NOP or after) then
> eventually ftrace will detect it and free the corresponding record. Kprobes
> however will keep on single-sepping the instruction it installed the probe on.
>
> These patches prevent freeing of records which have failed (except for init
> functions). This prevents re-recording of failed functions and allows tracking
> them so that they can be listed using a new debugfs file -> 'failures'. The
> main change here is preventing removal of ftrace records from ftace_hash. This
> way records can be looked up by their 'ip' at any time.
>

Hi Abhishek,

Have you also taken into account adding and removing of modules? This can
cause some funny behaviours with ftrace that can be rather dangerous.

-- Steve

Re: [PATCH 0/3][RFC] ftrace: track dynamic ftrace update failures

[Abhishek Sagar]

On Mon, Jun 2, 2008 at 11:26 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> Hi Abhishek,

Hi Steven,

Thanks for reviewing my recent patches.

> Have you also taken into account adding and removing of modules? This can
> cause some funny behaviours with ftrace that can be rather dangerous.

I've tried to accommodate them (fixed the wrong use of
kernel_text_address). This makes
init and module function records eligible to be freed. Only core
kernel function update failures now get tracked/remembered.

There might still be a need to find a way to track external modifiers
of mcount call-sites such as (but not limited to) kprobes. Consider
this (rather long) contrived scenario:

1. Function foo gets recorded by ftrace.
2. Mcount call site of foo is patched with a NOP by ftraced.
3. foo gets 'enabled' by some ftrace filter rule. NOP changes to a
call to the function tracer.
4. A kprobe is registered on the mcount call-site. It places a trap
generating instruction at the
mcount call site and stores the call-to-function-tracer instruction
elsewhere internally.
5. Tracing is disabled by end user.
6. Disabling of foo fails because Kprobe had placed a trap at the
mcount call site.
7. foo still keeps getting traced because Kprobes will keep
single-stepping the call-to-function-tracer instruction each time the
mcount call site is executed.

Not sure what to do with such cases...

--
Regards,
Abhishek Sagar

Re: [PATCH 0/3][RFC] ftrace: track dynamic ftrace update failures

[Steven Rostedt]

On Tue, 3 Jun 2008, Abhishek Sagar wrote:
>
> > Have you also taken into account adding and removing of modules? This can
> > cause some funny behaviours with ftrace that can be rather dangerous.
>
> I've tried to accommodate them (fixed the wrong use of
> kernel_text_address). This makes
> init and module function records eligible to be freed. Only core
> kernel function update failures now get tracked/remembered.
>
> There might still be a need to find a way to track external modifiers
> of mcount call-sites such as (but not limited to) kprobes. Consider
> this (rather long) contrived scenario:
>
> 1. Function foo gets recorded by ftrace.
> 2. Mcount call site of foo is patched with a NOP by ftraced.
> 3. foo gets 'enabled' by some ftrace filter rule. NOP changes to a
> call to the function tracer.
> 4. A kprobe is registered on the mcount call-site. It places a trap
> generating instruction at the
> mcount call site and stores the call-to-function-tracer instruction
> elsewhere internally.
> 5. Tracing is disabled by end user.
> 6. Disabling of foo fails because Kprobe had placed a trap at the
> mcount call site.
> 7. foo still keeps getting traced because Kprobes will keep
> single-stepping the call-to-function-tracer instruction each time the
> mcount call site is executed.
>
> Not sure what to do with such cases...

Can we add an API to kprobes or perhaps a way to register code
modification in ftrace, that ftrace can ignore? Let the two know about
each other, but in a way that they dont need to know the internals of one
another.

-- Steve

Re: [PATCH 0/3][RFC] ftrace: track dynamic ftrace update failures

[Abhishek Sagar]

On Tue, Jun 3, 2008 at 6:30 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> Can we add an API to kprobes or perhaps a way to register code
> modification in ftrace, that ftrace can ignore? Let the two know about
> each other, but in a way that they dont need to know the internals of one
> another.

There's a get_kprobe() already which can tell if an address's kprobed
or not. But still doesn't stop kprobe from forcing change on mcount
sites and fooling ftrace like mentioned previously. The
problem with maintaining a blacklist sort of thing for kprobed mcount
sites may also be an
overkill since it can permanently make that function un-ftraceable.

One kprobe specific measure can be to provide notifiers in
[un]register_kprobe(). Ftrace can
listen on register event to temporarily disable tracing of the
corresponding function by forcing
a NOP at the mcount site just before a kprobe is placed on it, if it's
already recorded/enabled.
The tracing can be enabled back when the corresponding unregister
event is received. Worth a
shot?

--
Regards,
Abhishek Sagar