Re: [PATCH] Linux Kernel Markers 0.5 for Linux 2.6.17 (with probe management)

[Mathieu Desnoyers <compudj@krystal.dyndns.org>]

Good morning Ingo,

* Ingo Molnar (mingo@elte.hu) wrote:
> 
> * Mathieu Desnoyers <compudj@krystal.dyndns.org> wrote:
> 
> > I clearly expressed my position in the previous emails, so did you. 
> > You argued about a use of tracing that is not relevant to my vision of 
> > reality, which is :
> > 
> > - Embedded systems developers won't want a breakpoint-based probe
> 
> are you arguing that i'm trying to force breakpoint-based probing on 
> you? I dont. In fact i explicitly mentioned that i'd accept and support 
> a 5-byte NOP in the body of the marker, in the following mails:
> 
>     "just go for [...] the 5-NOP variant"
>       http://marc.theaimsgroup.com/?l=linux-kernel&m=115859771924187&w=2
>         (my reply to your second proposal)
> 
>     "or at most one NOP"
>       http://marc.theaimsgroup.com/?l=linux-kernel&m=115865412332230&w=2
>         (my reply to your third proposal)
> 
>     "at most a NOP inserted"
>       http://marc.theaimsgroup.com/?l=linux-kernel&m=115886524224874&w=2
>         (my reply to your fifth proposal)
> 
> That enables the probe to be turned into a function call - not an INT3 
> breakpoint. Does it take some effort to implement that on your part? 
> Yes, of course, but getting code upstream is never easy, /especially/ in 
> cases where most of the users wont use a particular feature.
> 

Some details are worth to be mentioned :

- The 5-NOP variant will imply a replacement of 5 1 bytes instructions with 1 5
  bytes one, which is trickier. Masami Hiramatsu's proposal of 2 bytes near jump
  + 3 NOPS is nicer.
- Patching such a 5-bytes instruction memory region doesn't turn markers into a
  complete function call, which includes argument passing.
- The argument "most of the users wont use a particular feature" contradicts
  what you said earlier about every distribution wanting to enable a tracing
  mechanism for their users.

> > - High performance computing users won't want a breakpoint-based probe
> 
> I am not forcing breakpoint-based probing, at all. I dont want _static, 
> build-time function call based_ probing, and there is a big difference. 
> And one reason why i want to avoid "static, build-time function call 
> based probing" is because high-performance computing users dont want any 
> overhead at all in the kernel fastpath.
> 

I think that the performance benefits gained by using tracing information for
studying a system makes the overhead of a jump in the kernel fast path
insignificant. Having a stack setup + function call already put there by the
compiler has the following advantages :

- It is very robust (I could think of using it on a live server, which is not
  true of the djprobe approach).
- It is predictable on every architecture.
- The information extracted is _always_ coherent with the marked variables,
  because the compiler itself created the full function call (stack setup
  included).


> > - djprobe is far away from being in an acceptable state on 
> >   architectures with very inconvenient erratas (x86).
> 
> djprobes over a NOP marker are perfectly usable and safe: just add a 
> simple constraint to them to only allow a djprobes insertion if it 
> replaces a 5-byte NOP.
> 

2 bytes jump + 3 bytes nops.. Yes, it should modify it without causing an
illegal instruction, but how ? Are you aware that their approach has to :
- put an int3
- wait for _all_ the CPUs to execute this int3
- then change the 5 bytes instruction

I can think of a lot of cases where the CPUs will never execute this int3.
Probably that sending an IPI or launching a kernel thread on each CPU to make
sure that this int3 is executed could give more guarantees there. But my point
is not even there : I have seen very skillful teams work hard on those
hardware-caused problems for years and the result is still not usable. It looks
to me like a race between software developers and hardware manufacturers, where
the software guy is always one step behind. This kind of scenario happens when
you want to use an architecture in a way it was not designed and tested for.

As long as CPU manufacturers won't design for live instruction patching (and why
should they do that ? the in3 breakpoint is all what is needed from their
perspective), this will be a race where software developers will lose.


> > - kprobe and djprobe cannot access local variables in every cases
> 
> it is possible with the marker mechanism i outlined before:
> 
>   http://marc.theaimsgroup.com/?l=linux-kernel&m=115886524224874&w=2
> 
> have i missed to address any concern of yours?
> 

Interesting idea. That would make it possible to probe local variables at the
marker site. That's very good for use of kprobes on low rate debug-type markers,
but that doesn't solve my concern about the cat-and-mouse race expressed earlier
about live kernel polymorphic code.

I would be all in for this kind of combo :

If you can find a way to make a kprobe-based probe extract the variables from
such a variable-dependency marked site, that would be great for dynamic of low
event rate code paths. For the high event rate, and while we wait for such a
probe to exist, I think that the load+jump over a complete call is the lowest
cost, most robust, coherent, predictable and portable mechanism I have seen
so far.


Mathieu


OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68