Re: 2.4 Ready list - Kernel Hooks

Re: 2.4 Ready list - Kernel Hooks

[Richard J Moore]

Werner Almesberger wrote:

>Richard J Moore wrote:
>> Kernel Hooks is also ready,
>
>I'm a bit puzzled as to what those hooks accomplish. They look
>like a less flexible but a little faster and more portable
>variant of kprobes.
>
>Is this what they are ? If yes, does it really make sense to
>have two so similar mechanisms for tapping into execution flows
>in the kernel ?
>
>- Werner

Hello Werner, the two things are different:

kprobes
-------

This is kernel interface that allows kernel modules register to register
one or more probes.
A probes comprises a breakpoint location, a breakpoint handler and a post
single-step handler.
Why use the term probes? Because we don't intend to hijack the system,
merely register a location where we can seamlessly gather data and
continue.
The sequence of events that occurs when code containing a probepoint
executes are:
1) The associated probe handler is invoked.
2) The probe handler returns.
3) The probed instruction is single-stepped.
4) The post-single-step handler is called.
5) The post-single-step handler returns.
6) The probed code continues execution.

kprobes confines itself to kernel-space probepoints, which are implemented
using a breakpoint instruction.
There are three incremental patches that Vamsi submitted today which extend
krpobes as follows:
1) debug register management - provides a kernel interface for debug
register allocation and deallocation so debuggers can co-exists e.g
kprobes, ptrace, kdb etc..
2) kwatch points - allows probes to be set using debug registers. This
allows probes to fire on data accesses for example.
3) user space probes - this extends kprobes to be able to set probepoints
in user space. Note probes are tracked by inode and offset so that they are
global and relative to a module. This distinguishes kprobes user-space
probes from ptrace implemented breakpoints.


dprobes
-------
The four kprobes patches is almost equivalent to dprobes. Dprobes provides
a generic RPN interpreter in which to define probe handler actions. We
decided that RPN interpreter should be separated out from the breakpointing
mechanism. It's just an example probe handler and can exist outside the
kernel. Also having a set of callable interfaces is more flexible than just
having an RPN language to define probe handlers. The dprobes project has
evolved in to kprobes + a sample device driver that provide the generic RPN
probe handler.

kernel hooks
------------
This is nothing more than a call-back mechanism such as could be used by
LSM or LTT. The call-backs have to be statically coded into the source
unlike kprobes where the call-back to a probe handler is implemented via a
debug interrupt from a watchpoint or dynamically implanted INT3. We created
kernel hooks for exactly the same reasons that LSM needs hooks - to allow
ancillary function to exist outside the kernel, to avoid kernel bloat, to
allow more than one function to be called from a given call-back (think of
kdb and kprobes - both need to be called from do_debug).

Yes both kprobes and kernel hooks implement call-backs, but using INT3 to
call functions is not the most efficient call mechanism, whereas implanting
call back dynamically for debugging purposes is a tad more difficult if
done by patching in a jmp or call instruction.

It's a case of horses for courses. kprobes is a debugging facility; kernel
hooks is a static call-back mechanism.

Richard

Re: 2.4 Ready list - Kernel Hooks

[Greg KH]

On Wed, Oct 23, 2002 at 12:09:38AM +0100, Richard J Moore wrote:
> We created
> kernel hooks for exactly the same reasons that LSM needs hooks - to allow
> ancillary function to exist outside the kernel, to avoid kernel bloat, to
> allow more than one function to be called from a given call-back (think of
> kdb and kprobes - both need to be called from do_debug).

No, that is NOT the same reason LSM needs hooks!  LSM hooks are there to
mediate access to various kernel objects, from within the kernel itself.
Please do not confuse LSM with any of the above projects.

thanks,

greg k-h

Re: 2.4 Ready list - Kernel Hooks

[Werner Almesberger]

Richard J Moore wrote:
> This is nothing more than a call-back mechanism such as could be used by
> LSM or LTT.

Hmm, Greg has already voiced some violent disagreement regarding
LSM :-) That leaves LTT. Given the more exploratory nature of LTT,
I wonder if [dk]probes wouldn't be quite sufficient there, too.

Is the idea that people would deploy hooks locally, i.e. while
profiling or debugging, or that some hooks would be put permanently
in the kernel ? I can envision some rather nasty coding habits
developing if the latter would be used extensively. (INTERCAL has
"COME FROM", COBOL has "ALTER", ... ;-)

By the way, those hooks look like an excellent mechanism for
circumventing the GPL, so you might want to export them with
EXPORT_SYMBOL_GPL.

> Yes both kprobes and kernel hooks implement call-backs, but using INT3 to
> call functions is not the most efficient call mechanism,

Oh, you could probably have some "fast" probes by just checking
for a certain "anchor" pattern (e.g. a sequence of 5 nops on
i386), which could then be replaced with a direct call. This
optimization would have to be optional, in case some code yields
the anchor pattern such that it isn't also a basic block.

Hooks would still have the advantage of easier access to local
variables, of course.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: 2.4 Ready list - Kernel Hooks

[Karim Yaghmour]

Werner Almesberger wrote:
> Richard J Moore wrote:
> > This is nothing more than a call-back mechanism such as could be used by
> > LSM or LTT.
> 
> Hmm, Greg has already voiced some violent disagreement regarding
> LSM :-) That leaves LTT. Given the more exploratory nature of LTT,
> I wonder if [dk]probes wouldn't be quite sufficient there, too.

The whole point of tracing is that the system's behavior should not
be modified but only recorded. Generating int3 won't do.

> Oh, you could probably have some "fast" probes by just checking
> for a certain "anchor" pattern (e.g. a sequence of 5 nops on
> i386), which could then be replaced with a direct call. This
> optimization would have to be optional, in case some code yields
> the anchor pattern such that it isn't also a basic block.

If I remember correctly, the optimized arch-dependent code in kernel
hooks uses "compare immediate" and the value of the immediate is
edited to enable/disable hooking. Given modern branch-prediction the
cost should be quite close to an unconditional jump.

Karim

===================================================
                 Karim Yaghmour
               karim@opersys.com
      Embedded and Real-Time Linux Expert
===================================================

Re: 2.4 Ready list - Kernel Hooks

[Richard J Moore]

Greg KH wrote:
>Please read the LSM documentation for more information about this.  It
>can be found in the kernel at:
>            Documentation/DocBook/lsm.*
>and there are a number of USENIX and OLS papers about different aspects
>of the project at:


Thanks Greg. I'll check out the doc. I do remember posting the LSM mailing
list about kernel hooks, but as I recall there was no response. I assumed
that the hooking mechanism was not the focus of attention - that was
18months ago. A few weeks ago Suparna told me LSM had been enquiring about
kernel hooks - never heard the outcome though.

Richard

Re: 2.4 Ready list - Kernel Hooks

[Greg KH]

On Thu, Oct 24, 2002 at 05:38:12PM +0100, Richard J Moore wrote:
> 
> A few weeks ago Suparna told me LSM had been enquiring about kernel
> hooks - never heard the outcome though.

Wrong type of "hooks".  Ours would not work for what you are stating you
need to do, sorry.

thanks,

greg k-h

Re: 2.4 Ready list - Kernel Hooks

[Richard J Moore]

> Is the idea that people would deploy hooks locally, i.e. while
> profiling or debugging, or that some hooks would be put permanently
> in the kernel ?

Our principle reasons for using hooks is:

1) We simplify the integration of related facilities that would share a
number of common hook points, e.g. kdb, dprobes, ltt etc
2) We don't bloat the kernel with these feature but still have the ability
to turn them on dynamically when the need (or the pain) is sufficient for
us to do something about it.
2a) we can reduce the overhead of the extra function when dormant to almost
nil if it can be unhooked from the kernel.
3) We used them during development to extricate a function from the kernel
into a loadable module. This avoided many reboots and kernel builds.


>By the way, those hooks look like an excellent mechanism for
>circumventing the GPL, so you might want to export them with
>EXPORT_SYMBOL_GPL.

We already do that.

I don't envisage having an arbitrary set of hook points scattered
throughout the kernel. It's only when, for example, dprobes needed certain
hooks that we added them.



Richard

Re: 2.4 Ready list - Kernel Hooks

[Werner Almesberger]

Richard J Moore wrote:
>>EXPORT_SYMBOL_GPL.
> 
> We already do that.

Oops, missed that one, sorry ! I was looking at the interface
functions, but making the hooks themselves GPL-only is even
better.

> I don't envisage having an arbitrary set of hook points scattered
> throughout the kernel.

Let's hope you're right :-)

But wouldn't a small extension of kprobes get you pretty much
the same functionality/performance:

- at busy attachment points, add a "kprobe anchor", which
  translates to five NOPs [1,2], preceded by a global symbol
- when setting a kprobe, check if the five bytes starting
  at p->addr are NOPs [3]
- if yes, insert a call to kprobes_fastpath. if not, use
  the current double breakpoint mechanism
- kprobes_fastpath can just return to the caller, no code
  modification or single-stepping required

[1] Assuming i386.
[2] Or any sufficiently unlikely sequence of instructions that
    executes faster than NOPs.
[3] Or some other pattern - but a quick look at the kernel binary
    suggests that all strings of five or more NOPs are used for
    padding between function, so it would be safe to assume that
    any such sequence is a basic block.

The advantage over hooks would be that users of this mechanism
wouldn't have to choose between fast but intrusive (hooks) and
slow but flexible (probes).

Now, it's non-trivial to do a "return from caller" with
[kd]probes. I haven't looked at that part yet. Do you have the
infrastructure for this ?

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: 2.4 Ready list - Kernel Hooks

[Vamsi Krishna S .]

On Wed, Oct 23, 2002 at 09:50:15PM +0000, Werner Almesberger wrote:
> Oops, missed that one, sorry ! I was looking at the interface
> functions, but making the hooks themselves GPL-only is even
> better.
>
Yes, I have done that in the latest patch.
 
> > I don't envisage having an arbitrary set of hook points scattered
> > throughout the kernel.
> 
> Let's hope you're right :-)
> 
kernelhooks is similar to notifier lists (include/linux/notifier.h),
only much faster when there are no users. This patch does not
add any hooks itself, I am sure placement of each hook will be
critically reviewed.

> But wouldn't a small extension of kprobes get you pretty much
> the same functionality/performance:
>
> <snip nice idea>
>
Yes, this is possible, but I think using hooks is much cleaner.

> The advantage over hooks would be that users of this mechanism
> wouldn't have to choose between fast but intrusive (hooks) and
> slow but flexible (probes).
> 
As I see it, hooks should be used for allowing other kernel code
to tap into certain well defined paths in the kernel, say in
trap 3 or trap 1 handlers in the kernel to allow multiple 
kernel-level breakpointing tools. Or, certain well defined paths
(potentially fast paths) for traceing purposes, where it is 
necessary to ensure that for the most time there are no users 
of these hooks and their placement alone should place minimal 
overhead.

So, hooks are designed, placed at well thought-out locations.
Probes OTOH are mostly ad-hoc. While debugging a problem, if
you find the need to probe a specific code location for more
info, put a probe there, on the fly, with out going through 
the recompile and reboot cycle.

> Now, it's non-trivial to do a "return from caller" with
> [kd]probes. I haven't looked at that part yet. Do you have the
> infrastructure for this ?
> 
No, returning from caller will be much harder with [kd]probes.

Hope this clarifies the issue.

Thanks,
Vamsi.
-- 
Vamsi Krishna S.
Linux Technology Center,
IBM Software Lab, Bangalore.
Ph: +91 80 5044959
Internet: vamsi@in.ibm.com

Re: 2.4 Ready list - Kernel Hooks

[Werner Almesberger]

Vamsi Krishna S . wrote:
> So, hooks are designed, placed at well thought-out locations.
> Probes OTOH are mostly ad-hoc.

Yes, my point was that the same (general) mechanism should be
suitable for both types of use. However, ...

>> [kd]probes. I haven't looked at that part yet. Do you have the
>> infrastructure for this ?
>> 
> No, returning from caller will be much harder with [kd]probes.

... this seems to kill my grand unified hook/probe theory :-(

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: 2.4 Ready list - Kernel Hooks

[Richard J Moore]

>On Wed, Oct 23, 2002 at 12:09:38AM +0100, Richard J Moore wrote:
>> We created
>> kernel hooks for exactly the same reasons that LSM needs hooks - to
allow
>> ancillary function to exist outside the kernel, to avoid kernel bloat,
to
>> allow more than one function to be called from a given call-back (think
of
>> kdb and kprobes - both need to be called from do_debug).
>
>No, that is NOT the same reason LSM needs hooks!  LSM hooks are there to
>mediate access to various kernel objects, from within the kernel itself.
>Please do not confuse LSM with any of the above projects.
>
>thanks,
>
>greg k-h

I would have to understand what you meant by "mediate between various
kernel objects" to know whether LSM's need for hooks is radically different
to RAS needs. Can you explain further?


Richard

Re: 2.4 Ready list - Kernel Hooks

[Greg KH]

On Wed, Oct 23, 2002 at 09:10:13AM +0100, Richard J Moore wrote:
> 
> >On Wed, Oct 23, 2002 at 12:09:38AM +0100, Richard J Moore wrote:
> >> We created
> >> kernel hooks for exactly the same reasons that LSM needs hooks - to allow
> >> ancillary function to exist outside the kernel, to avoid kernel bloat, to
> >> allow more than one function to be called from a given call-back (think of
> >> kdb and kprobes - both need to be called from do_debug).
> >
> >No, that is NOT the same reason LSM needs hooks!  LSM hooks are there to
> >mediate access to various kernel objects, from within the kernel itself.
> >Please do not confuse LSM with any of the above projects.
> 
> I would have to understand what you meant by "mediate between various
> kernel objects" to know whether LSM's need for hooks is radically different
> to RAS needs. Can you explain further?

Please read the LSM documentation for more information about this.  It
can be found in the kernel at:
	Documentation/DocBook/lsm.*
and there are a number of USENIX and OLS papers about different aspects
of the project at:
	lsm.immunix.org

In the beginning of the LSM project, both the DProbes and LTT groups
came asking that we use their patches to implement LSM.  It was
quickly determined that the types of "hooks" these projects offered was
not what the LSM group needed (or wanted).  So the current
implementation was developed.

Hope this helps.  If you have any further questions, please feel free to
ask (after reading that documentation :)

thanks,

greg k-h

2.4 Ready list - Kernel Hooks

[Richard J Moore]

Kernel Hooks is also ready,
see: http://www-124.ibm.com/linux/projects/kernelhooks/


Richard
RAS Project Lead - IBM Linux Technology Centre

Re: 2.4 Ready list - Kernel Hooks

[Werner Almesberger]

Richard J Moore wrote:
> Kernel Hooks is also ready,

I'm a bit puzzled as to what those hooks accomplish. They look
like a less flexible but a little faster and more portable
variant of kprobes.

Is this what they are ? If yes, does it really make sense to
have two so similar mechanisms for tapping into execution flows
in the kernel ?

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/