Re: [TOOL] kprobestest : Kprobe stress test tool

[Masami Hiramatsu <mhiramat@redhat.com>]

Frederic Weisbecker wrote:
>> Most of them can be fixed just by adding __kprobes.
>> Some of them which are already in the another section, kprobes
>> should check the symbols are in the section.
>
>
> You mean the blacklist?
>
> I also fear that putting bad kprobed functions into the kprobe
> section or into the blacklist may hide some kprobe internal bugs.
>
> Doing so is indeed mandatory for functions that trigger tracing
> recursion of things like that, but what if kprobe has an internal
> bug that only triggers while probe a certain class of function.
>
> Ie: it would be nice to identify the reason of the crash for
> each culprit in these lists.
 >
 > That may even help to find the others in advance.

Indeed, actually I've found some bugs while making jump-optimization
patches by using this stress test.
But some of them are obviously what we just forget to add __kprobes,
since those will be called from kprobes int3 handling functions.

And also, many lock-related code has been changed. I think
kprobes should use raw_*_lock, or prohibit to probe lock monitoring
functions like lockdep, because it will cause recursive call.

>
> Also kprobes seems to be a very fragile feature (that's what
> this selftest unearthes at least for me).
> And it really needs a recursion detection that stops every kprobing
> while reaching a given threshold of recursion. Something
> that would dump the stack and the falling kprobe structure.

Hmm, kprobes already has recursion detection(kp->nmiss), so
maybe, we can check it.

>
> That would avoid such hard lockups and also help to identify
> the dangerous symbols to probe.
>
>
>
>>> The problem is that I don't have any serial line in this
>>> box then I can't catch any crash log.
>>> My K7 testbox also died in my arms this afternoon.
>>>
>>> But I still have two other testboxes (one P2 and one P3),
>>> hopefully I could reproduce the problem in these boxes
>>> in which I can connect a serial line.
>>
>> Thank you for helping me to find it!
>>
>>> I've pushed your patches in the following git tree:
>>>
>>> git://git.kernel.org/pub/scm/linux/kernel/git/fgrederic/random-tracing.git \
>>> 	tracing/kprobes
>>>
>>> So you can send patches on top of this one.
>>
>> Great! I've found another trivial bugs, so I'll fix those on it.
>
> Cool :)
>
> Btw, here is the result of your stress test in a PIII (attaching the log
> and the config).

Thanks, I'll check that.

Thank you,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com