Re: [RFC PATCH 0/4] utrace: new modular infrastructure for user debug/tracing

Re: [RFC PATCH 0/4] utrace: new modular infrastructure for user debug/tracing

[Chuck Ebbert]

In-Reply-To: <20060613231000.38B76180072@magilla.sf.frob.com>

On Tue, 13 Jun 2006 16:10:00 -0700, Roland McGrath wrote:

> I have been working on for a while, and imagining for much longer,
> replacing ptrace from the ground up.  This is what I've come up with so
> far, and I'm looking for some reactions on the direction.

At least three different sets of people want to extend the syscall
tracing.  Jeff Dike posted a patch that lets you supply a bitmask of
syscalls to trace.  Renzo Davoli posted one that lets you decide, after
trapping entrance to a syscall, whether to skip the trap that would
normally be done on exit from the same call.  Charles P. Wright also
had a similar patch.  I think this needs to be done at the utrace
level -- a tracing engine couldn't add that on its own (could it?)

Renzo Davoli also posted a patch to allow "batching" of ptrace requests
and Systemptap really needs this, too.  AFAICT this can be done by writing
a custom engine.

And BTW patches 1 and 2 never made it to the list.  The ones on your
server (http://redhat.com/~roland/utrace/) don't apply cleanly due to
whitespace damage but that can be fixed by stripping trailing whitespace
from the kernel files patch(1) complains about.

-- 
Chuck

Re: [RFC PATCH 0/4] utrace: new modular infrastructure for user debug/tracing

[Frank Ch. Eigler]

Chuck Ebbert <76306.1226@compuserve.com> writes:

> [...]  Renzo Davoli also posted a patch to allow "batching" of
> ptrace requests and Systemptap really needs this, too.  AFAICT this
> can be done by writing a custom engine. [...]

Indeed, I like what I see (at least those parts I understand) in
roland's utrace code.  One missed opportunity bit appears to be any
new support for something like per-thread breakpoints.

If I correctly understand how gdb etc. work, a hit breakpoint involves
stoppage of all other threads of a process, then the breakpoint
instruction is replaced by the original one, then the thread is
single-stepped, then the breakpoint is put back, then finally all
threads are resumed.  Could utrace API provide short-lived per-thread
page copies to execute the single-stepped original instruction out of,
and avoid stopping & resuming all other threads?

- FChE

Re: [RFC PATCH 0/4] utrace: new modular infrastructure for user debug/tracing

[Daniel Jacobowitz]

On Fri, Jun 16, 2006 at 10:42:00AM -0400, Frank Ch. Eigler wrote:
> Indeed, I like what I see (at least those parts I understand) in
> roland's utrace code.  One missed opportunity bit appears to be any
> new support for something like per-thread breakpoints.
> 
> If I correctly understand how gdb etc. work, a hit breakpoint involves
> stoppage of all other threads of a process, then the breakpoint
> instruction is replaced by the original one, then the thread is
> single-stepped, then the breakpoint is put back, then finally all
> threads are resumed.  Could utrace API provide short-lived per-thread
> page copies to execute the single-stepped original instruction out of,
> and avoid stopping & resuming all other threads?

FYI: I talked with Roland about this a while ago, and got the
impression that he was interested in implementing it, but wanted to get
utrace going first.  It doesn't really relate to utrace; but it would
need a new interface however it was implemented.

GDB really does crave this feature.

-- 
Daniel Jacobowitz
CodeSourcery

Re: [RFC PATCH 0/4] utrace: new modular infrastructure for user debug/tracing

[Roland McGrath]

> At least three different sets of people want to extend the syscall
> tracing.  Jeff Dike posted a patch that lets you supply a bitmask of
> syscalls to trace.  Renzo Davoli posted one that lets you decide, after
> trapping entrance to a syscall, whether to skip the trap that would
> normally be done on exit from the same call.  Charles P. Wright also
> had a similar patch.  I think this needs to be done at the utrace
> level -- a tracing engine couldn't add that on its own (could it?)

Indeed I do think a tracing engine can and should do those things itself.
An engine's report_syscall callback can look in a bitmap that it maintains
itself and decide to return quickly without doing anything.  That callback
is only a function call or two away from where a core-level check in a
bitmap would be.  It's my expectation that the overhead of an engine
getting a callback and bailing out quickly will not be troublesome.  (I'd
guess the biggest slowdown might be just from taking the slow path in the
assembly necessary to do any kind of syscall tracing.  To optimally avoid
that would require checking the bitmap in the assembly code, and I tend to
doubt that optimization would be worth its maintenance burden.)  If the
overhead inherent in engines doing their own filtering proves burdensome,
then we could add new calls to the utrace core for setting certain special
kinds of automatic filters like a syscall bitmap.  But that is an
optimization for later; if it proves desireable, it fits cleanly as a
compatible extension of the current interface.

The latter feature, of deciding on a syscall-entry event whether ot not to
see the next syscall-exit event, is already easy to implement in an engine
and about as optimally as any specialized utrace core addition might be.
An engine's report_syscall_entry callback can store a flag that is checked
by its report_syscall_exit callback, and that is pretty simple.  If you
don't want to use a flag bit of your own somewhere (engine->data or a data
structure whose pointer you store there), there is a second way that is
pretty straightforward as well.  Each syscall-exit event is preceded by a
syscall-entry event (except when you've just attached to a threa already in
the middle of a syscall).  So your report_syscall_entry callback call
utrace_set_flags every time to change the set of events of interest to
include or exclude the syscall-exit event, knowing each new setting will
only affect the immediate corresponding syscall exit and you'll be called
again to decide afresh for the next syscall.

> Renzo Davoli also posted a patch to allow "batching" of ptrace requests
> and Systemptap really needs this, too.  AFAICT this can be done by writing
> a custom engine.

That's the idea.  Batching is one of several obvious parts of a decent
user-level interface (in contrast to the whole ptrace model).  The purpose
of the utrace layer is to make it tractable to go and write these things,
and to experiment with many of them, small and large.

> And BTW patches 1 and 2 never made it to the list.  

The patches are rather large and might have been discarded by the mailing
list software because of that.  Having noticed the 40k limit on mailed
patches in Documentation/SubmittingPatches, perhaps I shouldn't be posting
them directly.

> The ones on your server (http://redhat.com/~roland/utrace/) don't apply
> cleanly due to whitespace damage but that can be fixed by stripping
> trailing whitespace from the kernel files patch(1) complains about.

Sorry about that.  (You can also apply them fine with patch -l, which is
why I managed not to notice the problem myself the first time around.)
I've fixed the way I generate the patches, and I've verified that the new
patches now my site do apply to 2.6.17.


Thanks,
Roland

Re: [RFC PATCH 0/4] utrace: new modular infrastructure for user debug/tracing

[Albert Cahalan]

Roland McGrath writes:

> This series of patches revamps the support for user process debugging
> from the ground up, replacing the old ptrace support completely.
>
> Two major problems with ptrace are its interface and its implementation.

Yay!

Having written a debugger for Linux, I can assure everyone that
this is long overdue. We've been putting it off for years because
the code is so horrible and nasty, yet not normally used prior to
the creation of user-mode Linux.

Doing serious work with ptrace() is a horrible joke involving
interfaces that are crufty, undocumented, and insufficient.

BTW: debuggers can't get and set the signal context (with cr2!),
can't get and set LDT entries, don't get events when unshare()
succeeds, can't call ptrace() from any desired thread, can't do
a stop or start that affects more than one task, can't adjust the
target's signal data (handlers,blocked,ignored,etc.), can't adjust
file pointers and flags in the target, etc.

With great difficulty, some of the above can be hacked around by
inserting code into the target and running it. This is horribly
painful. It puts writing a debugger out of reach of many people
who might otherwise have only very basic code insertion needs
or none at all.

> system call interface is clunky to use and to extend, and makes it
> difficult to reduce the overhead of doing several operations and of
> transferring large amounts of data.

See the FreeBSD ptrace() man page. We're just being dumb.

> There is no way for more than one
> party to trace a process.  The reporting model using SIGCHLD and wait4
> is tricky to use reliably from userland, and especially hard to
> integrate with other kinds of event loop.

We needlessly make wait4/waitid/waitpid fail to accept
equivalent flags. (some may need to be inverted sense)

We don't appear to have any way to get these events into a
select() or poll(). We could use this even for non-debugger apps.

[RFC PATCH 0/4] utrace: new modular infrastructure for user debug/tracing

[Roland McGrath]

I have been working on for a while, and imagining for much longer,
replacing ptrace from the ground up.  This is what I've come up with so
far, and I'm looking for some reactions on the direction.  What I'm
proposing here is a substrate for doing wonderful new things, and I don't
yet have something built on top of it to demonstrate wonderful things.
This is intended to make it easy for lots of folks to whip up new things
and show what fancy business is possible and worthwhile.

I've separated this into four successive patches in hopes it makes it
easier to read the patches.  The intent is that each intermediate patch
yields a kernel tree that compiles and works, though after the first patch
and before the last you get ENOSYS from any ptrace call.  Patch #1 wipes
out ptrace and its cruft from the bowels of the kernel.  Patch #2 converts
the architecture-specific ptrace guts into the architecture-specific guts
for the new thing.  Patch #3 is the crux of it, the new layer for writing
debugging interfaces.  Patch #4 implements ptrace with user ABI
compatibility in terms of that layer.

These patches are relative to 2.6.16.20; I will shortly rebase to the
current 2.6.17ish tree, and post updated patches.  I don't think the old
base will make it any harder to review the new code in the patches or its
core design and implementation choices.  

There are some known loose ends in the code.  I am working on those and
will have more updates along with the rebase in a few days.  I am seeking
feedback on the patches now to identify more issues I have not already
noted in the code.

These changes require some small porting work for each architecture.  In
these patches, I have done the architecture work only for i386, x86_64, and
powerpc.  The machine-specific work is quite small and straightforward for
anyone even mildly familiar with the architecture in question.  It consists
mainly of rearranging the existing architecture code used for ptrace into
some new functions.  There should be no need for new assembly hacking or
anything like that.  I was able to do the powerpc support in a couple of
hours, and am not any kind of expert on ppc (and took quite a bit longer
just to get myself able to build and boot ppc kernels).  Anyone interested
in doing the architecture support for another machine, please contact me
and I'll be glad to help.  The steps might already be fairly obvious from
reading the arch/ changes in these patches.

At http://redhat.com/~roland/utrace/ you can always find the current state
of this work.  There you can also find a small test suite I use, and an
example module demonstrating a novel feature implemented very cheaply using
the new infrastructure.  Please send feedback to me at <roland@redhat.com>.

I've tested the new kernel APIs directly with the aforementioned (tiny)
test suite, and tested them more thoroughly by testing the ptrace
compatibility implementation based on them.  That I've tested using the gdb
test suite, on x86_64 and ppc64 for both 32-bit and 64-bit userland
(including both 32-bit and 64-bit gdb's debugging 32-bit processes on the
64-bit kernel), and on i386.  I have not tested the ppc32 kernel, but it
builds and the changed code is mostly shared with the ppc64 code that does
work, so it has a decent chance.  (For other architectures, some work is
still required even to get the kernel building again.)

---

This series of patches revamps the support for user process debugging
from the ground up, replacing the old ptrace support completely.

Two major problems with ptrace are its interface and its implementation.

The low-level code for tracing core events is directly tied into the
implementation of the ptrace system call interface.  Machine-dependent
code for accessing registers and controlling single-stepping is
intermingled with core implementation details that are actually
machine-independent and repeated across arch directories.  ptrace
interferes with the normal parent-child linkage, introducing many
corner cases that have caused trouble in the past.

The shortcomings of ptrace as a user-mode interface for debugging are
many, and well-known to those who have worked with it from the userland
side, or been involved in fixing and maintaining it in the kernel.  The
system call interface is clunky to use and to extend, and makes it
difficult to reduce the overhead of doing several operations and of
transferring large amounts of data.  There is no way for more than one
party to trace a process.  The reporting model using SIGCHLD and wait4
is tricky to use reliably from userland, and especially hard to
integrate with other kinds of event loop.  Thread event reporting is
heavy-weight and not specified with good granularity: in practice a
traced thread stops for everything.

The old ptrace implementation is removed entirely and replaced with a
modular interface layer that provides user debugging and tracing
functionality separate from any single userland interface.  Rather than
trying to come up with a single new interface to replace ptrace, this
provides a platform for higher-level code in kernel modules to provide
userland interfaces for tracing and debugging.  The ptrace system call
is provided for compatibility, written on top of the new modular layer.

Currently there are these four patches:

[PATCH 1] utrace: tracehook (die, ptrace, die)
[PATCH 2] utrace: register sets
[PATCH 3] utrace core
[PATCH 4] utrace: ptrace compatibility


Thanks,
Roland