An autrace that follows forks

An autrace that follows forks

[John D. Ramsdell]

I plan to write a version of autrace that follows forks.  When run in
follow forks mode, it will ptrace the child.  Each time the child or
one of its descendents creates a new process, the program will use the
API provided by libaudit to add an audit rule for the freshly created
process.  If I hear of interest in this capability, I'll prepare a
patch against the most recent audit source release, and share.

Last summer, someone else in my company made a simple version of a
fork following tracer based on the audit library; however it does not
compile on my FC5 machine.  The reason is he was including the
kernel's headers for ptrace, which defines PTRACE_O_TRACEFORK and
other symbols not part of the standard ptrace header.  I don't like
the idea of requiring kernel headers to compile the extended version
autrace.

I studied the strace source code, in particular, strace.c and
process.c.  I ran the preprocessor on these files using the options
created on an FC5 machine, and my comments refer to the preprocessed
version of these source files.

When asked to follow forks, in addition to calling ptrace with the
PTRACE_TRACEME option in the child, strace uses procfs to find the
task ID of the process PID in the directory /proc/PID/task.
Apparently, it's the only file name in the directory that parses as a
positive number.  Using ptrace to attach to the task ID allows strace
to follow forks.  I'm guessing that I have to set a ptrace option so
that the parent is signaled on every descendent system call, and
inspect each system call to see if a new process was created.  I bet
following more of the strace logic will allow me to figure out these
details.  

I'm worried that blindly following strace's ways may lead me astray.
Is there someplace I can find high-level documentation on when to use
ptrace and when to use the procfs?  Should I also study the GDB
sources?

John

Re: An autrace that follows forks

[Steve Grubb]

On Wednesday 11 October 2006 16:06, John D. Ramsdell wrote:
> I plan to write a version of autrace that follows forks.

This is a problem that requires a kernel side implementation. We've talked 
about it in the past and I don't think we've got code yet. We did add audit 
by ppid which helps a little. But we need a generic way to say that we want 
to audit a specific program and all its descendants.

-Steve

Re: An autrace that follows forks

[John D. Ramsdell]

Steve Grubb <sgrubb@redhat.com> writes:

> On Wednesday 11 October 2006 16:06, John D. Ramsdell wrote:
> > I plan to write a version of autrace that follows forks.
> 
> This is a problem that requires a kernel side implementation.

Do you mean this is a problem that requires a kernel side
implementation to do it well?  Ptracing the descendants has the down
side of changing the behavior of an application due to all the tracing
signals, but until a kernel side implementation is available, the
ptracing solution seems to me to be the only way to get the audit data
we desire.  Or do you mean the idea of using ptrace to follow forks is
flawed for some reason, and will not work?

One quick question, I notice autrace.c invokes /sbin/auditctl to
change audit rules, but shouldn't it being using audit_add_rule and
friends instead?  I'll implement this change if you want me to.

John

Re: An autrace that follows forks

[Steve Grubb]

On Thursday 12 October 2006 06:51, John D. Ramsdell wrote:
> One quick question, I notice autrace.c invokes /sbin/auditctl to
> change audit rules, but shouldn't it being using audit_add_rule and
> friends instead?

It was using auditctl just for expedience when the program was written. I can 
add the code to switch it over.

-Steve

Re: An autrace that follows forks

[John D. Ramsdell]

Steve Grubb <sgrubb@redhat.com> writes:

> It was using auditctl just for expedience when the program was
> written. I can add the code to switch it over.

I wrote code that uses libaudit to add rules.  After performing "yum
install audit-libs-devel", I was pleased to find a manual page for
audit_add_rule, but the description of the return value says to refer
to audit_send.  However, I found no manual page for audit_send.
Studying code made me realize the error condition is <= 0, not < 0, so
it's important not to guess.

John

static int audit_sys; /* File descriptor for socket to audit system */

static int			/* Create a rule for a pid */
init_rule(struct audit_rule *r, pid_t pid)
{				/* Returns zero on success */
  char field[PID_FIELD_SIZE];
  if (snprintf(field, PID_FIELD_SIZE, "pid=%d", pid) >= PID_FIELD_SIZE) {
    fprintf(stderr, "Internal error in init_rule\n");
    exit(1);
  }
  memset(r, 0, sizeof(*r));
  if (audit_rule_syscallbyname(r, "all") < 0) {
    fprintf(stderr, "Illegal syscall name for audit rule for %d\n", pid);
    return 1;
  }
  if (audit_rule_fieldpair(r, field, AUDIT_FILTER_ENTRY)) {
    fprintf(stderr, "Cannot add field to audit rule for %d\n", pid);
    return 1;
  }
  return 0;
}

/* Equivalent to '/sbin/auditctl -a entry,always -F pid=%d -S all' */
static int
add_rule(pid_t pid)		/* Returns zero on success */
{
  int i;
  for (i = 0; i < nchildren; i++)
    if (pid == children[i])	/* Rule already present */
      return 0;
  if (nchildren >= MAX_CHILDREN) {
    fprintf(stderr, "Too many children\n");
    return 1;
  }
  struct audit_rule r[1];
  if (init_rule(r, pid))
    return 1;
  if (audit_add_rule(audit_sys, r, AUDIT_FILTER_ENTRY, AUDIT_ALWAYS) <= 0) {
    fprintf(stderr, "Cannot add an audit rule for %d\n", pid);
    return 1;
  }
  children[nchildren++] = pid;
  return 0;
}

Re: An autrace that follows forks

[James Antill]

[-- Attachment #1.1: Type: text/plain, Size: 1214 bytes --]

On Wed, 2006-10-11 at 16:06 -0400, John D. Ramsdell wrote:

> Last summer, someone else in my company made a simple version of a
> fork following tracer based on the audit library; however it does not
> compile on my FC5 machine.  The reason is he was including the
> kernel's headers for ptrace, which defines PTRACE_O_TRACEFORK and
> other symbols not part of the standard ptrace header.  I don't like
> the idea of requiring kernel headers to compile the extended version
> autrace.

 Note that although the FC5 kernel-headers package might well not be
updated, it's very likely that FC5 already has a kernel that supports
the PTRACE_O_ options[1]. So, personally, I'd just have something like:

#ifndef PTRACE_O_TRACEFORK
# define PTRACE_O_TRACEFORK 0x00000002 /* from kernel-2.6.x */
#endif

...and assuming you handle the error return from PTRACE_SETOPTIONS, it
should mostly just work on all relevant systems.

[1] git-annotate says that flag was added on: 2005-04-16 15:20:36

-- 
James Antill - <james.antill@redhat.com>
setsockopt(fd, IPPROTO_TCP, TCP_CONGESTION, ...);
setsockopt(fd, IPPROTO_TCP, TCP_DEFER_ACCEPT, ...);
setsockopt(fd, SOL_SOCKET,  SO_ATTACH_FILTER, ...);


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: An autrace that follows forks

[John D. Ramsdell]

James Antill <james.antill@redhat.com> writes:

>  Note that although the FC5 kernel-headers package might well not be
> updated, it's very likely that FC5 already has a kernel that supports
> the PTRACE_O_ options[1]. So, personally, I'd just have something like:
> 
> #ifndef PTRACE_O_TRACEFORK
> # define PTRACE_O_TRACEFORK 0x00000002 /* from kernel-2.6.x */
> #endif

Ah, and also do the same for PTRACE_EVENT_FORK and PTRACE_GETEVENTMSG.
I'm guessing using these options makes it so that signals are only
sent when a process is forked, not at every system call.

I grep'd the kernel documentation directory but didn't find much of
interest on ptrace.  I haven't looked at the kernel sources yet.
Googling on PTRACE_EVENT_FORK and the like keeps leading me to GDB, so
I plan to look at the sources.

http://www.cygwin.com/ml/gdb-patches/2003-08/msg00274.html

Thanks for your help.

John

Re: An autrace that follows forks

[John D. Ramsdell]

James Antill <james.antill@redhat.com> writes:

> So, personally, I'd just have something like:
> 
> #ifndef PTRACE_O_TRACEFORK
> # define PTRACE_O_TRACEFORK 0x00000002 /* from kernel-2.6.x */
> #endif
> 
> ...and assuming you handle the error return from PTRACE_SETOPTIONS, it
> should mostly just work on all relevant systems.

The file gdb/linux_nat.c shows that GDB uses this trick, and many
other useful ones.  Time to cut-and-paste.

John

Re: An autrace that follows forks

[John D. Ramsdell]

I realize others on this list aren't interested in a version of
autrace that follows forks using ptrace, but I thought I'd share
tricks I learned for following forks.  Once the traced child is
started, the parent goes into a loop waiting for signals.  The tricky
part seems to be that the SIGTRAP generated by the parent's immediate
child has to be converted to a SIGSTOP before continuing the child.

static int			/* Watch all children */
watch(pid_t pid)		/* This process' child is pid  */
{				/* Function returns an exit code */
  if (add_rule(pid))
    return 1;
  for (;;) {
    int status;
    pid = wait_for_it(&status);
    if (pid < 0) {
      if (errno == ECHILD)	/* No children to wait for */
	return 0;		/* Declare success */
      perror("wait");
      return 1;
    }
    if (WIFSTOPPED(status)) {
      int signal = WSTOPSIG(status);
      if (signal == SIGTRAP) {	/* Tracing causes this signal */
	unsigned long msg;
	if (geteventmsg(pid, &msg) < 0) {
	  perror("ptrace(PTRACE_GETEVENTMSG, ...)");
	  return 1;
	}
	pid_t child = (pid_t)msg;
	if (child) {
	  /* The child of each traced fork is noted here */
	  if (add_rule(child))
	    return 1;
	}
	/* Only this process' child gets to this location, and just
	   one time */
	else if (setoptions(pid, PTRACE_O_TRACEFORK) < 0) {
	  perror("ptrace(PTRACE_SETOPTIONS, ...)");
	  return 1;
	}
	else
	  signal = SIGSTOP;
      }
      if (restart(pid, signal) < 0) {
	perror("ptrace(PTRACE_CONT, ...)");
	return 1;
      }
    }
  }
}

Some of these tricks were learned from code written by someone who
until recently worked at my company.  There are no comments in his
code describing the origin of these ideas.

John

Re: An autrace that follows forks

[John D. Ramsdell]

ramsdell@mitre.org (John D. Ramsdell) writes:

> The tricky part seems to be that the SIGTRAP generated by the
> parent's immediate child has to be converted to a SIGSTOP before
> continuing the child.

If you trace a shell, you find out you must always convert a SIGTRAP
to a SIGSTOP.

John

> static int			/* Watch all children */
> watch(pid_t pid)		/* This process' child is pid  */
> {				/* Function returns an exit code */
>   if (add_rule(pid))
>     return 1;
>   for (;;) {
>     int status;
>     pid = wait_for_it(&status);
>     if (pid < 0) {
>       if (errno == ECHILD)	/* No children to wait for */
> 	return 0;		/* Declare success */
>       perror("wait");
>       return 1;
>     }
>     if (WIFSTOPPED(status)) {
>       int signal = WSTOPSIG(status);
>       if (signal == SIGTRAP) {	/* Tracing causes this signal */
        signal = SIGSTOP;
> 	unsigned long msg;
> 	if (geteventmsg(pid, &msg) < 0) {
> 	  perror("ptrace(PTRACE_GETEVENTMSG, ...)");
> 	  return 1;
> 	}
> 	pid_t child = (pid_t)msg;
> 	if (child) {
> 	  /* The child of each traced fork is noted here */
> 	  if (add_rule(child))
> 	    return 1;
> 	}
> 	/* Only this process' child gets to this location, and just
> 	   one time */
> 	else if (setoptions(pid, PTRACE_O_TRACEFORK) < 0) {
> 	  perror("ptrace(PTRACE_SETOPTIONS, ...)");
> 	  return 1;
> 	}
 	/* Wrong
>       else
> 	  signal = SIGSTOP;
        */
>       }
>       if (restart(pid, signal) < 0) {
> 	perror("ptrace(PTRACE_CONT, ...)");
> 	return 1;
>       }
>     }
>   }
> }