2.6.x Fork Problem?

2.6.x Fork Problem?

[Torin Ford]

All,
   We've got an application (we'll call it AppB) based on apache and a
proprietary module that we push to other Linux boxes.  The way we do this is
first push a proprietary executable (we'll call it AppA) to the Linux box
via rexec or some other mechanism.  We then send messages to AppA and tell
it to pull AppB from some place.  AppA will pull AppB and run its installer,
and then starts AppB via a shell script.  So process wise, we have a tree
like this:

init
  |
  |-AppA
  |
  |-AppB

Once AppB is running, the apache module will periodically call fork and one
of the execs to execute some other process.  AppB will execute code like
this to fork the process:

pid = fork();
switch (pid)
{
   case -1:
      blah; /* big trouble */
      break;
   case 0: /* Child */
      blah;
      blah;
      exec(some command here)
      blah; /* If we get here, we're in big trouble. */
      break;
   default: /* Parent */
      pid2 = waitpid(pid, &status, 0);
      if (pid2 == -1)
      {
         blah;  /* check out errno */
      }
}

On 2.4.x, this code works great.  But now with 2.6.x (stock SuSE 9.1 and FC2
kernels, as well as home grown kernels), waitpid will always return -1, and
errno will be 10 (ECHILD, No child processes).  We know for a fact that the
fork and exec calls succeed due to debug print outs.  Here's the real
kicker.  We only see this problem when we first push the software and start
it.  If we push the software and start it, stop it and start it again, then
allow it to try the fork/exec again, it succeeds.  We have the same problem
if we bundle the entire application (AppA and AppB) into an installer and
use that to install it on a machine.  The installer will start the
application, and the fork/exec calls will fail with ECHILD.  If we then stop
the App and start it again, everything is fine.

I've widdled the code down to just do this:

pid = fork();
switch (pid)
{
   case -1:
      blah; /* big trouble */
      break;
   case 0: /* Child */
      exit(1);
      break;
   default: /* Parent */
      pid2 = waitpid(pid, &status, 0);
      if (pid2 == -1)
      {
         blah;  /* check out errno */
      }
}

and I get the same results, so I now the exec has nothing to do with it.

Anyone have any ideas on why this would happen?

Thanks,

Torin Ford
Venturi Technology Partners

Re: 2.6.x Fork Problem?

[Alan Cox]

On Iau, 2004-08-12 at 01:01, Torin Ford wrote:
> I've widdled the code down to just do this:
> 
> pid = fork();
> switch (pid)
> {
>    case -1:
>       blah; /* big trouble */
>       break;
>    case 0: /* Child */
>       exit(1);
>       break;
>    default: /* Parent */
>       pid2 = waitpid(pid, &status, 0);
>       if (pid2 == -1)
>       {
>          blah;  /* check out errno */
>       }
> }
> 
> and I get the same results, so I now the exec has nothing to do with it.

Well I see two oddities in the example. You call exit() not _exit() so
the child will duplicate various queued stdio of the parent. Doesn't
seem to be relevant however.

Secondly and I suspect of importance you don't do anything with SIGCLD
so you are inheriting a random status. If the child signal is being
ignored then it will be cleared automatically. In that situation your
code functionality depends solely upon which thread runs first.

Re: 2.6.x Fork Problem?

[Jesse Pollard]

On Wednesday 11 August 2004 19:01, Torin Ford wrote:
> All,
>    We've got an application (we'll call it AppB) based on apache and a
> proprietary module that we push to other Linux boxes.  The way we do this
> is first push a proprietary executable (we'll call it AppA) to the Linux
> box via rexec or some other mechanism.  We then send messages to AppA and
> tell it to pull AppB from some place.  AppA will pull AppB and run its
> installer, and then starts AppB via a shell script.  So process wise, we
> have a tree like this:
>
> init
>
>   |-AppA
>   |
>   |-AppB
>
> Once AppB is running, the apache module will periodically call fork and one
> of the execs to execute some other process.  AppB will execute code like
> this to fork the process:
>
> pid = fork();
> switch (pid)
> {
>    case -1:
>       blah; /* big trouble */
>       break;
>    case 0: /* Child */
>       blah;
>       blah;
>       exec(some command here)
>       blah; /* If we get here, we're in big trouble. */
>       break;
>    default: /* Parent */
>       pid2 = waitpid(pid, &status, 0);
>       if (pid2 == -1)
>       {
>          blah;  /* check out errno */
>       }
> }
>
> On 2.4.x, this code works great.  But now with 2.6.x (stock SuSE 9.1 and
> FC2 kernels, as well as home grown kernels), waitpid will always return -1,
> and errno will be 10 (ECHILD, No child processes).  We know for a fact that
> the fork and exec calls succeed due to debug print outs.  Here's the real
> kicker.  We only see this problem when we first push the software and start
> it.  If we push the software and start it, stop it and start it again, then
> allow it to try the fork/exec again, it succeeds.  We have the same problem
> if we bundle the entire application (AppA and AppB) into an installer and
> use that to install it on a machine.  The installer will start the
> application, and the fork/exec calls will fail with ECHILD.  If we then
> stop the App and start it again, everything is fine.
>
> I've widdled the code down to just do this:
>
> pid = fork();
> switch (pid)
> {
>    case -1:
>       blah; /* big trouble */
>       break;
>    case 0: /* Child */
>       exit(1);
>       break;
>    default: /* Parent */
>       pid2 = waitpid(pid, &status, 0);
>       if (pid2 == -1)
>       {
>          blah;  /* check out errno */
>       }
> }
>
> and I get the same results, so I now the exec has nothing to do with it.
>
> Anyone have any ideas on why this would happen?

Yup - the parent process executed waitpid before the child process finished 
the setup. This can happen in a multi-cpu environment or even a single, if
the scheduler puts the parent process higher than the child in the queue.

I seem to remember there was a change in the fork handling in 2.5 that allowed
this to happen. It does allow for faster server response since the parent 
(assumed to be a server) can spin on the socket requests and fork handler 
servers without a delay between the fork and re-examining the incoming 
request socket.

The way I've been doing the delay is a signal handler (SIGCHLD) to catch the
signal, then a "while( 0 < (pid = waitpid(-1,&stat, WNOHANG)))..." loop to
catch any/all children. The parent process ends up sleeping in a select until
the signal occurs (and the handler executes). Then resumes.

This way the parent process is waiting for ANY child to terminate, not just
the one specified, and it waits for a signal - which can't be generated until
the/a process fully exists and terminates.

Re: 2.6.x Fork Problem?

[Frank van Maarseveen]

On Thu, Aug 12, 2004 at 09:26:27AM -0500, Jesse Pollard wrote:
> On Wednesday 11 August 2004 19:01, Torin Ford wrote:
> >
> > pid = fork();
> > switch (pid)
> > {
> >    case -1:
> >       blah; /* big trouble */
> >       break;
> >    case 0: /* Child */
> >       exit(1);
> >       break;
> >    default: /* Parent */
> >       pid2 = waitpid(pid, &status, 0);
> >       if (pid2 == -1)
> >       {
> >          blah;  /* check out errno */
> >       }
> > }
> 
> Yup - the parent process executed waitpid before the child process finished 
> the setup. This can happen in a multi-cpu environment or even a single, if
> the scheduler puts the parent process higher than the child in the queue.

ugh! I can follow the rationale for SMP.

But wouldn't this kind of behavior actually break most real world programs?

-- 
Frank

Re: 2.6.x Fork Problem?

[Richard B. Johnson]

On Fri, 13 Aug 2004, Frank van Maarseveen wrote:

> On Thu, Aug 12, 2004 at 09:26:27AM -0500, Jesse Pollard wrote:
> > On Wednesday 11 August 2004 19:01, Torin Ford wrote:
> > >
> > > pid = fork();
> > > switch (pid)
> > > {
> > >    case -1:
> > >       blah; /* big trouble */
> > >       break;
> > >    case 0: /* Child */
> > >       exit(1);
> > >       break;
> > >    default: /* Parent */
> > >       pid2 = waitpid(pid, &status, 0);
> > >       if (pid2 == -1)
> > >       {
> > >          blah;  /* check out errno */
> > >       }
> > > }
> >
> > Yup - the parent process executed waitpid before the child process finished
> > the setup. This can happen in a multi-cpu environment or even a single, if
> > the scheduler puts the parent process higher than the child in the queue.
>
> ugh! I can follow the rationale for SMP.
>
> But wouldn't this kind of behavior actually break most real world programs?
>
> --
> Frank

When fork() returns with a pid. The pid is valid. It cannot
be that the child doesn't exist yet. This would, as you said,
break everything. What seems to happening is the child calls
exit() before the parent gets the CPU. The child should wait
in exit() until somebody claims its status.

There has never been any guarantee that the child gets the CPU
sooner than the parent. If the parent and child need to synchronize
things, there are lots of ways to do it.

In the above code there is something missing. in the code shown,
the child __will__ wait in exit() until somebody claims its status.
However, the child probably did a setsid(), becoming a process-leader
or the parent set up a SIGCHLD handler before the fork. In these
cases, the exit() will quickly exit because somebody will claim
the exit status.

So, by the time the parent gets the CPU, the child is long gone.
The solution is to use the default SIGCHLD handler if the parent
expects to get the child's status and for the child to not execute
setsid(), which will allow init to reap its status.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: 2.6.x Fork Problem?

[Frank van Maarseveen]

On Fri, Aug 13, 2004 at 03:36:34PM -0400, Richard B. Johnson wrote:
> 
> In the above code there is something missing. in the code shown,
> the child __will__ wait in exit() until somebody claims its status.
> However, the child probably did a setsid(), becoming a process-leader
> or the parent set up a SIGCHLD handler before the fork. In these
> cases, the exit() will quickly exit because somebody will claim
> the exit status.
> 
> So, by the time the parent gets the CPU, the child is long gone.
> The solution is to use the default SIGCHLD handler if the parent
> expects to get the child's status and for the child to not execute
> setsid(), which will allow init to reap its status.

AFAIK a child doing setsid() has no effect whatsoever on any wait*()
done by the parent. It just sets a new session leader.

But SIGCHLD set to SIG_IGN instead of SIG_DFL is a perfect explanation.
Rereading alan's reply I suddenly got it: "random status" didn't refer
to the &status arg but to the signal status. SIG_IGN is inherited I
guess so a

	signal(SIGCHLD, SIG_DFL);

once before the fork() should fix it. Hmm, so actually our parent should
have reset SIGCHLD before exec'ing this code. This could cause more
problems.

-- 
Frank

Re: 2.6.x Fork Problem?

[Alan Cox]

On Gwe, 2004-08-13 at 20:09, Frank van Maarseveen wrote:
> > Yup - the parent process executed waitpid before the child process finished 
> > the setup. This can happen in a multi-cpu environment or even a single, if
> > the scheduler puts the parent process higher than the child in the queue.
> 
> ugh! I can follow the rationale for SMP.

Such a behaviour would not be rational, so we don't do anything like
that.

Re: 2.6.x Fork Problem?

[Torin Ford]

On Fri, 13 Aug 2004, Frank van Maarseveen wrote:

> On Fri, Aug 13, 2004 at 03:36:34PM -0400, Richard B. Johnson wrote:
> > 
> > In the above code there is something missing. in the code shown,
> > the child __will__ wait in exit() until somebody claims its status.
> > However, the child probably did a setsid(), becoming a process-leader
> > or the parent set up a SIGCHLD handler before the fork. In these
> > cases, the exit() will quickly exit because somebody will claim
> > the exit status.
> > 
> > So, by the time the parent gets the CPU, the child is long gone.
> > The solution is to use the default SIGCHLD handler if the parent
> > expects to get the child's status and for the child to not execute
> > setsid(), which will allow init to reap its status.
> 
> AFAIK a child doing setsid() has no effect whatsoever on any wait*()
> done by the parent. It just sets a new session leader.
> 
> But SIGCHLD set to SIG_IGN instead of SIG_DFL is a perfect explanation.
> Rereading alan's reply I suddenly got it: "random status" didn't refer
> to the &status arg but to the signal status. SIG_IGN is inherited I
> guess so a
> 
> 	signal(SIGCHLD, SIG_DFL);
> 
> once before the fork() should fix it. Hmm, so actually our parent should
> have reset SIGCHLD before exec'ing this code. This could cause more
> problems.
> 
> 

Thanks to everyone for their responses.  Alan and others were correct in 
that we needed to change the signal handler for SIGCHLD.  If I remember 
correctly, apache changes the signal handler for SIGCHLD, so we would have 
inherited those signal handlers from apache.  Adding a signal(SIGCHLD, 
SIG_DFL) before the fork fixed the problem for us.

Torin Ford
Venturi Technology Partners