Tentative "login:" loop fix

Tentative "login:" loop fix

[Jody Bruchon]

There's this piece of code with no comments in arch/i86/mm/malloc.c:

if (currentp->t_begstack > currentp->t_endbrk)
  if(len > currentp->t_endseg - 0x1000) {
     return -ENOMEM;
}

I don't understand why t_endseg is being reduced by 0x1000 before 
checking and can't find any justification for it anywhere else in the 
kernel code; fs/exec.c:sys_execve() sets t_endseg and only says "needed 
for sys_brk()" which is of no help. Documentation/text/bin_formats.txt 
mentions 0x1000 as part of an example while discussing the ELKS 
executable file format but it doesn't quite seem to be related. From 
what I can gather, ELKS uses 0x1000 as a data segment offset in some 
places but I can't see any of those being related to what's going on in 
sys_brk()...

One thing is for certain: this check is causing the frustrating problem 
with the "login:" prompt that never ends. Removing the offset in the 
check fixes the login loop problem (which has plagued ELKS for a long 
time) and I can't seem to trigger any bad behavior in any random 
userspace programs with the check removed.

The t_endseg check code was originally introduced when ELKS executable 
format support was put into CVS in 2003 or so. If anyone knows why the 
0x1000 reduction was there, please explain it to me. Otherwise I'm 
leaving the change I've already committed in place which fixes this 
problem for now so we can worry about other things.

-Jody

Re: Tentative "login:" loop fix

[Alan Cox]

On Tue, 03 Mar 2015 23:07:47 -0500
Jody Bruchon <jody@jodybruchon.com> wrote:

> There's this piece of code with no comments in arch/i86/mm/malloc.c:
> 
> if (currentp->t_begstack > currentp->t_endbrk)
>   if(len > currentp->t_endseg - 0x1000) {
>      return -ENOMEM;
> }
> 
> I don't understand why t_endseg is being reduced by 0x1000 before 
> checking and can't find any justification for it anywhere else in the 
> kernel code; fs/exec.c:sys_execve() sets t_endseg and only says "needed 
> for sys_brk()" which is of no help. Documentation/text/bin_formats.txt 
> mentions 0x1000 as part of an example while discussing the ELKS 
> executable file format but it doesn't quite seem to be related. From 
> what I can gather, ELKS uses 0x1000 as a data segment offset in some 
> places but I can't see any of those being related to what's going on in 
> sys_brk()...
> 
> One thing is for certain: this check is causing the frustrating problem 
> with the "login:" prompt that never ends. Removing the offset in the 
> check fixes the login loop problem (which has plagued ELKS for a long 
> time) and I can't seem to trigger any bad behavior in any random 
> userspace programs with the check removed.

Why does it cause the login prompt problem ?

Removing the check fixes it but you don't say why that cures it, just
that it does.

Alan