From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jody Bruchon Subject: Tentative "login:" loop fix Date: Tue, 03 Mar 2015 23:07:47 -0500 Message-ID: <54F68513.2050002@jodybruchon.com> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Return-path: Sender: linux-8086-owner@vger.kernel.org List-ID: Content-Type: text/plain; charset="us-ascii"; format="flowed" To: ELKS There's this piece of code with no comments in arch/i86/mm/malloc.c: if (currentp->t_begstack > currentp->t_endbrk) if(len > currentp->t_endseg - 0x1000) { return -ENOMEM; } I don't understand why t_endseg is being reduced by 0x1000 before checking and can't find any justification for it anywhere else in the kernel code; fs/exec.c:sys_execve() sets t_endseg and only says "needed for sys_brk()" which is of no help. Documentation/text/bin_formats.txt mentions 0x1000 as part of an example while discussing the ELKS executable file format but it doesn't quite seem to be related. From what I can gather, ELKS uses 0x1000 as a data segment offset in some places but I can't see any of those being related to what's going on in sys_brk()... One thing is for certain: this check is causing the frustrating problem with the "login:" prompt that never ends. Removing the offset in the check fixes the login loop problem (which has plagued ELKS for a long time) and I can't seem to trigger any bad behavior in any random userspace programs with the check removed. The t_endseg check code was originally introduced when ELKS executable format support was put into CVS in 2003 or so. If anyone knows why the 0x1000 reduction was there, please explain it to me. Otherwise I'm leaving the change I've already committed in place which fixes this problem for now so we can worry about other things. -Jody