Re: Segfault on the i386 enter instruction

[Bill Davidsen <davidsen@tmr.com>]

linux-os (Dick Johnson) wrote:
> On Fri, 12 May 2006, Tomasz Malesinski wrote:
> 
>> The code attached below segfaults on the enter instruction. It works
>> when a stack frame is created by the three commented out
>> instructions and also when the first operand of the enter instruction
>> is small (less than about 6500 on my system).
>>
>> AFAIK, the only difference between creating a stack frame with the
>> enter instruction or push/mov/sub is that enter checks if the new
>> value of esp is inside the stack segment limit.
>>
>> I tested it on a vanilla kernel 2.4.26 on Intel Celeron and also on
>> probably non-vanilla 2.6.16.13 running on 3 dual core AMD Opteron,
>> quite busy, server. It is working in 32-bit mode. Interestingly, on
>> the second machine sometimes the program worked correctly.
>>
>> I am not subscribed to the list. Please cc replies to me.
>>
>>
>> 	.file	"a.c"
>> 	.version	"01.01"
>> gcc2_compiled.:
>> .section	.rodata
>> .LC0:
>> 	.string	"asdf\n"
>> .text
>> 	.align 4
>> .globl main
>> 	.type	 main,@function
>> main:
>> 	enter $10008, $0
>> #	pushl %ebp
>> #	movl %esp,%ebp
>> #	subl $10008,%esp
>> 	addl $-12,%esp
>          ^^^^^^^^^^^^^^____________ WTF
>          adding a negative number is subtracting that positive value.
Right, adding -12 is the same as subtracting 12. I have no idea what 
you're getting at with the next two lines.
>          You just subtracted 0xfffffff3 (on a 32-bit machine) from
>          the stack pointer. It damn-well better seg-fault!
No, we subtracted 12. I'm not sure where that number came from, it's the 
1's complement of 12 but I'm dead sure Linux code isn't running on any 
1's comp machines.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me