sysenter cause a segmentation fault.

sysenter cause a segmentation fault.

[Ying Chu]

A simple code running on PIII 800,kernel 2.6.20 with SEP enabled
causes  a  segmentation fault.
The slice is as following.

#include <stdio.h>
#include <sys/syacall.h>

int main()
{
    unsigned long ret=0;
    __asm__ __volatile__ ("movl %1,%%eax\n\tpushl %%ecx\n\t"
     "pushl %%edx\n\tpush %%ebp\n\t"
     "movl %%esp,%%ebp\n\t"     "sysenter\n\t"
     "movl %%eax,%0"
     :"=m"(ret)
     :"b"(SYS_getuid));
   printf("%d\n",ret);
   return 0;
}

More, when i enabled Kdb and set "bp sysenter_entry",it went with a
catastrophic error errupted.

Re: sysenter cause a segmentation fault.

[Andi Kleen]

"Ying Chu" <xjtuychu@gmail.com> writes:

> A simple code running on PIII 800,kernel 2.6.20 with SEP enabled
> causes  a  segmentation fault.

SYSENTER has a hardcoded return address, that is why you cannot be used
anywhere but in the vsyscall trampoline.  In addition it doesn't work
on x86-64 for 32bit processes for AMD. The only system call method
supported without the vDSO is int 0x80. Better you just use the vDSO.

> More, when i enabled Kdb and set "bp sysenter_entry",it went with a
> catastrophic error errupted.

The stack and some other kernel state are not set up yet at this point.

-Andi