32-bit syscalls from 64-bit process on x86-64?

32-bit syscalls from 64-bit process on x86-64?

[Jeremy Fitzhardinge]

Hi Andi,

Is it possible for a 64-bit process to invoke the 32-bit syscall
compatibility layer?  I'm thinking this might be useful for Valgrind,
since if it is running on an x86-64 host, it can take advantage of
having more registers and a larger address space to do a better
emulation of plain ia32.  But this is only practical if I can reuse the
kernel's 32-bit emulation layer, since duplicating it in Valgrind would
be silly (particularly ioctls).

>From a quick look at the code, it seems to me that int 0x80 might still
work in 64-bit mode, but connect to 32-bit syscalls.  Is that right?  If
not, could it be made to be right?  Alternatively, something like adding
a constant offset to the syscall numbers would work for me (ie, 0-N are
64-bit syscalls, 0x10000-N are 32-bit).  Hm, no, it looks like int 0x80
just calls normal 64-bit syscalls...

And does the 32-bit layer keep any private state?  For example, if I
modify the signal state with 32-syscalls in one place, and 64-bit
syscalls elsewhere, will that cause a problem or inconsistencies?

Thanks,
	J

Re: 32-bit syscalls from 64-bit process on x86-64?

[Jeremy Fitzhardinge]

On Thu, 2004-12-02 at 08:22 -0800, I wrote:
> Hi Andi,

[ Or any other x86-64-heads. ]

	J

Re: 32-bit syscalls from 64-bit process on x86-64?

[Andi Kleen]

On Thu, Dec 02, 2004 at 08:22:00AM -0800, Jeremy Fitzhardinge wrote:
> Hi Andi,
> 
> Is it possible for a 64-bit process to invoke the 32-bit syscall
> compatibility layer?  I'm thinking this might be useful for Valgrind,
> since if it is running on an x86-64 host, it can take advantage of
> having more registers and a larger address space to do a better
> emulation of plain ia32.  But this is only practical if I can reuse the
> kernel's 32-bit emulation layer, since duplicating it in Valgrind would
> be silly (particularly ioctls).

It has been done actually (in the intel ia32el JIT)
> 
> >From a quick look at the code, it seems to me that int 0x80 might still
> work in 64-bit mode, but connect to 32-bit syscalls.  Is that right?  If
> not, could it be made to be right?  Alternatively, something like adding
> a constant offset to the syscall numbers would work for me (ie, 0-N are
> 64-bit syscalls, 0x10000-N are 32-bit).  Hm, no, it looks like int 0x80
> just calls normal 64-bit syscalls....

int 0x80 from 64bit will call 32bit syscalls. But some things
will not work properly, e.g. signals because they test the 32bit
flag in the task_struct. So you would still get 64bit signal frames.
There are some other such tests, but they probably wont affect you.

There were some plans at one point to allow to toggle the flag 
using personality or prctl, but so far it hasn't been implemented.
There is also no way to do 64bit calls from a 32bit executable.
> 
> And does the 32-bit layer keep any private state?  For example, if I
> modify the signal state with 32-syscalls in one place, and 64-bit
> syscalls elsewhere, will that cause a problem or inconsistencies?

64bit processes always get 64bit signals.

-Andi

Re: 32-bit syscalls from 64-bit process on x86-64?

[Jeremy Fitzhardinge]

On Fri, 2004-12-03 at 07:15 +0100, Andi Kleen wrote:
> int 0x80 from 64bit will call 32bit syscalls.

Hm, that's not what I found.  If I run int 0x80 with [re]ax=1, I get
exit with a 32-bit process and write with a 64-bit one.

>  But some things
> will not work properly, e.g. signals because they test the 32bit
> flag in the task_struct. So you would still get 64bit signal frames.

OK, I can deal with that, since signals are never directly delivered to
the client.

> There are some other such tests, but they probably wont affect you.
> 
> There were some plans at one point to allow to toggle the flag 
> using personality or prctl, but so far it hasn't been implemented.
> There is also no way to do 64bit calls from a 32bit executable.

What about my idea of using distinct ranges of syscall numbers to
disambiguate them?

	J

Re: 32-bit syscalls from 64-bit process on x86-64?

[Petr Vandrovec]

On Fri, Dec 03, 2004 at 03:16:29PM -0800, Jeremy Fitzhardinge wrote:
> On Fri, 2004-12-03 at 07:15 +0100, Andi Kleen wrote:
> > int 0x80 from 64bit will call 32bit syscalls.
> 
> Hm, that's not what I found.  If I run int 0x80 with [re]ax=1, I get
> exit with a 32-bit process and write with a 64-bit one.

Seems to work for me.  See second example below.  If it won't work, you
can always use reverse procedure to one I used in syscall.c - only
problem is that you'll have to allocate some memory below 4GB to
hold code & stack during 32bit syscall.
 
> >  But some things
> > will not work properly, e.g. signals because they test the 32bit
> > flag in the task_struct. So you would still get 64bit signal frames.
> 
> OK, I can deal with that, since signals are never directly delivered to
> the client.
> 
> > There are some other such tests, but they probably wont affect you.
> > 
> > There were some plans at one point to allow to toggle the flag 
> > using personality or prctl, but so far it hasn't been implemented.
> > There is also no way to do 64bit calls from a 32bit executable.
> 
> What about my idea of using distinct ranges of syscall numbers to
> disambiguate them?

What about this?  This does 64bit syscalls from 32bit executable.

Developed/tested on 2.6.10-rc2-mm2.  You may want to replace $0x33 with 
some other value on 2.4.x kernels, but otherwise it should work fine for 
you.  And well, you may want to do some additional coding so you can 
actually pass & retrieve 64bit values to/from syscalls.

When it works correctly, program should print something and exit with code
111.  If it sigsegvs or exits with 222, something went wrong.
							Petr

vana:~/64bit-test# more syscall.c
#include <stdio.h>
#include <string.h>

#define TOLM                            \
                "pushl %%cs\n"          \
                "pushl $91f\n"          \
                "ljmpl $0x33,$90f\n"    \
                ".code64\n"             \
                "90:\n"
#define FROMLM                          \
                "lretl\n"               \
                ".code32\n"             \
                "91:\n"

unsigned long x8664syscall1(unsigned long op, unsigned long value) {
        unsigned long result;

        asm volatile(
                TOLM
                "mov %1,%1\n"   /* Opteron does not need this, but em64t maybe needs */
                "mov %2,%2\n"
                "syscall\n"
                FROMLM
                : "=a"(result) : "0"(op), "D"(value));
        return result;
}

unsigned long x8664syscall3(unsigned long op, unsigned long v1, unsigned long v2, unsigned long v3) {
        unsigned long result;

        asm volatile(
                TOLM
                "mov %1,%1\n"
                "mov %2,%2\n"
                "mov %3,%3\n"
                "mov %4,%4\n"
                "syscall\n"
                FROMLM
                : "=a"(result) : "0"(op), "D"(v1), "S"(v2), "d"(v3));
        return result;
}

int main(int argc, char* argv[]) {
        printf("Write?\n");
        x8664syscall3(1 /* x86-64 __NR_write */, 1 /* stdout */, (unsigned long)"64-bit rules!\n", 14);
        printf("And now exit(111)\n");
        x8664syscall1(60 /* x86-64 __NR_exit */, 111);
        printf("Exit did not exit...\n");
        return 222;
}
vana:~/64bit-test# gcc -W -Wall -O2 -o syscall syscall.c
syscall.c: In function `main':
syscall.c:43: warning: unused parameter `argc'
syscall.c:43: warning: unused parameter `argv'
vana:~/64bit-test# ./syscall
Write?
64-bit rules!
And now exit(111)
vana:~/64bit-test# echo $?
111
vana:~/64bit-test# file syscall
syscall: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.0, dynamically linked (uses shared libs), not stripped



vana:~/64bit-test# more syscall64.c
#include <stdio.h>
#include <string.h>

unsigned long ia32syscall1(unsigned long op, unsigned long value) {
        unsigned long result;

        asm volatile(
                "int $0x80\n"
                : "=a"(result) : "0"(op), "b"(value));
        return result;
}

unsigned long ia32syscall3(unsigned long op, unsigned long v1, unsigned long v2, unsigned long v3) {
        unsigned long result;

        asm volatile(
                "int $0x80\n"
                : "=a"(result) : "0"(op), "b"(v1), "c"(v2), "d"(v3));
        return result;
}

int main(int argc, char* argv[]) {
        printf("Write?\n");
        ia32syscall3(4 /* 386 __NR_write */, 1 /* stdout */, (unsigned long)"32-bit rules!\n", 14);
        printf("And now exit(111)\n");
        ia32syscall1(1 /* 386 __NR_exit */, 111);
        printf("Exit did not exit...\n");
        return 222;
}
vana:~/64bit-test# x86_64-linux-gcc -W -Wall -O2 -o syscall64 syscall64.c
syscall64.c: In function `main':
syscall64.c:22: warning: unused parameter `argc'
syscall64.c:22: warning: unused parameter `argv'
vana:~/64bit-test# ./syscall64
Write?
32-bit rules!
And now exit(111)
vana:~/64bit-test# echo $?
111
vana:~/64bit-test# file ./syscall64
./syscall64: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped
vana:~/64bit-test#

Re: 32-bit syscalls from 64-bit process on x86-64?

[Jeremy Fitzhardinge]

On Sat, 2004-12-04 at 15:40 +0100, Petr Vandrovec wrote:
> On Fri, Dec 03, 2004 at 03:16:29PM -0800, Jeremy Fitzhardinge wrote:
> > On Fri, 2004-12-03 at 07:15 +0100, Andi Kleen wrote:
> > > int 0x80 from 64bit will call 32bit syscalls.
> > 
> > Hm, that's not what I found.  If I run int 0x80 with [re]ax=1, I get
> > exit with a 32-bit process and write with a 64-bit one.
> 
> Seems to work for me.  See second example below.  If it won't work, you
> can always use reverse procedure to one I used in syscall.c - only
> problem is that you'll have to allocate some memory below 4GB to
> hold code & stack during 32bit syscall.
>  
> > >  But some things
> > > will not work properly, e.g. signals because they test the 32bit
> > > flag in the task_struct. So you would still get 64bit signal frames.
> > 
> > OK, I can deal with that, since signals are never directly delivered to
> > the client.
> > 
> > > There are some other such tests, but they probably wont affect you.
> > > 
> > > There were some plans at one point to allow to toggle the flag 
> > > using personality or prctl, but so far it hasn't been implemented.
> > > There is also no way to do 64bit calls from a 32bit executable.
> > 
> > What about my idea of using distinct ranges of syscall numbers to
> > disambiguate them?
> 
> What about this?  This does 64bit syscalls from 32bit executable.
> 
> Developed/tested on 2.6.10-rc2-mm2.  You may want to replace $0x33 with 
> some other value on 2.4.x kernels, but otherwise it should work fine for 
> you.  And well, you may want to do some additional coding so you can 
> actually pass & retrieve 64bit values to/from syscalls.
> 
> When it works correctly, program should print something and exit with code
> 111.  If it sigsegvs or exits with 222, something went wrong.

Both of these work for me on a stock FC3 kernel (2.6.9-1.681_FC3).  Not
sure what I was doing wrong...

Is 32-from-64 syscalls considered to be a documented API - something
which I can rely on?

The 64-from-32 switch is also very interesting; I hadn't realized you
could switch modes that easily within a process.

	J

Re: 32-bit syscalls from 64-bit process on x86-64?

[Jeremy Fitzhardinge]

On Sat, 2004-12-04 at 15:40 +0100, Petr Vandrovec wrote:
> Seems to work for me.  See second example below.  If it won't work, you
> can always use reverse procedure to one I used in syscall.c - only
> problem is that you'll have to allocate some memory below 4GB to
> hold code & stack during 32bit syscall.

I've been playing with this.

If I have a 64-bit process, and I switch to the 32-bit segment with ljmp
it works fine - until I get a signal.  The kernel doesn't reload cs with
the 64-bit segment before calling the handler, which causes an obvious
mess.  I've been experimenting with setting the handler to this little
trampoline:

handler32:
.code32
	ljmp *1f
.code64
1:	.long handler	/* x86-64 handler */
	.word 0x33	/* USER_CS */

but this always gets a SIGSEGV on the ljmp.  I don't know why - there
are lots of reasons for getting a GPF, and there doesn't seem to be a
way to distinguish them.

(Hm, if I change this to the "ljmp $0x33, $handler" form, it gets to
handler, but then crashes because rsp has been truncated to 32-bits.
Hmm2, OK, so if I set up a 32-bit stack and a signal stack, it all works
OK.)

Any ideas?  Is this a kernel bug: should it always make sure that CS has
the right value before starting a signal handler?  I notice that the
ia32 kernel reloads cs before starting a signal handler.

	J

Re: 32-bit syscalls from 64-bit process on x86-64?

[Andi Kleen]

> #define TOLM                            \
>                 "pushl %%cs\n"          \
>                 "pushl $91f\n"          \
>                 "ljmpl $0x33,$90f\n"    \

It's useless, there is nothing in the kernel code that checks the 
32bit segment.

-Andi

Re: 32-bit syscalls from 64-bit process on x86-64?

[Petr Vandrovec]

On 14 Dec 04 at 8:45, Andi Kleen wrote:
> > #define TOLM                            \
> >                 "pushl %%cs\n"          \
> >                 "pushl $91f\n"          \
> >                 "ljmpl $0x33,$90f\n"    \
> 
> It's useless, there is nothing in the kernel code that checks the 
> 32bit segment.

???  Processor checks for 32bit/64bit segment.  It is impossible to load
upper 32bit of all registers with non-zero value or call 64bit
syscall entry point from 32bit mode.  As x86-64 kernel offers 64bit 
interface through syscall only, only way how to issue 64bit system call
is using syscall instruction in 64bit code.

Or are you trying to say that these samples do not work and you cannot
call 64bit entry point from 32bit app, or vice versa?  Then I'm afraid
that you are not completely right, as these samples do work...
                                                Petr Vandrovec

Re: 32-bit syscalls from 64-bit process on x86-64?

[Andi Kleen]

On Tue, Dec 14, 2004 at 11:01:12PM +0100, Petr Vandrovec wrote:
> On 14 Dec 04 at 8:45, Andi Kleen wrote:
> > > #define TOLM                            \
> > >                 "pushl %%cs\n"          \
> > >                 "pushl $91f\n"          \
> > >                 "ljmpl $0x33,$90f\n"    \
> > 
> > It's useless, there is nothing in the kernel code that checks the 
> > 32bit segment.
> 
> ???  Processor checks for 32bit/64bit segment.  It is impossible to load
> upper 32bit of all registers with non-zero value or call 64bit
> syscall entry point from 32bit mode.  As x86-64 kernel offers 64bit 
> interface through syscall only, only way how to issue 64bit system call
> is using syscall instruction in 64bit code.

Ah sorry. I misread the intention of your code. I thought you wanted
to do it the other way round - 32bit syscall from 64bit code.
I just wanted to point out that you can do it directly without
changing the code segment, as long as you use int $0x80.

>From 64bit-from-32bit the lcall is needed agreed. However as a 
warning it will not work for all calls since a few check a bit
in task_struct that says if the process is 32bit or 64bit
(rather rare though, most prominent is signal handling) 

> 
> Or are you trying to say that these samples do not work and you cannot
> call 64bit entry point from 32bit app, or vice versa?  Then I'm afraid
> that you are not completely right, as these samples do work...

I haven't ever tried it, but I see no reason it cannot work.

-Andi

Re: 32-bit syscalls from 64-bit process on x86-64?

[Jeremy Fitzhardinge]

On Wed, 2004-12-15 at 05:27 +0100, Andi Kleen wrote:
> From 64bit-from-32bit the lcall is needed agreed. However as a 
> warning it will not work for all calls since a few check a bit
> in task_struct that says if the process is 32bit or 64bit
> (rather rare though, most prominent is signal handling) 

When delivering a signal to a 64-bit process (ie, without TIF_IA32 set),
do you think it should always set cs to be USER_CS?  At the moment, if
cs is something else (ie, USER32_CS), it tries to deliver the signal
with that current...

	J

Re: 32-bit syscalls from 64-bit process on x86-64?

[Andi Kleen]

On Wed, Dec 15, 2004 at 02:50:07AM -0800, Jeremy Fitzhardinge wrote:
> On Wed, 2004-12-15 at 05:27 +0100, Andi Kleen wrote:
> > From 64bit-from-32bit the lcall is needed agreed. However as a 
> > warning it will not work for all calls since a few check a bit
> > in task_struct that says if the process is 32bit or 64bit
> > (rather rare though, most prominent is signal handling) 
> 
> When delivering a signal to a 64-bit process (ie, without TIF_IA32 set),
> do you think it should always set cs to be USER_CS?  At the moment, if
> cs is something else (ie, USER32_CS), it tries to deliver the signal
> with that current...

Hmm, in theory you could handle a 64bit signal frame from 32bit code
(just may need an assembly stub if you want the arguments). But it 
would be quite ugly agreed.

Perhaps it should force __USER_CS yes in this case, agreed.

There is a small risk of breaking someone, but it's very small.

I can do that change if you want.

BTW the long term plan is to get rid of the special cases to make
it easierto use the 32bit kernel ABI from a 64bit program.
This means signal handling will likely just check the code segment
at some  point to decide if it should set up 32bit or 64bit frames
and we'll probably do similar things with the other cases 
(except exec which needs to stay this way) 

If you're interested in this I guess that could be done sooner
with some patch submissions (hint hint ;)

-Andi

Re: 32-bit syscalls from 64-bit process on x86-64?

[Jeremy Fitzhardinge]

On Wed, 2004-12-15 at 11:55 +0100, Andi Kleen wrote:
> Hmm, in theory you could handle a 64bit signal frame from 32bit code
> (just may need an assembly stub if you want the arguments). But it 
> would be quite ugly agreed.

Yes.  I've tried this out, and it works OK, but it isn't pleasing.  One
of the main problems is that the stack is likely to be above 4G, so %esp
has no useful value, and when you switch to 64-bit mode, the top 32-bits
of %rsp become undefined.

> Perhaps it should force __USER_CS yes in this case, agreed.
> 
> There is a small risk of breaking someone, but it's very small.

Well, if they've got code which is already switching between 32 and 64
bit segments, then they need to cope with either cs being current at
delivery time.

> I can do that change if you want.
> 
> BTW the long term plan is to get rid of the special cases to make
> it easierto use the 32bit kernel ABI from a 64bit program.
> This means signal handling will likely just check the code segment
> at some  point to decide if it should set up 32bit or 64bit frames
> and we'll probably do similar things with the other cases 
> (except exec which needs to stay this way) 

At syscall time, rather than delivery time, I assume.  Hm, I'd prefer it
if it didn't look at the current segment, but at the syscall path.  Ie,
installing a handler with __NR_rt_sigaction via int 0x80 (or 32-bit
syscall/sysenter) should set up a 32-bit frame on delivery, but if the
handler was installed with the 64-bit syscall, it should be called with
a 64-bit frame.

> If you're interested in this I guess that could be done sooner
> with some patch submissions (hint hint ;)

I'll take a look.

	J

Re: 32-bit syscalls from 64-bit process on x86-64?

[Andi Kleen]

On Wed, Dec 15, 2004 at 12:58:05PM -0800, Jeremy Fitzhardinge wrote:
> On Wed, 2004-12-15 at 11:55 +0100, Andi Kleen wrote:
> > Hmm, in theory you could handle a 64bit signal frame from 32bit code
> > (just may need an assembly stub if you want the arguments). But it 
> > would be quite ugly agreed.
> 
> Yes.  I've tried this out, and it works OK, but it isn't pleasing.  One
> of the main problems is that the stack is likely to be above 4G, so %esp
> has no useful value, and when you switch to 64-bit mode, the top 32-bits
> of %rsp become undefined.

They seem to stay at the previous value in the current CPUs, but it's 
undefined behaviour yes.

> 
> > Perhaps it should force __USER_CS yes in this case, agreed.
> > 
> > There is a small risk of breaking someone, but it's very small.
> 
> Well, if they've got code which is already switching between 32 and 64
> bit segments, then they need to cope with either cs being current at
> delivery time.

Can you cope with that in valgrind? If I change it there will
be kernels with both behaviour around for a long time.

> 
> > I can do that change if you want.
> > 
> > BTW the long term plan is to get rid of the special cases to make
> > it easierto use the 32bit kernel ABI from a 64bit program.
> > This means signal handling will likely just check the code segment
> > at some  point to decide if it should set up 32bit or 64bit frames
> > and we'll probably do similar things with the other cases 
> > (except exec which needs to stay this way) 
> 
> At syscall time, rather than delivery time, I assume.  Hm, I'd prefer it
> if it didn't look at the current segment, but at the syscall path.  Ie,
> installing a handler with __NR_rt_sigaction via int 0x80 (or 32-bit
> syscall/sysenter) should set up a 32-bit frame on delivery, but if the
> handler was installed with the 64-bit syscall, it should be called with
> a 64-bit frame.

That would probably add some complications to the syscall exit path:
there is some code sharing between 32bit and 64bit, in particular 64bit
execve uses the IRET path to restore all registers and there can be a 
signal after that.

The only case that wouldn't be handled by checking the segment would
be that if someone defines a 64bit LDT code segment and uses it, it
will not work. But that won't work anyways from 64bit because SYSRET
always forces __USER_CS. If you want to do any 64bit system calls
you cannot use such custom segment anyways. 

So i don't see any drawbacks in just checking the segment for this.

-Andi