Is r25 saved across syscalls?

Is r25 saved across syscalls?

[Rich Felker]

Hi all,
The kernel syscall entry/exit code seems to always save and restore
r25. Is this stable/documented behavior I can rely on? If there's a
reason it _needs_ to be preserved, knowing that would help convince me
it's safe to assume it will always be done. The intended usage is to
be able to make syscalls (where the syscall # is not a constant that
could be loaded with lwi) without a stack frame, as in "move $2,$25 ;
syscall".

Rich

Re: Is r25 saved across syscalls?

[Ralf Baechle]

On Sun, Sep 09, 2012 at 03:30:08PM -0400, Rich Felker wrote:

> The kernel syscall entry/exit code seems to always save and restore
> r25. Is this stable/documented behavior I can rely on? If there's a
> reason it _needs_ to be preserved, knowing that would help convince me
> it's safe to assume it will always be done. The intended usage is to
> be able to make syscalls (where the syscall # is not a constant that
> could be loaded with lwi) without a stack frame, as in "move $2,$25 ;
> syscall".

The basic design idea is that syscalls use a calling convention similar
to subroutine calls.  $25 is $t9, so a temp register which is callee saved.

So if the kernel is saving $t9 and you've been relying on that, consider
yourself lucky - there's not guarantee for that.

  Ralf

Re: Is r25 saved across syscalls?

[Rich Felker]

On Mon, Sep 10, 2012 at 07:08:30PM +0200, Ralf Baechle wrote:
> On Sun, Sep 09, 2012 at 03:30:08PM -0400, Rich Felker wrote:
> 
> > The kernel syscall entry/exit code seems to always save and restore
> > r25. Is this stable/documented behavior I can rely on? If there's a
> > reason it _needs_ to be preserved, knowing that would help convince me
> > it's safe to assume it will always be done. The intended usage is to
> > be able to make syscalls (where the syscall # is not a constant that
> > could be loaded with lwi) without a stack frame, as in "move $2,$25 ;
> > syscall".
> 
> The basic design idea is that syscalls use a calling convention similar
> to subroutine calls.  $25 is $t9, so a temp register which is callee saved.
> 
> So if the kernel is saving $t9 and you've been relying on that, consider
> yourself lucky - there's not guarantee for that.

Is there any documentation of what the kernel does guarantee? All
existing syscall-making code I've seen depends at least on r4-r7 not
being clobbered when a signal interrupts a syscall and sets it up for
restart (since the arguments still need to be there when it's
restarted), and seems to also depend on r4-r6 not being clobbered when
the syscall successfully returns (since they're not listed in the
clobber list, e.g. in uClibc's inline syscall asm). These are
requirements beyond the normal function call convention (which does
not require the callee preserve the values of r4-r7).

As for my problem, I can use r7 as the temp ("move $2,$7 ; syscall")
for syscalls with 3 or fewer args, but for the 4-arg syscall, $7 is
occupied by an argument, and I'd need to spill the syscall number to
the stack to be able to restore it if $25 is not available...

Rich

Re: Is r25 saved across syscalls?

[David Daney]

On 09/10/2012 10:22 AM, Rich Felker wrote:
> On Mon, Sep 10, 2012 at 07:08:30PM +0200, Ralf Baechle wrote:
>> On Sun, Sep 09, 2012 at 03:30:08PM -0400, Rich Felker wrote:
>>
>>> The kernel syscall entry/exit code seems to always save and restore
>>> r25. Is this stable/documented behavior I can rely on? If there's a
>>> reason it _needs_ to be preserved, knowing that would help convince me
>>> it's safe to assume it will always be done. The intended usage is to
>>> be able to make syscalls (where the syscall # is not a constant that
>>> could be loaded with lwi) without a stack frame, as in "move $2,$25 ;
>>> syscall".
>>
>> The basic design idea is that syscalls use a calling convention similar
>> to subroutine calls.  $25 is $t9, so a temp register which is callee saved.
>>
>> So if the kernel is saving $t9 and you've been relying on that, consider
>> yourself lucky - there's not guarantee for that.
>
> Is there any documentation of what the kernel does guarantee?

Not really.  The glibc souces can be used as the canonical 
implementation as we cannot break it.  glibc assumes $25 is clobbered.

> All
> existing syscall-making code I've seen depends at least on r4-r7 not
> being clobbered when a signal interrupts a syscall

This is an internal kernel implementation detail. Relying on it in 
userspace is probably not a good idea.

> and sets it up for
> restart (since the arguments still need to be there when it's
> restarted), and seems to also depend on r4-r6 not being clobbered when
> the syscall successfully returns (since they're not listed in the
> clobber list, e.g. in uClibc's inline syscall asm).

Some versions of uClibc's inline syscall asm are buggy.  So they cannot 
be used as an indication of what is supported.

> These are
> requirements beyond the normal function call convention (which does
> not require the callee preserve the values of r4-r7).

I would assume these are clobbered (from glibc sources 
ports/sysdeps/unix/sysv/linux/mips/mips64/n64/sysdep.h):

"$1", "$3", "$10", "$11", "$12", "$13", "$14", "$15", "$24", "$25", 
"hi", "lo"


David Daney

Re: Is r25 saved across syscalls?

[Rich Felker]

On Mon, Sep 10, 2012 at 11:04:55AM -0700, David Daney wrote:
> On 09/10/2012 10:22 AM, Rich Felker wrote:
> >On Mon, Sep 10, 2012 at 07:08:30PM +0200, Ralf Baechle wrote:
> >>On Sun, Sep 09, 2012 at 03:30:08PM -0400, Rich Felker wrote:
> >>
> >>>The kernel syscall entry/exit code seems to always save and restore
> >>>r25. Is this stable/documented behavior I can rely on? If there's a
> >>>reason it _needs_ to be preserved, knowing that would help convince me
> >>>it's safe to assume it will always be done. The intended usage is to
> >>>be able to make syscalls (where the syscall # is not a constant that
> >>>could be loaded with lwi) without a stack frame, as in "move $2,$25 ;
> >>>syscall".
> >>
> >>The basic design idea is that syscalls use a calling convention similar
> >>to subroutine calls.  $25 is $t9, so a temp register which is callee saved.
> >>
> >>So if the kernel is saving $t9 and you've been relying on that, consider
> >>yourself lucky - there's not guarantee for that.
> >
> >Is there any documentation of what the kernel does guarantee?
> 
> Not really.  The glibc souces can be used as the canonical
> implementation as we cannot break it.  glibc assumes $25 is
> clobbered.

OK.

> >All
> >existing syscall-making code I've seen depends at least on r4-r7 not
> >being clobbered when a signal interrupts a syscall
> 
> This is an internal kernel implementation detail. Relying on it in
> userspace is probably not a good idea.

When a restartable system call is interrupted by a signal, the kernel
must arrange for it to restart after the signal handler returns.
While some other obscure variants with trampolines are conceivable,
the canonical way to do this is to set PC back to the syscall
instruction with all the relevant registers preserved. MIPS is a bit
peculiar in that the kernel sets PC back to the _previous_ instruction
and requires that instruction to reload $2. This requirement is part
of the syscall ABI in that failure of the application to properly
reload $2 in this slot will cause unpredictable behavior when a
syscall needs to be resumed after a signal.

While I asked about preserving $25 in general, my actual concern is
about the syscall restarting situation. I don't care if the value of
$25 (or $7 in my alternate version) is lost once the syscall returns;
I only care that the value is still there if the kernel decides to
point PC back at the instruction before syscall in order to implement
restarting.

> >and sets it up for
> >restart (since the arguments still need to be there when it's
> >restarted), and seems to also depend on r4-r6 not being clobbered when
> >the syscall successfully returns (since they're not listed in the
> >clobber list, e.g. in uClibc's inline syscall asm).
> 
> Some versions of uClibc's inline syscall asm are buggy.  So they
> cannot be used as an indication of what is supported.

The code I'm looking at seems to match what you cited from glibc.

> >These are
> >requirements beyond the normal function call convention (which does
> >not require the callee preserve the values of r4-r7).
> 
> I would assume these are clobbered (from glibc sources
> ports/sysdeps/unix/sysv/linux/mips/mips64/n64/sysdep.h):
> 
> "$1", "$3", "$10", "$11", "$12", "$13", "$14", "$15", "$24", "$25",
> "hi", "lo"

OK.

Rich

Re: Is r25 saved across syscalls?

[Maciej W. Rozycki]

On Mon, 10 Sep 2012, Rich Felker wrote:

> As for my problem, I can use r7 as the temp ("move $2,$7 ; syscall")
> for syscalls with 3 or fewer args, but for the 4-arg syscall, $7 is
> occupied by an argument, and I'd need to spill the syscall number to
> the stack to be able to restore it if $25 is not available...

 If performance or some other factors require you to avoid spilling the 
syscall number to the stack or other readily-accessible (e.g. GP-relative) 
memory and the number is not a constant you could load with LI, then you 
can always store it in a call-saved register, one of $s0-$s8, that are 
guaranteed by the syscall ABI to be preserved across.

 Relying on any call-clobbered registers, including $7 to be preserved 
across a syscall is risky, to say the least, as this is not guaranteed by 
the syscall ABI.  I do wonder however why we have these instructions to 
save/restore $25 in SAVE_SOME/RESTORE_SOME.  This dates back to 2.4 at the 
very least.

 Ralf, any insights?

  Maciej

Re: Is r25 saved across syscalls?

[Kevin D. Kissell]

On 09/10/2012 05:29 PM, Maciej W. Rozycki wrote:
> I do wonder however why we have these instructions to save/restore $25
> in SAVE_SOME/RESTORE_SOME. This dates back to 2.4 at the very least.
> Ralf, any insights?
Hi, guys.  Maybe the fact that it's used for dispatching PIC calls has
something to do with it?

/K.

Re: Is r25 saved across syscalls?

[David Daney]

On 09/10/2012 06:04 PM, Kevin D. Kissell wrote:
> On 09/10/2012 05:29 PM, Maciej W. Rozycki wrote:
>> I do wonder however why we have these instructions to save/restore $25
>> in SAVE_SOME/RESTORE_SOME. This dates back to 2.4 at the very least.
>> Ralf, any insights?
> Hi, guys.  Maybe the fact that it's used for dispatching PIC calls has
> something to do with it?
>

I don't think so.  It is call clobbered in all Linux ABIs.  So there is 
never a need to save it.  It even has the pseudonym of $t9 indicating 
that it will get clobbered.

The other call clobbered registers that happen to be saved are for the 
nefarious uses of the kernel itself (a0..a6 for system call restarting), 
I think this is just left over cruft.

David Daney


> /K.
>
>
>
>

Re: Is r25 saved across syscalls?

[Rich Felker]

On Tue, Sep 11, 2012 at 01:29:52AM +0100, Maciej W. Rozycki wrote:
> On Mon, 10 Sep 2012, Rich Felker wrote:
> 
> > As for my problem, I can use r7 as the temp ("move $2,$7 ; syscall")
> > for syscalls with 3 or fewer args, but for the 4-arg syscall, $7 is
> > occupied by an argument, and I'd need to spill the syscall number to
> > the stack to be able to restore it if $25 is not available...
> 
>  If performance or some other factors require you to avoid spilling the 
> syscall number to the stack or other readily-accessible (e.g. GP-relative) 
> memory and the number is not a constant you could load with LI, then you 
> can always store it in a call-saved register, one of $s0-$s8, that are 
> guaranteed by the syscall ABI to be preserved across.

That's not possible; you'd need to save the old contents of that
register somewhere else, and that requires spilling it to the stack.

>  Relying on any call-clobbered registers, including $7 to be preserved 
> across a syscall is risky, to say the least, as this is not guaranteed by 
> the syscall ABI.

Relying on them being preserved upon return from the syscall is
"unsafe", I agree. In reality, r4-r6 are preserved, and r7 is
clobbered with the syscall error flag. But there's no fundamental
reason r4-r6 have to be preserved in this case.

On the other hand, relying on them being preserved when the kernel
resets PC to the instruction before the syscall instruction in order
to restart as syscall after a signal interrupts it is completely safe.
If it didn't restore them, the restarted syscall would be executed
with the wrong arguments.

Of course the kernel design could change to point PC at the syscall
instruction rather than the previous instruction, and arrange for the
registers (including $2) to all have their correct values for the
syscall, and then the issue would become irrelevant because the
instruction "move $2,$7" would not be executed again on restart.

> I do wonder however why we have these instructions to 
> save/restore $25 in SAVE_SOME/RESTORE_SOME.  This dates back to 2.4 at the 
> very least.
> 
>  Ralf, any insights?

I would be interested in knowing too. It goes back further than 2.4.
It seems 2.0 saved and restored ALL registers, and 2.2 dropped it down
to the current set. This past change is why I'm hesitant to rely on
behavior that's not either documented or fundamentally required.

Rich

Re: Is r25 saved across syscalls?

[Ralf Baechle]

On Sun, Sep 09, 2012 at 03:30:08PM -0400, Rich Felker wrote:

> Hi all,
> The kernel syscall entry/exit code seems to always save and restore
> r25. Is this stable/documented behavior I can rely on? If there's a
> reason it _needs_ to be preserved, knowing that would help convince me
> it's safe to assume it will always be done. The intended usage is to
> be able to make syscalls (where the syscall # is not a constant that
> could be loaded with lwi) without a stack frame, as in "move $2,$25 ;
> syscall".

Since there is no place where the syscall interface is documented other
than in the code itself, I've written a new wiki article

  http://www.linux-mips.org/wiki/Syscall

as start.  It's still lacking on the more obscure points but it at least
should have have answered your question, had it already existed when you
asked.

  Ralf

Re: Is r25 saved across syscalls?

[Ralf Baechle]

On Mon, Sep 10, 2012 at 02:37:20PM -0400, Rich Felker wrote:

> When a restartable system call is interrupted by a signal, the kernel
> must arrange for it to restart after the signal handler returns.
> While some other obscure variants with trampolines are conceivable,
> the canonical way to do this is to set PC back to the syscall
> instruction with all the relevant registers preserved. MIPS is a bit
> peculiar in that the kernel sets PC back to the _previous_ instruction
> and requires that instruction to reload $2. This requirement is part
> of the syscall ABI in that failure of the application to properly
> reload $2 in this slot will cause unpredictable behavior when a
> syscall needs to be resumed after a signal.
> 
> While I asked about preserving $25 in general, my actual concern is
> about the syscall restarting situation. I don't care if the value of
> $25 (or $7 in my alternate version) is lost once the syscall returns;
> I only care that the value is still there if the kernel decides to
> point PC back at the instruction before syscall in order to implement
> restarting.

Yes.  The kernel keeps a backup copy of $a3 around and uses it to restore
the old content of $a3 before returning to userland, even in old kernels.

A recent signal.c contains:

        if (regs->regs[2] == ERESTART_RESTARTBLOCK) {
                regs->regs[2] = current->thread.abi->restart;
                regs->regs[7] = regs->regs[26];
                regs->cp0_epc -= 4;
        }

Note that c0_epc is made to point back to the SYSCALL instruction,
not the one preceeding the SYSCALL instructions since 8f5a00eb4 [MIPS:
Sanitize restart logics] which went in for 2.6.36.

Relying on userland to reload $v0 was something ugly that Linux inherited
from god knows where and I'm happy to have gotten rid of that.

> The code I'm looking at seems to match what you cited from glibc.
> 
> > >These are
> > >requirements beyond the normal function call convention (which does
> > >not require the callee preserve the values of r4-r7).
> > 
> > I would assume these are clobbered (from glibc sources
> > ports/sysdeps/unix/sysv/linux/mips/mips64/n64/sysdep.h):
> > 
> > "$1", "$3", "$10", "$11", "$12", "$13", "$14", "$15", "$24", "$25",
> > "hi", "lo"

Which is correct but also means that the _syscallX() macros that were in
<asm/unistd.h> up to 2.6.19 were broken; the were lacking clobbers for
$25, $hi and $lo.  Unfortunately these macros were copied into many
libraries and applications.

  Ralf

Re: Is r25 saved across syscalls?

[Ralf Baechle]

On Tue, Sep 11, 2012 at 01:29:52AM +0100, Maciej W. Rozycki wrote:

>  Relying on any call-clobbered registers, including $7 to be preserved 
> across a syscall is risky, to say the least, as this is not guaranteed by 
> the syscall ABI.  I do wonder however why we have these instructions to 
> save/restore $25 in SAVE_SOME/RESTORE_SOME.  This dates back to 2.4 at the 
> very least.
> 
>  Ralf, any insights?

It dates back to the initial commit in 36ea5120 from March 27, 1998 for
2.1.90 when for the sake of better lmbench syscall latency numbers I had
introduced the concept of partial saving of a register frame.  I think it
should rather have been in SAVE_TEMP/RESTORE_TEMP instead.

  Ralf

Re: Is r25 saved across syscalls?

[Rich Felker]

On Tue, Sep 11, 2012 at 10:48:04AM +0200, Ralf Baechle wrote:
> Note that c0_epc is made to point back to the SYSCALL instruction,
> not the one preceeding the SYSCALL instructions since 8f5a00eb4 [MIPS:
> Sanitize restart logics] which went in for 2.6.36.
> 
> Relying on userland to reload $v0 was something ugly that Linux inherited
> from god knows where and I'm happy to have gotten rid of that.

So basically my whole question/concern is irrelevant to anything but
pre-2.6.36 kernels, and all of those preserve $25, so it would have
been safe to keep using $25.

Thankfully I already found another solution using an "ir" constraint
and "addu $2,$0,%2"; this assembles to "li" whenever the compiler can
do constant propagation, and if CP fails or the syscall number is not
constant, it allocates a register (either an unused argument register
or a call-preserved register since all the others are already in the
clobberlist or used as inputs).

> > The code I'm looking at seems to match what you cited from glibc.
> > 
> > > >These are
> > > >requirements beyond the normal function call convention (which does
> > > >not require the callee preserve the values of r4-r7).
> > > 
> > > I would assume these are clobbered (from glibc sources
> > > ports/sysdeps/unix/sysv/linux/mips/mips64/n64/sysdep.h):
> > > 
> > > "$1", "$3", "$10", "$11", "$12", "$13", "$14", "$15", "$24", "$25",
> > > "hi", "lo"
> 
> Which is correct but also means that the _syscallX() macros that were in
> <asm/unistd.h> up to 2.6.19 were broken; the were lacking clobbers for
> $25, $hi and $lo.  Unfortunately these macros were copied into many
> libraries and applications.

I don't think the compiler will try to cache anything in $25 itself
anyway. Normally it seems to only get used for its role in the
function call ABI. But yes, in theory this is rather problematic,
moreso that my issue of using $25 to restore $2 on restart.

Rich

Re: Is r25 saved across syscalls?

[Rich Felker]

On Tue, Sep 11, 2012 at 10:12:56AM +0200, Ralf Baechle wrote:
> On Sun, Sep 09, 2012 at 03:30:08PM -0400, Rich Felker wrote:
> 
> > Hi all,
> > The kernel syscall entry/exit code seems to always save and restore
> > r25. Is this stable/documented behavior I can rely on? If there's a
> > reason it _needs_ to be preserved, knowing that would help convince me
> > it's safe to assume it will always be done. The intended usage is to
> > be able to make syscalls (where the syscall # is not a constant that
> > could be loaded with lwi) without a stack frame, as in "move $2,$25 ;
> > syscall".
> 
> Since there is no place where the syscall interface is documented other
> than in the code itself, I've written a new wiki article
> 
>   http://www.linux-mips.org/wiki/Syscall
> 
> as start.  It's still lacking on the more obscure points but it at least
> should have have answered your question, had it already existed when you
> asked.

Thanks!

Some comments... In the table,

    $a0 ... $a2/$a7 except $a3

is unclear. Do you mean to say $a0 ... $a2 on o32 and also $a4 ... $a7
on all other ABIs? If so I think it would make sense to put those
ranges as separate lines in the table, so it's clear that the second
group are not preserved on o32 (if they were, they would also have
solved my problem).

As for

    $a3  4th syscall argument   $a3 set to 0/1 for success/error

Does the kernel guarantee 0/1, or is it 0/nonzero? This could matter
to asm programmers using the syscall ABI who want to do bit twiddling.

    Syscall restarting is a special case where $v0, $v1 and $a3 will
    stay unmodified. Even the program counter will stay unmodified so
    the same syscall will be executed again. This is something that
    does not matter to application programmers but may become visible
    in debuggers. Syscall restarting is something that is used
    internally by the kernel, for example when during a large read(2)
    syscall the kernel receives a signal.

The way syscall restarting works does matter to userspace, although
only to very low-level code. In musl (http://www.etalabs.net/musl),
the syscall routine for cancellation-point syscalls uses labels in the
asm before checking the cancellation flag and immediately after the
syscall instruction so that the signal handler that processes thread
cancellation can examine the saved program counter in the ucontext_t
it receives and determine whether the interrupted code is at a
cancellable syscall or not, with no race conditions. The glibc/NPTL
approach of wrapping a plain syscall with code to change to async
cancellation mode and back has extremely dangerous race conditions,
and my approach in musl of examining the program counter and comparing
it against asm labels is the only solution I've seen that's race-free.

Rich