Adding(?) XI support to MIPS-Linux?

Adding(?) XI support to MIPS-Linux?

[Brian Foster]

Hello,

 The MIPS 4KSd core (at least) implements an XI (eXecute Inhibit)
 page protection bit.  XI is similar to the NX (No Execute) bit
 in the more recent AMD/Intel x86 families:  Attempts to I-fetch
 from a page with XI set cause an exception.

 I've been asked to look into the possibility/difficulty of adding
 support for XI to MIPS-Linux (at least for the 4KSd core), for
 both kernel-mode and user-land.  An admittedly very quick check
 of 2.6.25(-ish) suggests there is no support at all, at present,
 for XI for any MIPS core/configuration.  Nor has a search of the
 'Net found much of anything, with the possible exception of the
 PaX and grsecurity projects.

 Of course, I don't want to repeat work which has already been done.
 Has anyone studied adding, or better yet, already (tried to?) added,
 any support for XI?

 *Speculating*, and with the caveat I'm not completely up on MIPS
 TLB/memory-management/caches, I don't see kernel-mode as too much
 of an issue.  There's no (known to me) cases of code-in-data/stack?
 Whilst I know bugger-all about module loading/unloading, I rather
 doubt there's anything too tricky going on there?

 User-land is clearly more difficult due to the `sigreturn' and FPU
 emulation trampolines, which reside in user-land stack.  At the
 moment I am presuming a `vsyscall'-page scheme solves that problem,
 but have not researched the issue (esp. the FPU emulation) closely.
 A `vsyscall' which just contains the trampolines doesn't sound too
 difficult; but if you add to that (which I am NOT proposing!) the
 optimization of volatile time-of-day data (RO to user-land, RW to
 kernel-mode), then there would seem to be issues/gotchas with MIPS
 caching and/or page-protection?

 Does `gcc' for MIPS generate any trampolines in the stack/data?
 I keep seeing hints that `gcc' does in certain unspecified cases
 and/or architectures, but am at a lost just what is being talked
 about.  Wikipedia suggests this only happens in (some?) cases of
 nested functions; as such, for C, they don't seem necessary and
 my initial inclination is say "don't use 'em!".

 I'm hoping to be able to avoid any support at all for executable
 stacks:  I want all user-land code to have XI-set stacks.  This is
 partly for simplicity (I presume), but also because the target is
 the the "secure" embedded world, where simply being able to assert
 you cannot, never, execute code in the stack (or in data) is much
 easier to pass muster with various certification authorities and
 testing laboratories.  (PCI-PED is the area of immediate interest
 here.)  I'm aware mprotect(2), at the least, is an issue, because
 it can be used to set a page to be both writable and executable.
 (Hence, more generally, I want to be able to assert that no memory
 with the exception of kseg0/kseg1, is ever concurrently writable
 and executable.)

 Advice, suggestions, pointers, comments are very welcome.

cheers!
       -blf-

-- 
"How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools." |      http://www.stopesso.com

Re: Re: Adding(?) XI support to MIPS-Linux?

[Brian Foster]

> Date: Monday 09 June 2008
> From: Andrew Dyer <adyer@righthandtech.com>
>
> Brian Foster wrote:
>>  The MIPS 4KSd core (at least) implements an XI (eXecute Inhibit)
>>  page protection bit.  XI is similar to the NX (No Execute) bit
>>  in the more recent AMD/Intel x86 families:  Attempts to I-fetch
>>  from a page with XI set cause an exception.
>
> AFAIK this is the only part with this extension, so I doubt that
> much has been done with it.
>
> I know that OpenBSD has done a bunch of work with x86 implementing
> a similar protection scheme, you might do some searching on their
> lists to see if they have any information on GCC mods required.

Andrew,

 As far as I am aware, XI is part of the SMARTMIPS extension,
 and I _think_ SMARTMIPS is only implemented by the 4KS cores?
 Whilst other companies have licensed that core from MIPS, to
 the best of my knowledge the SoC I'm concerned with (Innova
 Card's USIP Pro) is the only one running Linux.  So I suspect
 you're quite correct:  Nothing's been done.

 I've been in contact with the PaX people, and they inform me
 "the experimental NX tweaks [ for MIPS ] didn't get anywhere
 due to lack of time/interest of the guy who started it."

 The "market segment" USIP is aimed at is sufficiently touchy
 about security I currently believe it's plausible (assuming
 it's technologically possible) to simply forbid (not support)
 concurrently executable-and-writable memory.  As such, certain
 programs won't work.  Tough.  There's no call to support the
 (broken) JVM's et al. that "require" it, and I'm hoping that
 nested C/C++ functions are rare (ideally non-existent in the
 code which normally runs on USIP).  It'd be nice if such code
 either fails to compile and/or fails to link/load, but that's
 some (highly useful) porcelain.

 Broadly, what I'm trying to say is I don't want to touch gcc
 (and/or binutils) and am unconvinced I have to.  But I'm very
 much open to correction here!

 The x86 (including amd64) and, AFAIK, SuperH (sh) Linux kernels
 now support NX or equivalent; indeed, a test on my 2.6.22(-ish)
 amd64 workstation (Kubuntu 7.10) has a non-executable stack.
 As such, those could be a model worth studying/following, but
 I understand they have support for specially-marked binaries to
 have executable stacks (i.e., binutils/gcc mods, which I want to
 avoid).  It's probably worth having a look at OpenBSD (thanks
 for the suggestion!).

 Advice, suggestions, pointers, comments, corrections are very
 welcome.

cheers!
	-blf-

-- 
"How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools." |      http://www.stopesso.com

Re: Adding(?) XI support to MIPS-Linux?

[Kevin D. Kissell]

Brian Foster wrote:
> Andrew,
>
>  As far as I am aware, XI is part of the SMARTMIPS extension,
>  and I _think_ SMARTMIPS is only implemented by the 4KS cores?
>   
That is correct, though there has long been interest in having XI/RI as 
an option
for non-SmartMIPS cores and I would not be surprised if sooner or later it
became more generally available.
>  Whilst other companies have licensed that core from MIPS, to
>  the best of my knowledge the SoC I'm concerned with (Innova
>  Card's USIP Pro) is the only one running Linux.  So I suspect
>  you're quite correct:  Nothing's been done.
>   
I believe that there is at least one other 4KS-family customer working with
Linux, but they haven't been nearly as active as InnovaCard.  I had some
email exchanges with someone who was working on their kernel a couple
of years back about this very topic.  Indeed, I thought that they submitted
some patches for basic RI/XI support at one point.  Scan the linux-mips.org
archives, if they survived the rehosting.
>  I've been in contact with the PaX people, and they inform me
>  "the experimental NX tweaks [ for MIPS ] didn't get anywhere
>  due to lack of time/interest of the guy who started it."
>
>  The "market segment" USIP is aimed at is sufficiently touchy
>  about security I currently believe it's plausible (assuming
>  it's technologically possible) to simply forbid (not support)
>  concurrently executable-and-writable memory.  As such, certain
>  programs won't work.  Tough.  There's no call to support the
>  (broken) JVM's et al. that "require" it, and I'm hoping that
>  nested C/C++ functions are rare (ideally non-existent in the
>  code which normally runs on USIP).  It'd be nice if such code
>  either fails to compile and/or fails to link/load, but that's
>  some (highly useful) porcelain.
>
>  Broadly, what I'm trying to say is I don't want to touch gcc
>  (and/or binutils) and am unconvinced I have to.  But I'm very
>  much open to correction here!
>
>  The x86 (including amd64) and, AFAIK, SuperH (sh) Linux kernels
>  now support NX or equivalent; indeed, a test on my 2.6.22(-ish)
>  amd64 workstation (Kubuntu 7.10) has a non-executable stack.
>  As such, those could be a model worth studying/following, but
>  I understand they have support for specially-marked binaries to
>  have executable stacks (i.e., binutils/gcc mods, which I want to
>  avoid).
Well, strictly speaking, you wouldn't actually *need* to modify binutils
to make specially tagged binaries.  You could borrow an unused bit in
the ELF header somewhere, have the kernel recognize it, and write your
own little tool that only turns that bit on/off in an ELF file.  In the 
longer
term, I'd argue that if there's support for appropriate binary tagging in
the x86 tools, that support should simply be enabled for MIPS targets
and any other non-x86 archiectures with such support (e.g. Alpha, if
anyone still uses them).

          Regards,

          Kevin K.

Re: Adding(?) XI support to MIPS-Linux?

[Thiemo Seufer]

Kevin D. Kissell wrote:
[snip]
>>  Broadly, what I'm trying to say is I don't want to touch gcc
>>  (and/or binutils) and am unconvinced I have to.  But I'm very
>>  much open to correction here!
>>
>>  The x86 (including amd64) and, AFAIK, SuperH (sh) Linux kernels
>>  now support NX or equivalent; indeed, a test on my 2.6.22(-ish)
>>  amd64 workstation (Kubuntu 7.10) has a non-executable stack.
>>  As such, those could be a model worth studying/following, but
>>  I understand they have support for specially-marked binaries to
>>  have executable stacks (i.e., binutils/gcc mods, which I want to
>>  avoid).
> Well, strictly speaking, you wouldn't actually *need* to modify binutils
> to make specially tagged binaries.  You could borrow an unused bit in
> the ELF header somewhere, have the kernel recognize it, and write your
> own little tool that only turns that bit on/off in an ELF file.

This exists already in ld's -z execstack/noexecstack feature. It is
not used by default because too many things depend on executable
stacks on MIPS.


Thiemo

Re: Adding(?) XI support to MIPS-Linux?

[Brian Foster]

 1) Date: Mon, 09 Jun 2008 21:32:59 +0200
 1) From: "Kevin D. Kissell" <KevinK@paralogos.com>
 1)
 1) Brian Foster wrote:
 1) >  As far as I am aware, XI is part of the SMARTMIPS extension,
 1) >  and I _think_ SMARTMIPS is only implemented by the 4KS cores?
 1)
 1) That is correct, though there has long been interest in having XI/RI
 1) as an option for non-SmartMIPS cores and I would not be surprised if
 1) sooner or later it became more generally available.

Ok, thanks for the confirmation.
I'll keep this in mind w.r.t. to any proposed patches.

 1) >  Whilst other companies have licensed that core from MIPS, to
 1) >  the best of my knowledge the SoC I'm concerned with (Innova
 1) >  Card's USIP Pro) is the only one running Linux.  [ ... ]
 1)
 1) I believe that there is at least one other 4KS-family customer working
 1) with Linux, but they haven't been nearly as active as InnovaCard.
 1) I had some email exchanges with someone who was working on their kernel
 1) a couple of years back about this very topic.  Indeed, I thought that
 1) they submitted some patches for basic RI/XI support at one point.
 1) Scan the linux-mips.org archives, if they survived the rehosting.

I just spent a fair bit of the morning searching the archives,
and cannot find anything related to 4KS, SmartMIPS, XI, or RI
(except odd mentions here and there, none(?) of substance (on
this subject), and all seeming to involve Innova Card).  I'm
*very* interested in seeing/learning what others have done or
tried, so if you or anyone else has any information/whatever
that you can share, please let me know!  Thanks.

 1)[ ... ]
 1) >  Broadly, what I'm trying to say is I don't want to touch gcc
 1) >  (and/or binutils) and am unconvinced I have to.  But I'm very
 1) >  much open to correction here!
 1) >
 1) >  The x86 (including amd64) and, AFAIK, SuperH (sh) Linux kernels
 1) >  now support NX or equivalent [ ... ].
 1) >  I understand they have support for specially-marked binaries to
 1) >  have executable stacks (i.e., binutils/gcc mods, which I want to
 1) >  avoid).
 1)
 1) Well, strictly speaking, you wouldn't actually *need* to modify
 1) binutils to make specially tagged binaries.  [ ... ]

Agreed.  I was speaking loosely.

 1) In the longer term, I'd argue that if there's support for appropriate
 1) binary tagging in the x86 tools, that support should simply be enabled
 1) for MIPS targets and any other non-x86 archiectures with such support
 1) (e.g. Alpha, if anyone still uses them).

Agreed.  This is all *speculative* ATM.  There seems to be
a rational argument that for the specific situation I'm
concerned with, an absolute imposition of non-executable
stack(+data) would be "better".  In the more general
situation, including the longer term, tagged binaries are
(very probably), perhaps unfortunately, desirable.

 2) Date: Mon, 9 Jun 2008 21:46:27 +0100
 2) From: Thiemo Seufer <ths@networkno.de>
 2)
 2) Kevin D. Kissell wrote:
 2)[ ... ]
 2) > Well, strictly speaking, you wouldn't actually *need* to modify
 2) > binutils to make specially tagged binaries.  [ ... ]
 2)
 2) This exists already in ld's -z execstack/noexecstack feature.

Good point.  Thanks for the reminder.

 2) It is not used by default because too many things depend on executable
 2) stacks on MIPS.

Ah!  Can you be more specific please?  At the present time
I'm only aware of three situations where executable stacks
are magically used ("magic" meaning it's being done without
the programmer explicitly coding it):

  1. sigreturn.
  2. something to do with FPU emulation?
  3. pointer to a nested function (gcc extension).

And, significantly, I am do not know of any need for the
kernel-mode stacks to be executable.  Except, perhaps,
for case 3, the above are (should be?) user-land only.

There are also "non-magic" users (JIT in some JVMs is,
I believe, the usual example).  These deliberate users,
and case 3, I want to be able to argue I can blow off
in the specific circumstance I am concerned about.
That is, they simply fail.  Always.  But as said above,
in general and for the longer term, that's presumably
not acceptable (i.e., marked binaries are needed).

cheers!
	-blf-

-- 
"How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools." |      http://www.stopesso.com

Re: Adding(?) XI support to MIPS-Linux?

[Thiemo Seufer]

Brian Foster wrote:
[snip]
>  2) Kevin D. Kissell wrote:
>  2)[ ... ]
>  2) > Well, strictly speaking, you wouldn't actually *need* to modify
>  2) > binutils to make specially tagged binaries.  [ ... ]
>  2)
>  2) This exists already in ld's -z execstack/noexecstack feature.
> 
> Good point.  Thanks for the reminder.
> 
>  2) It is not used by default because too many things depend on executable
>  2) stacks on MIPS.
> 
> Ah!  Can you be more specific please?  At the present time
> I'm only aware of three situations where executable stacks
> are magically used ("magic" meaning it's being done without
> the programmer explicitly coding it):
> 
>   1. sigreturn.
>   2. something to do with FPU emulation?
>   3. pointer to a nested function (gcc extension).

Those, plus manually coded trampolines in e.g. foreign function
interfacing (which are typically hidden in some library). I don't
know if you can ignore that completely. :-)

> And, significantly, I am do not know of any need for the
> kernel-mode stacks to be executable.  Except, perhaps,
> for case 3, the above are (should be?) user-land only.

AFAIK nested functions are frowned upon in kernelspace.


Thiemo

Re: Adding(?) XI support to MIPS-Linux?

[David Daney]

Thiemo Seufer wrote:
> Brian Foster wrote:
> [snip]
>   
>>  2) Kevin D. Kissell wrote:
>>  2)[ ... ]
>>  2) > Well, strictly speaking, you wouldn't actually *need* to modify
>>  2) > binutils to make specially tagged binaries.  [ ... ]
>>  2)
>>  2) This exists already in ld's -z execstack/noexecstack feature.
>>
>> Good point.  Thanks for the reminder.
>>
>>  2) It is not used by default because too many things depend on executable
>>  2) stacks on MIPS.
>>
>> Ah!  Can you be more specific please?  At the present time
>> I'm only aware of three situations where executable stacks
>> are magically used ("magic" meaning it's being done without
>> the programmer explicitly coding it):
>>
>>   1. sigreturn.
>>   2. something to do with FPU emulation?
>>   3. pointer to a nested function (gcc extension).
>>     
>
> Those, plus manually coded trampolines in e.g. foreign function
> interfacing (which are typically hidden in some library). I don't
> know if you can ignore that completely. :-)
>
>   
The trampolines in libffi are user allocated, so there is a choice of 
where to place them.  In libgcj (which uses the libffi trampolines) the 
trampolines are allocated on the heap and care is taken to set the 
execute permissions on the memory in question.  Other users may have 
problems, but by now most code should work as XI support has been 
present on x86 for quite some time now.

As long as there is a mechanism to make user space stacks (and heap) 
executable, there should be no problem.  People running code that 
requires it can switch off the XI support.

David Daney
>> And, significantly, I am do not know of any need for the
>> kernel-mode stacks to be executable.  Except, perhaps,
>> for case 3, the above are (should be?) user-land only.
>>     
>
> AFAIK nested functions are frowned upon in kernelspace.
>
>
> Thiemo
>
>   

Re: Adding(?) XI support to MIPS-Linux?

[Ralf Baechle]

On Mon, Jun 09, 2008 at 09:32:59PM +0200, Kevin D. Kissell wrote:

> That is correct, though there has long been interest in having XI/RI as an 
> option
> for non-SmartMIPS cores and I would not be surprised if sooner or later it
> became more generally available.

Cavium has it in their 64-bit core.  I haven't verified this in the docs
but apparently it is meant to be compatible with the old SmartMIPS ASE
for MIPS32.

  Ralf

Re: Adding(?) XI support to MIPS-Linux?

[Kevin D. Kissell]

Ralf Baechle wrote:
> On Mon, Jun 09, 2008 at 09:32:59PM +0200, Kevin D. Kissell wrote:
>
>   
>> That is correct, though there has long been interest in having XI/RI as an option for non-SmartMIPS cores and I would not be surprised if sooner or later it became more generally available.
>>     
>
> Cavium has it in their 64-bit core.  I haven't verified this in the docs
> but apparently it is meant to be compatible with the old SmartMIPS ASE
> for MIPS32.
>   
Do check the documentation.  I can't comment officially, but I can 
observe that,
in the hypothetical case where you'd want XI/RI semantics in a 64-bit 
processor,
you might use exactly the same semantics (and therefore the same kernel 
C code
support), but you might want to use different bits  for XI/RI in a 
64-bit TLB entry
than in a 32-bit TLB entry (and therefore different header file 
definitions).

          Regards,

          Kevin K.

Re: Adding(?) XI support to MIPS-Linux?

[Brian Foster]

David Daney wrote:
> Thiemo Seufer wrote:
> > Brian Foster wrote:
> >>  2) Kevin D. Kissell wrote:
> >>  2)[... ‘ld -z noexecstack’ ] is not used by default because too many
> >>  2) things depend on executable stacks on MIPS.
> >> 
> >> Ah!  Can you be more specific please?  At the present time
> >> I'm only aware of three situations where executable stacks
> >> are magically used ("magic" meaning it's being done without
> >> the programmer explicitly coding it):
> >> 
> >>   1. sigreturn.
> >>   2. something to do with FPU emulation?
> >>   3. pointer to a nested function (gcc extension).
> > 
> > Those, plus manually coded trampolines in e.g. foreign function
> > interfacing (which are typically hidden in some library).  I don't
> > know if you can ignore that completely. :-)
> 
> The trampolines in libffi are user allocated, so there is a choice of
> where to place them.  In libgcj (which uses the libffi trampolines) the
> trampolines are allocated on the heap and care is taken to set the
> execute permissions on the memory in question.  Other users may have
> problems, but by now most code should work as XI support has been
> present on x86 for quite some time now.

 David, thanks for clarifying Thiemo's point; I wasn't
 quite sure what he meant by “foreign functions” albeit
 (as it happens) apparently guessed correctly.  And for
 the specific example of ‘libffi’ (and ‘libgcj’); that's
 a new library (to me).

> As long as there is a mechanism to make user space stacks (and heap)
> executable, there should be no problem.  People running code that
> requires it can switch off the XI support.

 Agreed.  It is (alas?) important for the general case and
 the longer term.  But it's plausible that for the specific
 case I have, it's not important and maybe not even wanted.
 (I'm working in a security paranoid/sensitive area .... .)
 Please note this is rather *speculative* ATM!

 It's case 2 (above), the trampoline that has “something
 to do with FPU emulation”, which has me concerned ATM.
 The 4KSd core does not have an FPU.  That encourages the
 use of ‘-msoft-float’ (at least for performance), but does
 not require it.  (Albeit I wonder if, in the restricted
 world I'm playing in, if it could be “required” (assuming
 it doesn't have an issue?)?  Hum .... .)

 The quick summary (which I'm sure others on this list can
 clarify/correct) is the FP trampoline, which is pushed on
 the user-land stack is, unlike sigreturn, not fixed code.
 It varies on a per-instance per-thread basis.  Hence the
 simple ‘vsyscall’ mechanism ((to be?) used for sigreturn)
 is inappropriate.

 The trampoline is only used to execute a non-FP instruction
 (<instr>) in the delay slot of an FP-instruction:

     <instr>  # Non-FP instruction to execute in user-land
     BADINST  # Bad instruction forcing return to FP emulator
     COOKIE   # Bad instruction (not executed) for verification
     <epc>    # Where to resume execution after <instr>

 Belch! ;-\  Whilst I can think of a few things that may work
 (temporarily change page permissions;  or go ahead and use
 the ‘vsyscall’ page with some interlocking magic;  or a new
 new dedicated per-thread page;  or ...?) none seem appealing.

 Suggestions?  Comments?  Prior art to study?

thanks & cheers!
	-blf-

( I'm experimenting with a new technique for posting to the
 list to save me some hassle.  It *should* work, but .... ! )
-- 
“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com

Re: Adding(?) XI support to MIPS-Linux?

[Kevin D. Kissell]

Brian Foster wrote:
>  It's case 2 (above), the trampoline that has “something
>  to do with FPU emulation”, which has me concerned ATM.
>  The 4KSd core does not have an FPU.  That encourages the
>  use of ‘-msoft-float’ (at least for performance), but does
>  not require it.  (Albeit I wonder if, in the restricted
>  world I'm playing in, if it could be “required” (assuming
>  it doesn't have an issue?)?  Hum .... .)
>   
The use of -msoft-float historically required (and as far as I know 
still requires)
a completely different ground-up userland build, so it gets used less 
than you
might think.
>  The quick summary (which I'm sure others on this list can
>  clarify/correct) is the FP trampoline, which is pushed on
>  the user-land stack is, unlike sigreturn, not fixed code.
>  It varies on a per-instance per-thread basis.  Hence the
>  simple ‘vsyscall’ mechanism ((to be?) used for sigreturn)
>  is inappropriate.
>
>  The trampoline is only used to execute a non-FP instruction
>  (<instr>) in the delay slot of an FP-instruction:
>
>      <instr>  # Non-FP instruction to execute in user-land
>      BADINST  # Bad instruction forcing return to FP emulator
>      COOKIE   # Bad instruction (not executed) for verification
>      <epc>    # Where to resume execution after <instr>
>
>  Belch! ;-\  Whilst I can think of a few things that may work
>  (temporarily change page permissions;  or go ahead and use
>  the ‘vsyscall’ page with some interlocking magic;  or a new
>  new dedicated per-thread page;  or ...?) none seem appealing.
>
>  Suggestions?  Comments?  Prior art to study?
>   
As the jerk who originally bolted the FP emulator into the MIPS kernel
and came up with the stack trampoline hack, I can explain why it seemed
sane at the time.  If an FP branch is emulated and to be taken, we have to
find a way for the instruction in the delay slot to be executed prior to the
transfer of control to the branch target.  It has to execute with the user's
permissions.  Putting it on the user's stack and building a trampoline was
the fairly classical way of doing it, but note that it's architecturally 
illegal
to put a branch in a branch delay slot (floating point or otherwise), so
there's no possibility of recursion. So one only needs 3-4 words (one
could substitute another means of validation for the cookie) per
thread.  It just has to be part of the user's address space.  I suppose
that instead of using a few words just above the stack, one could use
a few words just below the current "brk()" point, or, better still (but
far more invasive) pad the text segment, which should always be
executable, with 4 words that the kernel can find in a hurry.

          Regards,

          Kevin K.

Re: Adding(?) XI support to MIPS-Linux?

[Brian Foster]

On Wednesday 11 June 2008 Kevin D. Kissell wrote:
> Brian Foster wrote:
> >[ ...  the FPU emulation ] trampoline, which is pushed on
> >  the user-land stack is, unlike sigreturn, not fixed code.
> >  It varies on a per-instance per-thread basis.  Hence the
> >  simple ‘vsyscall’ mechanism ((to be?) used for sigreturn)
> >  is inappropriate.
> >
> >  The trampoline is only used to execute a non-FP instruction
> >  (<instr>) in the delay slot of an FP-instruction  [ ... ]
> >
> >  Belch! ;-\  Whilst I can think of a few things that may work
> >  (temporarily change page permissions;  or go ahead and use
> >  the ‘vsyscall’ page with some interlocking magic;  or a new
> >  new dedicated per-thread page;  or ...?) none seem appealing.
>[ ... ]
> As the jerk who originally bolted the FP emulator into the MIPS kernel
> and came up with the stack trampoline hack, I can explain why it seemed
> sane at the time.  If an FP branch is emulated and to be taken, we have to
> find a way for the instruction in the delay slot to be executed prior to the
> transfer of control to the branch target.  It has to execute with the user's
> permissions.  Putting it on the user's stack and building a trampoline was
> the fairly classical way of doing it, but note that it's architecturally
> illegal to put a branch in a branch delay slot (floating point or otherwise),
> so there's no possibility of recursion.  So one only needs 3-4 words (one
> could substitute another means of validation for the cookie) per thread.

Jerk,  ;-)

 Yes, once I worked out what it was doing it all seemed cute
 (albeit I don't quite see what the danger is with recursion?).
 My “Belch!” was referring to the problems it now causes with
 non-executable stacks.

> It just has to be part of the user's address space.  I suppose
> that instead of using a few words just above the stack, one could use
> a few words just below the current "brk()" point, or, better still (but
> far more invasive) pad the text segment, which should always be
> executable, with 4 words that the kernel can find in a hurry.

 First, you need to really careful about multithreaded code
 concurrently doing FPU stuff.  That is, it's possible there
 may be more than one “live” emulated FPU delay slot in the
 same address space.  So stuffing the code into text, or
 near the brk()-point, or similar, all has concurrency issues.

 This is what makes the current on-the-stack approach neat;
 the stack _is_ per-thread so there's no concurrency mess.

 As for putting the trampoline near the brk()-point, besides
 the concurrency problem, there's also the issue that the
 containing page would have to be made user-executable (if
 temporarily).  Unless I'm confused, that page is nominally
 data (heap-ish).  With the addition of XI support, I would
 expect data to nominally also be non-executable.

cheers!
	-blf-

-- 
“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com

Re: Adding(?) XI support to MIPS-Linux?

[Brian Foster]

On Wednesday 11 June 2008 15:16:56 Brian Foster wrote:
>[ ... ]
>  It's [ ... ] the trampoline that has “something to do with
>  FPU emulation”, which has me concerned ATM.  [ ... ]
> 
>  The quick summary [ ... ] is the FP trampoline, which is pushed
>  on the user-land stack is, unlike sigreturn, not fixed code.
>  It varies on a per-instance per-thread basis.  Hence the
>  simple ‘vsyscall’ mechanism ((to be?) used for sigreturn)
>  is inappropriate.
> 
>  The trampoline is only used to execute a non-FP instruction
>  (<instr>) in the delay slot of an FP-instruction:
> 
      [ point A "before" <instr> ]
>      <instr>  # Non-FP instruction to execute in user-land
      [ point B "after" <instr> "before" BADINST ]
>      BADINST  # Bad instruction forcing return to FP emulator
>      COOKIE   # Bad instruction (not executed) for verification
>      <epc>    # Where to resume execution after <instr>
      [ user-land stack-pointer points here(-ish) ]

 Whilst thinking about the problem and possible solutions,
 it occurred to me there could be a defect in the current
 trampoline:  Suppose there is a signal, either at point A,
 due to <instr> itself, or at point B, which is caught on
 this stack, and the user-land signal-handler ‘return’s.

 Doesn't the signal-handler/sigreturn stack-frame overwrite
 the FP trampoline?   In which case, when the signal-hander
 returns, more-or-less anything could happen.  (And very
 unlikely to be what's wanted!)

cheers!
	-blf-

-- 
“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com

Re: Adding(?) XI support to MIPS-Linux?

[Kevin D. Kissell]

Brian Foster wrote:
>  Whilst thinking about the problem and possible solutions,
>  it occurred to me there could be a defect in the current
>  trampoline:  Suppose there is a signal, either at point A,
>  due to <instr> itself, or at point B, which is caught on
>  this stack, and the user-land signal-handler ‘return’s.
>
>  Doesn't the signal-handler/sigreturn stack-frame overwrite
>  the FP trampoline?   In which case, when the signal-hander
>  returns, more-or-less anything could happen.  (And very
>  unlikely to be what's wanted!)
>   
When I first integrated the FP emulator into the kernel, back in 2.2.x, 
I seem to
recall that someone found this problem and that I came up with a tweak 
to signal
stack setup that protected the FP branch delay slot trampoline.  Maybe 
I'm mistaken,
or maybe the tweak was lost?

          Regards,

          Kevin K.

Re: Adding(?) XI support to MIPS-Linux?

[Brian Foster]

On Wednesday 18 June 2008 11:36:53 Kevin D. Kissell wrote:
> Brian Foster wrote:
> >  Whilst thinking about the problem and possible solutions,
> >  it occurred to me there could be a defect in the current
> >  trampoline:  Suppose there is a signal, either at point A,
> >  due to <instr> itself, or at point B, which is caught on
> >  this stack, and the user-land signal-handler ‘return’s.
> >
> >  Doesn't the signal-handler/sigreturn stack-frame overwrite
> >  the FP trampoline?   [ ... ]
> 
> When I first integrated the FP emulator into the kernel, back in 2.2.x,
> I seem to recall that someone found this problem and that I came up with
> a tweak to signal stack setup that protected the FP branch delay slot
> trampoline.  Maybe I'm mistaken, or maybe the tweak was lost?

 The error is mine:  I overlooked the tweak.
 Now that you mention it / remind me of it,
 I distinctly recall it; in fact, that was
 what first alerted me to the existance of
 the FP trampoline.

sorry & cheers!
	-blf-
-- 
“How many surrealists does it take to   | Brian Foster
 change a lightbulb? Three. One calms   | somewhere in south of France
 the warthog, and two fill the bathtub  |   Stop E$$o (ExxonMobil)!
 with brightly-coloured machine tools.” |      http://www.stopesso.com