Brian Foster wrote: 
On Friday 24 April 2009 17:30:35 David Daney wrote:
  
Kevin D. Kissell wrote:
    
Brian Foster wrote:
      
On Wednesday 22 April 2009 20:01:44 David Daney wrote:
        
Kevin D. Kissell wrote:
          
David Daney wrote:
            
This is a preliminary patch to add a vdso to all user processes.
[ ... ]
              
Note that for FPU-less CPUs, the kernel FP emulator also uses a user
stack trampoline to execute instructions in the delay slots of emulated
FP branches.  [ ... ]
            
   As David says, this is a Very Ugly Problem.  Each FP trampoline
  is effectively per-(runtime-)instance per-thread [ ... ]
        
I haven't reviewed David's code in detail, but from his description, I 
thought that there was a vdso page per task/thread.  If there's only one 
per processor, then, yes, that poses a challenge to porting the FPU 
emulation code to use it, since, as you observe, the instruction 
sequence to be executed may differ for each delay slot emulation.  It 
should still be possible, though.  [ ... ]
      
Kevin is right, this is ugly.

My current plan is to map an anonymous page with execute permission for 
each vma (process) and place all FP trampolines there.  Each thread that 
needs a trampoline will allocate a piece of this page and write the 
trampoline.  We can arrange it so that the only way a thread can exit 
the trampoline is by taking some sort of fault (currently this is true 
for the normal case), or exiting.
    

David,

   The above is the bit which has always stumped me.
  Having a per-process(or similar) page for the FP
  trampoline(s) is the “obvious” approach, but what
  has had me going around in circles is how to know
  when an allocated slot/trampoline can be freed.
  As you imply, in the normal case, it seems trivial.
  It's the not-normal cases which aren't clear (or at
  least aren't clear to me!).

   You say (EMPHASIS added) “We can arrange it so
  that the ONLY way a thread can exit the trampoline
  is by taking some sort of fault ... or exiting”,
  which if true, could solve the issue.  Could you
  elucidate on this point, please?
Well, he's *almost* right about that. The delay slot emulation function executes a single instruction off the user stack/vdso slot, which is followed in memory by an instruction that provokes an address exception.  The address exception handler detects the special case (and it should be noted that detecting the special case could be made simpler and more reliable if a vdso-type region were used), cleans up, and restores normal stack behavior.  That "clean up" could, of course, include any necessary vdso slot management.  But what about cases that won't get to the magic alignment trap?

As the instruction being executed is extracted from a branch delay slot, we know it's not legal for it to be any sort of branch or jump instruction.  But it *could* be a trap or system call instruction, or a load/store that would provoke a TLB exception.  In the usual cases, however, as I believe David was alluding, either the exception will ultimately unwind to return to execute the magic alignment trap, or the thread will exit, and could free the emulation slot as part of general cleanup.

But there's a case that isn't handled in this model, and that's the case of an exception (or interrupt that falls in the 2-instruction window) resulting in a signal that is caught and dispatched, and where either the signal handler does a longjmp and restarts FP computation, or where the signal handler itself contains a FP branch with yet another delay slot to be emulated. One *could* get alarm signal before the original delay slot instruction is executed, so recycling the same vdso cache line would be premature.  It's hard to get away from something distinctly stack-like if one wants to cover these cases.

My short-term suggestion would be to leave FP emulator delay slot handling on the (executable) user stack, even if signal trampolines use the vdso.  Longer term, we might consider what sorts of crockery would be necessary to deal with delay slot abandonment and recursion.  That might mean adding cruft to the signal dispatch logic to detect that we're in mid-delay-slot-emulation and defer the signal until after the alignment trap cleanup is done (adds annoying run-time overhead, but is probably the smallest increase in footprint and complexity), or it might mean changing the delay slot emulation paradigm completely and bolting a full instruction set emulator into the FP emulator, so that the delay slot instruction is simulated in kernel mode, rather than requiring execution in user mode.  I rejected that idea out-of-hand when I first did the FP emulator integration with the kernel, years ago, but maybe the constraints have changed...

          Regards,

          Kevin K.