Re: [BK-2.5] Move "used FPU status" into new non-atomic thread_info->status field.

Re: [BK-2.5] Move "used FPU status" into new non-atomic thread_info->status field.

[David S. Miller]

   From: Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
   Date: Mon, 10 Mar 2003 18:20:35 +0000
   	
Linus said, in a recent BK changelog:

   	Also, fix x86 FP state after fork() by making sure the FP is unlazied
   	_before_ we copy the state information. Otherwise, if a process did a
   	fork() while holding the FP state lazily in the registers, the child
   	would incorrectly unlazy bogus state.
   
At least on sparc{32,64}, we consider FPU state to be clobbered coming
into system calls, this eliminates a lot of hair wrt. FPU state
restoring in cases such as fork().

Are you preserving FPU state across fork() on x86?  If so, what do you
think might rely on this?

Re: [BK-2.5] Move "used FPU status" into new non-atomic thread_info->status field.

[Linus Torvalds]

On Mon, 10 Mar 2003, David S. Miller wrote:
>    
> At least on sparc{32,64}, we consider FPU state to be clobbered coming
> into system calls, this eliminates a lot of hair wrt. FPU state
> restoring in cases such as fork().

We could _probably_ do it on x86 too. The standard C calling convention on 
x86 says FPU register state is clobbered, if I remember correctly. 
However, some of the state is "long-term", like rounding modes, exception 
masking etc, and even if we didn't save the register state we would have 
to save that part.

And once you save that part, you're better off saving the registers too, 
since it's all loaded and saved with the same fxsave/fxrestor instruction 
(ie we'd actually have to do _more_ work to save only part of the FP 
state).

> Are you preserving FPU state across fork() on x86?  If so, what do you
> think might rely on this?

Probably nothing per se. HOWEVER, we'd still need to save the state for 
rounding etc, so we might as well save it all.

As it was, the x86 state was pretty much random after fork(), and that can 
definitely lead to problems for real programs if they depend on things 
like silent underflow etc. 

(Now, in _practice_ all processes on the machine tends to use the same
rounding and exception control, so the "random" state wasn't actually very
random, and would not lead to problems. It's a security issue, though).

		Linus

Re: [BK-2.5] Move "used FPU status" into new non-atomic thread_info->status field.

[David S. Miller]

   From: Linus Torvalds <torvalds@transmeta.com>
   Date: Mon, 10 Mar 2003 11:25:55 -0800 (PST)
   
   > Are you preserving FPU state across fork() on x86?  If so, what do you
   > think might rely on this?
   
   Probably nothing per se. HOWEVER, we'd still need to save the state for 
   rounding etc, so we might as well save it all.
   
I see.

We preserve the rounding/etc. modes on sparc, we merely zap the actual
FPU registers around the system call.

And that's like 4 L2 cache lines of registers on sparc64, so there
really is a benefit from only saving the mode register across a system
call.

Re: [BK-2.5] Move "used FPU status" into new non-atomic thread_info->status field.

[Chris Friesen]

Linus Torvalds wrote:
> On Mon, 10 Mar 2003, David S. Miller wrote:
> 
>>   
>>At least on sparc{32,64}, we consider FPU state to be clobbered coming
>>into system calls, this eliminates a lot of hair wrt. FPU state
>>restoring in cases such as fork().
>>
> 
> We could _probably_ do it on x86 too. The standard C calling convention on 
> x86 says FPU register state is clobbered, if I remember correctly. 
> However, some of the state is "long-term", like rounding modes, exception 
> masking etc, and even if we didn't save the register state we would have 
> to save that part.
> 
> And once you save that part, you're better off saving the registers too, 
> since it's all loaded and saved with the same fxsave/fxrestor instruction 
> (ie we'd actually have to do _more_ work to save only part of the FP 
> state).

Does this open the door for using FP in the kernel?

Chris


-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: [BK-2.5] Move "used FPU status" into new non-atomic thread_info->status field.

[Linus Torvalds]

On Mon, 10 Mar 2003, Chris Friesen wrote:
> > 
> > And once you save that part, you're better off saving the registers too, 
> > since it's all loaded and saved with the same fxsave/fxrestor instruction 
> > (ie we'd actually have to do _more_ work to save only part of the FP 
> > state).
> 
> Does this open the door for using FP in the kernel?

Not any wider than it already is.

For a while now, x86-specific optimizations (and all such stuff is by 
nature very much architecture-specific) have been able to do

	kernel_fpu_begin();
	...
	kernel_fpu_end();

and use the FP state in between. It generally sucks if the user-mode 
process had touched FP state (we'll force it saved), but most of the time 
that isn't true, and the only thing it does is to temporarily clear the 
TS bit so that the FPU works again (and then sets it again in fpu_end, 
although if this was a common thing we _could_ make that be a "work" 
thing that is only done at return-to-user-mode).

Of course, clearing TS isn't exactly fast, so this really only works if 
you have tons of stuff that you _really_ want to use the FPU for. And 
since the FP cache is per-CPU, the whole region in question is 
non-preemptible, so this can only be used for non-blocking stuff.

In other words: it's still very much a special case, and if the question 
was "can I just use FP in the kernel" then the answer is still a 
resounding NO, since other architectures may not support it AT ALL.

		Linus

Re: [BK-2.5] Move "used FPU status" into new non-atomic thread_info->status field.

[Andi Kleen]

Linus Torvalds <torvalds@transmeta.com> writes:

> (Now, in _practice_ all processes on the machine tends to use the same
> rounding and exception control, so the "random" state wasn't actually very
> random, and would not lead to problems. It's a security issue, though).

Oh it does. Together with Marcus Meissner I just tracked down a 32bit
emulation problem on x86-64 with Wine today. The program running in
Wine would randomly crash on a flds with an floating point exception.  

Turned out the 32bit ptrace unlazy FPU path shared two lines too many
with with the 32bit signal FPU saving path and was resetting the
used_fpu flag. Result was that the FPU state of the child could be
reinitialized in some circumstances on ptrace accesses.  Wine actually
does use ptrace between the Wine server and the emulated process for
some complicated calls. It did one unlucky ptrace and then the FPCR was
at the linux defaults again - program crashed.

-Andi

Re: [BK-2.5] Move "used FPU status" into new non-atomic thread_info->status field.

[David S. Miller]

   From: Andi Kleen <ak@suse.de>
   Date: 10 Mar 2003 22:01:17 +0100
   
   Turned out the 32bit ptrace unlazy FPU path shared two lines too many
   with with the 32bit signal FPU saving path and was resetting the
   used_fpu flag. Result was that the FPU state of the child could be
   reinitialized in some circumstances on ptrace accesses.

So what it depended upon was the FP control register state,
not the state of the individual FPU registers, across fork()
right?

Re: [BK-2.5] Move "used FPU status" into new non-atomic thread_info->status field.

[Andi Kleen]

On Mon, Mar 10, 2003 at 12:45:02PM -0800, David S. Miller wrote:
>    Turned out the 32bit ptrace unlazy FPU path shared two lines too many
>    with with the 32bit signal FPU saving path and was resetting the
>    used_fpu flag. Result was that the FPU state of the child could be
>    reinitialized in some circumstances on ptrace accesses.
> 
> So what it depended upon was the FP control register state,
> not the state of the individual FPU registers, across fork()
> right?

Yes. The IA32 ABI says the FPU registers are clobbered in a function
call. And fork is a function call. Same with the SSE registers.

Unfortunately it is much more expensive to save individual registers
(SSE and x87 stack) than to just save everything using FXSAVE. 
FXSAVE uses lazy saving and saves only the x87 registers that are
actually uses.

For SSE registers it may make sense, but then FXSAVE does that already
too and you always have to handle the x87 register stack too.

I doubt it would be a good idea to not use FXSAVE on i386. The microcode
can do a better job here because it has more information. In addition
it also promises to handle future new Intel registers transparently.

x86-64 ABIs have similar semantics.

-Andi 

Re: [BK-2.5] Move "used FPU status" into new non-atomic thread_info->status field.

[Mikael Pettersson]

On Mon, 10 Mar 2003 11:25:55 -0800 (PST), Linus Torvalds wrote:
>We could _probably_ do it on x86 too. The standard C calling convention on 
>x86 says FPU register state is clobbered, if I remember correctly. 
>However, some of the state is "long-term", like rounding modes, exception 
>masking etc, and even if we didn't save the register state we would have 
>to save that part.
...
>As it was, the x86 state was pretty much random after fork(), and that can 
>definitely lead to problems for real programs if they depend on things 
>like silent underflow etc. 

Do you mean x87 control or the x87 stack here?

>(Now, in _practice_ all processes on the machine tends to use the same
>rounding and exception control, so the "random" state wasn't actually very
>random, and would not lead to problems. It's a security issue, though).

Sorry for being dense, but can you clarify: will current 2.{2,4,5}
kernels preserve or destroy the parent process' FPU control at fork()?

We're using unmasked FPU exceptions on x86 (and Solaris/SPARC) in the
runtime system for the Erlang telecom systems programming language.
This gives a noticeable performance improvement, but it relies on
the FPU control not changing beneath it: the FPU control is only
initialised at startup and when SIGFPU has occurred.

/Mikael

Re: [BK-2.5] Move "used FPU status" into new non-atomic thread_info->status field.

[Linus Torvalds]

On Tue, 11 Mar 2003, Mikael Pettersson wrote:

> On Mon, 10 Mar 2003 11:25:55 -0800 (PST), Linus Torvalds wrote:
> ...
> >As it was, the x86 state was pretty much random after fork(), and that can 
> >definitely lead to problems for real programs if they depend on things 
> >like silent underflow etc. 
> 
> Do you mean x87 control or the x87 stack here?

everything was random - the child would (if the fork() happened while the 
parent had a lazy pending x87 state) get a random state from somebody 
else.

HOWEVER: this only happens if the fork() actually has pending state on
that CPU, which is often not the case (if it has slept at all since the
last x87 op, or has been timesliced away etc, it won't have any pending
state on the CPU).

> >(Now, in _practice_ all processes on the machine tends to use the same
> >rounding and exception control, so the "random" state wasn't actually very
> >random, and would not lead to problems. It's a security issue, though).
> 
> Sorry for being dense, but can you clarify: will current 2.{2,4,5}
> kernels preserve or destroy the parent process' FPU control at fork()?

They're guaranteed to preserve the control state (it has to: you can't 
just change the exception mask over a function call). However, that was 
buggy at least in 2.5.x, and very possibly in 2.4.x too - haven't checked.

And since preserving the exception mask basically implies doing the whole 
fxsave/fxrestor thing, in practice _everything_ is saved over fork() into 
the child.

		Linus

Re: [BK-2.5] Move "used FPU status" into new non-atomic thread_info->status field.

[Mikael Pettersson]

Linus Torvalds writes:
 > > Sorry for being dense, but can you clarify: will current 2.{2,4,5}
 > > kernels preserve or destroy the parent process' FPU control at fork()?
 > 
 > They're guaranteed to preserve the control state (it has to: you can't 
 > just change the exception mask over a function call). However, that was 
 > buggy at least in 2.5.x, and very possibly in 2.4.x too - haven't checked.

Thanks. Our use of unmasked FPU exceptions should be safe then, unless
2.4 also has the bug you fixed in 2.5.

/Mikael