Restoring FPU exception state

Restoring FPU exception state

[David Vrabel]

The FPU exception state includes 4 registers:

- 64-bit FIP
- 16-bit FCS
- 64-bit FDP
- 16-bit FDS

When a CPU takes an FPU exception in long mode, all 4 registers are
fully updated.  This state can be saved with a combination of REX.W
prefixed XSAVE and FNSTENV.  This state cannot be restored with any
combination of instructions as those that restore the 64-bit FIP/FDP
clear FCS and FDS; and those that restore FCS and FDS clear the upper
32-bits of FIP and FDP [1].

This causes problems when running Microsoft's Driver Verifier in a
64-bit Windows guest (seen with Windows 7 SP1, but other versions may
also be affected).

The Driver Verifier prior to calling a driver's interrupt handler will
save the FPU state, after the handler is called it will save the state
again and do a byte-by-byte compare to verify the state has not changed.
 A 0x3D (INTERRUPT_EXCEPTION_NOT_HANDLED) BugCheck is raised if the
state does not match.

Windows uses XSAVE to save the FPU state, but it does not use a REX.W
prefixed XSAVE, and saves only the lower 32-bits of FIP/FDP and FCS/FDS.

If the VCPU is descheduled between these two checks, the contents of
FCS/FDS is lost, Windows will notice and BugCheck.

When saving a VCPUs FPU state, Xen first uses a REX.W prefixed XSAVE and
notices that FIP/FDP[64:32] is non-zero and assumes are REX.W prefixed
XRSTOR is required to restore the full 64-bits of FIP/FDP.  This clears
FCS/FDS.

On processors with FPCSDS[2] (bit 13) set in CPUID leaf 0x7, sub-leaf
0x0, do not save FCS/FDS (they always write zeros) and this problem does
not occur, because FCS/FDS never needs to be restored.

Does anyone any thoughts of a solution for processors without the FPCSDS
feature?

David

[1] Intel SDM vol 1. section 8.1.8.
[2] http://www.sandpile.org/x86/cpuid.htm#level_0000_0007h

Re: Restoring FPU exception state

[Jan Beulich]

>>> On 17.02.16 at 14:08, <david.vrabel@citrix.com> wrote:
> The FPU exception state includes 4 registers:
> 
> - 64-bit FIP
> - 16-bit FCS
> - 64-bit FDP
> - 16-bit FDS
> 
> When a CPU takes an FPU exception in long mode, all 4 registers are
> fully updated.  This state can be saved with a combination of REX.W
> prefixed XSAVE and FNSTENV.  This state cannot be restored with any
> combination of instructions as those that restore the 64-bit FIP/FDP
> clear FCS and FDS; and those that restore FCS and FDS clear the upper
> 32-bits of FIP and FDP [1].
> 
> This causes problems when running Microsoft's Driver Verifier in a
> 64-bit Windows guest (seen with Windows 7 SP1, but other versions may
> also be affected).
> 
> The Driver Verifier prior to calling a driver's interrupt handler will
> save the FPU state, after the handler is called it will save the state
> again and do a byte-by-byte compare to verify the state has not changed.
>  A 0x3D (INTERRUPT_EXCEPTION_NOT_HANDLED) BugCheck is raised if the
> state does not match.
> 
> Windows uses XSAVE to save the FPU state, but it does not use a REX.W
> prefixed XSAVE, and saves only the lower 32-bits of FIP/FDP and FCS/FDS.

Oh, you say that's the case even in 64-bit Windows? That's rather
unexpected.

> If the VCPU is descheduled between these two checks, the contents of
> FCS/FDS is lost, Windows will notice and BugCheck.
> 
> When saving a VCPUs FPU state, Xen first uses a REX.W prefixed XSAVE and
> notices that FIP/FDP[64:32] is non-zero and assumes are REX.W prefixed
> XRSTOR is required to restore the full 64-bits of FIP/FDP.  This clears
> FCS/FDS.
> 
> On processors with FPCSDS[2] (bit 13) set in CPUID leaf 0x7, sub-leaf
> 0x0, do not save FCS/FDS (they always write zeros) and this problem does
> not occur, because FCS/FDS never needs to be restored.
> 
> Does anyone any thoughts of a solution for processors without the FPCSDS
> feature?

One would assume they have a solution to this problem on Hyper-V,
but then again their solution may simply be that they don't use REX
prefixes anywhere (i.e. namely also not in context switch code). In
which case, though, they'd corrupt state of their non-Windows guests.

But to answer your question: I have no idea. The handling of these
FPU code/data pointers has been an unloved child from the very
beginning of the x86-64 architecture. All I could see us doing would
be to add a per-domain control to override the auto-detection in the
XSAVE code path.

Jan

Re: Restoring FPU exception state

[Tian, Kevin]

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Wednesday, February 17, 2016 9:46 PM
> 
> >>> On 17.02.16 at 14:08, <david.vrabel@citrix.com> wrote:
> > The FPU exception state includes 4 registers:
> >
> > - 64-bit FIP
> > - 16-bit FCS
> > - 64-bit FDP
> > - 16-bit FDS
> >
> > When a CPU takes an FPU exception in long mode, all 4 registers are
> > fully updated.  This state can be saved with a combination of REX.W
> > prefixed XSAVE and FNSTENV.  This state cannot be restored with any
> > combination of instructions as those that restore the 64-bit FIP/FDP
> > clear FCS and FDS; and those that restore FCS and FDS clear the upper
> > 32-bits of FIP and FDP [1].
> >
> > This causes problems when running Microsoft's Driver Verifier in a
> > 64-bit Windows guest (seen with Windows 7 SP1, but other versions may
> > also be affected).
> >
> > The Driver Verifier prior to calling a driver's interrupt handler will
> > save the FPU state, after the handler is called it will save the state
> > again and do a byte-by-byte compare to verify the state has not changed.
> >  A 0x3D (INTERRUPT_EXCEPTION_NOT_HANDLED) BugCheck is raised if the
> > state does not match.
> >
> > Windows uses XSAVE to save the FPU state, but it does not use a REX.W
> > prefixed XSAVE, and saves only the lower 32-bits of FIP/FDP and FCS/FDS.
> 
> Oh, you say that's the case even in 64-bit Windows? That's rather
> unexpected.
> 
> > If the VCPU is descheduled between these two checks, the contents of
> > FCS/FDS is lost, Windows will notice and BugCheck.
> >
> > When saving a VCPUs FPU state, Xen first uses a REX.W prefixed XSAVE and
> > notices that FIP/FDP[64:32] is non-zero and assumes are REX.W prefixed
> > XRSTOR is required to restore the full 64-bits of FIP/FDP.  This clears
> > FCS/FDS.
> >
> > On processors with FPCSDS[2] (bit 13) set in CPUID leaf 0x7, sub-leaf
> > 0x0, do not save FCS/FDS (they always write zeros) and this problem does
> > not occur, because FCS/FDS never needs to be restored.
> >
> > Does anyone any thoughts of a solution for processors without the FPCSDS
> > feature?
> 
> One would assume they have a solution to this problem on Hyper-V,
> but then again their solution may simply be that they don't use REX
> prefixes anywhere (i.e. namely also not in context switch code). In
> which case, though, they'd corrupt state of their non-Windows guests.
> 
> But to answer your question: I have no idea. The handling of these
> FPU code/data pointers has been an unloved child from the very
> beginning of the x86-64 architecture. All I could see us doing would
> be to add a per-domain control to override the auto-detection in the
> XSAVE code path.
> 

Interesting. Let me also have a check internally whether there is other
architectural alternative. BTW, not quite related. Could I think finally
Xen may allow user to specify OS type as a general per-domain control,
and then Xen can do free optimizations underlying based on that type?
I don't expect we want to expose raw FPU related per-domain control
since it's difficult for users to correctly set it up...

Thanks
Kevin

Re: Restoring FPU exception state

[Jan Beulich]

>>> On 18.02.16 at 07:30, <kevin.tian@intel.com> wrote:
> Interesting. Let me also have a check internally whether there is other
> architectural alternative. BTW, not quite related. Could I think finally
> Xen may allow user to specify OS type as a general per-domain control,
> and then Xen can do free optimizations underlying based on that type?
> I don't expect we want to expose raw FPU related per-domain control
> since it's difficult for users to correctly set it up...

So what if then, with us implying some kind of workaround here
from the OS type, someone set the OS type to Windows, and
subsequently MS decided to fix their issue?

Jan

Re: Restoring FPU exception state

[Paul Durrant]

> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: 18 February 2016 08:16
> To: Kevin Tian
> Cc: Andrew Cooper; David Vrabel; Paul Durrant; Feng Wu; xen-devel
> Subject: RE: Restoring FPU exception state
> 
> >>> On 18.02.16 at 07:30, <kevin.tian@intel.com> wrote:
> > Interesting. Let me also have a check internally whether there is other
> > architectural alternative. BTW, not quite related. Could I think finally
> > Xen may allow user to specify OS type as a general per-domain control,
> > and then Xen can do free optimizations underlying based on that type?
> > I don't expect we want to expose raw FPU related per-domain control
> > since it's difficult for users to correctly set it up...
> 
> So what if then, with us implying some kind of workaround here
> from the OS type, someone set the OS type to Windows, and
> subsequently MS decided to fix their issue?

We already get full OS version information from Windows via the VIRIDIAN_MSR_GUEST_OS_ID MSR so it would be quite straightforward to work around the issue for all Windows at the moment and then make the check more specific if Microsoft choose to avoid the problem by some other means.

  Paul

> 
> Jan

Re: Restoring FPU exception state

[Jan Beulich]

>>> On 18.02.16 at 09:41, <Paul.Durrant@citrix.com> wrote:
>>  -----Original Message-----
>> From: Jan Beulich [mailto:JBeulich@suse.com]
>> Sent: 18 February 2016 08:16
>> To: Kevin Tian
>> Cc: Andrew Cooper; David Vrabel; Paul Durrant; Feng Wu; xen-devel
>> Subject: RE: Restoring FPU exception state
>> 
>> >>> On 18.02.16 at 07:30, <kevin.tian@intel.com> wrote:
>> > Interesting. Let me also have a check internally whether there is other
>> > architectural alternative. BTW, not quite related. Could I think finally
>> > Xen may allow user to specify OS type as a general per-domain control,
>> > and then Xen can do free optimizations underlying based on that type?
>> > I don't expect we want to expose raw FPU related per-domain control
>> > since it's difficult for users to correctly set it up...
>> 
>> So what if then, with us implying some kind of workaround here
>> from the OS type, someone set the OS type to Windows, and
>> subsequently MS decided to fix their issue?
> 
> We already get full OS version information from Windows via the 
> VIRIDIAN_MSR_GUEST_OS_ID MSR so it would be quite straightforward to work 
> around the issue for all Windows at the moment and then make the check more 
> specific if Microsoft choose to avoid the problem by some other means.

That's only if "viridian=1" in the guest config, isn't it? Plus - how would
we know MS chose to fix their apparent issue?

Jan

Re: Restoring FPU exception state

[Paul Durrant]

> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: 18 February 2016 08:50
> To: Paul Durrant; Kevin Tian
> Cc: Andrew Cooper; David Vrabel; Feng Wu; xen-devel
> Subject: RE: Restoring FPU exception state
> 
> >>> On 18.02.16 at 09:41, <Paul.Durrant@citrix.com> wrote:
> >>  -----Original Message-----
> >> From: Jan Beulich [mailto:JBeulich@suse.com]
> >> Sent: 18 February 2016 08:16
> >> To: Kevin Tian
> >> Cc: Andrew Cooper; David Vrabel; Paul Durrant; Feng Wu; xen-devel
> >> Subject: RE: Restoring FPU exception state
> >>
> >> >>> On 18.02.16 at 07:30, <kevin.tian@intel.com> wrote:
> >> > Interesting. Let me also have a check internally whether there is other
> >> > architectural alternative. BTW, not quite related. Could I think finally
> >> > Xen may allow user to specify OS type as a general per-domain control,
> >> > and then Xen can do free optimizations underlying based on that type?
> >> > I don't expect we want to expose raw FPU related per-domain control
> >> > since it's difficult for users to correctly set it up...
> >>
> >> So what if then, with us implying some kind of workaround here
> >> from the OS type, someone set the OS type to Windows, and
> >> subsequently MS decided to fix their issue?
> >
> > We already get full OS version information from Windows via the
> > VIRIDIAN_MSR_GUEST_OS_ID MSR so it would be quite straightforward to
> work
> > around the issue for all Windows at the moment and then make the check
> more
> > specific if Microsoft choose to avoid the problem by some other means.
> 
> That's only if "viridian=1" in the guest config, isn't it?

Yes, but most people are going to run with that on for HVM guests; in general there's no penalty for doing so unless you're running some old and buggy OS. If it's turned off then there's nothing Xen can do other than continue to behave as it does now.

> Plus - how would
> we know MS chose to fix their apparent issue?
> 

Well either we'd hear about it or we can run tests on each new version of Windows (it wouldn't be hard to add a specific test for this and add it XenServer's automated suite). 

  Paul

> Jan

Re: Restoring FPU exception state

[Jan Beulich]

>>> On 18.02.16 at 09:54, <Paul.Durrant@citrix.com> wrote:
>>  -----Original Message-----
>> From: Jan Beulich [mailto:JBeulich@suse.com]
>> Sent: 18 February 2016 08:50
>> To: Paul Durrant; Kevin Tian
>> Cc: Andrew Cooper; David Vrabel; Feng Wu; xen-devel
>> Subject: RE: Restoring FPU exception state
>> 
>> >>> On 18.02.16 at 09:41, <Paul.Durrant@citrix.com> wrote:
>> >>  -----Original Message-----
>> >> From: Jan Beulich [mailto:JBeulich@suse.com]
>> >> Sent: 18 February 2016 08:16
>> >> To: Kevin Tian
>> >> Cc: Andrew Cooper; David Vrabel; Paul Durrant; Feng Wu; xen-devel
>> >> Subject: RE: Restoring FPU exception state
>> >>
>> >> >>> On 18.02.16 at 07:30, <kevin.tian@intel.com> wrote:
>> >> > Interesting. Let me also have a check internally whether there is other
>> >> > architectural alternative. BTW, not quite related. Could I think finally
>> >> > Xen may allow user to specify OS type as a general per-domain control,
>> >> > and then Xen can do free optimizations underlying based on that type?
>> >> > I don't expect we want to expose raw FPU related per-domain control
>> >> > since it's difficult for users to correctly set it up...
>> >>
>> >> So what if then, with us implying some kind of workaround here
>> >> from the OS type, someone set the OS type to Windows, and
>> >> subsequently MS decided to fix their issue?
>> >
>> > We already get full OS version information from Windows via the
>> > VIRIDIAN_MSR_GUEST_OS_ID MSR so it would be quite straightforward to
>> work
>> > around the issue for all Windows at the moment and then make the check
>> more
>> > specific if Microsoft choose to avoid the problem by some other means.
>> 
>> That's only if "viridian=1" in the guest config, isn't it?
> 
> Yes, but most people are going to run with that on for HVM guests;

That view may be pretty XenServer centric.

> in 
> general there's no penalty for doing so unless you're running some old and 
> buggy OS. If it's turned off then there's nothing Xen can do other than 
> continue to behave as it does now.
> 
>> Plus - how would
>> we know MS chose to fix their apparent issue?
>> 
> 
> Well either we'd hear about it or we can run tests on each new version of 
> Windows (it wouldn't be hard to add a specific test for this and add it 
> XenServer's automated suite). 

Well, if that also covers kernel updates for existing versions of
Windows.

But taking both together, defaulting the behavior based on the
Viridian flag is certainly an option, just that we'd need an override
both ways nevertheless.

Jan

Re: Restoring FPU exception state

[Paul Durrant]

> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: 18 February 2016 09:24
> To: Paul Durrant
> Cc: Andrew Cooper; David Vrabel; Feng Wu; Kevin Tian; xen-devel
> Subject: RE: Restoring FPU exception state
> 
> >>> On 18.02.16 at 09:54, <Paul.Durrant@citrix.com> wrote:
> >>  -----Original Message-----
> >> From: Jan Beulich [mailto:JBeulich@suse.com]
> >> Sent: 18 February 2016 08:50
> >> To: Paul Durrant; Kevin Tian
> >> Cc: Andrew Cooper; David Vrabel; Feng Wu; xen-devel
> >> Subject: RE: Restoring FPU exception state
> >>
> >> >>> On 18.02.16 at 09:41, <Paul.Durrant@citrix.com> wrote:
> >> >>  -----Original Message-----
> >> >> From: Jan Beulich [mailto:JBeulich@suse.com]
> >> >> Sent: 18 February 2016 08:16
> >> >> To: Kevin Tian
> >> >> Cc: Andrew Cooper; David Vrabel; Paul Durrant; Feng Wu; xen-devel
> >> >> Subject: RE: Restoring FPU exception state
> >> >>
> >> >> >>> On 18.02.16 at 07:30, <kevin.tian@intel.com> wrote:
> >> >> > Interesting. Let me also have a check internally whether there is
> other
> >> >> > architectural alternative. BTW, not quite related. Could I think finally
> >> >> > Xen may allow user to specify OS type as a general per-domain
> control,
> >> >> > and then Xen can do free optimizations underlying based on that
> type?
> >> >> > I don't expect we want to expose raw FPU related per-domain
> control
> >> >> > since it's difficult for users to correctly set it up...
> >> >>
> >> >> So what if then, with us implying some kind of workaround here
> >> >> from the OS type, someone set the OS type to Windows, and
> >> >> subsequently MS decided to fix their issue?
> >> >
> >> > We already get full OS version information from Windows via the
> >> > VIRIDIAN_MSR_GUEST_OS_ID MSR so it would be quite straightforward
> to
> >> work
> >> > around the issue for all Windows at the moment and then make the
> check
> >> more
> >> > specific if Microsoft choose to avoid the problem by some other means.
> >>
> >> That's only if "viridian=1" in the guest config, isn't it?
> >
> > Yes, but most people are going to run with that on for HVM guests;
> 
> That view may be pretty XenServer centric.

It may be but really shouldn't be. Setting viridian on by default for HVM guests has been pretty safe for a long time.

> 
> > in
> > general there's no penalty for doing so unless you're running some old and
> > buggy OS. If it's turned off then there's nothing Xen can do other than
> > continue to behave as it does now.
> >
> >> Plus - how would
> >> we know MS chose to fix their apparent issue?
> >>
> >
> > Well either we'd hear about it or we can run tests on each new version of
> > Windows (it wouldn't be hard to add a specific test for this and add it
> > XenServer's automated suite).
> 
> Well, if that also covers kernel updates for existing versions of
> Windows.
> 

It should do; there's build number and service pack info in the OS id. I'd be disappointed if at least one of these didn't change with a kernel update.

> But taking both together, defaulting the behavior based on the
> Viridian flag is certainly an option, just that we'd need an override
> both ways nevertheless.
> 

Yes, there should definitely be some way to override the default choice.

  Paul

> Jan

Re: Restoring FPU exception state

[Tian, Kevin]

> From: Tian, Kevin
> Sent: Thursday, February 18, 2016 2:31 PM
> 
> >
> > > If the VCPU is descheduled between these two checks, the contents of
> > > FCS/FDS is lost, Windows will notice and BugCheck.
> > >
> > > When saving a VCPUs FPU state, Xen first uses a REX.W prefixed XSAVE and
> > > notices that FIP/FDP[64:32] is non-zero and assumes are REX.W prefixed
> > > XRSTOR is required to restore the full 64-bits of FIP/FDP.  This clears
> > > FCS/FDS.
> > >
> > > On processors with FPCSDS[2] (bit 13) set in CPUID leaf 0x7, sub-leaf
> > > 0x0, do not save FCS/FDS (they always write zeros) and this problem does
> > > not occur, because FCS/FDS never needs to be restored.
> > >
> > > Does anyone any thoughts of a solution for processors without the FPCSDS
> > > feature?
> >
> > One would assume they have a solution to this problem on Hyper-V,
> > but then again their solution may simply be that they don't use REX
> > prefixes anywhere (i.e. namely also not in context switch code). In
> > which case, though, they'd corrupt state of their non-Windows guests.
> >
> > But to answer your question: I have no idea. The handling of these
> > FPU code/data pointers has been an unloved child from the very
> > beginning of the x86-64 architecture. All I could see us doing would
> > be to add a per-domain control to override the auto-detection in the
> > XSAVE code path.
> >
> 
> Interesting. Let me also have a check internally whether there is other
> architectural alternative.

Looks there is no other good alternative. FPCSDS is right introduced
to address this issue, but then it means only for newer platform.

Thanks
Kevin

Re: Restoring FPU exception state

[Jan Beulich]

>>> On 24.02.16 at 11:33, <kevin.tian@intel.com> wrote:
>>  From: Tian, Kevin
>> Sent: Thursday, February 18, 2016 2:31 PM
>> 
>> >
>> > > If the VCPU is descheduled between these two checks, the contents of
>> > > FCS/FDS is lost, Windows will notice and BugCheck.
>> > >
>> > > When saving a VCPUs FPU state, Xen first uses a REX.W prefixed XSAVE and
>> > > notices that FIP/FDP[64:32] is non-zero and assumes are REX.W prefixed
>> > > XRSTOR is required to restore the full 64-bits of FIP/FDP.  This clears
>> > > FCS/FDS.
>> > >
>> > > On processors with FPCSDS[2] (bit 13) set in CPUID leaf 0x7, sub-leaf
>> > > 0x0, do not save FCS/FDS (they always write zeros) and this problem does
>> > > not occur, because FCS/FDS never needs to be restored.
>> > >
>> > > Does anyone any thoughts of a solution for processors without the FPCSDS
>> > > feature?
>> >
>> > One would assume they have a solution to this problem on Hyper-V,
>> > but then again their solution may simply be that they don't use REX
>> > prefixes anywhere (i.e. namely also not in context switch code). In
>> > which case, though, they'd corrupt state of their non-Windows guests.
>> >
>> > But to answer your question: I have no idea. The handling of these
>> > FPU code/data pointers has been an unloved child from the very
>> > beginning of the x86-64 architecture. All I could see us doing would
>> > be to add a per-domain control to override the auto-detection in the
>> > XSAVE code path.
>> >
>> 
>> Interesting. Let me also have a check internally whether there is other
>> architectural alternative.
> 
> Looks there is no other good alternative. FPCSDS is right introduced
> to address this issue, but then it means only for newer platform.

Except that it breaks migration from older hosts (storing these fields)
to newer hosts (storing zero instead). I really think this feature
should have been an opt-in one (i.e. to be enabled via some control
register bit), instead of one thats always on.

Jan