Re: XSAVE save/restore shortcomings

[Paolo Bonzini <pbonzini@redhat.com>]

[adding H. Peter Anvin... the context is whether the layout of the
XSAVE/XRSTOR area is fixed, including the offset of each separate
Ext_SAVE_Area].

Il 06/09/2013 09:45, Zhang, Yang Z ha scritto:
>>>> >>> So here we are: Paolo quotes the doc stating that the layout is
>>>> >>> fixed
>>> >> Layout is not equal to the offset.
>> > 
>> > Sorry, I don't follow. What does "The layout of the XSAVE/XRSTOR save
>> > area is fixed" mean to you then?
> My understanding is that "fixed" means "the first area always is
> FPU/SSE(offset 0, size 512), the second area always is header(offset
> 512, size64) and for other areas, you must check the CPUID to know the
> details". Not the offset of each feature is fixed.

The sentence is "The XSAVE/XRSTOR save area is not compacted if some
features are not saved or are not supported by the processor and/or by
system software".  This means four things:

- the XSAVE/XRSTOR save area is not compacted if some features are not
saved by the processor (my guess: this is for XSAVEOPT)

- the XSAVE/XRSTOR save area is not compacted if some features are not
saved by system software (my guess: bits are not set in EDX:EAX)

- the the XSAVE/XRSTOR save area is not compacted if some features are
not supported by system software (my guess: bits are not set in XCR0)

- the the XSAVE/XRSTOR save area is not compacted if some features are
not supported by the processor (???)

I cannot give any sensible interpretation to the last case except "the
offsets are fixed".  In theory the lengths may vary, I suppose, though
that would not make much sense.

BTW, in my copy of the manual (325383-045US, January 2013) the
Ext_SaveAreas are in consecutive order in "Figure 13-2. Future Layout of
XSAVE/XRSTOR Area and XSTATE_BV with Five Sets of Processor State
Extensions".

And in the AVX-512/MPX manual, the vector states in Ext_SAVE_Area_2 to 7
have documented (thus fixed) offsets, while the MPX states in
Ext_SAVE_Area_3 and 4 do not have documented offsets.  And MPX is
exactly where documented offsets would be most handy, since BNDCFGU and
BNDSTATUS are only accessible via XSAVE/XRSTOR.

Linux publishes the XSAVE blob to userspace as an NT_X86_XSTATE note in
core dumps, but does not include CPUID data.  If offsets change, it will
not be possible to examine a core dump without knowing the processor on
which it was produced.

For our beloved hypervisors, if offsets can change it will be
_impossible_ to migrate from a machine to another if they have different
offsets.  It will be impossible (lest you disable XSAVE and thus all
processor features starting with AVX) to have clusters of heterogeneous
virtualization hosts.  This is because (a) the guest might have stored
XSAVE areas at arbitrary places in the source, and the destination will
not be able to restore them; (b) there are no vmexits for
XSAVE/XSAVEOPT/XRSTOR, and anyway they would be too slow.

So please Intel, pretty please do not modify the XSAVE offsets, and
clarify this as soon as possible.

Paolo


> See the Table 4-20 in Intel SDM volume 2, it even puts the area3 behind area4.
>