Problem with KVM when using XSAVES in host

Problem with KVM when using XSAVES in host

[Nadav Amit]

I encountered an interesting and annoying problem when KVM uses
XSAVES/XRSTORS.

The problem results from the fact XSAVES does not save the exact value of
XINUSE[1]. See Intel SDM 13.10 “Operation of XSAVES”: “...if RFBM[1] = 1 and
MXCSR does not have the value 1F80H, XSAVEC writes XSTATE_BV[1] as 1 even if
XINUSE[1] = 0”. [ there is a typo in the SDM; they refer to XSAVES in this
section, and probably copy-pasted the sentence]. XINUSE[1] marks whether the
SSE state is in its initial state and not saved in the save-area.

The result of this behaviour, is that when KVM uses XSAVES and then XRSTORS,
it may not restore XINUSE[1] correctly, i.e. restore it to 1, although it
was 0.

This behaviour hurts the “equivalence property” - the VM does not behave as
bare-metal system. Moreover, it may hurt the VM performance if the VM uses
XSAVEOPT (and not XSAVES), has MXCSR with value different than the reset
value of 1F80H and has all SSE registers set to zero. In such case, the VM
would save/restore SSE registers unnecessarily. I don’t know whether such
scenario happens in real workloads.

tl;dr - hypervisors which use XSAVES (and XSAVEC) mess the VM state and may
hurt VM performance. Perhaps KVM should use XSAVE/XSAVEOPT instead.

Regards,
Nadav

Re: Problem with KVM when using XSAVES in host

[Paolo Bonzini]

On 14/12/2014 15:31, Nadav Amit wrote:
> This behaviour hurts the “equivalence property” - the VM does not behave as
> bare-metal system. Moreover, it may hurt the VM performance if the VM uses
> XSAVEOPT (and not XSAVES), has MXCSR with value different than the reset
> value of 1F80H and has all SSE registers set to zero. In such case, the VM
> would save/restore SSE registers unnecessarily. I don’t know whether such
> scenario happens in real workloads.
> 
> tl;dr - hypervisors which use XSAVES (and XSAVEC) mess the VM state and may
> hurt VM performance. Perhaps KVM should use XSAVE/XSAVEOPT instead.

Unfortunately KVM uses whatever the kernel uses.  KVM does not
save/restore itself the FPU state.

The same problem above can happen if the kernel uses XSAVES/XRSTORS and
userspace uses XSAVEOPT.  So it's a processor bug.

Paolo