Re: [PATCH] x86/kvm: Account for fpstate->user_xfeatures changes

[Tyler Stachecki <stachecki.tyler@gmail.com>]

On Fri, Sep 15, 2023 at 04:41:20AM -0300, Leonardo Bras wrote:
> Other than that, all I can think of is removing the features from guest:
> 
> As you commented, there may be some features that would not be a problem 
> to be removed, and also there may be features which are not used by the 
> workload, and could be removed. But this would depend on the feature, and 
> the workload, beind a custom solution for every case.

Yes, the "fixup back" should be refined to pointed and verified cases.
 
> For this (removing guest features), from kernel side, I would suggest using 
> SystemTap (and eBPF, IIRC). The procedures should be something like:
> - Try to migrate VM from host with older kernel: fail
> - Look at qemu error, which features are missing?
> - Are those features safely removable from guest ? 
>   - If so, get an SystemTap / eBPF script masking out the undesired bits.
>   - Try the migration again, it should succeed.
> 
> IIRC, this could also be done in qemu side, with a custom qemu:
> - Try to migrate VM from host with older kernel: fail
> - Look at qemu error, which features are missing?
> - Are those features safely removable from guest ?
>   - If so, get a custom qemu which mask-out the desired flags before the VM 
>     starts
>   - Live migrate (can be inside the source host) to the custom qemu
>   - Live migrate from custom qemu to target host.
> - The custom qemu could be on a auxiliary host, and used only for this
> 
> Yes, it's hard, takes time, and may not solve every case, but it gets a 
> higher chance of the VM surviving in the long run.

Thank you for taking the time to throughly consider the issue and suggest some
ways out - I really appreciate it.

> But keep in mind this is a hack.
> Taking features from a live guest is not supported in any way, and has a 
> high chance of crashing the VM.

OK - if there's no interest in the below, I will not push for including this
patch in the kernel tree any longer. I do think the specific case below is what
a vast majority of KVM users will struggle with in the near future, though:

I have a test environment with Broadwell-based (have only AVX-256) guests
running under Skylake (PKRU, AVX512, ...) hypervisors.

I added some pr_debug statements to a guest kernel running under a hypervisor,
with said hypervisor containing neither your nor my patches, and printed the
guests view of `fpu_kernel_cfg.max_features` at boot. It was 0x7, or:
  XFEATURE_MASK_FP, XFEATURE_MASK_SSE, XFEATURE_MASK_YMM

Thus, I'm pretty sure that all that's happening here is that the guest's FP
context is having PKRU/ZMM. saved and restored needlessly by the hypervisor.
Stripping it on a live-migration does not seem to have any ill-effects in
all the testing I have done.

Cheers,
Tyler