From mboxrd@z Thu Jan  1 00:00:00 1970
From: Rik van Riel <riel@redhat.com>
Subject: Re: [PATCH 1/2] x86,kvm: move qemu/guest FPU switching out to
 vcpu_run
Date: Wed, 15 Nov 2017 09:40:46 -0500
Message-ID: <1510756846.21121.289.camel@redhat.com>
References: <20171114215424.32214-1-riel@redhat.com>
         <20171114215424.32214-2-riel@redhat.com>
         <CANRm+Cyfd-A-8Q+o+Mgqs7yACsP65x7ETT5f5dqW1K4iaA2Taw@mail.gmail.com>
         <1510715030.21121.287.camel@redhat.com>
         <CANRm+Cxet5FDQWy+aNb21KZBCqbLiRvZM1T6G64j=S7b8_9kfw@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Cc: Paolo Bonzini <pbonzini@redhat.com>, kvm <kvm@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        David Hildenbrand <david@redhat.com>,
        Christian Borntraeger <borntraeger@de.ibm.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Radim Krcmar <rkrcmar@redhat.com>
To: Wanpeng Li <kernellwp@gmail.com>
Return-path: <kvm-owner@vger.kernel.org>
Received: from mx1.redhat.com ([209.132.183.28]:59798 "EHLO mx1.redhat.com"
        rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
        id S1754013AbdKOOku (ORCPT <rfc822;kvm@vger.kernel.org>);
        Wed, 15 Nov 2017 09:40:50 -0500
In-Reply-To: <CANRm+Cxet5FDQWy+aNb21KZBCqbLiRvZM1T6G64j=S7b8_9kfw@mail.gmail.com>
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

On Wed, 2017-11-15 at 12:33 +0800, Wanpeng Li wrote:
> 2017-11-15 11:03 GMT+08:00 Rik van Riel <riel@redhat.com>:
> > On Wed, 2017-11-15 at 08:47 +0800, Wanpeng Li wrote:
> > > 2017-11-15 5:54 GMT+08:00  <riel@redhat.com>:
> > > > From: Rik van Riel <riel@redhat.com>
> > > > 
> > > > Currently, every time a VCPU is scheduled out, the host kernel
> > > > will
> > > > first save the guest FPU/xstate context, then load the qemu
> > > > userspace
> > > > FPU context, only to then immediately save the qemu userspace
> > > > FPU
> > > > context back to memory. When scheduling in a VCPU, the same
> > > > extraneous
> > > > FPU loads and saves are done.
> > > > 
> > > > This could be avoided by moving from a model where the guest
> > > > FPU is
> > > > loaded and stored with preemption disabled, to a model where
> > > > the
> > > > qemu userspace FPU is swapped out for the guest FPU context for
> > > > the duration of the KVM_RUN ioctl.
> > > 
> > > What will happen if CONFIG_PREEMPT is enabled?
> > 
> > The scheduler will save the guest FPU context when a
> > VCPU thread is preempted, and restore it when it is
> > scheduled back in.
> 
> I mean all the involved processes will use fpu. Before patch if
> kernel
> preempt occur:
> 
> context_switch
>   -> prepare_task_switch
>         -> fire_sched_out_preempt_notifiers
>               -> kvm_sched_out
>                     -> kvm_arch_vcpu_put
>                           -> kvm_put_guest_fpu
>                                -> copy_fpregs_to_fpstate(&vcpu-
> >arch.guest_fpu)
>                                     save xsave area to guest fpu
> buffer
>                                -> __kernel_fpu_end
>                                      ->
> copy_kernel_to_fpregs(&current->thread.fpu.state)
>                                          restore prev vCPU qemu
> userspace FPU to the xsave area
>   -> switch_to
>         -> __switch_to
>             -> switch_fpu_prepare
>                   -> copy_fpregs_to_fpstate => save xsave area to
> prev
> vCPU qemu userspace FPU
>             -> switch_fpu_finish
>                   -> copy_kernel_to_fpgregs => restore next task FPU
> to xsave area
> 
> 
> After the patch:
> 
> context_switch
>   -> prepare_task_switch
>         -> fire_sched_out_preempt_notifiers
>               -> kvm_sched_out
> 
>  -> switch_to
>         -> __switch_to
>             -> switch_fpu_prepare
>                   -> copy_fpregs_to_fpstate         => Oops
>                   save xsave area to prev vCPU qemu userspace FPU,
> actually the guest FPU buffer is loaded in xsave area, you transmit
> guest FPU in xsave area into the prev vCPU qemu userspace FPU

When entering kvm_arch_vcpu_ioctl_run we save the qemu userspace
FPU context in &vcpu->arch.user_fpu, and we restore that before
leaving kvm_arch_vcpu_ioctl_run.

Userspace should always see the userspace FPU context, no?

Am I overlooking anything?