[PATCH v1 0/2] KVM: arm64: Fix bugs related to mp_state updates

[PATCH v1 0/2] KVM: arm64: Fix bugs related to mp_state updates

[Reiji Watanabe]

This series adds fixes that were missing in the patch [1].

The patch [1] added the mp_state_lock to serialize writes to
kvm_vcpu_arch::{mp_state, reset_state}, and promoted all
accessors of mp_state to {READ,WRITE}_ONCE() as readers do not
acquire the mp_state_lock.

Since the patch [1] didn't fix all the relevant code, fix the
code that weren't addressed yet.

This series is based on v6.3-rc7 with the series [2] applied.

[1] https://lore.kernel.org/all/20230327164747.2466958-2-oliver.upton@linux.dev/
[2] https://lore.kernel.org/all/20230327164747.2466958-1-oliver.upton@linux.dev/

Reiji Watanabe (2):
  KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()
  KVM: arm64: Have kvm_psci_vcpu_on() use WRITE_ONCE() to update
    mp_state

 arch/arm64/kvm/arm.c  | 5 ++++-
 arch/arm64/kvm/psci.c | 2 +-
 2 files changed, 5 insertions(+), 2 deletions(-)

-- 
2.40.0.396.gfff15efe05-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v1 1/2] KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()

[Reiji Watanabe]

kvm_arch_vcpu_ioctl_vcpu_init() doesn't acquire mp_state_lock
when setting the mp_state to KVM_MP_STATE_RUNNABLE. Fix the
code to acquire the lock.

Signed-off-by: Reiji Watanabe <reijiw@google.com>
---
 arch/arm64/kvm/arm.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index fbafcbbcc463..388aa4f18f21 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1244,8 +1244,11 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
 	 */
 	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
 		kvm_arm_vcpu_power_off(vcpu);
-	else
+	else {
+		spin_lock(&vcpu->arch.mp_state_lock);
 		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
+		spin_unlock(&vcpu->arch.mp_state_lock);
+	}
 
 	return 0;
 }
-- 
2.40.0.396.gfff15efe05-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v1 2/2] KVM: arm64: Have kvm_psci_vcpu_on() use WRITE_ONCE() to update mp_state

[Reiji Watanabe]

All accessors of kvm_vcpu_arch::mp_state should be {READ,WRITE}_ONCE(),
since readers of the mp_state don't acquire the mp_state_lock.
Nonetheless, kvm_psci_vcpu_on() updates the mp_state without using
WRITE_ONCE(). So, fix the code to update the mp_state using WRITE_ONCE.

Signed-off-by: Reiji Watanabe <reijiw@google.com>
---
 arch/arm64/kvm/psci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/psci.c b/arch/arm64/kvm/psci.c
index 5767e6baa61a..d046e82e3723 100644
--- a/arch/arm64/kvm/psci.c
+++ b/arch/arm64/kvm/psci.c
@@ -110,7 +110,7 @@ static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
 	 */
 	smp_wmb();
 
-	vcpu->arch.mp_state.mp_state = KVM_MP_STATE_RUNNABLE;
+	WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
 	kvm_vcpu_wake_up(vcpu);
 
 out_unlock:
-- 
2.40.0.396.gfff15efe05-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v1 1/2] KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()

[Marc Zyngier]

On Wed, 19 Apr 2023 03:18:51 +0100,
Reiji Watanabe <reijiw@google.com> wrote:
> 
> kvm_arch_vcpu_ioctl_vcpu_init() doesn't acquire mp_state_lock
> when setting the mp_state to KVM_MP_STATE_RUNNABLE. Fix the
> code to acquire the lock.
> 
> Signed-off-by: Reiji Watanabe <reijiw@google.com>
> ---
>  arch/arm64/kvm/arm.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index fbafcbbcc463..388aa4f18f21 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -1244,8 +1244,11 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
>  	 */
>  	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
>  		kvm_arm_vcpu_power_off(vcpu);
> -	else
> +	else {
> +		spin_lock(&vcpu->arch.mp_state_lock);
>  		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
> +		spin_unlock(&vcpu->arch.mp_state_lock);
> +	}
>  
>  	return 0;
>  }

I'm not entirely convinced that this fixes anything. What does the
lock hazard against given that the write is atomic? But maybe a
slightly more readable of this would be to expand the critical section
this way:

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 4ec888fdd4f7..bb21d0c25de7 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1246,11 +1246,15 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
 	/*
 	 * Handle the "start in power-off" case.
 	 */
+	spin_lock(&vcpu->arch.mp_state_lock);
+
 	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
-		kvm_arm_vcpu_power_off(vcpu);
+		__kvm_arm_vcpu_power_off(vcpu);
 	else
 		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
 
+	spin_unlock(&vcpu->arch.mp_state_lock);
+
 	return 0;
 }

Thoughts?

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v1 1/2] KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()

[Reiji Watanabe]

Hi Marc,

On Wed, Apr 19, 2023 at 08:12:45AM +0100, Marc Zyngier wrote:
> On Wed, 19 Apr 2023 03:18:51 +0100,
> Reiji Watanabe <reijiw@google.com> wrote:
> > kvm_arch_vcpu_ioctl_vcpu_init() doesn't acquire mp_state_lock
> > when setting the mp_state to KVM_MP_STATE_RUNNABLE. Fix the
> > code to acquire the lock.
> > 
> > Signed-off-by: Reiji Watanabe <reijiw@google.com>
> > ---
> >  arch/arm64/kvm/arm.c | 5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> > 
> > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> > index fbafcbbcc463..388aa4f18f21 100644
> > --- a/arch/arm64/kvm/arm.c
> > +++ b/arch/arm64/kvm/arm.c
> > @@ -1244,8 +1244,11 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
> >  	 */
> >  	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
> >  		kvm_arm_vcpu_power_off(vcpu);
> > -	else
> > +	else {
> > +		spin_lock(&vcpu->arch.mp_state_lock);
> >  		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
> > +		spin_unlock(&vcpu->arch.mp_state_lock);
> > +	}
> >  
> >  	return 0;
> >  }
> 
> I'm not entirely convinced that this fixes anything. What does the
> lock hazard against given that the write is atomic? But maybe a

It appears that kvm_psci_vcpu_on() expects the vCPU's mp_state
to not be changed by holding the lock.  Although I don't think this
code practically causes any real issues now, I am a little concerned
about leaving one instance that updates mpstate without acquiring the
lock, in terms of future maintenance, as holding the lock won't prevent
mp_state from being updated.

What do you think ?

> slightly more readable of this would be to expand the critical section
> this way:
> 
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index 4ec888fdd4f7..bb21d0c25de7 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -1246,11 +1246,15 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
>  	/*
>  	 * Handle the "start in power-off" case.
>  	 */
> +	spin_lock(&vcpu->arch.mp_state_lock);
> +
>  	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
> -		kvm_arm_vcpu_power_off(vcpu);
> +		__kvm_arm_vcpu_power_off(vcpu);
>  	else
>  		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
>  
> +	spin_unlock(&vcpu->arch.mp_state_lock);
> +
>  	return 0;
>  }
> 
> Thoughts?

Yes, it looks better!

Thank you,
Reiji

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v1 0/2] KVM: arm64: Fix bugs related to mp_state updates

[Marc Zyngier]

On Tue, 18 Apr 2023 19:18:50 -0700, Reiji Watanabe wrote:
> This series adds fixes that were missing in the patch [1].
> 
> The patch [1] added the mp_state_lock to serialize writes to
> kvm_vcpu_arch::{mp_state, reset_state}, and promoted all
> accessors of mp_state to {READ,WRITE}_ONCE() as readers do not
> acquire the mp_state_lock.
> 
> [...]

Applied to next, thanks!

[1/2] KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()
      commit: 4ff910be01c0ca28c2ea8b354dd47a3a17524489
[2/2] KVM: arm64: Have kvm_psci_vcpu_on() use WRITE_ONCE() to update mp_state
      commit: a189884bdc9238aeba941c50f02e25eb584fafed

Cheers,

	M.
-- 
Without deviation from the norm, progress is not possible.



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v1 1/2] KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()

[Marc Zyngier]

On Thu, 20 Apr 2023 03:13:02 +0100,
Reiji Watanabe <reijiw@google.com> wrote:
> 
> Hi Marc,
> 
> On Wed, Apr 19, 2023 at 08:12:45AM +0100, Marc Zyngier wrote:
> > On Wed, 19 Apr 2023 03:18:51 +0100,
> > Reiji Watanabe <reijiw@google.com> wrote:
> > > kvm_arch_vcpu_ioctl_vcpu_init() doesn't acquire mp_state_lock
> > > when setting the mp_state to KVM_MP_STATE_RUNNABLE. Fix the
> > > code to acquire the lock.
> > > 
> > > Signed-off-by: Reiji Watanabe <reijiw@google.com>
> > > ---
> > >  arch/arm64/kvm/arm.c | 5 ++++-
> > >  1 file changed, 4 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> > > index fbafcbbcc463..388aa4f18f21 100644
> > > --- a/arch/arm64/kvm/arm.c
> > > +++ b/arch/arm64/kvm/arm.c
> > > @@ -1244,8 +1244,11 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
> > >  	 */
> > >  	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
> > >  		kvm_arm_vcpu_power_off(vcpu);
> > > -	else
> > > +	else {
> > > +		spin_lock(&vcpu->arch.mp_state_lock);
> > >  		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
> > > +		spin_unlock(&vcpu->arch.mp_state_lock);
> > > +	}
> > >  
> > >  	return 0;
> > >  }
> > 
> > I'm not entirely convinced that this fixes anything. What does the
> > lock hazard against given that the write is atomic? But maybe a
> 
> It appears that kvm_psci_vcpu_on() expects the vCPU's mp_state
> to not be changed by holding the lock.  Although I don't think this
> code practically causes any real issues now, I am a little concerned
> about leaving one instance that updates mpstate without acquiring the
> lock, in terms of future maintenance, as holding the lock won't prevent
> mp_state from being updated.
> 
> What do you think ?

Right, fair enough. It is probably better to take the lock and not
have to think of this sort of things... I'm becoming more lazy by the
minute!

> 
> > slightly more readable of this would be to expand the critical section
> > this way:
> > 
> > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> > index 4ec888fdd4f7..bb21d0c25de7 100644
> > --- a/arch/arm64/kvm/arm.c
> > +++ b/arch/arm64/kvm/arm.c
> > @@ -1246,11 +1246,15 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
> >  	/*
> >  	 * Handle the "start in power-off" case.
> >  	 */
> > +	spin_lock(&vcpu->arch.mp_state_lock);
> > +
> >  	if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
> > -		kvm_arm_vcpu_power_off(vcpu);
> > +		__kvm_arm_vcpu_power_off(vcpu);
> >  	else
> >  		WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
> >  
> > +	spin_unlock(&vcpu->arch.mp_state_lock);
> > +
> >  	return 0;
> >  }
> > 
> > Thoughts?
> 
> Yes, it looks better!

Cool. I've applied this change to your patch, applied the series to
the lock inversion branch, and remerged the branch in -next.

We're getting there! ;-)

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v1 1/2] KVM: arm64: Acquire mp_state_lock in kvm_arch_vcpu_ioctl_vcpu_init()

[Reiji Watanabe]

On Thu, Apr 20, 2023 at 1:16 AM Marc Zyngier <maz@kernel.org> wrote:
>
> On Thu, 20 Apr 2023 03:13:02 +0100,
> Reiji Watanabe <reijiw@google.com> wrote:
> >
> > Hi Marc,
> >
> > On Wed, Apr 19, 2023 at 08:12:45AM +0100, Marc Zyngier wrote:
> > > On Wed, 19 Apr 2023 03:18:51 +0100,
> > > Reiji Watanabe <reijiw@google.com> wrote:
> > > > kvm_arch_vcpu_ioctl_vcpu_init() doesn't acquire mp_state_lock
> > > > when setting the mp_state to KVM_MP_STATE_RUNNABLE. Fix the
> > > > code to acquire the lock.
> > > >
> > > > Signed-off-by: Reiji Watanabe <reijiw@google.com>
> > > > ---
> > > >  arch/arm64/kvm/arm.c | 5 ++++-
> > > >  1 file changed, 4 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> > > > index fbafcbbcc463..388aa4f18f21 100644
> > > > --- a/arch/arm64/kvm/arm.c
> > > > +++ b/arch/arm64/kvm/arm.c
> > > > @@ -1244,8 +1244,11 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
> > > >    */
> > > >   if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
> > > >           kvm_arm_vcpu_power_off(vcpu);
> > > > - else
> > > > + else {
> > > > +         spin_lock(&vcpu->arch.mp_state_lock);
> > > >           WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
> > > > +         spin_unlock(&vcpu->arch.mp_state_lock);
> > > > + }
> > > >
> > > >   return 0;
> > > >  }
> > >
> > > I'm not entirely convinced that this fixes anything. What does the
> > > lock hazard against given that the write is atomic? But maybe a
> >
> > It appears that kvm_psci_vcpu_on() expects the vCPU's mp_state
> > to not be changed by holding the lock.  Although I don't think this
> > code practically causes any real issues now, I am a little concerned
> > about leaving one instance that updates mpstate without acquiring the
> > lock, in terms of future maintenance, as holding the lock won't prevent
> > mp_state from being updated.
> >
> > What do you think ?
>
> Right, fair enough. It is probably better to take the lock and not
> have to think of this sort of things... I'm becoming more lazy by the
> minute!
>
> >
> > > slightly more readable of this would be to expand the critical section
> > > this way:
> > >
> > > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> > > index 4ec888fdd4f7..bb21d0c25de7 100644
> > > --- a/arch/arm64/kvm/arm.c
> > > +++ b/arch/arm64/kvm/arm.c
> > > @@ -1246,11 +1246,15 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu,
> > >     /*
> > >      * Handle the "start in power-off" case.
> > >      */
> > > +   spin_lock(&vcpu->arch.mp_state_lock);
> > > +
> > >     if (test_bit(KVM_ARM_VCPU_POWER_OFF, vcpu->arch.features))
> > > -           kvm_arm_vcpu_power_off(vcpu);
> > > +           __kvm_arm_vcpu_power_off(vcpu);
> > >     else
> > >             WRITE_ONCE(vcpu->arch.mp_state.mp_state, KVM_MP_STATE_RUNNABLE);
> > >
> > > +   spin_unlock(&vcpu->arch.mp_state_lock);
> > > +
> > >     return 0;
> > >  }
> > >
> > > Thoughts?
> >
> > Yes, it looks better!
>
> Cool. I've applied this change to your patch, applied the series to
> the lock inversion branch, and remerged the branch in -next.
>
> We're getting there! ;-)

Thank you, Marc!
Reiji

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel