[PATCH v5 0/4] ARM: vfp: Introduce vfp_lock() for VFP locking.

[PATCH v5 0/4] ARM: vfp: Introduce vfp_lock() for VFP locking.

[Sebastian Andrzej Siewior]

There was a bug report on PREEMPT_RT which made me look into VFP locking
on ARM. The usage of local_bh_disable() to ensure exclusive access to
the VFP unit is not working on PREEMPT_RT because the softirq context is
preemptible.

This series introduces vfp_lock() which does the right thing.

v3…v4:
    - Repost.

v2…v3: https://lore.kernel.org/all/20230926125502.1127015-1-bigeasy@linutronix.de
    - Repost with Ard's Reviewed-by tag.

v1…v2: https://lore.kernel.org/all/20230628080516.798032-1-bigeasy@linutronix.de
    - Split the last patch into two patches (3 and 4 of this series).
      Suggested by Ard.

v1: https://lore.kernel.org/all/20230615101656.1308942-1-bigeasy@linutronix.de

Sebastian

[PATCH v5 1/4] ARM: vfp: Provide vfp_lock() for VFP locking.

[Sebastian Andrzej Siewior]

kernel_neon_begin() uses local_bh_disable() to ensure exclusive access
to the VFP unit. This is broken on PREEMPT_RT because a BH disabled
section remains preemptible on PREEMPT_RT.

Introduce vfp_lock() which uses local_bh_disable() and preempt_disable()
on PREEMPT_RT. Since softirqs are processed always in thread context,
disabling preemption is enough to ensure that the current context won't
get interrupted by something that is using the VFP. Use it in
kernel_neon_begin().

Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 arch/arm/vfp/vfpmodule.c | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)

diff --git a/arch/arm/vfp/vfpmodule.c b/arch/arm/vfp/vfpmodule.c
index b68efe643a12c..7d234fc82ebf6 100644
--- a/arch/arm/vfp/vfpmodule.c
+++ b/arch/arm/vfp/vfpmodule.c
@@ -55,6 +55,34 @@ extern unsigned int VFP_arch_feroceon __alias(VFP_arch);
  */
 union vfp_state *vfp_current_hw_state[NR_CPUS];
 
+/*
+ * Claim ownership of the VFP unit.
+ *
+ * The caller may change VFP registers until vfp_unlock() is called.
+ *
+ * local_bh_disable() is used to disable preemption and to disable VFP
+ * processing in softirq context. On PREEMPT_RT kernels local_bh_disable() is
+ * not sufficient because it only serializes soft interrupt related sections
+ * via a local lock, but stays preemptible. Disabling preemption is the right
+ * choice here as bottom half processing is always in thread context on RT
+ * kernels so it implicitly prevents bottom half processing as well.
+ */
+static void vfp_lock(void)
+{
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+		local_bh_disable();
+	else
+		preempt_disable();
+}
+
+static void vfp_unlock(void)
+{
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+		local_bh_enable();
+	else
+		preempt_enable();
+}
+
 /*
  * Is 'thread's most up to date state stored in this CPUs hardware?
  * Must be called from non-preemptible context.
@@ -837,7 +865,7 @@ void kernel_neon_begin(void)
 	unsigned int cpu;
 	u32 fpexc;
 
-	local_bh_disable();
+	vfp_lock();
 
 	/*
 	 * Kernel mode NEON is only allowed outside of hardirq context with
@@ -868,7 +896,7 @@ void kernel_neon_end(void)
 {
 	/* Disable the NEON/VFP unit. */
 	fmxr(FPEXC, fmrx(FPEXC) & ~FPEXC_EN);
-	local_bh_enable();
+	vfp_unlock();
 }
 EXPORT_SYMBOL(kernel_neon_end);
 
-- 
2.45.2

[PATCH v5 2/4] ARM: vfp: Use vfp_lock() in vfp_sync_hwstate().

[Sebastian Andrzej Siewior]

vfp_sync_hwstate() uses preempt_disable() followed by local_bh_disable()
to ensure that it won't get interrupted while checking the VFP state.
This harms PREEMPT_RT because softirq handling can get preempted and
local_bh_disable() synchronizes the related section with a sleeping lock
which does not work with disabled preemption.

Use the vfp_lock() to synchronize the access.

Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 arch/arm/vfp/vfpmodule.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/arch/arm/vfp/vfpmodule.c b/arch/arm/vfp/vfpmodule.c
index 7d234fc82ebf6..49ccd205ad780 100644
--- a/arch/arm/vfp/vfpmodule.c
+++ b/arch/arm/vfp/vfpmodule.c
@@ -540,11 +540,9 @@ static inline void vfp_pm_init(void) { }
  */
 void vfp_sync_hwstate(struct thread_info *thread)
 {
-	unsigned int cpu = get_cpu();
+	vfp_lock();
 
-	local_bh_disable();
-
-	if (vfp_state_in_hw(cpu, thread)) {
+	if (vfp_state_in_hw(raw_smp_processor_id(), thread)) {
 		u32 fpexc = fmrx(FPEXC);
 
 		/*
@@ -555,8 +553,7 @@ void vfp_sync_hwstate(struct thread_info *thread)
 		fmxr(FPEXC, fpexc);
 	}
 
-	local_bh_enable();
-	put_cpu();
+	vfp_unlock();
 }
 
 /* Ensure that the thread reloads the hardware VFP state on the next use. */
-- 
2.45.2

[PATCH v5 3/4] ARM: vfp: Use vfp_lock() in vfp_support_entry().

[Sebastian Andrzej Siewior]

vfp_entry() is invoked from exception handler and is fully preemptible.
It uses local_bh_disable() to remain uninterrupted while checking the
VFP state.
This is not working on PREEMPT_RT because local_bh_disable()
synchronizes the relevant section but the context remains fully
preemptible.

Use vfp_lock() for uninterrupted access.

Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 arch/arm/vfp/vfpmodule.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm/vfp/vfpmodule.c b/arch/arm/vfp/vfpmodule.c
index 49ccd205ad780..6ebd5d4b8fa29 100644
--- a/arch/arm/vfp/vfpmodule.c
+++ b/arch/arm/vfp/vfpmodule.c
@@ -708,7 +708,7 @@ static int vfp_support_entry(struct pt_regs *regs, u32 trigger)
 	if (!user_mode(regs))
 		return vfp_kmode_exception(regs, trigger);
 
-	local_bh_disable();
+	vfp_lock();
 	fpexc = fmrx(FPEXC);
 
 	/*
@@ -787,7 +787,7 @@ static int vfp_support_entry(struct pt_regs *regs, u32 trigger)
 			if (!(fpscr & FPSCR_IXE)) {
 				if (!(fpscr & FPSCR_LENGTH_MASK)) {
 					pr_debug("not VFP\n");
-					local_bh_enable();
+					vfp_unlock();
 					return -ENOEXEC;
 				}
 				fpexc |= FPEXC_DEX;
@@ -797,7 +797,7 @@ bounce:		regs->ARM_pc += 4;
 		VFP_bounce(trigger, fpexc, regs);
 	}
 
-	local_bh_enable();
+	vfp_unlock();
 	return 0;
 }
 
-- 
2.45.2

[PATCH v5 4/4] ARM: vfp: Move sending signals outside of vfp_lock()ed section.

[Sebastian Andrzej Siewior]

VFP_bounce() is invoked from within vfp_support_entry() and may send a
signal. Sending a signal uses spinlock_t which becomes a sleeping lock
on PREEMPT_RT and must not be acquired within a preempt-disabled
section.

Move the vfp_raise_sigfpe() block outside of the vfp_lock() section.

Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 arch/arm/vfp/vfpmodule.c | 29 ++++++++++++++++++-----------
 1 file changed, 18 insertions(+), 11 deletions(-)

diff --git a/arch/arm/vfp/vfpmodule.c b/arch/arm/vfp/vfpmodule.c
index 6ebd5d4b8fa29..48745a3c52618 100644
--- a/arch/arm/vfp/vfpmodule.c
+++ b/arch/arm/vfp/vfpmodule.c
@@ -268,7 +268,7 @@ static void vfp_panic(char *reason, u32 inst)
 /*
  * Process bitmask of exception conditions.
  */
-static void vfp_raise_exceptions(u32 exceptions, u32 inst, u32 fpscr, struct pt_regs *regs)
+static int vfp_raise_exceptions(u32 exceptions, u32 inst, u32 fpscr)
 {
 	int si_code = 0;
 
@@ -276,8 +276,7 @@ static void vfp_raise_exceptions(u32 exceptions, u32 inst, u32 fpscr, struct pt_
 
 	if (exceptions == VFP_EXCEPTION_ERROR) {
 		vfp_panic("unhandled bounce", inst);
-		vfp_raise_sigfpe(FPE_FLTINV, regs);
-		return;
+		return FPE_FLTINV;
 	}
 
 	/*
@@ -305,8 +304,7 @@ static void vfp_raise_exceptions(u32 exceptions, u32 inst, u32 fpscr, struct pt_
 	RAISE(FPSCR_OFC, FPSCR_OFE, FPE_FLTOVF);
 	RAISE(FPSCR_IOC, FPSCR_IOE, FPE_FLTINV);
 
-	if (si_code)
-		vfp_raise_sigfpe(si_code, regs);
+	return si_code;
 }
 
 /*
@@ -352,6 +350,8 @@ static u32 vfp_emulate_instruction(u32 inst, u32 fpscr, struct pt_regs *regs)
 static void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs)
 {
 	u32 fpscr, orig_fpscr, fpsid, exceptions;
+	int si_code2 = 0;
+	int si_code = 0;
 
 	pr_debug("VFP: bounce: trigger %08x fpexc %08x\n", trigger, fpexc);
 
@@ -397,8 +397,8 @@ static void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs)
 		 * unallocated VFP instruction but with FPSCR.IXE set and not
 		 * on VFP subarch 1.
 		 */
-		 vfp_raise_exceptions(VFP_EXCEPTION_ERROR, trigger, fpscr, regs);
-		return;
+		si_code = vfp_raise_exceptions(VFP_EXCEPTION_ERROR, trigger, fpscr);
+		goto exit;
 	}
 
 	/*
@@ -422,14 +422,14 @@ static void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs)
 	 */
 	exceptions = vfp_emulate_instruction(trigger, fpscr, regs);
 	if (exceptions)
-		vfp_raise_exceptions(exceptions, trigger, orig_fpscr, regs);
+		si_code2 = vfp_raise_exceptions(exceptions, trigger, orig_fpscr);
 
 	/*
 	 * If there isn't a second FP instruction, exit now. Note that
 	 * the FPEXC.FP2V bit is valid only if FPEXC.EX is 1.
 	 */
 	if ((fpexc & (FPEXC_EX | FPEXC_FP2V)) != (FPEXC_EX | FPEXC_FP2V))
-		return;
+		goto exit;
 
 	/*
 	 * The barrier() here prevents fpinst2 being read
@@ -441,7 +441,13 @@ static void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs)
  emulate:
 	exceptions = vfp_emulate_instruction(trigger, orig_fpscr, regs);
 	if (exceptions)
-		vfp_raise_exceptions(exceptions, trigger, orig_fpscr, regs);
+		si_code = vfp_raise_exceptions(exceptions, trigger, orig_fpscr);
+exit:
+	vfp_unlock();
+	if (si_code2)
+		vfp_raise_sigfpe(si_code2, regs);
+	if (si_code)
+		vfp_raise_sigfpe(si_code, regs);
 }
 
 static void vfp_enable(void *unused)
@@ -773,6 +779,7 @@ static int vfp_support_entry(struct pt_regs *regs, u32 trigger)
 		 * replay the instruction that trapped.
 		 */
 		fmxr(FPEXC, fpexc);
+		vfp_unlock();
 	} else {
 		/* Check for synchronous or asynchronous exceptions */
 		if (!(fpexc & (FPEXC_EX | FPEXC_DEX))) {
@@ -794,10 +801,10 @@ static int vfp_support_entry(struct pt_regs *regs, u32 trigger)
 			}
 		}
 bounce:		regs->ARM_pc += 4;
+		/* VFP_bounce() will invoke vfp_unlock() */
 		VFP_bounce(trigger, fpexc, regs);
 	}
 
-	vfp_unlock();
 	return 0;
 }
 
-- 
2.45.2

Re: [PATCH v5 0/4] ARM: vfp: Introduce vfp_lock() for VFP locking.

[Russell King (Oracle)]

On Fri, Jun 28, 2024 at 03:46:15PM +0200, Sebastian Andrzej Siewior wrote:
> There was a bug report on PREEMPT_RT which made me look into VFP locking
> on ARM. The usage of local_bh_disable() to ensure exclusive access to
> the VFP unit is not working on PREEMPT_RT because the softirq context is
> preemptible.
> 
> This series introduces vfp_lock() which does the right thing.

I disagre with the term "lock" here - it isn't about any kind of
locking, it's about:

1. preventing the thread moving onto a different CPU
2. preventing kernel-mode usage of VFP while a VFP fault is being
   handled (and thus changing the VFP state.)

The proble with "lock" is that it makes it look like this code is
single-threaded which it isn't. Many CPUs can be processing a VFP
fault at the same time - remembering that each CPU has their own
VFP unit.

Maybe name it vfp_state_hold() ... vfp_state_release() instead?
Hold as in "don't let anything interfere with the state we are
going to be accessing".

I'm open to better names if anyone can think of any!

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: [PATCH v5 0/4] ARM: vfp: Introduce vfp_lock() for VFP locking.

[Sebastian Andrzej Siewior]

On 2024-06-28 15:07:37 [+0100], Russell King (Oracle) wrote:
> On Fri, Jun 28, 2024 at 03:46:15PM +0200, Sebastian Andrzej Siewior wrote:
> > There was a bug report on PREEMPT_RT which made me look into VFP locking
> > on ARM. The usage of local_bh_disable() to ensure exclusive access to
> > the VFP unit is not working on PREEMPT_RT because the softirq context is
> > preemptible.
> > 
> > This series introduces vfp_lock() which does the right thing.
> 
> I disagre with the term "lock" here - it isn't about any kind of
> locking, it's about:
> 
> 1. preventing the thread moving onto a different CPU
> 2. preventing kernel-mode usage of VFP while a VFP fault is being
>    handled (and thus changing the VFP state.)
> 
> The proble with "lock" is that it makes it look like this code is
> single-threaded which it isn't. Many CPUs can be processing a VFP
> fault at the same time - remembering that each CPU has their own
> VFP unit.

Yes, from that point of view it looks bad. Given that every CPU has a
VFP unit it is more of a per-CPU lock. But this isn't obvious from just
looking at it.

> Maybe name it vfp_state_hold() ... vfp_state_release() instead?
> Hold as in "don't let anything interfere with the state we are
> going to be accessing".
> 
> I'm open to better names if anyone can think of any!

No, I'm fine with it.

Sebastian