[PATCH dovetail 6.11 v2 0/3] RISC-V: IRQ pipelining core

[PATCH dovetail 6.11 v2 0/3] RISC-V: IRQ pipelining core

[Tobias Schaffner]

Hi all,

This introduces IRQ pipelining for RISC-V into Dovetail. The patches
are on top of v6.11-dovetail-rebase. Feel free to ask for other targets
if needed.

I thought about splitting the patch up into smaller parts, but decided
to stick to the structure of the analogous patches of the other
architectures. Maybe this will make the rebasing process easier.

Testing was done in QEMU with IRQ pipeline torture tests, both with and
without SMP enabled.

Changes since v1:
* Removed some dovetail specific changes. These will be readded in a
  later dovetail patch series. Thanks for the kind review @rpm!
* Added a patch that guards _TIF_RETUSER with an CONFIG_DOVETAIL ifdef
  to allow building with IRQ pipelining but without dovetail being
  implemented.
* Fix unneeded reordering in riscv irqflags.h to make the diff nicer.

Tobias Schaffner (3):
  dovetail: Guard _TIF_RETUSER with CONFIG_DOVETAIL
  clocksource/timer-riscv: irq_pipeline: enable pipelined clock events
  RISC-V: irq_pipeline: add IRQ pipelining core

 arch/riscv/Kconfig                         |   1 +
 arch/riscv/include/asm/irq_pipeline.h      | 143 +++++++++++++++++++++
 arch/riscv/include/asm/irqflags.h          |  25 ++--
 arch/riscv/include/asm/thread_info.h       |   8 ++
 arch/riscv/kernel/Makefile                 |   1 +
 arch/riscv/kernel/irq_pipeline.c           |  26 ++++
 arch/riscv/kernel/smp.c                    |  93 +++++++++++++-
 arch/riscv/kernel/smpboot.c                |   2 +-
 arch/riscv/kernel/traps.c                  |  51 +++++++-
 arch/riscv/mm/fault.c                      |  83 ++++++++++--
 drivers/clocksource/timer-riscv.c          |  11 +-
 drivers/irqchip/irq-riscv-aplic-direct.c   |   3 +-
 drivers/irqchip/irq-riscv-aplic-msi.c      |   3 +-
 drivers/irqchip/irq-riscv-imsic-platform.c |   3 +-
 drivers/irqchip/irq-riscv-intc.c           |   2 +
 drivers/irqchip/irq-sifive-plic.c          |   3 +-
 include/linux/entry-common.h               |   4 +-
 17 files changed, 420 insertions(+), 42 deletions(-)
 create mode 100644 arch/riscv/include/asm/irq_pipeline.h
 create mode 100644 arch/riscv/kernel/irq_pipeline.c

-- 
2.43.0

[PATCH v2 1/3] dovetail: Guard _TIF_RETUSER with CONFIG_DOVETAIL

[Tobias Schaffner]

_TIF_RETUSER has to be guarded by CONFIG_DOVETAIL to allow building
CONFIG_IRQ_PIPELINE without a dovetail implementation.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 include/linux/entry-common.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h
index 900c6e303ab3..8a84f57bfde1 100644
--- a/include/linux/entry-common.h
+++ b/include/linux/entry-common.h
@@ -350,13 +350,15 @@ unsigned long exit_to_user_mode_loop(struct pt_regs *regs,
 
 static inline bool do_retuser(unsigned long ti_work)
 {
-	if (dovetailing() && (ti_work & _TIF_RETUSER)) {
+#ifdef CONFIG_DOVETAIL
+	if (ti_work & _TIF_RETUSER) {
 		hard_local_irq_enable();
 		inband_retuser_notify();
 		hard_local_irq_disable();
 		/* RETUSER might have switched oob */
 		return running_inband();
 	}
+#endif
 
 	return false;
 }
-- 
2.43.0

[PATCH v2 2/3] clocksource/timer-riscv: irq_pipeline: enable pipelined clock events

[Tobias Schaffner]

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 drivers/clocksource/timer-riscv.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/drivers/clocksource/timer-riscv.c b/drivers/clocksource/timer-riscv.c
index 48ce50c5f5e6..2f014b1c745f 100644
--- a/drivers/clocksource/timer-riscv.c
+++ b/drivers/clocksource/timer-riscv.c
@@ -70,7 +70,8 @@ static int riscv_clock_shutdown(struct clock_event_device *evt)
 static unsigned int riscv_clock_event_irq;
 static DEFINE_PER_CPU(struct clock_event_device, riscv_clock_event) = {
 	.name			= "riscv_timer_clockevent",
-	.features		= CLOCK_EVT_FEAT_ONESHOT,
+	.features		= CLOCK_EVT_FEAT_ONESHOT |
+				  CLOCK_EVT_FEAT_PIPELINE,
 	.rating			= 100,
 	.set_next_event		= riscv_clock_next_event,
 	.set_state_shutdown	= riscv_clock_shutdown,
@@ -143,7 +144,7 @@ static irqreturn_t riscv_timer_interrupt(int irq, void *dev_id)
 	struct clock_event_device *evdev = this_cpu_ptr(&riscv_clock_event);
 
 	riscv_clock_event_stop();
-	evdev->event_handler(evdev);
+	clockevents_handle_event(evdev);
 
 	return IRQ_HANDLED;
 }
@@ -175,9 +176,9 @@ static int __init riscv_timer_init_common(void)
 
 	sched_clock_register(riscv_sched_clock, 64, riscv_timebase);
 
-	error = request_percpu_irq(riscv_clock_event_irq,
-				    riscv_timer_interrupt,
-				    "riscv-timer", &riscv_clock_event);
+	error = __request_percpu_irq(riscv_clock_event_irq,
+				     riscv_timer_interrupt, IRQF_TIMER,
+				     "riscv-timer", &riscv_clock_event);
 	if (error) {
 		pr_err("registering percpu irq failed [%d]\n", error);
 		return error;
-- 
2.43.0

[PATCH v2 3/3] RISC-V: irq_pipeline: add IRQ pipelining core

[Tobias Schaffner]

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 arch/riscv/Kconfig                         |   1 +
 arch/riscv/include/asm/irq_pipeline.h      | 143 +++++++++++++++++++++
 arch/riscv/include/asm/irqflags.h          |  25 ++--
 arch/riscv/include/asm/thread_info.h       |   8 ++
 arch/riscv/kernel/Makefile                 |   1 +
 arch/riscv/kernel/irq_pipeline.c           |  26 ++++
 arch/riscv/kernel/smp.c                    |  93 +++++++++++++-
 arch/riscv/kernel/smpboot.c                |   2 +-
 arch/riscv/kernel/traps.c                  |  51 +++++++-
 arch/riscv/mm/fault.c                      |  83 ++++++++++--
 drivers/irqchip/irq-riscv-aplic-direct.c   |   3 +-
 drivers/irqchip/irq-riscv-aplic-msi.c      |   3 +-
 drivers/irqchip/irq-riscv-imsic-platform.c |   3 +-
 drivers/irqchip/irq-riscv-intc.c           |   2 +
 drivers/irqchip/irq-sifive-plic.c          |   3 +-
 15 files changed, 411 insertions(+), 36 deletions(-)
 create mode 100644 arch/riscv/include/asm/irq_pipeline.h
 create mode 100644 arch/riscv/kernel/irq_pipeline.c

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 0f3cd7c3a436..d9ae490e015a 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -131,6 +131,7 @@ config RISCV
 	select HAVE_ARCH_TRANSPARENT_HUGEPAGE if 64BIT && MMU
 	select HAVE_ARCH_USERFAULTFD_MINOR if 64BIT && USERFAULTFD
 	select HAVE_ARCH_VMAP_STACK if MMU && 64BIT
+	select HAVE_IRQ_PIPELINE
 	select HAVE_ASM_MODVERSIONS
 	select HAVE_CONTEXT_TRACKING_USER
 	select HAVE_DEBUG_KMEMLEAK
diff --git a/arch/riscv/include/asm/irq_pipeline.h b/arch/riscv/include/asm/irq_pipeline.h
new file mode 100644
index 000000000000..66f141ea23f8
--- /dev/null
+++ b/arch/riscv/include/asm/irq_pipeline.h
@@ -0,0 +1,143 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * IRQ Pipelining adapted from the ARM version.
+ *
+ * Copyright (C) 2024 Siemens AG
+ * Author:       Tobias Schaffner <tobias.schaffner@siemens.com>.
+ */
+
+#ifndef _ASM_RISCV_IRQ_PIPELINE_H
+#define _ASM_RISCV_IRQ_PIPELINE_H
+
+/*
+ * Multiplex In-band IPI messages over software interrupt 0
+ * as we do in the arm implementation.
+ */
+#define OOB_NR_IPI		3
+#define OOB_IPI_OFFSET		1 /* Software interrupt 1 */
+#define TIMER_OOB_IPI		(ipi_virq_base + OOB_IPI_OFFSET)
+#define RESCHEDULE_OOB_IPI	(TIMER_OOB_IPI + 1)
+#define CALL_FUNCTION_OOB_IPI	(RESCHEDULE_OOB_IPI + 1)
+
+extern int ipi_virq_base;
+
+#ifdef CONFIG_IRQ_PIPELINE
+
+#include <asm/ptrace.h>
+
+/* NOTE: Any bit should be fine as long as we don't hit SR_SIE or SR_MIE. */
+#define IRQMASK_i_POS   31
+
+static inline notrace
+unsigned long arch_irqs_virtual_to_native_flags(int stalled)
+{
+	if (!stalled)
+		return SR_IE;
+	return 0;
+}
+
+static inline notrace
+unsigned long arch_irqs_native_to_virtual_flags(unsigned long flags)
+{
+	return (!!native_irqs_disabled_flags(flags)) << IRQMASK_i_POS;
+}
+
+static inline int arch_irqs_disabled_flags(unsigned long flags)
+{
+	return native_irqs_disabled_flags(flags);
+}
+
+static inline notrace void arch_local_irq_enable(void)
+{
+	barrier();
+	inband_irq_enable();
+}
+
+static inline notrace void arch_local_irq_disable(void)
+{
+	inband_irq_disable();
+	barrier();
+}
+
+static inline notrace unsigned long arch_local_save_flags(void)
+{
+	int stalled = inband_irqs_disabled();
+
+	barrier();
+	return arch_irqs_virtual_to_native_flags(stalled);
+}
+
+/* test hardware interrupt enable bit */
+static inline int arch_irqs_disabled(void)
+{
+	return arch_irqs_disabled_flags(arch_local_save_flags());
+}
+
+static inline notrace unsigned long arch_local_irq_save(void)
+{
+	int stalled = inband_irq_save();
+
+	barrier();
+	return arch_irqs_virtual_to_native_flags(stalled);
+}
+
+/* set interrupt enabled status */
+static inline void arch_local_irq_restore(unsigned long flags)
+{
+	inband_irq_restore(arch_irqs_disabled_flags(flags));
+	barrier();
+}
+
+static inline
+void arch_save_timer_regs(struct pt_regs *dst, struct pt_regs *src)
+{
+	dst->status = src->status;
+}
+
+#else /* !CONFIG_IRQ_PIPELINE */
+
+static inline unsigned long arch_local_irq_save(void)
+{
+	return native_irq_save();
+}
+
+static inline void arch_local_irq_enable(void)
+{
+	native_irq_enable();
+}
+
+static inline void arch_local_irq_disable(void)
+{
+	native_irq_disable();
+}
+
+static inline unsigned long arch_local_save_flags(void)
+{
+	return native_save_flags();
+}
+
+static inline void arch_local_irq_restore(unsigned long flags)
+{
+	native_irq_restore(flags);
+}
+
+static inline int arch_irqs_disabled_flags(unsigned long flags)
+{
+	return native_irqs_disabled_flags(flags);
+}
+
+#endif /* !CONFIG_IRQ_PIPELINE */
+
+extern void (*handle_arch_irq)(struct pt_regs *);
+
+static inline void arch_handle_irq_pipelined(struct pt_regs *regs)
+{
+	handle_arch_irq(regs);
+}
+
+static inline int arch_enable_oob_stage(void)
+{
+	return 0;
+}
+
+#endif /* _ASM_RISCV_IRQ_PIPELINE_H */
diff --git a/arch/riscv/include/asm/irqflags.h b/arch/riscv/include/asm/irqflags.h
index 6fd8cbfcfcc7..839aaadfc036 100644
--- a/arch/riscv/include/asm/irqflags.h
+++ b/arch/riscv/include/asm/irqflags.h
@@ -10,45 +10,54 @@
 #include <asm/csr.h>
 
 /* read interrupt enabled status */
-static inline unsigned long arch_local_save_flags(void)
+static inline unsigned long native_save_flags(void)
 {
 	return csr_read(CSR_STATUS);
 }
 
 /* unconditionally enable interrupts */
-static inline void arch_local_irq_enable(void)
+static inline void native_irq_enable(void)
 {
 	csr_set(CSR_STATUS, SR_IE);
 }
 
 /* unconditionally disable interrupts */
-static inline void arch_local_irq_disable(void)
+static inline void native_irq_disable(void)
 {
 	csr_clear(CSR_STATUS, SR_IE);
 }
 
 /* get status and disable interrupts */
-static inline unsigned long arch_local_irq_save(void)
+static inline unsigned long native_irq_save(void)
 {
 	return csr_read_clear(CSR_STATUS, SR_IE);
 }
 
 /* test flags */
-static inline int arch_irqs_disabled_flags(unsigned long flags)
+static inline int native_irqs_disabled_flags(unsigned long flags)
 {
 	return !(flags & SR_IE);
 }
 
 /* test hardware interrupt enable bit */
-static inline int arch_irqs_disabled(void)
+static inline bool native_irqs_disabled(void)
 {
-	return arch_irqs_disabled_flags(arch_local_save_flags());
+	return native_irqs_disabled_flags(native_save_flags());
 }
 
 /* set interrupt enabled status */
-static inline void arch_local_irq_restore(unsigned long flags)
+static inline void native_irq_restore(unsigned long flags)
 {
 	csr_set(CSR_STATUS, flags & SR_IE);
 }
 
+#include <asm/irq_pipeline.h>
+
+static inline void native_irq_sync(void)
+{
+	native_irq_disable();
+	asm volatile("nop" : : : "memory");
+	native_irq_enable();
+}
+
 #endif /* _ASM_RISCV_IRQFLAGS_H */
diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h
index fca5c6be2b81..1ed22edef1fc 100644
--- a/arch/riscv/include/asm/thread_info.h
+++ b/arch/riscv/include/asm/thread_info.h
@@ -47,6 +47,7 @@
  */
 struct thread_info {
 	unsigned long		flags;		/* low level flags */
+	__u32			local_flags;	/* local (synchronous) flags */
 	int                     preempt_count;  /* 0=>preemptible, <0=>BUG */
 	/*
 	 * These stack pointers are overwritten on every system call or
@@ -83,6 +84,8 @@ struct thread_info {
 	INIT_SCS				\
 }
 
+#define ti_local_flags(__ti)    ((__ti)->local_flags)
+
 void arch_release_task_struct(struct task_struct *tsk);
 int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 
@@ -116,4 +119,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 	(_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \
 	 _TIF_NOTIFY_SIGNAL | _TIF_UPROBE)
 
+/*
+ * Local (synchronous) thread flags.
+ */
+#define _TLF_OOB		0x0001
+
 #endif /* _ASM_RISCV_THREAD_INFO_H */
diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile
index 06d407f1b30b..f6827b2432d1 100644
--- a/arch/riscv/kernel/Makefile
+++ b/arch/riscv/kernel/Makefile
@@ -81,6 +81,7 @@ obj-$(CONFIG_MODULES)		+= module.o
 obj-$(CONFIG_MODULE_SECTIONS)	+= module-sections.o
 
 obj-$(CONFIG_CPU_PM)		+= suspend_entry.o suspend.o
+obj-$(CONFIG_IRQ_PIPELINE)	+= irq_pipeline.o
 obj-$(CONFIG_HIBERNATION)	+= hibernate.o hibernate-asm.o
 
 obj-$(CONFIG_FUNCTION_TRACER)	+= mcount.o ftrace.o
diff --git a/arch/riscv/kernel/irq_pipeline.c b/arch/riscv/kernel/irq_pipeline.c
new file mode 100644
index 000000000000..4eeb4394434f
--- /dev/null
+++ b/arch/riscv/kernel/irq_pipeline.c
@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * IRQ Pipelining implementation adapted from the ARM version.
+ *
+ * Copyright (C) 2024 Siemens AG
+ * Author:       Tobias Schaffner <tobias.schaffner@siemens.com>.
+ */
+#include <linux/irq.h>
+#include <linux/irq_pipeline.h>
+
+void arch_do_IRQ_pipelined(struct irq_desc *desc)
+{
+	struct pt_regs *regs = raw_cpu_ptr(&irq_pipeline.tick_regs);
+	struct pt_regs *old_regs = set_irq_regs(regs);
+
+	irq_enter();
+	handle_irq_desc(desc);
+	irq_exit();
+
+	set_irq_regs(old_regs);
+}
+
+void __init arch_irq_pipeline_init(void)
+{
+	/* no per-arch init. */
+}
diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c
index 8e6eb64459af..61ce611aa4a2 100644
--- a/arch/riscv/kernel/smp.c
+++ b/arch/riscv/kernel/smp.c
@@ -20,12 +20,15 @@
 #include <linux/seq_file.h>
 #include <linux/delay.h>
 #include <linux/irq.h>
+#include <linux/irq_pipeline.h>
 #include <linux/irq_work.h>
 
 #include <asm/tlbflush.h>
 #include <asm/cacheflush.h>
 #include <asm/cpu_ops.h>
 
+#include <trace/events/ipi.h>
+
 enum ipi_message_type {
 	IPI_RESCHEDULE,
 	IPI_CALL_FUNC,
@@ -45,8 +48,8 @@ void __init smp_setup_processor_id(void)
 	cpuid_to_hartid_map(0) = boot_cpu_hartid;
 }
 
-static DEFINE_PER_CPU_READ_MOSTLY(int, ipi_dummy_dev);
-static int ipi_virq_base __ro_after_init;
+//static DEFINE_PER_CPU_READ_MOSTLY(int, ipi_dummy_dev);
+int ipi_virq_base __ro_after_init;
 static int nr_ipi __ro_after_init = IPI_MAX;
 static struct irq_desc *ipi_desc[IPI_MAX] __read_mostly;
 
@@ -77,7 +80,7 @@ static inline void ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs)
 
 	atomic_dec(&waiting_for_crash_ipi);
 
-	local_irq_disable();
+	hard_local_irq_disable();
 
 #ifdef CONFIG_HOTPLUG_CPU
 	if (cpu_has_hotplug(cpu))
@@ -94,6 +97,47 @@ static inline void ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs)
 }
 #endif
 
+#ifdef CONFIG_IRQ_PIPELINE
+
+static DEFINE_PER_CPU(unsigned long, ipi_messages);
+
+static DEFINE_PER_CPU(unsigned int [IPI_MAX], ipi_counts);
+
+void irq_send_oob_ipi(unsigned int irq,
+		const struct cpumask *cpumask)
+{
+	unsigned int sgi = irq - ipi_virq_base;
+
+	if (WARN_ON(irq_pipeline_debug() &&
+		    (sgi < OOB_IPI_OFFSET ||
+		     sgi >= OOB_IPI_OFFSET + OOB_NR_IPI)))
+		return;
+
+	__ipi_send_mask(ipi_desc[sgi], cpumask);
+}
+EXPORT_SYMBOL_GPL(irq_send_oob_ipi);
+
+static void send_ipi_mask(const struct cpumask *mask, enum ipi_message_type op)
+{
+	unsigned int cpu;
+
+	for_each_cpu(cpu, mask)
+		set_bit(op, &per_cpu(ipi_messages, cpu));
+
+	wmb();
+	__ipi_send_mask(ipi_desc[0], mask);
+}
+
+static void send_ipi_single(int cpu, enum ipi_message_type op)
+{
+	set_bit(op, &per_cpu(ipi_messages, cpu));
+
+	wmb();
+	__ipi_send_mask(ipi_desc[0], cpumask_of(cpu));
+}
+
+#else /* !CONFIG_IRQ_PIPELINE */
+
 static void send_ipi_mask(const struct cpumask *mask, enum ipi_message_type op)
 {
 	__ipi_send_mask(ipi_desc[op], mask);
@@ -104,6 +148,8 @@ static void send_ipi_single(int cpu, enum ipi_message_type op)
 	__ipi_send_mask(ipi_desc[op], cpumask_of(cpu));
 }
 
+#endif /* !CONFIG_IRQ_PIPELINE */
+
 #ifdef CONFIG_IRQ_WORK
 void arch_irq_work_raise(void)
 {
@@ -144,6 +190,26 @@ static irqreturn_t handle_IPI(int irq, void *data)
 	return IRQ_HANDLED;
 }
 
+#ifdef CONFIG_IRQ_PIPELINE
+
+static irqreturn_t ipi_handler(int irq, void *data)
+{
+	unsigned long *pmsg;
+	unsigned int ipinr;
+
+	pmsg = raw_cpu_ptr(&ipi_messages);
+	while (*pmsg) {
+		ipinr = ffs(*pmsg) - 1;
+		clear_bit(ipinr, pmsg);
+		__this_cpu_inc(ipi_counts[ipinr]);
+		handle_IPI(ipinr + ipi_virq_base, data);
+	}
+
+	return IRQ_HANDLED;
+}
+
+#endif /* !CONFIG_IRQ_PIPELINE */
+
 void riscv_ipi_enable(void)
 {
 	int i;
@@ -173,20 +239,33 @@ bool riscv_ipi_have_virq_range(void)
 
 void riscv_ipi_set_virq_range(int virq, int nr)
 {
-	int i, err;
+	int i, inband_nr_ipi;
 
 	if (WARN_ON(ipi_virq_base))
 		return;
 
 	WARN_ON(nr < IPI_MAX);
+
 	nr_ipi = min(nr, IPI_MAX);
+
+	/*
+	 * irq_pipeline: the in-band stage traps SGI0 only,
+	 * over which IPI messages are mutiplexed. Other SGIs
+	 * are available for exchanging out-of-band IPIs.
+	 */
+	inband_nr_ipi = irqs_pipelined() ? 1 : nr_ipi;
+
 	ipi_virq_base = virq;
 
 	/* Request IPIs */
 	for (i = 0; i < nr_ipi; i++) {
-		err = request_percpu_irq(ipi_virq_base + i, handle_IPI,
-					 "IPI", &ipi_dummy_dev);
-		WARN_ON(err);
+		if (i < inband_nr_ipi) {
+			int err;
+
+			err = request_percpu_irq(ipi_virq_base + i, ipi_handler,
+						 "IPI", &irq_stat);
+			WARN_ON(err);
+		}
 
 		ipi_desc[i] = irq_to_desc(ipi_virq_base + i);
 		irq_set_status_flags(ipi_virq_base + i, IRQ_HIDDEN);
diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c
index 0f8f1c95ac38..c80673c6e8cf 100644
--- a/arch/riscv/kernel/smpboot.c
+++ b/arch/riscv/kernel/smpboot.c
@@ -246,6 +246,6 @@ asmlinkage __visible void smp_callin(void)
 	 * Disable preemption before enabling interrupts, so we don't try to
 	 * schedule a CPU that hasn't actually started yet.
 	 */
-	local_irq_enable();
+	local_irq_enable_full();
 	cpu_startup_entry(CPUHP_AP_ONLINE_IDLE);
 }
diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c
index 05a16b1f0aee..75bd020d231e 100644
--- a/arch/riscv/kernel/traps.c
+++ b/arch/riscv/kernel/traps.c
@@ -111,10 +111,30 @@ void die(struct pt_regs *regs, const char *str)
 		make_task_dead(SIGSEGV);
 }
 
+static __always_inline
+bool mark_trap_entry(int signo, struct pt_regs *regs)
+{
+	if (likely(running_inband())) {
+		hard_cond_local_irq_enable();
+		return true;
+	}
+
+	return false;
+}
+
+static __always_inline
+void mark_trap_exit(int signo, struct pt_regs *regs)
+{
+	hard_cond_local_irq_disable();
+}
+
 void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr)
 {
 	struct task_struct *tsk = current;
 
+	if (!mark_trap_entry(signo, regs))
+		return;
+
 	if (show_unhandled_signals && unhandled_signal(tsk, signo)
 	    && printk_ratelimit()) {
 		pr_info("%s[%d]: unhandled signal %d code 0x%x at 0x" REG_FMT,
@@ -126,6 +146,8 @@ void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr)
 	}
 
 	force_sig_fault(signo, code, (void __user *)addr);
+
+	mark_trap_exit(signo, regs);
 }
 
 static void do_trap_error(struct pt_regs *regs, int signo, int code,
@@ -174,11 +196,17 @@ asmlinkage __visible __trap_section void do_trap_insn_illegal(struct pt_regs *re
 	if (user_mode(regs)) {
 		irqentry_enter_from_user_mode(regs);
 
-		local_irq_enable();
+		if (running_inband())
+			local_irq_enable_full();
+		else
+			hard_local_irq_enable();
 
 		handled = riscv_v_first_use_handler(regs);
 
-		local_irq_disable();
+		if (running_inband())
+			local_irq_disable_full();
+		else
+			hard_local_irq_disable();
 
 		if (!handled)
 			do_trap_error(regs, SIGILL, ILL_ILLOPC, regs->epc,
@@ -341,7 +369,7 @@ void do_trap_ecall_u(struct pt_regs *regs)
 		 * The resulting 6 bits of entropy is seen in SP[9:4].
 		 */
 		choose_random_kstack_offset(get_random_u16());
-
+done:
 		syscall_exit_to_user_mode(regs);
 	} else {
 		irqentry_state_t state = irqentry_nmi_enter(regs);
@@ -361,12 +389,24 @@ asmlinkage __visible noinstr void do_page_fault(struct pt_regs *regs)
 
 	handle_page_fault(regs);
 
-	local_irq_disable();
+	if (running_inband())
+		local_irq_disable_full();
+	else
+		hard_local_irq_disable();
 
 	irqentry_exit(regs, state);
 }
 #endif
 
+#ifdef CONFIG_IRQ_PIPELINE
+static void noinstr handle_riscv_irq(struct pt_regs *regs)
+{
+	struct pt_regs *old_regs = set_irq_regs(regs);
+
+	handle_irq_pipelined(regs);
+	set_irq_regs(old_regs);
+}
+#else
 static void noinstr handle_riscv_irq(struct pt_regs *regs)
 {
 	struct pt_regs *old_regs;
@@ -377,6 +417,9 @@ static void noinstr handle_riscv_irq(struct pt_regs *regs)
 	set_irq_regs(old_regs);
 	irq_exit_rcu();
 }
+#endif
+
+extern void (*handle_arch_irq)(struct pt_regs *);
 
 asmlinkage void noinstr do_irq(struct pt_regs *regs)
 {
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index a9f2b4af8f3f..7a09fcfb534c 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -22,9 +22,60 @@
 
 #include "../kernel/head.h"
 
+#ifdef CONFIG_IRQ_PIPELINE
+/*
+ * We need to synchronize the virtual interrupt state with the hard
+ * interrupt state we received on entry, then turn hardirqs back on to
+ * allow code which does not require strict serialization to be
+ * preempted by an out-of-band activity.
+ */
+static inline
+unsigned long fault_entry(struct pt_regs *regs)
+{
+	unsigned long flags;
+
+	flags = hard_local_save_flags();
+
+	if (raw_irqs_disabled_flags(flags)) {
+		stall_inband();
+		trace_hardirqs_off();
+	}
+
+	hard_local_irq_enable();
+
+	if (running_inband())
+		local_irq_enable();
+
+	return flags;
+}
+
+static inline void fault_exit(unsigned long flags)
+{
+	WARN_ON_ONCE(irq_pipeline_debug() && hard_irqs_disabled());
+
+	/*
+	 * We expect kentry_exit_pipelined() to clear the stall bit if
+	 * kentry_enter_pipelined() observed it that way.
+	 */
+	hard_local_irq_restore(flags);
+}
+
+#else	/* !CONFIG_IRQ_PIPELINE */
+
+static inline
+unsigned long fault_entry(struct pt_regs *regs)
+{
+	return 0;
+}
+
+static inline void fault_exit(unsigned long x) { }
+
+#endif	/* !CONFIG_IRQ_PIPELINE */
+
 static void die_kernel_fault(const char *msg, unsigned long addr,
 		struct pt_regs *regs)
 {
+	irq_pipeline_oops();
 	bust_spinlocks(1);
 
 	pr_alert("Unable to handle kernel %s at virtual address " REG_FMT "\n", msg,
@@ -61,6 +112,8 @@ static inline void no_context(struct pt_regs *regs, unsigned long addr)
 
 static inline void mm_fault_error(struct pt_regs *regs, unsigned long addr, vm_fault_t fault)
 {
+	unsigned long irqflags;
+
 	if (!user_mode(regs)) {
 		no_context(regs, addr);
 		return;
@@ -71,7 +124,9 @@ static inline void mm_fault_error(struct pt_regs *regs, unsigned long addr, vm_f
 		 * We ran out of memory, call the OOM killer, and return the userspace
 		 * (which will retry the fault, or kill us if we got oom-killed).
 		 */
+		irqflags = fault_entry(regs);
 		pagefault_out_of_memory();
+		fault_exit(irqflags);
 		return;
 	} else if (fault & (VM_FAULT_SIGBUS | VM_FAULT_HWPOISON | VM_FAULT_HWPOISON_LARGE)) {
 		/* Kernel mode? Handle exceptions or die */
@@ -229,6 +284,9 @@ void handle_page_fault(struct pt_regs *regs)
 	unsigned int flags = FAULT_FLAG_DEFAULT;
 	int code = SEGV_MAPERR;
 	vm_fault_t fault;
+	unsigned long irqflags;
+
+	irqflags = fault_entry(regs);
 
 	cause = regs->cause;
 	addr = regs->badaddr;
@@ -237,7 +295,7 @@ void handle_page_fault(struct pt_regs *regs)
 	mm = tsk->mm;
 
 	if (kprobe_page_fault(regs, cause))
-		return;
+		goto out;
 
 	/*
 	 * Fault-in kernel-space virtual memory on-demand.
@@ -251,12 +309,12 @@ void handle_page_fault(struct pt_regs *regs)
 	if ((!IS_ENABLED(CONFIG_MMU) || !IS_ENABLED(CONFIG_64BIT)) &&
 	    unlikely(addr >= VMALLOC_START && addr < VMALLOC_END)) {
 		vmalloc_fault(regs, code, addr);
-		return;
+		goto out;
 	}
 
 	/* Enable interrupts if they were enabled in the parent context. */
 	if (!regs_irqs_disabled(regs))
-		local_irq_enable();
+		hard_local_irq_enable();
 
 	/*
 	 * If we're in an interrupt, have no user context, or are running
@@ -265,7 +323,7 @@ void handle_page_fault(struct pt_regs *regs)
 	if (unlikely(faulthandler_disabled() || !mm)) {
 		tsk->thread.bad_cause = cause;
 		no_context(regs, addr);
-		return;
+		goto out;
 	}
 
 	if (user_mode(regs))
@@ -273,7 +331,7 @@ void handle_page_fault(struct pt_regs *regs)
 
 	if (!user_mode(regs) && addr < TASK_SIZE && unlikely(!(regs->status & SR_SUM))) {
 		if (fixup_exception(regs))
-			return;
+			goto out;
 
 		die_kernel_fault("access to user memory without uaccess routines", addr, regs);
 	}
@@ -296,7 +354,7 @@ void handle_page_fault(struct pt_regs *regs)
 		count_vm_vma_lock_event(VMA_LOCK_SUCCESS);
 		tsk->thread.bad_cause = cause;
 		bad_area_nosemaphore(regs, SEGV_ACCERR, addr);
-		return;
+		goto out;
 	}
 
 	fault = handle_mm_fault(vma, addr, flags | FAULT_FLAG_VMA_LOCK, regs);
@@ -314,7 +372,7 @@ void handle_page_fault(struct pt_regs *regs)
 	if (fault_signal_pending(fault, regs)) {
 		if (!user_mode(regs))
 			no_context(regs, addr);
-		return;
+		goto out;
 	}
 lock_mmap:
 
@@ -323,7 +381,7 @@ void handle_page_fault(struct pt_regs *regs)
 	if (unlikely(!vma)) {
 		tsk->thread.bad_cause = cause;
 		bad_area_nosemaphore(regs, code, addr);
-		return;
+		goto out;
 	}
 
 	/*
@@ -335,7 +393,7 @@ void handle_page_fault(struct pt_regs *regs)
 	if (unlikely(access_error(cause, vma))) {
 		tsk->thread.bad_cause = cause;
 		bad_area(regs, mm, code, addr);
-		return;
+		goto out;
 	}
 
 	/*
@@ -353,12 +411,12 @@ void handle_page_fault(struct pt_regs *regs)
 	if (fault_signal_pending(fault, regs)) {
 		if (!user_mode(regs))
 			no_context(regs, addr);
-		return;
+		goto out;
 	}
 
 	/* The fault is fully completed (including releasing mmap lock) */
 	if (fault & VM_FAULT_COMPLETED)
-		return;
+		goto out;
 
 	if (unlikely(fault & VM_FAULT_RETRY)) {
 		flags |= FAULT_FLAG_TRIED;
@@ -377,7 +435,8 @@ void handle_page_fault(struct pt_regs *regs)
 	if (unlikely(fault & VM_FAULT_ERROR)) {
 		tsk->thread.bad_cause = cause;
 		mm_fault_error(regs, addr, fault);
-		return;
 	}
+out:
+	fault_exit(irqflags);
 	return;
 }
diff --git a/drivers/irqchip/irq-riscv-aplic-direct.c b/drivers/irqchip/irq-riscv-aplic-direct.c
index 4a3ffe856d6c..ae0d31f2d32c 100644
--- a/drivers/irqchip/irq-riscv-aplic-direct.c
+++ b/drivers/irqchip/irq-riscv-aplic-direct.c
@@ -87,7 +87,8 @@ static struct irq_chip aplic_direct_chip = {
 #endif
 	.flags		= IRQCHIP_SET_TYPE_MASKED |
 			  IRQCHIP_SKIP_SET_WAKE |
-			  IRQCHIP_MASK_ON_SUSPEND,
+			  IRQCHIP_MASK_ON_SUSPEND |
+			  IRQCHIP_PIPELINE_SAFE,
 };
 
 static int aplic_direct_irqdomain_translate(struct irq_domain *d, struct irq_fwspec *fwspec,
diff --git a/drivers/irqchip/irq-riscv-aplic-msi.c b/drivers/irqchip/irq-riscv-aplic-msi.c
index 028444af48bd..213679771828 100644
--- a/drivers/irqchip/irq-riscv-aplic-msi.c
+++ b/drivers/irqchip/irq-riscv-aplic-msi.c
@@ -138,7 +138,8 @@ static const struct msi_domain_template aplic_msi_template = {
 		.irq_write_msi_msg	= aplic_msi_write_msg,
 		.flags			= IRQCHIP_SET_TYPE_MASKED |
 					  IRQCHIP_SKIP_SET_WAKE |
-					  IRQCHIP_MASK_ON_SUSPEND,
+					  IRQCHIP_MASK_ON_SUSPEND |
+					  IRQCHIP_PIPELINE_SAFE,
 	},
 
 	.ops = {
diff --git a/drivers/irqchip/irq-riscv-imsic-platform.c b/drivers/irqchip/irq-riscv-imsic-platform.c
index 11723a763c10..1bfb7cc94f82 100644
--- a/drivers/irqchip/irq-riscv-imsic-platform.c
+++ b/drivers/irqchip/irq-riscv-imsic-platform.c
@@ -137,7 +137,8 @@ static struct irq_chip imsic_irq_base_chip = {
 	.irq_retrigger		= imsic_irq_retrigger,
 	.irq_compose_msi_msg	= imsic_irq_compose_msg,
 	.flags			= IRQCHIP_SKIP_SET_WAKE |
-				  IRQCHIP_MASK_ON_SUSPEND,
+				  IRQCHIP_MASK_ON_SUSPEND |
+				  IRQCHIP_PIPELINE_SAFE,
 };
 
 static int imsic_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
diff --git a/drivers/irqchip/irq-riscv-intc.c b/drivers/irqchip/irq-riscv-intc.c
index 47f3200476da..2df18b7d36a1 100644
--- a/drivers/irqchip/irq-riscv-intc.c
+++ b/drivers/irqchip/irq-riscv-intc.c
@@ -111,6 +111,7 @@ static struct irq_chip riscv_intc_chip = {
 	.irq_mask = riscv_intc_irq_mask,
 	.irq_unmask = riscv_intc_irq_unmask,
 	.irq_eoi = riscv_intc_irq_eoi,
+	.flags = IRQCHIP_PIPELINE_SAFE,
 };
 
 static struct irq_chip andes_intc_chip = {
@@ -118,6 +119,7 @@ static struct irq_chip andes_intc_chip = {
 	.irq_mask	= andes_intc_irq_mask,
 	.irq_unmask	= andes_intc_irq_unmask,
 	.irq_eoi	= riscv_intc_irq_eoi,
+	.flags		= IRQCHIP_PIPELINE_SAFE,
 };
 
 static int riscv_intc_domain_map(struct irq_domain *d, unsigned int irq,
diff --git a/drivers/irqchip/irq-sifive-plic.c b/drivers/irqchip/irq-sifive-plic.c
index 9e22f7e378f5..f61db2ae0ca4 100644
--- a/drivers/irqchip/irq-sifive-plic.c
+++ b/drivers/irqchip/irq-sifive-plic.c
@@ -197,7 +197,8 @@ static struct irq_chip plic_edge_chip = {
 #endif
 	.irq_set_type	= plic_irq_set_type,
 	.flags		= IRQCHIP_SKIP_SET_WAKE |
-			  IRQCHIP_AFFINITY_PRE_STARTUP,
+			  IRQCHIP_AFFINITY_PRE_STARTUP |
+			  IRQCHIP_PIPELINE_SAFE,
 };
 
 static struct irq_chip plic_chip = {
-- 
2.43.0

Re: [PATCH v2 1/3] dovetail: Guard _TIF_RETUSER with CONFIG_DOVETAIL

[Pierre FICHEUX]

Hi

Thanks a lot for the answer. I've just booked oct 16 on my agenda.

 I think I'm already on the new list.

You were unable to be subscribed to the list because you are already
subscribed.

regards

Le ven. 4 oct. 2024 à 22:02, Tobias Schaffner
<tobias.schaffner@siemens.com> a écrit :
>
> _TIF_RETUSER has to be guarded by CONFIG_DOVETAIL to allow building
> CONFIG_IRQ_PIPELINE without a dovetail implementation.
>
> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> ---
>  include/linux/entry-common.h | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h
> index 900c6e303ab3..8a84f57bfde1 100644
> --- a/include/linux/entry-common.h
> +++ b/include/linux/entry-common.h
> @@ -350,13 +350,15 @@ unsigned long exit_to_user_mode_loop(struct pt_regs *regs,
>
>  static inline bool do_retuser(unsigned long ti_work)
>  {
> -       if (dovetailing() && (ti_work & _TIF_RETUSER)) {
> +#ifdef CONFIG_DOVETAIL
> +       if (ti_work & _TIF_RETUSER) {
>                 hard_local_irq_enable();
>                 inband_retuser_notify();
>                 hard_local_irq_disable();
>                 /* RETUSER might have switched oob */
>                 return running_inband();
>         }
> +#endif
>
>         return false;
>  }
> --
> 2.43.0
>
>


-- 

Pierre FICHEUX -/- CTO Smile ECS, France -\- pierre.ficheux@smile.fr
                             http://www.smile.fr
                             https://smile.eu/fr/offres/embarque-iot
I would love to change the world, but they won't give me the source code

Re: [PATCH v2 1/3] dovetail: Guard _TIF_RETUSER with CONFIG_DOVETAIL

[Schaffner, Tobias]

On Sat, 2024-10-05 at 12:16 +0200, Pierre FICHEUX wrote:
> Hi
> 
> Thanks a lot for the answer. I've just booked oct 16 on my agenda.
> 

Wrong thread. ;)

>  I think I'm already on the new list.
> 
> You were unable to be subscribed to the list because you are already
> subscribed.
> 
> regards
> 
> Le ven. 4 oct. 2024 à 22:02, Tobias Schaffner
> <tobias.schaffner@siemens.com> a écrit :
> > 
> > _TIF_RETUSER has to be guarded by CONFIG_DOVETAIL to allow building
> > CONFIG_IRQ_PIPELINE without a dovetail implementation.
> > 
> > Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> > ---
> >  include/linux/entry-common.h | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> > 
> > diff --git a/include/linux/entry-common.h b/include/linux/entry-
> > common.h
> > index 900c6e303ab3..8a84f57bfde1 100644
> > --- a/include/linux/entry-common.h
> > +++ b/include/linux/entry-common.h
> > @@ -350,13 +350,15 @@ unsigned long exit_to_user_mode_loop(struct
> > pt_regs *regs,
> > 
> >  static inline bool do_retuser(unsigned long ti_work)
> >  {
> > -       if (dovetailing() && (ti_work & _TIF_RETUSER)) {
> > +#ifdef CONFIG_DOVETAIL
> > +       if (ti_work & _TIF_RETUSER) {
> >                 hard_local_irq_enable();
> >                 inband_retuser_notify();
> >                 hard_local_irq_disable();
> >                 /* RETUSER might have switched oob */
> >                 return running_inband();
> >         }
> > +#endif
> > 
> >         return false;
> >  }
> > --
> > 2.43.0
> > 
> > 
> 
> 

Best,
Tobias

Re: [PATCH dovetail 6.11 v2 0/3] RISC-V: IRQ pipelining core

[Philippe Gerum]

Hi,

I started a Dovetail branch [1] for the RISC-V port, forward porting
your series (since it's SMP-enabled already) to 6.12-rc6. The code was
split in few more patches, including a couple of fixes we need as
well. The latter may be folded into the core pipeline support later
on. From that point, we could then pick the upper layer of the Dovetail
code already ported by Shanmu in order to enable EVL on top.

I recommend enabling the following switches when working on this port,
so that we can detect any issue with the virtual interrupt state:

CONFIG_PROVE_LOCKING
CONFIG_PREEMPT_RT
CONFIG_DEBUG_PREEMPT
CONFIG_RISCV_ISA_V_PREEMPTIVE

Here we go, so far:

root@homelab-qemu-riscv64:~# uname -a
Linux homelab-qemu-riscv64 6.12.0-rc6-00145-gb4167b69ed0e #20 SMP PREEMPT_RT IRQ_PIPELINE Mon Nov  4 10:26:04 CET 2024 riscv64 GNU/Linux

PS: I switched from RISC-V: to riscv: in the shortlogs to follow the
(most) common scheme used so far, which is likely to reuse the name of
the architecture folder.

[1] https://source.denx.de/Xenomai/linux-dovetail/-/tree/wip/dovetail-riscv?ref_type=heads

Tobias Schaffner <tobias.schaffner@siemens.com> writes:

> Hi all,
>
> This introduces IRQ pipelining for RISC-V into Dovetail. The patches
> are on top of v6.11-dovetail-rebase. Feel free to ask for other targets
> if needed.
>
> I thought about splitting the patch up into smaller parts, but decided
> to stick to the structure of the analogous patches of the other
> architectures. Maybe this will make the rebasing process easier.
>
> Testing was done in QEMU with IRQ pipeline torture tests, both with and
> without SMP enabled.
>
> Changes since v1:
> * Removed some dovetail specific changes. These will be readded in a
>   later dovetail patch series. Thanks for the kind review @rpm!
> * Added a patch that guards _TIF_RETUSER with an CONFIG_DOVETAIL ifdef
>   to allow building with IRQ pipelining but without dovetail being
>   implemented.
> * Fix unneeded reordering in riscv irqflags.h to make the diff nicer.
>
> Tobias Schaffner (3):
>   dovetail: Guard _TIF_RETUSER with CONFIG_DOVETAIL
>   clocksource/timer-riscv: irq_pipeline: enable pipelined clock events
>   RISC-V: irq_pipeline: add IRQ pipelining core
>
>  arch/riscv/Kconfig                         |   1 +
>  arch/riscv/include/asm/irq_pipeline.h      | 143 +++++++++++++++++++++
>  arch/riscv/include/asm/irqflags.h          |  25 ++--
>  arch/riscv/include/asm/thread_info.h       |   8 ++
>  arch/riscv/kernel/Makefile                 |   1 +
>  arch/riscv/kernel/irq_pipeline.c           |  26 ++++
>  arch/riscv/kernel/smp.c                    |  93 +++++++++++++-
>  arch/riscv/kernel/smpboot.c                |   2 +-
>  arch/riscv/kernel/traps.c                  |  51 +++++++-
>  arch/riscv/mm/fault.c                      |  83 ++++++++++--
>  drivers/clocksource/timer-riscv.c          |  11 +-
>  drivers/irqchip/irq-riscv-aplic-direct.c   |   3 +-
>  drivers/irqchip/irq-riscv-aplic-msi.c      |   3 +-
>  drivers/irqchip/irq-riscv-imsic-platform.c |   3 +-
>  drivers/irqchip/irq-riscv-intc.c           |   2 +
>  drivers/irqchip/irq-sifive-plic.c          |   3 +-
>  include/linux/entry-common.h               |   4 +-
>  17 files changed, 420 insertions(+), 42 deletions(-)
>  create mode 100644 arch/riscv/include/asm/irq_pipeline.h
>  create mode 100644 arch/riscv/kernel/irq_pipeline.c

-- 
Philippe.