[PATCH dovetail v4 00/12] riscv: Add dovetail support

[PATCH dovetail v4 00/12] riscv: Add dovetail support

[Tobias Schaffner]

Hi all,

this series introduces initial support for Dovetail on RISC-V.

Feedback, suggestions, and especially reviews are very welcome.

Changes since v1:
 * Disable independent irq/softirq stack usage when pipelining
 * Drop KVM changes as long as they can not be properly tested

Changes since v2:
 * Rework trap handling

Changes since v3:
 * Included the IRQ pipelining patches to have all riscv specific
   patches included
 * Rebased to dovetail/v6.18.y-dovetail-rebase
 * Merged most of the fixes
 * Fixed IRQ Stack handling and enabled IRQ_STACKS again

Best,
Tobias

Tobias Schaffner (11):
  riscv: irq_pipeline: add IRQ pipelining core
  riscv: irq_pipeline: fix irq stack handling
  riscv: irq_pipeline: synchronize IRQs on exit to user mode
  irqchip/riscv-aplic-direct: enable pipelined interrupt control
  irqchip/irq-riscv-aplic-msi: enable pipelined interrupt control
  irqchip/irq-riscv-imsic-platform: enable pipelined interrupt control
  irqchip/irq-riscv-intc: enable pipelined interrupt control
  irqchip/irq-sifive-plic: enable pipelined interrupt control
  clocksource/timer-riscv: irq_pipeline: enable pipelined clock events
  riscv: add initial dovetail co-kernel skeleton
  riscv: add out-of-band aware trap handling

shannmu (1):
  riscv: add dovetail-aware memory management

 arch/riscv/Kconfig                         |   3 +
 arch/riscv/include/asm/dovetail.h          |  23 +++
 arch/riscv/include/asm/entry-common.h      |  28 ++++
 arch/riscv/include/asm/irq_pipeline.h      | 148 +++++++++++++++++
 arch/riscv/include/asm/irqflags.h          |  32 ++--
 arch/riscv/include/asm/mmu_context.h       |   2 +
 arch/riscv/include/asm/syscall.h           |   6 +
 arch/riscv/include/asm/thread_info.h       |  17 ++
 arch/riscv/kernel/Makefile                 |   1 +
 arch/riscv/kernel/irq_pipeline.c           |  26 +++
 arch/riscv/kernel/kernel_mode_vector.c     |   2 +-
 arch/riscv/kernel/sbi-ipi.c                |  10 +-
 arch/riscv/kernel/smp.c                    |  97 ++++++++---
 arch/riscv/kernel/smpboot.c                |   2 +-
 arch/riscv/kernel/traps.c                  | 180 ++++++++++++++++++---
 arch/riscv/mm/cacheflush.c                 |   5 +-
 arch/riscv/mm/context.c                    |  18 ++-
 arch/riscv/mm/fault.c                      |  75 +++++++--
 arch/riscv/mm/tlbflush.c                   |   5 +-
 drivers/clocksource/timer-clint.c          |   8 +
 drivers/clocksource/timer-riscv.c          |  11 +-
 drivers/irqchip/irq-riscv-aplic-direct.c   |   3 +-
 drivers/irqchip/irq-riscv-aplic-msi.c      |   3 +-
 drivers/irqchip/irq-riscv-imsic-platform.c |   4 +-
 drivers/irqchip/irq-riscv-intc.c           |   2 +
 drivers/irqchip/irq-sifive-plic.c          |   3 +-
 26 files changed, 632 insertions(+), 82 deletions(-)
 create mode 100644 arch/riscv/include/asm/dovetail.h
 create mode 100644 arch/riscv/include/asm/irq_pipeline.h
 create mode 100644 arch/riscv/kernel/irq_pipeline.c

-- 
2.43.0

[PATCH dovetail v4 01/12] riscv: irq_pipeline: add IRQ pipelining core

[Tobias Schaffner]

This patchset integrates IRQ pipelining into the RISC-V architecture,
bringing it in line with the Dovetail/IRQ pipeline model used on other
architectures. It adds the core pipelining infrastructure, updates the
interrupt and trap entry/exit paths, and adapts low-level primitives to
cleanly separate in-band and out-of-band interrupt handling.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
Co-authored-by: shannmu <shanmu1901@gmail.com>
Co-authored-by: Philippe Gerum <rpm@xenomai.org>
---
 arch/riscv/Kconfig                     |   1 +
 arch/riscv/include/asm/irq_pipeline.h  | 148 +++++++++++++++++++++++++
 arch/riscv/include/asm/irqflags.h      |  32 ++++--
 arch/riscv/include/asm/thread_info.h   |   9 ++
 arch/riscv/kernel/Makefile             |   1 +
 arch/riscv/kernel/irq_pipeline.c       |  26 +++++
 arch/riscv/kernel/kernel_mode_vector.c |   2 +-
 arch/riscv/kernel/sbi-ipi.c            |  10 +-
 arch/riscv/kernel/smp.c                |  97 ++++++++++++----
 arch/riscv/kernel/smpboot.c            |   2 +-
 arch/riscv/kernel/traps.c              |  67 +++++++++--
 arch/riscv/mm/fault.c                  |  51 ++++++++-
 drivers/clocksource/timer-clint.c      |   8 ++
 13 files changed, 408 insertions(+), 46 deletions(-)
 create mode 100644 arch/riscv/include/asm/irq_pipeline.h
 create mode 100644 arch/riscv/kernel/irq_pipeline.c

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index fadec20b87a8..e93a0b6f9f2b 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -148,6 +148,7 @@ config RISCV
 	select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if 64BIT && MMU
 	select HAVE_ARCH_USERFAULTFD_MINOR if 64BIT && USERFAULTFD
 	select HAVE_ARCH_VMAP_STACK if MMU && 64BIT
+	select HAVE_IRQ_PIPELINE
 	select HAVE_ASM_MODVERSIONS
 	select HAVE_CONTEXT_TRACKING_USER
 	select HAVE_DEBUG_KMEMLEAK
diff --git a/arch/riscv/include/asm/irq_pipeline.h b/arch/riscv/include/asm/irq_pipeline.h
new file mode 100644
index 000000000000..c479ec44b3a5
--- /dev/null
+++ b/arch/riscv/include/asm/irq_pipeline.h
@@ -0,0 +1,148 @@
+/* SPDX-License-Identifier: GPL-2.0
+ *
+ * IRQ Pipelining adapted from the ARM version.
+ *
+ * Copyright (C) 2024 Siemens AG
+ * Author:       Tobias Schaffner <tobias.schaffner@siemens.com>.
+ */
+
+#ifndef _ASM_RISCV_IRQ_PIPELINE_H
+#define _ASM_RISCV_IRQ_PIPELINE_H
+
+#define OOB_NR_IPI		3
+
+
+extern int ipi_virq_base;
+extern int NR_CALL_FUNCTION_OOB_IPI;
+extern int NR_RESCHEDULE_OOB_IPI;
+extern int NR_TIMER_OOB_IPI;
+extern int ipi_max;
+
+#define CALL_FUNCTION_OOB_IPI (ipi_virq_base + NR_CALL_FUNCTION_OOB_IPI)
+#define RESCHEDULE_OOB_IPI (ipi_virq_base + NR_RESCHEDULE_OOB_IPI)
+#define TIMER_OOB_IPI (ipi_virq_base + NR_TIMER_OOB_IPI)
+#define PIPELINED_IPI_MAX ipi_max
+
+#ifdef CONFIG_IRQ_PIPELINE
+
+#include <asm/ptrace.h>
+
+/* NOTE: Any bit should be fine as long as we don't hit SR_SIE or SR_MIE. */
+#define IRQMASK_i_POS   31
+
+static inline notrace
+unsigned long arch_irqs_virtual_to_native_flags(int stalled)
+{
+	if (!stalled)
+		return SR_IE;
+	return 0;
+}
+
+static inline notrace
+unsigned long arch_irqs_native_to_virtual_flags(unsigned long flags)
+{
+	return (!!native_irqs_disabled_flags(flags)) << IRQMASK_i_POS;
+}
+
+static inline int arch_irqs_disabled_flags(unsigned long flags)
+{
+	return native_irqs_disabled_flags(flags);
+}
+
+static inline notrace void arch_local_irq_enable(void)
+{
+	barrier();
+	inband_irq_enable();
+}
+
+static inline notrace void arch_local_irq_disable(void)
+{
+	inband_irq_disable();
+	barrier();
+}
+
+static inline notrace unsigned long arch_local_save_flags(void)
+{
+	int stalled = inband_irqs_disabled();
+
+	barrier();
+	return arch_irqs_virtual_to_native_flags(stalled);
+}
+
+static inline notrace unsigned long arch_local_irq_save(void)
+{
+	int stalled = inband_irq_save();
+
+	barrier();
+	return arch_irqs_virtual_to_native_flags(stalled);
+}
+
+/* set interrupt enabled status */
+static inline void arch_local_irq_restore(unsigned long flags)
+{
+	inband_irq_restore(arch_irqs_disabled_flags(flags));
+	barrier();
+}
+
+static inline
+void arch_save_timer_regs(struct pt_regs *dst, struct pt_regs *src)
+{
+	dst->status = src->status;
+	dst->epc = src->epc;
+}
+
+#else /* !CONFIG_IRQ_PIPELINE */
+
+static inline unsigned long arch_local_irq_save(void)
+{
+	return native_irq_save();
+}
+
+static inline void arch_local_irq_enable(void)
+{
+	native_irq_enable();
+}
+
+static inline void arch_local_irq_disable(void)
+{
+	native_irq_disable();
+}
+
+static inline unsigned long arch_local_save_flags(void)
+{
+	return native_save_flags();
+}
+
+static inline void arch_local_irq_restore(unsigned long flags)
+{
+	native_irq_restore(flags);
+}
+
+static inline int arch_irqs_disabled_flags(unsigned long flags)
+{
+	return native_irqs_disabled_flags(flags);
+}
+
+#endif /* !CONFIG_IRQ_PIPELINE */
+
+/* test hardware interrupt enable bit */
+static inline int arch_irqs_disabled(void)
+{
+	return arch_irqs_disabled_flags(arch_local_save_flags());
+}
+
+struct pt_regs;
+
+extern void (*handle_arch_irq)(struct pt_regs *);
+
+static inline void arch_handle_irq_pipelined(struct pt_regs *regs)
+{
+	handle_arch_irq(regs);
+}
+
+static inline int arch_enable_oob_stage(void)
+{
+	return 0;
+}
+
+#endif /* _ASM_RISCV_IRQ_PIPELINE_H */
diff --git a/arch/riscv/include/asm/irqflags.h b/arch/riscv/include/asm/irqflags.h
index 6fd8cbfcfcc7..c3087b74752b 100644
--- a/arch/riscv/include/asm/irqflags.h
+++ b/arch/riscv/include/asm/irqflags.h
@@ -10,45 +10,57 @@
 #include <asm/csr.h>
 
 /* read interrupt enabled status */
-static inline unsigned long arch_local_save_flags(void)
+static inline unsigned long native_save_flags(void)
 {
-	return csr_read(CSR_STATUS);
+	return csr_read(CSR_STATUS) & SR_IE;
 }
 
 /* unconditionally enable interrupts */
-static inline void arch_local_irq_enable(void)
+static inline void native_irq_enable(void)
 {
 	csr_set(CSR_STATUS, SR_IE);
 }
 
 /* unconditionally disable interrupts */
-static inline void arch_local_irq_disable(void)
+static inline void native_irq_disable(void)
 {
 	csr_clear(CSR_STATUS, SR_IE);
 }
 
 /* get status and disable interrupts */
-static inline unsigned long arch_local_irq_save(void)
+static inline unsigned long native_irq_save(void)
 {
 	return csr_read_clear(CSR_STATUS, SR_IE);
 }
 
 /* test flags */
-static inline int arch_irqs_disabled_flags(unsigned long flags)
+static inline int native_irqs_disabled_flags(unsigned long flags)
 {
 	return !(flags & SR_IE);
 }
 
 /* test hardware interrupt enable bit */
-static inline int arch_irqs_disabled(void)
+static inline bool native_irqs_disabled(void)
 {
-	return arch_irqs_disabled_flags(arch_local_save_flags());
+	return native_irqs_disabled_flags(native_save_flags());
 }
 
 /* set interrupt enabled status */
-static inline void arch_local_irq_restore(unsigned long flags)
+static inline void native_irq_restore(unsigned long flags)
 {
-	csr_set(CSR_STATUS, flags & SR_IE);
+	if (flags & SR_IE)
+		csr_set(CSR_STATUS, SR_IE);
+	else
+		csr_clear(CSR_STATUS, SR_IE);
+}
+
+#include <asm/irq_pipeline.h>
+
+static inline void native_irq_sync(void)
+{
+	native_irq_enable();
+	asm volatile("nop" : : : "memory");
+	native_irq_disable();
 }
 
 #endif /* _ASM_RISCV_IRQFLAGS_H */
diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h
index 836d80dd2921..9e4c9b884b5a 100644
--- a/arch/riscv/include/asm/thread_info.h
+++ b/arch/riscv/include/asm/thread_info.h
@@ -52,6 +52,10 @@
  */
 struct thread_info {
 	unsigned long		flags;		/* low level flags */
+#ifdef CONFIG_IRQ_PIPELINE
+	__u32			local_flags;	/* local (synchronous) flags */
+#define ti_local_flags(__ti)    ((__ti)->local_flags)
+#endif  
 	int                     preempt_count;  /* 0=>preemptible, <0=>BUG */
 	/*
 	 * These stack pointers are overwritten on every system call or
@@ -121,4 +125,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 
 #define _TIF_RISCV_V_DEFER_RESTORE	BIT(TIF_RISCV_V_DEFER_RESTORE)
 
+/*
+ * Local (synchronous) thread flags.
+ */
+#define _TLF_OOB		0x0001
+
 #endif /* _ASM_RISCV_THREAD_INFO_H */
diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile
index f60fce69b725..e4be4ae40b30 100644
--- a/arch/riscv/kernel/Makefile
+++ b/arch/riscv/kernel/Makefile
@@ -91,6 +91,7 @@ obj-$(CONFIG_MODULES)		+= module.o
 obj-$(CONFIG_MODULE_SECTIONS)	+= module-sections.o
 
 obj-$(CONFIG_CPU_PM)		+= suspend_entry.o suspend.o
+obj-$(CONFIG_IRQ_PIPELINE)	+= irq_pipeline.o
 obj-$(CONFIG_HIBERNATION)	+= hibernate.o hibernate-asm.o
 
 obj-$(CONFIG_FUNCTION_TRACER)	+= mcount.o ftrace.o
diff --git a/arch/riscv/kernel/irq_pipeline.c b/arch/riscv/kernel/irq_pipeline.c
new file mode 100644
index 000000000000..4eeb4394434f
--- /dev/null
+++ b/arch/riscv/kernel/irq_pipeline.c
@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * IRQ Pipelining implementation adapted from the ARM version.
+ *
+ * Copyright (C) 2024 Siemens AG
+ * Author:       Tobias Schaffner <tobias.schaffner@siemens.com>.
+ */
+#include <linux/irq.h>
+#include <linux/irq_pipeline.h>
+
+void arch_do_IRQ_pipelined(struct irq_desc *desc)
+{
+	struct pt_regs *regs = raw_cpu_ptr(&irq_pipeline.tick_regs);
+	struct pt_regs *old_regs = set_irq_regs(regs);
+
+	irq_enter();
+	handle_irq_desc(desc);
+	irq_exit();
+
+	set_irq_regs(old_regs);
+}
+
+void __init arch_irq_pipeline_init(void)
+{
+	/* no per-arch init. */
+}
diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c
index 99972a48e86b..c4444577449c 100644
--- a/arch/riscv/kernel/kernel_mode_vector.c
+++ b/arch/riscv/kernel/kernel_mode_vector.c
@@ -171,7 +171,7 @@ asmlinkage void riscv_v_context_nesting_end(struct pt_regs *regs)
 	struct __riscv_v_ext_state *vstate = &current->thread.kernel_vstate;
 	u32 depth;
 
-	WARN_ON(!irqs_disabled());
+	WARN_ON(!hard_irqs_disabled());
 
 	if (!riscv_preempt_v_started(current))
 		return;
diff --git a/arch/riscv/kernel/sbi-ipi.c b/arch/riscv/kernel/sbi-ipi.c
index 0cc5559c08d8..176f4ad3c427 100644
--- a/arch/riscv/kernel/sbi-ipi.c
+++ b/arch/riscv/kernel/sbi-ipi.c
@@ -12,6 +12,7 @@
 #include <linux/irqchip/chained_irq.h>
 #include <linux/irqdomain.h>
 #include <asm/sbi.h>
+#include <asm/irq_pipeline.h>
 
 DEFINE_STATIC_KEY_FALSE(riscv_sbi_for_rfence);
 EXPORT_SYMBOL_GPL(riscv_sbi_for_rfence);
@@ -57,7 +58,11 @@ void __init sbi_ipi_init(void)
 		return;
 	}
 
+#ifdef CONFIG_IRQ_PIPELINE
+	virq = ipi_mux_create(PIPELINED_IPI_MAX, sbi_send_ipi);
+#else
 	virq = ipi_mux_create(BITS_PER_BYTE, sbi_send_ipi);
+#endif
 	if (virq <= 0) {
 		pr_err("unable to create muxed IPIs\n");
 		irq_dispose_mapping(sbi_ipi_virq);
@@ -74,8 +79,11 @@ void __init sbi_ipi_init(void)
 	cpuhp_setup_state(CPUHP_AP_IRQ_RISCV_SBI_IPI_STARTING,
 			  "irqchip/sbi-ipi:starting",
 			  sbi_ipi_starting_cpu, NULL);
-
+#ifdef CONFIG_IRQ_PIPELINE
+	riscv_ipi_set_virq_range(virq, PIPELINED_IPI_MAX);
+#else
 	riscv_ipi_set_virq_range(virq, BITS_PER_BYTE);
+#endif
 	pr_info("providing IPIs using SBI IPI extension\n");
 
 	/*
diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c
index 5ed5095320e6..ab404da52b46 100644
--- a/arch/riscv/kernel/smp.c
+++ b/arch/riscv/kernel/smp.c
@@ -8,6 +8,7 @@
  * Copyright (C) 2017 SiFive
  */
 
+#include "linux/compiler.h"
 #include <linux/cpu.h>
 #include <linux/clockchips.h>
 #include <linux/interrupt.h>
@@ -21,6 +22,7 @@
 #include <linux/seq_file.h>
 #include <linux/delay.h>
 #include <linux/irq.h>
+#include <linux/irq_pipeline.h>
 #include <linux/irq_work.h>
 #include <linux/nmi.h>
 
@@ -28,6 +30,8 @@
 #include <asm/cacheflush.h>
 #include <asm/cpu_ops.h>
 
+#include <trace/events/ipi.h>
+
 enum ipi_message_type {
 	IPI_RESCHEDULE,
 	IPI_CALL_FUNC,
@@ -37,7 +41,13 @@ enum ipi_message_type {
 	IPI_TIMER,
 	IPI_CPU_BACKTRACE,
 	IPI_KGDB_ROUNDUP,
-	IPI_MAX
+#ifdef CONFIG_IRQ_PIPELINE
+	OOB_TIMER_IPI,
+	OOB_RESCHEDULE_IPI,
+	OOB_CALL_FUNCTION_IPI,
+#endif
+	IPI_MAX,
+
 };
 
 static const char * const ipi_names[] = {
@@ -49,8 +59,21 @@ static const char * const ipi_names[] = {
 	[IPI_TIMER]		= "Timer broadcast interrupts",
 	[IPI_CPU_BACKTRACE]     = "CPU backtrace interrupts",
 	[IPI_KGDB_ROUNDUP]	= "KGDB roundup interrupts",
+#ifdef CONFIG_IRQ_PIPELINE
+	[OOB_TIMER_IPI]		= "OOB timer interrupts",
+	[OOB_RESCHEDULE_IPI]	= "OOB reschedule interrupts",
+	[OOB_CALL_FUNCTION_IPI]	= "OOB call function interrupts",
+#endif
 };
 
+#ifdef CONFIG_IRQ_PIPELINE
+#define INBAND_IPI_MAX (IPI_MAX - OOB_NR_IPI)
+int ipi_max __ro_after_init = IPI_MAX;
+int NR_TIMER_OOB_IPI __ro_after_init = OOB_TIMER_IPI;
+int NR_RESCHEDULE_OOB_IPI __ro_after_init = OOB_RESCHEDULE_IPI;
+int NR_CALL_FUNCTION_OOB_IPI __ro_after_init = OOB_CALL_FUNCTION_IPI;
+#endif
+
 unsigned long __cpuid_to_hartid_map[NR_CPUS] __ro_after_init = {
 	[0 ... NR_CPUS-1] = INVALID_HARTID
 };
@@ -63,11 +86,11 @@ void __init smp_setup_processor_id(void)
 	pr_info("Booting Linux on hartid %lu\n", boot_cpu_hartid);
 }
 
-static DEFINE_PER_CPU_READ_MOSTLY(int, ipi_dummy_dev);
-static int ipi_virq_base __ro_after_init;
+int ipi_virq_base __ro_after_init;
 static int nr_ipi __ro_after_init = IPI_MAX;
 static struct irq_desc *ipi_desc[IPI_MAX] __read_mostly;
 
+
 int riscv_hartid_to_cpuid(unsigned long hartid)
 {
 	int i;
@@ -95,7 +118,7 @@ static inline void ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs)
 
 	atomic_dec(&waiting_for_crash_ipi);
 
-	local_irq_disable();
+	hard_local_irq_disable();
 
 #ifdef CONFIG_HOTPLUG_CPU
 	if (cpu_has_hotplug(cpu))
@@ -112,23 +135,6 @@ static inline void ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs)
 }
 #endif
 
-static void send_ipi_mask(const struct cpumask *mask, enum ipi_message_type op)
-{
-	__ipi_send_mask(ipi_desc[op], mask);
-}
-
-static void send_ipi_single(int cpu, enum ipi_message_type op)
-{
-	__ipi_send_mask(ipi_desc[op], cpumask_of(cpu));
-}
-
-#ifdef CONFIG_IRQ_WORK
-void arch_irq_work_raise(void)
-{
-	send_ipi_single(smp_processor_id(), IPI_IRQ_WORK);
-}
-#endif
-
 static irqreturn_t handle_IPI(int irq, void *data)
 {
 	unsigned int cpu = smp_processor_id();
@@ -169,6 +175,45 @@ static irqreturn_t handle_IPI(int irq, void *data)
 	return IRQ_HANDLED;
 }
 
+#ifdef CONFIG_IRQ_PIPELINE
+
+void irq_send_oob_ipi(unsigned int irq,
+		const struct cpumask *cpumask)
+{
+	unsigned int op = irq - ipi_virq_base;
+
+	if (WARN_ON(irq_pipeline_debug() &&
+		    (op < INBAND_IPI_MAX ||
+		     op >= IPI_MAX)))
+		return;
+
+	/* Init oob ipis at first involve*/
+	if (unlikely(ipi_desc[op] == NULL))
+		ipi_desc[op] = irq_to_desc(irq);
+
+	__ipi_send_mask(ipi_desc[op], cpumask);
+}
+EXPORT_SYMBOL_GPL(irq_send_oob_ipi);
+
+#endif
+
+static void send_ipi_mask(const struct cpumask *mask, enum ipi_message_type op)
+{
+	__ipi_send_mask(ipi_desc[op], mask);
+}
+
+static void send_ipi_single(int cpu, enum ipi_message_type op)
+{
+	__ipi_send_mask(ipi_desc[op], cpumask_of(cpu));
+}
+
+#ifdef CONFIG_IRQ_WORK
+void arch_irq_work_raise(void)
+{
+	send_ipi_single(smp_processor_id(), IPI_IRQ_WORK);
+}
+#endif
+
 void riscv_ipi_enable(void)
 {
 	int i;
@@ -204,13 +249,17 @@ void riscv_ipi_set_virq_range(int virq, int nr)
 		return;
 
 	WARN_ON(nr < IPI_MAX);
-	nr_ipi = min(nr, IPI_MAX);
+
 	ipi_virq_base = virq;
 
 	/* Request IPIs */
 	for (i = 0; i < nr_ipi; i++) {
-		err = request_percpu_irq(ipi_virq_base + i, handle_IPI,
-					 ipi_names[i], &ipi_dummy_dev);
+#ifdef CONFIG_IRQ_PIPELINE
+		if (i < INBAND_IPI_MAX)
+#endif
+			err = request_percpu_irq(ipi_virq_base + i, handle_IPI,
+						 ipi_names[i], &irq_stat);
+
 		WARN_ON(err);
 
 		ipi_desc[i] = irq_to_desc(ipi_virq_base + i);
diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c
index 601a321e0f17..9cdd5908edf4 100644
--- a/arch/riscv/kernel/smpboot.c
+++ b/arch/riscv/kernel/smpboot.c
@@ -248,6 +248,6 @@ asmlinkage __visible void smp_callin(void)
 	 * Disable preemption before enabling interrupts, so we don't try to
 	 * schedule a CPU that hasn't actually started yet.
 	 */
-	local_irq_enable();
+	local_irq_enable_full();
 	cpu_startup_entry(CPUHP_AP_ONLINE_IDLE);
 }
diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c
index 80230de167de..44b627d762f9 100644
--- a/arch/riscv/kernel/traps.c
+++ b/arch/riscv/kernel/traps.c
@@ -112,10 +112,36 @@ void die(struct pt_regs *regs, const char *str)
 		make_task_dead(SIGSEGV);
 }
 
+static __always_inline
+bool mark_trap_entry(int signo, struct pt_regs *regs)
+{
+	/*
+	 * Dovetail: irqentry_enter*() already synchronized the
+	 * virtual and real interrupt states for us. If running
+	 * in-band, we may re-enable hard irqs to allow oob events in
+	 * while we perform the regular trap handling.
+	 */
+	if (likely(running_inband())) {
+		hard_cond_local_irq_enable();
+		return true;
+	}
+
+	return false;
+}
+
+static __always_inline
+void mark_trap_exit(int signo, struct pt_regs *regs)
+{
+	hard_cond_local_irq_disable();
+}
+
 void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr)
 {
 	struct task_struct *tsk = current;
 
+	if (!mark_trap_entry(signo, regs))
+		return;
+
 	if (show_unhandled_signals && unhandled_signal(tsk, signo)
 	    && printk_ratelimit()) {
 		pr_info("%s[%d]: unhandled signal %d code 0x%x at 0x" REG_FMT,
@@ -127,6 +153,8 @@ void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr)
 	}
 
 	force_sig_fault(signo, code, (void __user *)addr);
+
+	mark_trap_exit(signo, regs);
 }
 
 static void do_trap_error(struct pt_regs *regs, int signo, int code,
@@ -137,6 +165,12 @@ static void do_trap_error(struct pt_regs *regs, int signo, int code,
 	if (user_mode(regs)) {
 		do_trap(regs, signo, code, addr);
 	} else {
+		/*
+		 * Dovetail: If we trapped from kernel space, either
+		 * we can fix up the situation, or we can't and we may
+		 * happily crash with hard irqs off. Either way, don't
+		 * bother.
+		 */
 		if (!fixup_exception(regs))
 			die(regs, str);
 	}
@@ -176,9 +210,19 @@ asmlinkage __visible __trap_section void do_trap_insn_illegal(struct pt_regs *re
 
 	if (user_mode(regs)) {
 		irqentry_enter_from_user_mode(regs);
-		local_irq_enable();
+
+		if (running_inband())
+			local_irq_enable_full();
+		else
+			hard_local_irq_enable();
 
 		handled = riscv_v_first_use_handler(regs);
+
+		if (running_inband())
+			local_irq_disable_full();
+		else
+			hard_local_irq_disable();
+
 		if (!handled)
 			do_trap_error(regs, SIGILL, ILL_ILLOPC, regs->epc,
 				      "Oops - illegal instruction");
@@ -373,7 +417,10 @@ asmlinkage __visible noinstr void do_page_fault(struct pt_regs *regs)
 
 	handle_page_fault(regs);
 
-	local_irq_disable();
+	if (running_inband())
+		local_irq_disable_full();
+	else
+		hard_local_irq_disable();
 
 	irqentry_exit(regs, state);
 }
@@ -383,13 +430,19 @@ static void noinstr handle_riscv_irq(struct pt_regs *regs)
 {
 	struct pt_regs *old_regs;
 
-	irq_enter_rcu();
-	old_regs = set_irq_regs(regs);
-	handle_arch_irq(regs);
-	set_irq_regs(old_regs);
-	irq_exit_rcu();
+	if (irqs_pipelined()) {
+		handle_irq_pipelined(regs);
+	} else {
+		irq_enter_rcu();
+		old_regs = set_irq_regs(regs);
+		handle_arch_irq(regs);
+		set_irq_regs(old_regs);
+		irq_exit_rcu();
+	}
 }
 
+extern void (*handle_arch_irq)(struct pt_regs *);
+
 asmlinkage void noinstr do_irq(struct pt_regs *regs)
 {
 	irqentry_state_t state = irqentry_enter(regs);
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 04ed6f8acae4..d64d01874a79 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -25,6 +25,52 @@
 
 #include "../kernel/head.h"
 
+#ifdef CONFIG_IRQ_PIPELINE
+/*
+ * We need to synchronize the virtual interrupt state with the hard
+ * interrupt state we received on entry, then turn hardirqs back on to
+ * allow code which does not require strict serialization to be
+ * preempted by an out-of-band activity.
+ */
+static inline
+unsigned long fault_entry(struct pt_regs *regs)
+{
+	unsigned long flags;
+
+	flags = hard_local_save_flags();
+
+	if (raw_irqs_disabled_flags(flags)) {
+		stall_inband();
+		trace_hardirqs_off();
+	}
+
+	hard_local_irq_enable();
+
+	if (running_inband())
+		local_irq_enable();
+
+	return flags;
+}
+
+static inline void fault_exit(unsigned long flags)
+{
+	WARN_ON_ONCE(irq_pipeline_debug() && hard_irqs_disabled());
+
+	hard_local_irq_restore(flags);
+}
+
+#else	/* !CONFIG_IRQ_PIPELINE */
+
+static inline
+unsigned long fault_entry(struct pt_regs *regs)
+{
+	return 0;
+}
+
+static inline void fault_exit(unsigned long x) { }
+
+#endif	/* !CONFIG_IRQ_PIPELINE */
+
 static void show_pte(unsigned long addr)
 {
 	pgd_t *pgdp, pgd;
@@ -79,6 +125,7 @@ static void show_pte(unsigned long addr)
 static void die_kernel_fault(const char *msg, unsigned long addr,
 		struct pt_regs *regs)
 {
+	irq_pipeline_oops();
 	bust_spinlocks(1);
 
 	pr_alert("Unable to handle kernel %s at virtual address " REG_FMT "\n", msg,
@@ -315,8 +362,8 @@ void handle_page_fault(struct pt_regs *regs)
 	}
 
 	/* Enable interrupts if they were enabled in the parent context. */
-	if (!regs_irqs_disabled(regs))
-		local_irq_enable();
+	if (!regs_irqs_disabled(regs) && running_inband())
+		local_irq_enable_full();
 
 	/*
 	 * If we're in an interrupt, have no user context, or are running
diff --git a/drivers/clocksource/timer-clint.c b/drivers/clocksource/timer-clint.c
index 0bdd9d7ec545..5345e4d373ac 100644
--- a/drivers/clocksource/timer-clint.c
+++ b/drivers/clocksource/timer-clint.c
@@ -243,7 +243,11 @@ static int __init clint_timer_init_dt(struct device_node *np)
 	}
 
 #ifdef CONFIG_SMP
+#ifdef CONFIG_IRQ_PIPELINE
+	rc = ipi_mux_create(PIPELINED_IPI_MAX, clint_send_ipi);
+#else
 	rc = ipi_mux_create(BITS_PER_BYTE, clint_send_ipi);
+#endif
 	if (rc <= 0) {
 		pr_err("unable to create muxed IPIs\n");
 		rc = (rc < 0) ? rc : -ENODEV;
@@ -251,7 +255,11 @@ static int __init clint_timer_init_dt(struct device_node *np)
 	}
 
 	irq_set_chained_handler(clint_ipi_irq, clint_ipi_interrupt);
+#ifdef CONFIG_IRQ_PIPELINE
+	riscv_ipi_set_virq_range(rc, PIPELINED_IPI_MAX);
+#else
 	riscv_ipi_set_virq_range(rc, BITS_PER_BYTE);
+#endif
 	clint_clear_ipi();
 #endif
 
-- 
2.43.0

Re: [PATCH dovetail v4 01/12] riscv: irq_pipeline: add IRQ pipelining core

[Philippe Gerum]

Tobias Schaffner <tobias.schaffner@siemens.com> writes:

> This patchset integrates IRQ pipelining into the RISC-V architecture,
> bringing it in line with the Dovetail/IRQ pipeline model used on other
> architectures. It adds the core pipelining infrastructure, updates the
> interrupt and trap entry/exit paths, and adapts low-level primitives to
> cleanly separate in-band and out-of-band interrupt handling.
>
> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> Co-authored-by: shannmu <shanmu1901@gmail.com>
> Co-authored-by: Philippe Gerum <rpm@xenomai.org>
> ---
>  arch/riscv/Kconfig                     |   1 +
>  arch/riscv/include/asm/irq_pipeline.h  | 148 +++++++++++++++++++++++++
>  arch/riscv/include/asm/irqflags.h      |  32 ++++--
>  arch/riscv/include/asm/thread_info.h   |   9 ++
>  arch/riscv/kernel/Makefile             |   1 +
>  arch/riscv/kernel/irq_pipeline.c       |  26 +++++
>  arch/riscv/kernel/kernel_mode_vector.c |   2 +-
>  arch/riscv/kernel/sbi-ipi.c            |  10 +-
>  arch/riscv/kernel/smp.c                |  97 ++++++++++++----
>  arch/riscv/kernel/smpboot.c            |   2 +-
>  arch/riscv/kernel/traps.c              |  67 +++++++++--
>  arch/riscv/mm/fault.c                  |  51 ++++++++-
>  drivers/clocksource/timer-clint.c      |   8 ++
>  13 files changed, 408 insertions(+), 46 deletions(-)
>  create mode 100644 arch/riscv/include/asm/irq_pipeline.h
>  create mode 100644 arch/riscv/kernel/irq_pipeline.c
>
> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index fadec20b87a8..e93a0b6f9f2b 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -148,6 +148,7 @@ config RISCV
>  	select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD if 64BIT && MMU
>  	select HAVE_ARCH_USERFAULTFD_MINOR if 64BIT && USERFAULTFD
>  	select HAVE_ARCH_VMAP_STACK if MMU && 64BIT
> +	select HAVE_IRQ_PIPELINE
>  	select HAVE_ASM_MODVERSIONS
>  	select HAVE_CONTEXT_TRACKING_USER
>  	select HAVE_DEBUG_KMEMLEAK
> diff --git a/arch/riscv/include/asm/irq_pipeline.h b/arch/riscv/include/asm/irq_pipeline.h
> new file mode 100644
> index 000000000000..c479ec44b3a5
> --- /dev/null
> +++ b/arch/riscv/include/asm/irq_pipeline.h
> @@ -0,0 +1,148 @@
> +/* SPDX-License-Identifier: GPL-2.0
> + *
> + * IRQ Pipelining adapted from the ARM version.
> + *
> + * Copyright (C) 2024 Siemens AG
> + * Author:       Tobias Schaffner <tobias.schaffner@siemens.com>.
> + */
> +
> +#ifndef _ASM_RISCV_IRQ_PIPELINE_H
> +#define _ASM_RISCV_IRQ_PIPELINE_H
> +
> +#define OOB_NR_IPI		3
> +
> +
> +extern int ipi_virq_base;
> +extern int NR_CALL_FUNCTION_OOB_IPI;
> +extern int NR_RESCHEDULE_OOB_IPI;
> +extern int NR_TIMER_OOB_IPI;
> +extern int ipi_max;
> +
> +#define CALL_FUNCTION_OOB_IPI (ipi_virq_base + NR_CALL_FUNCTION_OOB_IPI)
> +#define RESCHEDULE_OOB_IPI (ipi_virq_base + NR_RESCHEDULE_OOB_IPI)
> +#define TIMER_OOB_IPI (ipi_virq_base + NR_TIMER_OOB_IPI)
> +#define PIPELINED_IPI_MAX ipi_max

Is there any reason for not having the above oob-specific stuff guarded
by CONFIG_IRQ_PIPELINE as below?

> +
> +#ifdef CONFIG_IRQ_PIPELINE
> +
> +#include <asm/ptrace.h>
> +
> +/* NOTE: Any bit should be fine as long as we don't hit SR_SIE or SR_MIE. */
> +#define IRQMASK_i_POS   31
> +
> +static inline notrace
> +unsigned long arch_irqs_virtual_to_native_flags(int stalled)
> +{
> +	if (!stalled)
> +		return SR_IE;
> +	return 0;
> +}
> +
> +static inline notrace
> +unsigned long arch_irqs_native_to_virtual_flags(unsigned long flags)
> +{
> +	return (!!native_irqs_disabled_flags(flags)) << IRQMASK_i_POS;
> +}
> +
> +static inline int arch_irqs_disabled_flags(unsigned long flags)
> +{
> +	return native_irqs_disabled_flags(flags);
> +}
> +
> +static inline notrace void arch_local_irq_enable(void)
> +{
> +	barrier();
> +	inband_irq_enable();
> +}
> +
> +static inline notrace void arch_local_irq_disable(void)
> +{
> +	inband_irq_disable();
> +	barrier();
> +}
> +
> +static inline notrace unsigned long arch_local_save_flags(void)
> +{
> +	int stalled = inband_irqs_disabled();
> +
> +	barrier();
> +	return arch_irqs_virtual_to_native_flags(stalled);
> +}
> +
> +static inline notrace unsigned long arch_local_irq_save(void)
> +{
> +	int stalled = inband_irq_save();
> +
> +	barrier();
> +	return arch_irqs_virtual_to_native_flags(stalled);
> +}
> +
> +/* set interrupt enabled status */
> +static inline void arch_local_irq_restore(unsigned long flags)
> +{
> +	inband_irq_restore(arch_irqs_disabled_flags(flags));
> +	barrier();
> +}
> +
> +static inline
> +void arch_save_timer_regs(struct pt_regs *dst, struct pt_regs *src)
> +{
> +	dst->status = src->status;
> +	dst->epc = src->epc;
> +}
> +
> +#else /* !CONFIG_IRQ_PIPELINE */
> +
> +static inline unsigned long arch_local_irq_save(void)
> +{
> +	return native_irq_save();
> +}
> +
> +static inline void arch_local_irq_enable(void)
> +{
> +	native_irq_enable();
> +}
> +
> +static inline void arch_local_irq_disable(void)
> +{
> +	native_irq_disable();
> +}
> +
> +static inline unsigned long arch_local_save_flags(void)
> +{
> +	return native_save_flags();
> +}
> +
> +static inline void arch_local_irq_restore(unsigned long flags)
> +{
> +	native_irq_restore(flags);
> +}
> +
> +static inline int arch_irqs_disabled_flags(unsigned long flags)
> +{
> +	return native_irqs_disabled_flags(flags);
> +}
> +
> +#endif /* !CONFIG_IRQ_PIPELINE */
> +
> +/* test hardware interrupt enable bit */
> +static inline int arch_irqs_disabled(void)
> +{
> +	return arch_irqs_disabled_flags(arch_local_save_flags());
> +}
> +
> +struct pt_regs;
> +
> +extern void (*handle_arch_irq)(struct pt_regs *);
> +
> +static inline void arch_handle_irq_pipelined(struct pt_regs *regs)
> +{
> +	handle_arch_irq(regs);
> +}
> +
> +static inline int arch_enable_oob_stage(void)
> +{
> +	return 0;
> +}
> +
> +#endif /* _ASM_RISCV_IRQ_PIPELINE_H */
> diff --git a/arch/riscv/include/asm/irqflags.h b/arch/riscv/include/asm/irqflags.h
> index 6fd8cbfcfcc7..c3087b74752b 100644
> --- a/arch/riscv/include/asm/irqflags.h
> +++ b/arch/riscv/include/asm/irqflags.h
> @@ -10,45 +10,57 @@
>  #include <asm/csr.h>
>  
>  /* read interrupt enabled status */
> -static inline unsigned long arch_local_save_flags(void)
> +static inline unsigned long native_save_flags(void)
>  {
> -	return csr_read(CSR_STATUS);
> +	return csr_read(CSR_STATUS) & SR_IE;
>  }
>  
>  /* unconditionally enable interrupts */
> -static inline void arch_local_irq_enable(void)
> +static inline void native_irq_enable(void)
>  {
>  	csr_set(CSR_STATUS, SR_IE);
>  }
>  
>  /* unconditionally disable interrupts */
> -static inline void arch_local_irq_disable(void)
> +static inline void native_irq_disable(void)
>  {
>  	csr_clear(CSR_STATUS, SR_IE);
>  }
>  
>  /* get status and disable interrupts */
> -static inline unsigned long arch_local_irq_save(void)
> +static inline unsigned long native_irq_save(void)
>  {
>  	return csr_read_clear(CSR_STATUS, SR_IE);
>  }
>  
>  /* test flags */
> -static inline int arch_irqs_disabled_flags(unsigned long flags)
> +static inline int native_irqs_disabled_flags(unsigned long flags)
>  {
>  	return !(flags & SR_IE);
>  }
>  
>  /* test hardware interrupt enable bit */
> -static inline int arch_irqs_disabled(void)
> +static inline bool native_irqs_disabled(void)
>  {
> -	return arch_irqs_disabled_flags(arch_local_save_flags());
> +	return native_irqs_disabled_flags(native_save_flags());
>  }
>  
>  /* set interrupt enabled status */
> -static inline void arch_local_irq_restore(unsigned long flags)
> +static inline void native_irq_restore(unsigned long flags)
>  {
> -	csr_set(CSR_STATUS, flags & SR_IE);
> +	if (flags & SR_IE)
> +		csr_set(CSR_STATUS, SR_IE);
> +	else
> +		csr_clear(CSR_STATUS, SR_IE);
> +}
> +
> +#include <asm/irq_pipeline.h>
> +
> +static inline void native_irq_sync(void)
> +{
> +	native_irq_enable();
> +	asm volatile("nop" : : : "memory");
> +	native_irq_disable();
>  }
>  
>  #endif /* _ASM_RISCV_IRQFLAGS_H */
> diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h
> index 836d80dd2921..9e4c9b884b5a 100644
> --- a/arch/riscv/include/asm/thread_info.h
> +++ b/arch/riscv/include/asm/thread_info.h
> @@ -52,6 +52,10 @@
>   */
>  struct thread_info {
>  	unsigned long		flags;		/* low level flags */
> +#ifdef CONFIG_IRQ_PIPELINE
> +	__u32			local_flags;	/* local (synchronous) flags */
> +#define ti_local_flags(__ti)    ((__ti)->local_flags)
> +#endif  
>  	int                     preempt_count;  /* 0=>preemptible, <0=>BUG */
>  	/*
>  	 * These stack pointers are overwritten on every system call or
> @@ -121,4 +125,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
>  
>  #define _TIF_RISCV_V_DEFER_RESTORE	BIT(TIF_RISCV_V_DEFER_RESTORE)
>  
> +/*
> + * Local (synchronous) thread flags.
> + */
> +#define _TLF_OOB		0x0001
> +
>  #endif /* _ASM_RISCV_THREAD_INFO_H */
> diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile
> index f60fce69b725..e4be4ae40b30 100644
> --- a/arch/riscv/kernel/Makefile
> +++ b/arch/riscv/kernel/Makefile
> @@ -91,6 +91,7 @@ obj-$(CONFIG_MODULES)		+= module.o
>  obj-$(CONFIG_MODULE_SECTIONS)	+= module-sections.o
>  
>  obj-$(CONFIG_CPU_PM)		+= suspend_entry.o suspend.o
> +obj-$(CONFIG_IRQ_PIPELINE)	+= irq_pipeline.o
>  obj-$(CONFIG_HIBERNATION)	+= hibernate.o hibernate-asm.o
>  
>  obj-$(CONFIG_FUNCTION_TRACER)	+= mcount.o ftrace.o
> diff --git a/arch/riscv/kernel/irq_pipeline.c b/arch/riscv/kernel/irq_pipeline.c
> new file mode 100644
> index 000000000000..4eeb4394434f
> --- /dev/null
> +++ b/arch/riscv/kernel/irq_pipeline.c
> @@ -0,0 +1,26 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * IRQ Pipelining implementation adapted from the ARM version.
> + *
> + * Copyright (C) 2024 Siemens AG
> + * Author:       Tobias Schaffner <tobias.schaffner@siemens.com>.
> + */
> +#include <linux/irq.h>
> +#include <linux/irq_pipeline.h>
> +
> +void arch_do_IRQ_pipelined(struct irq_desc *desc)
> +{
> +	struct pt_regs *regs = raw_cpu_ptr(&irq_pipeline.tick_regs);
> +	struct pt_regs *old_regs = set_irq_regs(regs);
> +
> +	irq_enter();
> +	handle_irq_desc(desc);
> +	irq_exit();
> +
> +	set_irq_regs(old_regs);
> +}
> +
> +void __init arch_irq_pipeline_init(void)
> +{
> +	/* no per-arch init. */
> +}
> diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c
> index 99972a48e86b..c4444577449c 100644
> --- a/arch/riscv/kernel/kernel_mode_vector.c
> +++ b/arch/riscv/kernel/kernel_mode_vector.c
> @@ -171,7 +171,7 @@ asmlinkage void riscv_v_context_nesting_end(struct pt_regs *regs)
>  	struct __riscv_v_ext_state *vstate = &current->thread.kernel_vstate;
>  	u32 depth;
>  
> -	WARN_ON(!irqs_disabled());
> +	WARN_ON(!hard_irqs_disabled());
>  
>  	if (!riscv_preempt_v_started(current))
>  		return;
> diff --git a/arch/riscv/kernel/sbi-ipi.c b/arch/riscv/kernel/sbi-ipi.c
> index 0cc5559c08d8..176f4ad3c427 100644
> --- a/arch/riscv/kernel/sbi-ipi.c
> +++ b/arch/riscv/kernel/sbi-ipi.c
> @@ -12,6 +12,7 @@
>  #include <linux/irqchip/chained_irq.h>
>  #include <linux/irqdomain.h>
>  #include <asm/sbi.h>
> +#include <asm/irq_pipeline.h>
>  
>  DEFINE_STATIC_KEY_FALSE(riscv_sbi_for_rfence);
>  EXPORT_SYMBOL_GPL(riscv_sbi_for_rfence);
> @@ -57,7 +58,11 @@ void __init sbi_ipi_init(void)
>  		return;
>  	}
>  
> +#ifdef CONFIG_IRQ_PIPELINE
> +	virq = ipi_mux_create(PIPELINED_IPI_MAX, sbi_send_ipi);
> +#else
>  	virq = ipi_mux_create(BITS_PER_BYTE, sbi_send_ipi);
> +#endif
>  	if (virq <= 0) {
>  		pr_err("unable to create muxed IPIs\n");
>  		irq_dispose_mapping(sbi_ipi_virq);
> @@ -74,8 +79,11 @@ void __init sbi_ipi_init(void)
>  	cpuhp_setup_state(CPUHP_AP_IRQ_RISCV_SBI_IPI_STARTING,
>  			  "irqchip/sbi-ipi:starting",
>  			  sbi_ipi_starting_cpu, NULL);
> -
> +#ifdef CONFIG_IRQ_PIPELINE
> +	riscv_ipi_set_virq_range(virq, PIPELINED_IPI_MAX);
> +#else
>  	riscv_ipi_set_virq_range(virq, BITS_PER_BYTE);
> +#endif
>  	pr_info("providing IPIs using SBI IPI extension\n");
>

Maybe we could tame down the #ifdeffery a bit by defining a single
symbol (e.g. IRQ_RISCV_IPI_MAX or whatever fits there) with a
conditional value, which we could use everywhere BITS_PER_BYTE has this
meaning?

#ifdef CONFIG_IRQ_PIPELINE
#define IRQ_RISCV_IPI_MAX ipi_max
#else
#define IRQ_RISCV_IPI_MAX BITS_PER_BYTE
#endif

>  	/*
> diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c
> index 5ed5095320e6..ab404da52b46 100644
> --- a/arch/riscv/kernel/smp.c
> +++ b/arch/riscv/kernel/smp.c
> @@ -8,6 +8,7 @@
>   * Copyright (C) 2017 SiFive
>   */
>  
> +#include "linux/compiler.h"
>  #include <linux/cpu.h>
>  #include <linux/clockchips.h>
>  #include <linux/interrupt.h>
> @@ -21,6 +22,7 @@
>  #include <linux/seq_file.h>
>  #include <linux/delay.h>
>  #include <linux/irq.h>
> +#include <linux/irq_pipeline.h>
>  #include <linux/irq_work.h>
>  #include <linux/nmi.h>
>  
> @@ -28,6 +30,8 @@
>  #include <asm/cacheflush.h>
>  #include <asm/cpu_ops.h>
>  
> +#include <trace/events/ipi.h>
> +
>  enum ipi_message_type {
>  	IPI_RESCHEDULE,
>  	IPI_CALL_FUNC,
> @@ -37,7 +41,13 @@ enum ipi_message_type {
>  	IPI_TIMER,
>  	IPI_CPU_BACKTRACE,
>  	IPI_KGDB_ROUNDUP,
> -	IPI_MAX
> +#ifdef CONFIG_IRQ_PIPELINE
> +	OOB_TIMER_IPI,
> +	OOB_RESCHEDULE_IPI,
> +	OOB_CALL_FUNCTION_IPI,
> +#endif
> +	IPI_MAX,
> +
>  };
>  
>  static const char * const ipi_names[] = {
> @@ -49,8 +59,21 @@ static const char * const ipi_names[] = {
>  	[IPI_TIMER]		= "Timer broadcast interrupts",
>  	[IPI_CPU_BACKTRACE]     = "CPU backtrace interrupts",
>  	[IPI_KGDB_ROUNDUP]	= "KGDB roundup interrupts",
> +#ifdef CONFIG_IRQ_PIPELINE
> +	[OOB_TIMER_IPI]		= "OOB timer interrupts",
> +	[OOB_RESCHEDULE_IPI]	= "OOB reschedule interrupts",
> +	[OOB_CALL_FUNCTION_IPI]	= "OOB call function interrupts",
> +#endif
>  };
>  
> +#ifdef CONFIG_IRQ_PIPELINE
> +#define INBAND_IPI_MAX (IPI_MAX - OOB_NR_IPI)
> +int ipi_max __ro_after_init = IPI_MAX;
> +int NR_TIMER_OOB_IPI __ro_after_init = OOB_TIMER_IPI;
> +int NR_RESCHEDULE_OOB_IPI __ro_after_init = OOB_RESCHEDULE_IPI;
> +int NR_CALL_FUNCTION_OOB_IPI __ro_after_init = OOB_CALL_FUNCTION_IPI;
> +#endif
> +
>  unsigned long __cpuid_to_hartid_map[NR_CPUS] __ro_after_init = {
>  	[0 ... NR_CPUS-1] = INVALID_HARTID
>  };
> @@ -63,11 +86,11 @@ void __init smp_setup_processor_id(void)
>  	pr_info("Booting Linux on hartid %lu\n", boot_cpu_hartid);
>  }
>  
> -static DEFINE_PER_CPU_READ_MOSTLY(int, ipi_dummy_dev);
> -static int ipi_virq_base __ro_after_init;
> +int ipi_virq_base __ro_after_init;
>  static int nr_ipi __ro_after_init = IPI_MAX;
>  static struct irq_desc *ipi_desc[IPI_MAX] __read_mostly;
>  
> +
>  int riscv_hartid_to_cpuid(unsigned long hartid)
>  {
>  	int i;
> @@ -95,7 +118,7 @@ static inline void ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs)
>  
>  	atomic_dec(&waiting_for_crash_ipi);
>  
> -	local_irq_disable();
> +	hard_local_irq_disable();
>  
>  #ifdef CONFIG_HOTPLUG_CPU
>  	if (cpu_has_hotplug(cpu))
> @@ -112,23 +135,6 @@ static inline void ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs)
>  }
>  #endif
>  
> -static void send_ipi_mask(const struct cpumask *mask, enum ipi_message_type op)
> -{
> -	__ipi_send_mask(ipi_desc[op], mask);
> -}
> -
> -static void send_ipi_single(int cpu, enum ipi_message_type op)
> -{
> -	__ipi_send_mask(ipi_desc[op], cpumask_of(cpu));
> -}
> -
> -#ifdef CONFIG_IRQ_WORK
> -void arch_irq_work_raise(void)
> -{
> -	send_ipi_single(smp_processor_id(), IPI_IRQ_WORK);
> -}
> -#endif
> -
>  static irqreturn_t handle_IPI(int irq, void *data)
>  {
>  	unsigned int cpu = smp_processor_id();
> @@ -169,6 +175,45 @@ static irqreturn_t handle_IPI(int irq, void *data)
>  	return IRQ_HANDLED;
>  }
>  
> +#ifdef CONFIG_IRQ_PIPELINE
> +
> +void irq_send_oob_ipi(unsigned int irq,
> +		const struct cpumask *cpumask)
> +{
> +	unsigned int op = irq - ipi_virq_base;
> +
> +	if (WARN_ON(irq_pipeline_debug() &&
> +		    (op < INBAND_IPI_MAX ||
> +		     op >= IPI_MAX)))
> +		return;
> +
> +	/* Init oob ipis at first involve*/
> +	if (unlikely(ipi_desc[op] == NULL))
> +		ipi_desc[op] = irq_to_desc(irq);
> +
> +	__ipi_send_mask(ipi_desc[op], cpumask);
> +}
> +EXPORT_SYMBOL_GPL(irq_send_oob_ipi);
> +
> +#endif

Again, generally speaking, I would really try to group pipeline-specific
changes to only a few sections in the file. Using neutral/empty
placeholders when no work is required when IRQ_PIPELINE/DOVETAIL is off
would preserve readability.

> +
> +static void send_ipi_mask(const struct cpumask *mask, enum ipi_message_type op)
> +{
> +	__ipi_send_mask(ipi_desc[op], mask);
> +}
> +
> +static void send_ipi_single(int cpu, enum ipi_message_type op)
> +{
> +	__ipi_send_mask(ipi_desc[op], cpumask_of(cpu));
> +}
> +
> +#ifdef CONFIG_IRQ_WORK
> +void arch_irq_work_raise(void)
> +{
> +	send_ipi_single(smp_processor_id(), IPI_IRQ_WORK);
> +}
> +#endif
> +
>  void riscv_ipi_enable(void)
>  {
>  	int i;
> @@ -204,13 +249,17 @@ void riscv_ipi_set_virq_range(int virq, int nr)
>  		return;
>  
>  	WARN_ON(nr < IPI_MAX);
> -	nr_ipi = min(nr, IPI_MAX);
> +
>  	ipi_virq_base = virq;
>  
>  	/* Request IPIs */
>  	for (i = 0; i < nr_ipi; i++) {
> -		err = request_percpu_irq(ipi_virq_base + i, handle_IPI,
> -					 ipi_names[i], &ipi_dummy_dev);
> +#ifdef CONFIG_IRQ_PIPELINE
> +		if (i < INBAND_IPI_MAX)
> +#endif

On a general note, you may want to use the irqs_pipelined() predicate
(or dovetailed() if you need to check for CONFIG_DOVETAIL=y) instead of
guarding with CONFIG_IRQ_PIPELINE. Granted, you would need to provide a
neutral equivalent which the compiler would not complain about when
CONFIG_IRQ_PIPELINE=n, but that would be much easier to read, keeping
the original code and intent clear and the purpose of the
dovetail-related addition obvious (well, sort of).

e.g.

Some global conditional section:

#ifdef CONFIG_IRQ_PIPELINE

static inline bool foo(void)
{
        if (unlikely(strange_pipeline_work() == 42))
            return false;

        return true;
}

#else

static inline bool foo(void)
{
        return true;
}

#endif

...

int bar(void)
{
        if (!foo())
            return -EBADLUCK;

        return 0;
}

> +			err = request_percpu_irq(ipi_virq_base + i, handle_IPI,
> +						 ipi_names[i], &irq_stat);
> +
>  		WARN_ON(err);
>  
>  		ipi_desc[i] = irq_to_desc(ipi_virq_base + i);
> diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c
> index 601a321e0f17..9cdd5908edf4 100644
> --- a/arch/riscv/kernel/smpboot.c
> +++ b/arch/riscv/kernel/smpboot.c
> @@ -248,6 +248,6 @@ asmlinkage __visible void smp_callin(void)
>  	 * Disable preemption before enabling interrupts, so we don't try to
>  	 * schedule a CPU that hasn't actually started yet.
>  	 */
> -	local_irq_enable();
> +	local_irq_enable_full();
>  	cpu_startup_entry(CPUHP_AP_ONLINE_IDLE);
>  }
> diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c
> index 80230de167de..44b627d762f9 100644
> --- a/arch/riscv/kernel/traps.c
> +++ b/arch/riscv/kernel/traps.c
> @@ -112,10 +112,36 @@ void die(struct pt_regs *regs, const char *str)
>  		make_task_dead(SIGSEGV);
>  }
>  
> +static __always_inline
> +bool mark_trap_entry(int signo, struct pt_regs *regs)
> +{
> +	/*
> +	 * Dovetail: irqentry_enter*() already synchronized the
> +	 * virtual and real interrupt states for us. If running
> +	 * in-band, we may re-enable hard irqs to allow oob events in
> +	 * while we perform the regular trap handling.
> +	 */
> +	if (likely(running_inband())) {
> +		hard_cond_local_irq_enable();
> +		return true;
> +	}
> +
> +	return false;
> +}
> +
> +static __always_inline
> +void mark_trap_exit(int signo, struct pt_regs *regs)
> +{
> +	hard_cond_local_irq_disable();
> +}
> +
>  void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr)
>  {
>  	struct task_struct *tsk = current;
>  
> +	if (!mark_trap_entry(signo, regs))
> +		return;
> +

Strictly speaking, we don't have to make a trap entry until we actually
run a companion core, because this is only useful when hitting a trap
from the oob stage, and this cannot/should not happen unless such core
is allowing threads to run on that stage. Iff that happens, then we do
need the companion core to do whatever it takes to switch us back to the
in-band execution stage, so that we can run the recovery code. So, if
CONFIG_DOVETAIL=n, we should never have to mark an entry. If we do, then
we have a serious problem, because that would mean that we trapped over
the oob stage when handling an IRQ. In this case, mark_trap_entry()
returning false would bypass the recovery code, which would not help.

IOW, I believe that any injecting of mark_trap_entry() and friends
should happen in a separate commit, so that it is possible to debug the
pipeline core without any interaction from such code, so that any
suspicion that introducing such code might cause a problem is ruled out.

>  	if (show_unhandled_signals && unhandled_signal(tsk, signo)
>  	    && printk_ratelimit()) {
>  		pr_info("%s[%d]: unhandled signal %d code 0x%x at 0x" REG_FMT,
> @@ -127,6 +153,8 @@ void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr)
>  	}
>  
>  	force_sig_fault(signo, code, (void __user *)addr);
> +
> +	mark_trap_exit(signo, regs);
>  }
>  
>  static void do_trap_error(struct pt_regs *regs, int signo, int code,
> @@ -137,6 +165,12 @@ static void do_trap_error(struct pt_regs *regs, int signo, int code,
>  	if (user_mode(regs)) {
>  		do_trap(regs, signo, code, addr);
>  	} else {
> +		/*
> +		 * Dovetail: If we trapped from kernel space, either
> +		 * we can fix up the situation, or we can't and we may
> +		 * happily crash with hard irqs off. Either way, don't
> +		 * bother.
> +		 */
>  		if (!fixup_exception(regs))
>  			die(regs, str);
>  	}
> @@ -176,9 +210,19 @@ asmlinkage __visible __trap_section void do_trap_insn_illegal(struct pt_regs *re
>  
>  	if (user_mode(regs)) {
>  		irqentry_enter_from_user_mode(regs);
> -		local_irq_enable();
> +
> +		if (running_inband())
> +			local_irq_enable_full();
> +		else
> +			hard_local_irq_enable();
>  
>  		handled = riscv_v_first_use_handler(regs);
> +
> +		if (running_inband())
> +			local_irq_disable_full();
> +		else
> +			hard_local_irq_disable();
> +
>  		if (!handled)
>  			do_trap_error(regs, SIGILL, ILL_ILLOPC, regs->epc,
>  				      "Oops - illegal instruction");
> @@ -373,7 +417,10 @@ asmlinkage __visible noinstr void do_page_fault(struct pt_regs *regs)
>  
>  	handle_page_fault(regs);
>  
> -	local_irq_disable();
> +	if (running_inband())
> +		local_irq_disable_full();
> +	else
> +		hard_local_irq_disable();
>  
>  	irqentry_exit(regs, state);
>  }
> @@ -383,13 +430,19 @@ static void noinstr handle_riscv_irq(struct pt_regs *regs)
>  {
>  	struct pt_regs *old_regs;
>  
> -	irq_enter_rcu();
> -	old_regs = set_irq_regs(regs);
> -	handle_arch_irq(regs);
> -	set_irq_regs(old_regs);
> -	irq_exit_rcu();
> +	if (irqs_pipelined()) {
> +		handle_irq_pipelined(regs);
> +	} else {
> +		irq_enter_rcu();
> +		old_regs = set_irq_regs(regs);
> +		handle_arch_irq(regs);
> +		set_irq_regs(old_regs);
> +		irq_exit_rcu();
> +	}
>  }
>  
> +extern void (*handle_arch_irq)(struct pt_regs *);
> +
>  asmlinkage void noinstr do_irq(struct pt_regs *regs)
>  {
>  	irqentry_state_t state = irqentry_enter(regs);
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index 04ed6f8acae4..d64d01874a79 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -25,6 +25,52 @@
>  
>  #include "../kernel/head.h"
>  
> +#ifdef CONFIG_IRQ_PIPELINE
> +/*
> + * We need to synchronize the virtual interrupt state with the hard
> + * interrupt state we received on entry, then turn hardirqs back on to
> + * allow code which does not require strict serialization to be
> + * preempted by an out-of-band activity.
> + */
> +static inline
> +unsigned long fault_entry(struct pt_regs *regs)
> +{
> +	unsigned long flags;
> +
> +	flags = hard_local_save_flags();
> +
> +	if (raw_irqs_disabled_flags(flags)) {
> +		stall_inband();
> +		trace_hardirqs_off();
> +	}
> +
> +	hard_local_irq_enable();
> +
> +	if (running_inband())
> +		local_irq_enable();
> +
> +	return flags;
> +}
> +
> +static inline void fault_exit(unsigned long flags)
> +{
> +	WARN_ON_ONCE(irq_pipeline_debug() && hard_irqs_disabled());
> +
> +	hard_local_irq_restore(flags);
> +}
> +
> +#else	/* !CONFIG_IRQ_PIPELINE */
> +
> +static inline
> +unsigned long fault_entry(struct pt_regs *regs)
> +{
> +	return 0;
> +}
> +
> +static inline void fault_exit(unsigned long x) { }
> +
> +#endif	/* !CONFIG_IRQ_PIPELINE */
> +
>  static void show_pte(unsigned long addr)
>  {
>  	pgd_t *pgdp, pgd;
> @@ -79,6 +125,7 @@ static void show_pte(unsigned long addr)
>  static void die_kernel_fault(const char *msg, unsigned long addr,
>  		struct pt_regs *regs)
>  {
> +	irq_pipeline_oops();
>  	bust_spinlocks(1);
>  
>  	pr_alert("Unable to handle kernel %s at virtual address " REG_FMT "\n", msg,
> @@ -315,8 +362,8 @@ void handle_page_fault(struct pt_regs *regs)
>  	}
>  
>  	/* Enable interrupts if they were enabled in the parent context. */
> -	if (!regs_irqs_disabled(regs))
> -		local_irq_enable();
> +	if (!regs_irqs_disabled(regs) && running_inband())
> +		local_irq_enable_full();
>  
>  	/*
>  	 * If we're in an interrupt, have no user context, or are running
> diff --git a/drivers/clocksource/timer-clint.c b/drivers/clocksource/timer-clint.c
> index 0bdd9d7ec545..5345e4d373ac 100644
> --- a/drivers/clocksource/timer-clint.c
> +++ b/drivers/clocksource/timer-clint.c
> @@ -243,7 +243,11 @@ static int __init clint_timer_init_dt(struct device_node *np)
>  	}
>  
>  #ifdef CONFIG_SMP
> +#ifdef CONFIG_IRQ_PIPELINE
> +	rc = ipi_mux_create(PIPELINED_IPI_MAX, clint_send_ipi);
> +#else
>  	rc = ipi_mux_create(BITS_PER_BYTE, clint_send_ipi);
> +#endif
>  	if (rc <= 0) {
>  		pr_err("unable to create muxed IPIs\n");
>  		rc = (rc < 0) ? rc : -ENODEV;
> @@ -251,7 +255,11 @@ static int __init clint_timer_init_dt(struct device_node *np)
>  	}
>  
>  	irq_set_chained_handler(clint_ipi_irq, clint_ipi_interrupt);
> +#ifdef CONFIG_IRQ_PIPELINE
> +	riscv_ipi_set_virq_range(rc, PIPELINED_IPI_MAX);
> +#else
>  	riscv_ipi_set_virq_range(rc, BITS_PER_BYTE);
> +#endif
>  	clint_clear_ipi();
>  #endif

-- 
Philippe.

[PATCH dovetail v4 02/12] riscv: irq_pipeline: fix irq stack handling

[Tobias Schaffner]

We can not rely on the task pointer with the companion core potentially
switching contetexts over the irq stack.

Track the nesting depth instead like arm64 does.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 arch/riscv/kernel/traps.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c
index 44b627d762f9..d0400086a075 100644
--- a/arch/riscv/kernel/traps.c
+++ b/arch/riscv/kernel/traps.c
@@ -443,14 +443,21 @@ static void noinstr handle_riscv_irq(struct pt_regs *regs)
 
 extern void (*handle_arch_irq)(struct pt_regs *);
 
+DEFINE_PER_CPU(int, irq_nesting);
+
 asmlinkage void noinstr do_irq(struct pt_regs *regs)
 {
 	irqentry_state_t state = irqentry_enter(regs);
 
-	if (IS_ENABLED(CONFIG_IRQ_STACKS) && on_thread_stack())
+#ifdef CONFIG_IRQ_STACKS
+	if (this_cpu_inc_return(irq_nesting) == 1)
 		call_on_irq_stack(regs, handle_riscv_irq);
 	else
 		handle_riscv_irq(regs);
+	this_cpu_dec(irq_nesting);
+#else
+	handle_riscv_irq(regs);
+#endif /* CONFIG_IRQ_STACKS */
 
 	irqentry_exit(regs, state);
 }
-- 
2.43.0

Re: [PATCH dovetail v4 02/12] riscv: irq_pipeline: fix irq stack handling

[Philippe Gerum]

Tobias Schaffner <tobias.schaffner@siemens.com> writes:

> We can not rely on the task pointer with the companion core potentially
> switching contetexts over the irq stack.
>
> Track the nesting depth instead like arm64 does.
>

This change is specific to CONFIG_DOVETAIL=y, so this should go to
"riscv: dovetail: add core support".

> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> ---
>  arch/riscv/kernel/traps.c | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c
> index 44b627d762f9..d0400086a075 100644
> --- a/arch/riscv/kernel/traps.c
> +++ b/arch/riscv/kernel/traps.c
> @@ -443,14 +443,21 @@ static void noinstr handle_riscv_irq(struct pt_regs *regs)
>  
>  extern void (*handle_arch_irq)(struct pt_regs *);
>  
> +DEFINE_PER_CPU(int, irq_nesting);
> +
>  asmlinkage void noinstr do_irq(struct pt_regs *regs)
>  {
>  	irqentry_state_t state = irqentry_enter(regs);
>  
> -	if (IS_ENABLED(CONFIG_IRQ_STACKS) && on_thread_stack())
> +#ifdef CONFIG_IRQ_STACKS
> +	if (this_cpu_inc_return(irq_nesting) == 1)

This is Dovetail-specific, we may want this code to be compiled out when
CONFIG_IRQ_PIPELINE=n. See the related section in v6.18 for arm64.
Nitpicking: actually we could even make this Dovetail-specific, because
the only way to have an oob task preempt an in-band task is when
CONFIG_DOVETAIL=y. The arm64 pipeline implementation is a bit sloppy on
this too.

>  		call_on_irq_stack(regs, handle_riscv_irq);
>  	else
>  		handle_riscv_irq(regs);
> +	this_cpu_dec(irq_nesting);
> +#else
> +	handle_riscv_irq(regs);
> +#endif /* CONFIG_IRQ_STACKS */
>  
>  	irqentry_exit(regs, state);
>  }

-- 
Philippe.

[PATCH dovetail v4 03/12] riscv: irq_pipeline: synchronize IRQs on exit to user mode

[Tobias Schaffner]

Ensure that any pending in-band IRQs are processed before returning
to user mode.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 arch/riscv/include/asm/entry-common.h | 28 +++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h
index b28ccc6cdeea..c56e2597ea01 100644
--- a/arch/riscv/include/asm/entry-common.h
+++ b/arch/riscv/include/asm/entry-common.h
@@ -7,6 +7,10 @@
 #include <asm/thread_info.h>
 #include <asm/vector.h>
 
+#ifdef CONFIG_IRQ_PIPELINE
+#include <linux/irqstage.h>
+#endif
+
 static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs,
 						  unsigned long ti_work)
 {
@@ -22,6 +26,30 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs,
 
 #define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare
 
+#ifdef CONFIG_IRQ_PIPELINE
+
+/*
+ * This is an ugly hack to fixup irqs sometimes getting lost in the log.
+ */
+static __always_inline void arch_exit_to_user_mode(void)
+{
+	struct irq_stage_data *p;
+	unsigned long flags;
+
+	if (!running_inband())
+		return;
+
+	p = this_inband_staged();
+	if (unlikely(stage_irqs_pending(p) && !in_pipeline())) {
+		flags = hard_local_irq_save();
+		sync_current_irq_stage();
+		hard_local_irq_restore(flags);
+	}
+}
+
+#define arch_exit_to_user_mode arch_exit_to_user_mode
+#endif
+
 void handle_page_fault(struct pt_regs *regs);
 void handle_break(struct pt_regs *regs);
 
-- 
2.43.0

[PATCH dovetail v4 04/12] irqchip/riscv-aplic-direct: enable pipelined interrupt control

[Tobias Schaffner]

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 drivers/irqchip/irq-riscv-aplic-direct.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/irqchip/irq-riscv-aplic-direct.c b/drivers/irqchip/irq-riscv-aplic-direct.c
index c2a75bf3d20c..c3b2b6b655cd 100644
--- a/drivers/irqchip/irq-riscv-aplic-direct.c
+++ b/drivers/irqchip/irq-riscv-aplic-direct.c
@@ -88,7 +88,8 @@ static struct irq_chip aplic_direct_chip = {
 #endif
 	.flags		= IRQCHIP_SET_TYPE_MASKED |
 			  IRQCHIP_SKIP_SET_WAKE |
-			  IRQCHIP_MASK_ON_SUSPEND,
+			  IRQCHIP_MASK_ON_SUSPEND |
+			  IRQCHIP_PIPELINE_SAFE,
 };
 
 static int aplic_direct_irqdomain_translate(struct irq_domain *d, struct irq_fwspec *fwspec,
-- 
2.43.0

[PATCH dovetail v4 05/12] irqchip/irq-riscv-aplic-msi: enable pipelined interrupt control

[Tobias Schaffner]

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 drivers/irqchip/irq-riscv-aplic-msi.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/irqchip/irq-riscv-aplic-msi.c b/drivers/irqchip/irq-riscv-aplic-msi.c
index fb8d1838609f..25de8929ea37 100644
--- a/drivers/irqchip/irq-riscv-aplic-msi.c
+++ b/drivers/irqchip/irq-riscv-aplic-msi.c
@@ -156,7 +156,8 @@ static const struct msi_domain_template aplic_msi_template = {
 		.irq_write_msi_msg	= aplic_msi_write_msg,
 		.flags			= IRQCHIP_SET_TYPE_MASKED |
 					  IRQCHIP_SKIP_SET_WAKE |
-					  IRQCHIP_MASK_ON_SUSPEND,
+					  IRQCHIP_MASK_ON_SUSPEND |
+					  IRQCHIP_PIPELINE_SAFE,
 	},
 
 	.ops = {
-- 
2.43.0

[PATCH dovetail v4 06/12] irqchip/irq-riscv-imsic-platform: enable pipelined interrupt control

[Tobias Schaffner]

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 drivers/irqchip/irq-riscv-imsic-platform.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/irqchip/irq-riscv-imsic-platform.c b/drivers/irqchip/irq-riscv-imsic-platform.c
index 643c8e459611..5d7ac88b7d1b 100644
--- a/drivers/irqchip/irq-riscv-imsic-platform.c
+++ b/drivers/irqchip/irq-riscv-imsic-platform.c
@@ -219,7 +219,9 @@ static struct irq_chip imsic_irq_base_chip = {
 	.irq_retrigger			= imsic_irq_retrigger,
 	.irq_ack			= imsic_irq_ack,
 	.irq_compose_msi_msg		= imsic_irq_compose_msg,
-	.flags				= IRQCHIP_SKIP_SET_WAKE | IRQCHIP_MASK_ON_SUSPEND,
+	.flags				= IRQCHIP_SKIP_SET_WAKE |
+				  	  IRQCHIP_MASK_ON_SUSPEND |
+				  	  IRQCHIP_PIPELINE_SAFE,
 };
 
 static int imsic_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
-- 
2.43.0

[PATCH dovetail v4 07/12] irqchip/irq-riscv-intc: enable pipelined interrupt control

[Tobias Schaffner]

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 drivers/irqchip/irq-riscv-intc.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/irqchip/irq-riscv-intc.c b/drivers/irqchip/irq-riscv-intc.c
index 70290b35b317..2aed498d68f6 100644
--- a/drivers/irqchip/irq-riscv-intc.c
+++ b/drivers/irqchip/irq-riscv-intc.c
@@ -111,6 +111,7 @@ static struct irq_chip riscv_intc_chip = {
 	.irq_mask = riscv_intc_irq_mask,
 	.irq_unmask = riscv_intc_irq_unmask,
 	.irq_eoi = riscv_intc_irq_eoi,
+	.flags = IRQCHIP_PIPELINE_SAFE,
 };
 
 static struct irq_chip andes_intc_chip = {
@@ -118,6 +119,7 @@ static struct irq_chip andes_intc_chip = {
 	.irq_mask	= andes_intc_irq_mask,
 	.irq_unmask	= andes_intc_irq_unmask,
 	.irq_eoi	= riscv_intc_irq_eoi,
+	.flags		= IRQCHIP_PIPELINE_SAFE,
 };
 
 static int riscv_intc_domain_map(struct irq_domain *d, unsigned int irq,
-- 
2.43.0

[PATCH dovetail v4 08/12] irqchip/irq-sifive-plic: enable pipelined interrupt control

[Tobias Schaffner]

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 drivers/irqchip/irq-sifive-plic.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/irqchip/irq-sifive-plic.c b/drivers/irqchip/irq-sifive-plic.c
index cbd7697bc148..94c9e9936042 100644
--- a/drivers/irqchip/irq-sifive-plic.c
+++ b/drivers/irqchip/irq-sifive-plic.c
@@ -204,7 +204,8 @@ static struct irq_chip plic_edge_chip = {
 #endif
 	.irq_set_type	= plic_irq_set_type,
 	.flags		= IRQCHIP_SKIP_SET_WAKE |
-			  IRQCHIP_AFFINITY_PRE_STARTUP,
+			  IRQCHIP_AFFINITY_PRE_STARTUP |
+			  IRQCHIP_PIPELINE_SAFE,
 };
 
 static struct irq_chip plic_chip = {
-- 
2.43.0

[PATCH dovetail v4 09/12] clocksource/timer-riscv: irq_pipeline: enable pipelined clock events

[Tobias Schaffner]

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 drivers/clocksource/timer-riscv.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/drivers/clocksource/timer-riscv.c b/drivers/clocksource/timer-riscv.c
index 4d7cf338824a..8d215ad82e77 100644
--- a/drivers/clocksource/timer-riscv.c
+++ b/drivers/clocksource/timer-riscv.c
@@ -70,7 +70,8 @@ static int riscv_clock_shutdown(struct clock_event_device *evt)
 static unsigned int riscv_clock_event_irq;
 static DEFINE_PER_CPU(struct clock_event_device, riscv_clock_event) = {
 	.name			= "riscv_timer_clockevent",
-	.features		= CLOCK_EVT_FEAT_ONESHOT,
+	.features		= CLOCK_EVT_FEAT_ONESHOT |
+				  CLOCK_EVT_FEAT_PIPELINE,
 	.rating			= 100,
 	.set_next_event		= riscv_clock_next_event,
 	.set_state_shutdown	= riscv_clock_shutdown,
@@ -149,7 +150,7 @@ static irqreturn_t riscv_timer_interrupt(int irq, void *dev_id)
 	struct clock_event_device *evdev = this_cpu_ptr(&riscv_clock_event);
 
 	riscv_clock_event_stop();
-	evdev->event_handler(evdev);
+	clockevents_handle_event(evdev);
 
 	return IRQ_HANDLED;
 }
@@ -181,9 +182,9 @@ static int __init riscv_timer_init_common(void)
 
 	sched_clock_register(riscv_sched_clock, 64, riscv_timebase);
 
-	error = request_percpu_irq(riscv_clock_event_irq,
-				    riscv_timer_interrupt,
-				    "riscv-timer", &riscv_clock_event);
+	error = __request_percpu_irq(riscv_clock_event_irq,
+				     riscv_timer_interrupt, IRQF_TIMER,
+				     "riscv-timer", &riscv_clock_event);
 	if (error) {
 		pr_err("registering percpu irq failed [%d]\n", error);
 		return error;
-- 
2.43.0

[PATCH dovetail v4 10/12] riscv: add initial dovetail co-kernel skeleton

[Tobias Schaffner]

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 arch/riscv/Kconfig                   |  2 ++
 arch/riscv/include/asm/dovetail.h    | 23 +++++++++++++++++++++++
 arch/riscv/include/asm/mmu_context.h |  4 ++++
 arch/riscv/include/asm/syscall.h     |  6 ++++++
 arch/riscv/include/asm/thread_info.h |  8 ++++++++
 5 files changed, 43 insertions(+)
 create mode 100644 arch/riscv/include/asm/dovetail.h

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index e93a0b6f9f2b..abec407aced5 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -149,6 +149,7 @@ config RISCV
 	select HAVE_ARCH_USERFAULTFD_MINOR if 64BIT && USERFAULTFD
 	select HAVE_ARCH_VMAP_STACK if MMU && 64BIT
 	select HAVE_IRQ_PIPELINE
+	select HAVE_DOVETAIL
 	select HAVE_ASM_MODVERSIONS
 	select HAVE_CONTEXT_TRACKING_USER
 	select HAVE_DEBUG_KMEMLEAK
@@ -377,6 +378,7 @@ config AS_HAS_OPTION_ARCH
 
 source "arch/riscv/Kconfig.socs"
 source "arch/riscv/Kconfig.errata"
+source "kernel/Kconfig.dovetail"
 
 menu "Platform type"
 
diff --git a/arch/riscv/include/asm/dovetail.h b/arch/riscv/include/asm/dovetail.h
new file mode 100644
index 000000000000..bb342e9eb18a
--- /dev/null
+++ b/arch/riscv/include/asm/dovetail.h
@@ -0,0 +1,23 @@
+/*
+* SPDX-License-Identifier: GPL-2.0
+*
+* Copyright (C) 2024 Tobias Schaffner
+*/
+#ifndef _ASM_RISCV_DOVETAIL_H
+#define _ASM_RISCV_DOVETAIL_H
+
+#if !defined(__ASSEMBLY__)
+#ifdef CONFIG_DOVETAIL
+
+static inline void arch_dovetail_exec_prepare(void)
+{ }
+
+static inline void arch_dovetail_switch_prepare(bool leave_inband)
+{ }
+
+static inline void arch_dovetail_switch_finish(bool enter_inband)
+{ }
+
+#endif /* CONFIG_DOVETAIL */
+#endif /* !__ASSEMBLY__ */
+#endif /* _ASM_RISCV_DOVETAIL_H */
diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/mmu_context.h
index 8c4bc49a3a0f..444136d7b253 100644
--- a/arch/riscv/include/asm/mmu_context.h
+++ b/arch/riscv/include/asm/mmu_context.h
@@ -38,6 +38,10 @@ static inline int init_new_context(struct task_struct *tsk,
 	return 0;
 }
 
+static inline void
+switch_oob_mm(struct mm_struct *prev, struct mm_struct *next,
+	      struct task_struct *tsk) { }
+
 DECLARE_STATIC_KEY_FALSE(use_asid_allocator);
 
 #ifdef CONFIG_RISCV_ISA_SUPM
diff --git a/arch/riscv/include/asm/syscall.h b/arch/riscv/include/asm/syscall.h
index 34313387f977..6ae989aafbcd 100644
--- a/arch/riscv/include/asm/syscall.h
+++ b/arch/riscv/include/asm/syscall.h
@@ -117,6 +117,12 @@ static inline bool arch_syscall_is_vdso_sigreturn(struct pt_regs *regs)
 	return false;
 }
 
+static inline unsigned long syscall_get_arg0(struct task_struct *task,
+					     struct pt_regs *regs)
+{
+	return regs->orig_a0;
+}
+
 asmlinkage long sys_riscv_flush_icache(uintptr_t, uintptr_t, uintptr_t);
 
 asmlinkage long sys_riscv_hwprobe(struct riscv_hwprobe *, size_t, size_t,
diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h
index 9e4c9b884b5a..6c40e29f6445 100644
--- a/arch/riscv/include/asm/thread_info.h
+++ b/arch/riscv/include/asm/thread_info.h
@@ -41,6 +41,7 @@
 
 #include <asm/processor.h>
 #include <asm/csr.h>
+#include <dovetail/thread_info.h>
 
 /*
  * low level task data that entry.S needs immediate access to
@@ -77,6 +78,7 @@ struct thread_info {
 	 */
 	unsigned long		a0, a1, a2;
 #endif
+	struct oob_thread_state oob_state;      /* co-kernel thread state */
 };
 
 #ifdef CONFIG_SHADOW_CALL_STACK
@@ -120,6 +122,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 
 #include <asm-generic/thread_info_tif.h>
 
+#define TIF_MAYDAY			14      /* emergency trap pending */
+#define _TIF_MAYDAY			(1 << TIF_MAYDAY)
+
 #define TIF_32BIT			16	/* compat-mode 32bit process */
 #define TIF_RISCV_V_DEFER_RESTORE	17	/* restore Vector before returing to user */
 
@@ -129,5 +134,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
  * Local (synchronous) thread flags.
  */
 #define _TLF_OOB		0x0001
+#define _TLF_DOVETAIL		0x0002
+#define _TLF_OFFSTAGE		0x0004
+#define _TLF_OOBTRAP		0x0008
 
 #endif /* _ASM_RISCV_THREAD_INFO_H */
-- 
2.43.0

Re: [PATCH dovetail v4 10/12] riscv: add initial dovetail co-kernel skeleton

[Philippe Gerum]

This patch is specific to CONFIG_DOVETAIL=y, so this should go to
"riscv: dovetail: add core support".

Tobias Schaffner <tobias.schaffner@siemens.com> writes:

> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> ---
>  arch/riscv/Kconfig                   |  2 ++
>  arch/riscv/include/asm/dovetail.h    | 23 +++++++++++++++++++++++
>  arch/riscv/include/asm/mmu_context.h |  4 ++++
>  arch/riscv/include/asm/syscall.h     |  6 ++++++
>  arch/riscv/include/asm/thread_info.h |  8 ++++++++
>  5 files changed, 43 insertions(+)
>  create mode 100644 arch/riscv/include/asm/dovetail.h
>
> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index e93a0b6f9f2b..abec407aced5 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -149,6 +149,7 @@ config RISCV
>  	select HAVE_ARCH_USERFAULTFD_MINOR if 64BIT && USERFAULTFD
>  	select HAVE_ARCH_VMAP_STACK if MMU && 64BIT
>  	select HAVE_IRQ_PIPELINE
> +	select HAVE_DOVETAIL
>  	select HAVE_ASM_MODVERSIONS
>  	select HAVE_CONTEXT_TRACKING_USER
>  	select HAVE_DEBUG_KMEMLEAK
> @@ -377,6 +378,7 @@ config AS_HAS_OPTION_ARCH
>  
>  source "arch/riscv/Kconfig.socs"
>  source "arch/riscv/Kconfig.errata"
> +source "kernel/Kconfig.dovetail"
>  
>  menu "Platform type"
>  
> diff --git a/arch/riscv/include/asm/dovetail.h b/arch/riscv/include/asm/dovetail.h
> new file mode 100644
> index 000000000000..bb342e9eb18a
> --- /dev/null
> +++ b/arch/riscv/include/asm/dovetail.h
> @@ -0,0 +1,23 @@
> +/*
> +* SPDX-License-Identifier: GPL-2.0
> +*
> +* Copyright (C) 2024 Tobias Schaffner
> +*/
> +#ifndef _ASM_RISCV_DOVETAIL_H
> +#define _ASM_RISCV_DOVETAIL_H
> +
> +#if !defined(__ASSEMBLY__)
> +#ifdef CONFIG_DOVETAIL
> +
> +static inline void arch_dovetail_exec_prepare(void)
> +{ }
> +
> +static inline void arch_dovetail_switch_prepare(bool leave_inband)
> +{ }
> +
> +static inline void arch_dovetail_switch_finish(bool enter_inband)
> +{ }
> +
> +#endif /* CONFIG_DOVETAIL */
> +#endif /* !__ASSEMBLY__ */
> +#endif /* _ASM_RISCV_DOVETAIL_H */
> diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/mmu_context.h
> index 8c4bc49a3a0f..444136d7b253 100644
> --- a/arch/riscv/include/asm/mmu_context.h
> +++ b/arch/riscv/include/asm/mmu_context.h
> @@ -38,6 +38,10 @@ static inline int init_new_context(struct task_struct *tsk,
>  	return 0;
>  }
>  
> +static inline void
> +switch_oob_mm(struct mm_struct *prev, struct mm_struct *next,
> +	      struct task_struct *tsk) { }
> +
>  DECLARE_STATIC_KEY_FALSE(use_asid_allocator);
>  
>  #ifdef CONFIG_RISCV_ISA_SUPM
> diff --git a/arch/riscv/include/asm/syscall.h b/arch/riscv/include/asm/syscall.h
> index 34313387f977..6ae989aafbcd 100644
> --- a/arch/riscv/include/asm/syscall.h
> +++ b/arch/riscv/include/asm/syscall.h
> @@ -117,6 +117,12 @@ static inline bool arch_syscall_is_vdso_sigreturn(struct pt_regs *regs)
>  	return false;
>  }
>  
> +static inline unsigned long syscall_get_arg0(struct task_struct *task,
> +					     struct pt_regs *regs)
> +{
> +	return regs->orig_a0;
> +}
> +
>  asmlinkage long sys_riscv_flush_icache(uintptr_t, uintptr_t, uintptr_t);
>  
>  asmlinkage long sys_riscv_hwprobe(struct riscv_hwprobe *, size_t, size_t,
> diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h
> index 9e4c9b884b5a..6c40e29f6445 100644
> --- a/arch/riscv/include/asm/thread_info.h
> +++ b/arch/riscv/include/asm/thread_info.h
> @@ -41,6 +41,7 @@
>  
>  #include <asm/processor.h>
>  #include <asm/csr.h>
> +#include <dovetail/thread_info.h>
>  
>  /*
>   * low level task data that entry.S needs immediate access to
> @@ -77,6 +78,7 @@ struct thread_info {
>  	 */
>  	unsigned long		a0, a1, a2;
>  #endif
> +	struct oob_thread_state oob_state;      /* co-kernel thread state */
>  };
>  
>  #ifdef CONFIG_SHADOW_CALL_STACK
> @@ -120,6 +122,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
>  
>  #include <asm-generic/thread_info_tif.h>
>  
> +#define TIF_MAYDAY			14      /* emergency trap pending */
> +#define _TIF_MAYDAY			(1 << TIF_MAYDAY)
> +
>  #define TIF_32BIT			16	/* compat-mode 32bit process */
>  #define TIF_RISCV_V_DEFER_RESTORE	17	/* restore Vector before returing to user */
>  
> @@ -129,5 +134,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
>   * Local (synchronous) thread flags.
>   */
>  #define _TLF_OOB		0x0001
> +#define _TLF_DOVETAIL		0x0002
> +#define _TLF_OFFSTAGE		0x0004
> +#define _TLF_OOBTRAP		0x0008
>  
>  #endif /* _ASM_RISCV_THREAD_INFO_H */

-- 
Philippe.

[PATCH dovetail v4 11/12] riscv: add out-of-band aware trap handling

[Tobias Schaffner]

Introduce trap handling hooks for the Dovetail co-kernel, integrating
out-of-band (OOB) notification and unwind support.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 arch/riscv/kernel/traps.c | 110 ++++++++++++++++++++++++++++++++------
 arch/riscv/mm/fault.c     |  21 +++++---
 2 files changed, 108 insertions(+), 23 deletions(-)

diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c
index d0400086a075..51e1407ee066 100644
--- a/arch/riscv/kernel/traps.c
+++ b/arch/riscv/kernel/traps.c
@@ -3,6 +3,8 @@
  * Copyright (C) 2012 Regents of the University of California
  */
 
+#include <asm-generic/signal.h>
+#include <linux/compiler_attributes.h>
 #include <linux/cpu.h>
 #include <linux/kernel.h>
 #include <linux/init.h>
@@ -115,6 +117,8 @@ void die(struct pt_regs *regs, const char *str)
 static __always_inline
 bool mark_trap_entry(int signo, struct pt_regs *regs)
 {
+	oob_trap_notify(signo, regs);
+
 	/*
 	 * Dovetail: irqentry_enter*() already synchronized the
 	 * virtual and real interrupt states for us. If running
@@ -126,21 +130,40 @@ bool mark_trap_entry(int signo, struct pt_regs *regs)
 		return true;
 	}
 
+	oob_trap_unwind(signo, regs);
+
 	return false;
 }
 
 static __always_inline
 void mark_trap_exit(int signo, struct pt_regs *regs)
 {
+	oob_trap_unwind(signo, regs);
 	hard_cond_local_irq_disable();
 }
 
-void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr)
+static __always_inline
+bool mark_trap_entry_raw(int trapnr, struct pt_regs *regs)
 {
-	struct task_struct *tsk = current;
+	oob_trap_notify(trapnr, regs);
 
-	if (!mark_trap_entry(signo, regs))
-		return;
+	if (running_oob()) {
+		oob_trap_unwind(trapnr, regs);
+		return false;
+	}
+
+	return true;
+}
+
+static __always_inline
+void mark_trap_exit_raw(int trapnr, struct pt_regs *regs)
+{
+	oob_trap_unwind(trapnr, regs);
+}
+
+static void do_trap_raw(struct pt_regs *regs, int signo, int code, unsigned long addr)
+{
+	struct task_struct *tsk = current;
 
 	if (show_unhandled_signals && unhandled_signal(tsk, signo)
 	    && printk_ratelimit()) {
@@ -153,6 +176,14 @@ void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr)
 	}
 
 	force_sig_fault(signo, code, (void __user *)addr);
+}
+
+void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr)
+{
+	if(!mark_trap_entry(signo, regs))
+		return;
+
+	do_trap_raw(regs, signo, code, addr);
 
 	mark_trap_exit(signo, regs);
 }
@@ -163,7 +194,7 @@ static void do_trap_error(struct pt_regs *regs, int signo, int code,
 	current->thread.bad_cause = regs->cause;
 
 	if (user_mode(regs)) {
-		do_trap(regs, signo, code, addr);
+		do_trap_raw(regs, signo, code, addr);
 	} else {
 		/*
 		 * Dovetail: If we trapped from kernel space, either
@@ -181,9 +212,12 @@ static void do_trap_error(struct pt_regs *regs, int signo, int code,
 #else
 #define __trap_section noinstr
 #endif
-#define DO_ERROR_INFO(name, signo, code, str)					\
+#define DO_ERROR_INFO(name, signo, code, str, trapnr)				\
 asmlinkage __visible __trap_section void name(struct pt_regs *regs)		\
 {										\
+	if(!mark_trap_entry(trapnr, regs))					\
+		return;								\
+										\
 	if (user_mode(regs)) {							\
 		irqentry_enter_from_user_mode(regs);				\
 		local_irq_enable();						\
@@ -195,19 +229,24 @@ asmlinkage __visible __trap_section void name(struct pt_regs *regs)		\
 		do_trap_error(regs, signo, code, regs->epc, "Oops - " str);	\
 		irqentry_nmi_exit(regs, state);					\
 	}									\
+										\
+	mark_trap_exit(trapnr, regs);						\
 }
 
 DO_ERROR_INFO(do_trap_unknown,
-	SIGILL, ILL_ILLTRP, "unknown exception");
+	SIGILL, ILL_ILLTRP, "unknown exception", EXC_INST_ILLEGAL);
 DO_ERROR_INFO(do_trap_insn_misaligned,
-	SIGBUS, BUS_ADRALN, "instruction address misaligned");
+	SIGBUS, BUS_ADRALN, "instruction address misaligned", EXC_INST_MISALIGNED);
 DO_ERROR_INFO(do_trap_insn_fault,
-	SIGSEGV, SEGV_ACCERR, "instruction access fault");
+	SIGSEGV, SEGV_ACCERR, "instruction access fault", EXC_INST_ACCESS);
 
 asmlinkage __visible __trap_section void do_trap_insn_illegal(struct pt_regs *regs)
 {
 	bool handled;
 
+	if(!mark_trap_entry(EXC_INST_ILLEGAL, regs))
+		return;
+
 	if (user_mode(regs)) {
 		irqentry_enter_from_user_mode(regs);
 
@@ -237,10 +276,11 @@ asmlinkage __visible __trap_section void do_trap_insn_illegal(struct pt_regs *re
 
 		irqentry_nmi_exit(regs, state);
 	}
+	mark_trap_exit(EXC_INST_ILLEGAL, regs);
 }
 
 DO_ERROR_INFO(do_trap_load_fault,
-	SIGSEGV, SEGV_ACCERR, "load access fault");
+	SIGSEGV, SEGV_ACCERR, "load access fault", EXC_LOAD_ACCESS);
 
 enum misaligned_access_type {
 	MISALIGNED_STORE,
@@ -264,6 +304,9 @@ static void do_trap_misaligned(struct pt_regs *regs, enum misaligned_access_type
 {
 	irqentry_state_t state;
 
+	if(!mark_trap_entry(EXC_LOAD_MISALIGNED, regs))
+		return;
+
 	if (user_mode(regs)) {
 		irqentry_enter_from_user_mode(regs);
 		local_irq_enable();
@@ -281,6 +324,8 @@ static void do_trap_misaligned(struct pt_regs *regs, enum misaligned_access_type
 	} else {
 		irqentry_nmi_exit(regs, state);
 	}
+
+	mark_trap_exit(EXC_LOAD_MISALIGNED, regs);
 }
 
 asmlinkage __visible __trap_section void do_trap_load_misaligned(struct pt_regs *regs)
@@ -290,15 +335,36 @@ asmlinkage __visible __trap_section void do_trap_load_misaligned(struct pt_regs
 
 asmlinkage __visible __trap_section void do_trap_store_misaligned(struct pt_regs *regs)
 {
-	do_trap_misaligned(regs, MISALIGNED_STORE);
+	if(!mark_trap_entry(EXC_STORE_MISALIGNED, regs))
+		return;
+
+	if (user_mode(regs)) {
+		irqentry_enter_from_user_mode(regs);
+
+		if (handle_misaligned_store(regs))
+			do_trap_error(regs, SIGBUS, BUS_ADRALN, regs->epc,
+				"Oops - store (or AMO) address misaligned");
+
+		irqentry_exit_to_user_mode(regs);
+	} else {
+		irqentry_state_t state = irqentry_nmi_enter(regs);
+
+		if (handle_misaligned_store(regs))
+			do_trap_error(regs, SIGBUS, BUS_ADRALN, regs->epc,
+				"Oops - store (or AMO) address misaligned");
+
+		irqentry_nmi_exit(regs, state);
+	}
+
+	mark_trap_exit(EXC_STORE_MISALIGNED, regs);
 }
 
 DO_ERROR_INFO(do_trap_store_fault,
-	SIGSEGV, SEGV_ACCERR, "store (or AMO) access fault");
+	SIGSEGV, SEGV_ACCERR, "store (or AMO) access fault", EXC_STORE_ACCESS);
 DO_ERROR_INFO(do_trap_ecall_s,
-	SIGILL, ILL_ILLTRP, "environment call from S-mode");
+	SIGILL, ILL_ILLTRP, "environment call from S-mode", EXC_SYSCALL);
 DO_ERROR_INFO(do_trap_ecall_m,
-	SIGILL, ILL_ILLTRP, "environment call from M-mode");
+	SIGILL, ILL_ILLTRP, "environment call from M-mode", EXC_SUPERVISOR_SYSCALL);
 
 static inline unsigned long get_break_insn_length(unsigned long pc)
 {
@@ -350,6 +416,9 @@ void handle_break(struct pt_regs *regs)
 
 asmlinkage __visible __trap_section void do_trap_break(struct pt_regs *regs)
 {
+	if(!mark_trap_entry_raw(EXC_BREAKPOINT, regs))
+		return;
+
 	if (user_mode(regs)) {
 		irqentry_enter_from_user_mode(regs);
 		local_irq_enable();
@@ -365,6 +434,8 @@ asmlinkage __visible __trap_section void do_trap_break(struct pt_regs *regs)
 
 		irqentry_nmi_exit(regs, state);
 	}
+
+	mark_trap_exit_raw(EXC_BREAKPOINT, regs);
 }
 
 asmlinkage __visible __trap_section  __no_stack_protector
@@ -381,6 +452,15 @@ void do_trap_ecall_u(struct pt_regs *regs)
 
 		syscall = syscall_enter_from_user_mode(regs, syscall);
 
+		if(dovetailing()) {
+			if (syscall == EXIT_SYSCALL_OOB) {
+				hard_local_irq_disable();
+				return;
+			}
+			if (syscall == EXIT_SYSCALL_TAIL)
+				goto done_inband;
+		}
+
 		add_random_kstack_offset();
 
 		if (syscall >= 0 && syscall < NR_syscalls)
@@ -398,6 +478,7 @@ void do_trap_ecall_u(struct pt_regs *regs)
 		 */
 		choose_random_kstack_offset(get_random_u16());
 
+done_inband:
 		syscall_exit_to_user_mode(regs);
 	} else {
 		irqentry_state_t state = irqentry_nmi_enter(regs);
@@ -407,7 +488,6 @@ void do_trap_ecall_u(struct pt_regs *regs)
 
 		irqentry_nmi_exit(regs, state);
 	}
-
 }
 
 #ifdef CONFIG_MMU
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index d64d01874a79..42450c47ee12 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -385,6 +385,10 @@ void handle_page_fault(struct pt_regs *regs)
 		die_kernel_fault("access to user memory without uaccess routines", addr, regs);
 	}
 
+	oob_trap_notify(EXC_INST_PAGE_FAULT, regs);
+	if (!running_inband())
+		goto out;
+
 	perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr);
 
 	if (cause == EXC_STORE_PAGE_FAULT)
@@ -403,7 +407,7 @@ void handle_page_fault(struct pt_regs *regs)
 		count_vm_vma_lock_event(VMA_LOCK_SUCCESS);
 		tsk->thread.bad_cause = cause;
 		bad_area_nosemaphore(regs, SEGV_ACCERR, addr);
-		return;
+		goto out;
 	}
 
 	fault = handle_mm_fault(vma, addr, flags | FAULT_FLAG_VMA_LOCK, regs);
@@ -421,7 +425,7 @@ void handle_page_fault(struct pt_regs *regs)
 	if (fault_signal_pending(fault, regs)) {
 		if (!user_mode(regs))
 			no_context(regs, addr);
-		return;
+		goto out;
 	}
 lock_mmap:
 
@@ -430,7 +434,7 @@ void handle_page_fault(struct pt_regs *regs)
 	if (unlikely(!vma)) {
 		tsk->thread.bad_cause = cause;
 		bad_area_nosemaphore(regs, code, addr);
-		return;
+		goto out;
 	}
 
 	/*
@@ -442,7 +446,7 @@ void handle_page_fault(struct pt_regs *regs)
 	if (unlikely(access_error(cause, vma))) {
 		tsk->thread.bad_cause = cause;
 		bad_area(regs, mm, code, addr);
-		return;
+		goto out;
 	}
 
 	/*
@@ -460,12 +464,12 @@ void handle_page_fault(struct pt_regs *regs)
 	if (fault_signal_pending(fault, regs)) {
 		if (!user_mode(regs))
 			no_context(regs, addr);
-		return;
+		goto out;
 	}
 
 	/* The fault is fully completed (including releasing mmap lock) */
 	if (fault & VM_FAULT_COMPLETED)
-		return;
+		goto out;
 
 	if (unlikely(fault & VM_FAULT_RETRY)) {
 		flags |= FAULT_FLAG_TRIED;
@@ -484,7 +488,8 @@ void handle_page_fault(struct pt_regs *regs)
 	if (unlikely(fault & VM_FAULT_ERROR)) {
 		tsk->thread.bad_cause = cause;
 		mm_fault_error(regs, addr, fault);
-		return;
 	}
-	return;
+
+out:
+	oob_trap_unwind(EXC_INST_PAGE_FAULT, regs);
 }
-- 
2.43.0

Re: [PATCH dovetail v4 11/12] riscv: add out-of-band aware trap handling

[Philippe Gerum]

"riscv: dovetail: add core support" as well.

Tobias Schaffner <tobias.schaffner@siemens.com> writes:

> Introduce trap handling hooks for the Dovetail co-kernel, integrating
> out-of-band (OOB) notification and unwind support.
>
> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> ---
>  arch/riscv/kernel/traps.c | 110 ++++++++++++++++++++++++++++++++------
>  arch/riscv/mm/fault.c     |  21 +++++---
>  2 files changed, 108 insertions(+), 23 deletions(-)
>
> diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c
> index d0400086a075..51e1407ee066 100644
> --- a/arch/riscv/kernel/traps.c
> +++ b/arch/riscv/kernel/traps.c
> @@ -3,6 +3,8 @@
>   * Copyright (C) 2012 Regents of the University of California
>   */
>  
> +#include <asm-generic/signal.h>
> +#include <linux/compiler_attributes.h>
>  #include <linux/cpu.h>
>  #include <linux/kernel.h>
>  #include <linux/init.h>
> @@ -115,6 +117,8 @@ void die(struct pt_regs *regs, const char *str)
>  static __always_inline
>  bool mark_trap_entry(int signo, struct pt_regs *regs)
>  {
> +	oob_trap_notify(signo, regs);
> +
>  	/*
>  	 * Dovetail: irqentry_enter*() already synchronized the
>  	 * virtual and real interrupt states for us. If running
> @@ -126,21 +130,40 @@ bool mark_trap_entry(int signo, struct pt_regs *regs)
>  		return true;
>  	}
>  
> +	oob_trap_unwind(signo, regs);
> +
>  	return false;
>  }
>  
>  static __always_inline
>  void mark_trap_exit(int signo, struct pt_regs *regs)
>  {
> +	oob_trap_unwind(signo, regs);
>  	hard_cond_local_irq_disable();
>  }
>  
> -void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr)
> +static __always_inline
> +bool mark_trap_entry_raw(int trapnr, struct pt_regs *regs)
>  {
> -	struct task_struct *tsk = current;
> +	oob_trap_notify(trapnr, regs);
>  
> -	if (!mark_trap_entry(signo, regs))
> -		return;
> +	if (running_oob()) {
> +		oob_trap_unwind(trapnr, regs);
> +		return false;
> +	}
> +
> +	return true;
> +}
> +
> +static __always_inline
> +void mark_trap_exit_raw(int trapnr, struct pt_regs *regs)
> +{
> +	oob_trap_unwind(trapnr, regs);
> +}
> +
> +static void do_trap_raw(struct pt_regs *regs, int signo, int code, unsigned long addr)
> +{
> +	struct task_struct *tsk = current;
>  
>  	if (show_unhandled_signals && unhandled_signal(tsk, signo)
>  	    && printk_ratelimit()) {
> @@ -153,6 +176,14 @@ void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr)
>  	}
>  
>  	force_sig_fault(signo, code, (void __user *)addr);
> +}
> +
> +void do_trap(struct pt_regs *regs, int signo, int code, unsigned long addr)
> +{
> +	if(!mark_trap_entry(signo, regs))
> +		return;
> +
> +	do_trap_raw(regs, signo, code, addr);
>  
>  	mark_trap_exit(signo, regs);
>  }
> @@ -163,7 +194,7 @@ static void do_trap_error(struct pt_regs *regs, int signo, int code,
>  	current->thread.bad_cause = regs->cause;
>  
>  	if (user_mode(regs)) {
> -		do_trap(regs, signo, code, addr);
> +		do_trap_raw(regs, signo, code, addr);
>  	} else {
>  		/*
>  		 * Dovetail: If we trapped from kernel space, either
> @@ -181,9 +212,12 @@ static void do_trap_error(struct pt_regs *regs, int signo, int code,
>  #else
>  #define __trap_section noinstr
>  #endif
> -#define DO_ERROR_INFO(name, signo, code, str)					\
> +#define DO_ERROR_INFO(name, signo, code, str, trapnr)				\
>  asmlinkage __visible __trap_section void name(struct pt_regs *regs)		\
>  {										\
> +	if(!mark_trap_entry(trapnr, regs))					\
> +		return;								\
> +										\
>  	if (user_mode(regs)) {							\
>  		irqentry_enter_from_user_mode(regs);				\
>  		local_irq_enable();						\
> @@ -195,19 +229,24 @@ asmlinkage __visible __trap_section void name(struct pt_regs *regs)		\
>  		do_trap_error(regs, signo, code, regs->epc, "Oops - " str);	\
>  		irqentry_nmi_exit(regs, state);					\
>  	}									\
> +										\
> +	mark_trap_exit(trapnr, regs);						\
>  }
>  
>  DO_ERROR_INFO(do_trap_unknown,
> -	SIGILL, ILL_ILLTRP, "unknown exception");
> +	SIGILL, ILL_ILLTRP, "unknown exception", EXC_INST_ILLEGAL);
>  DO_ERROR_INFO(do_trap_insn_misaligned,
> -	SIGBUS, BUS_ADRALN, "instruction address misaligned");
> +	SIGBUS, BUS_ADRALN, "instruction address misaligned", EXC_INST_MISALIGNED);
>  DO_ERROR_INFO(do_trap_insn_fault,
> -	SIGSEGV, SEGV_ACCERR, "instruction access fault");
> +	SIGSEGV, SEGV_ACCERR, "instruction access fault", EXC_INST_ACCESS);
>  
>  asmlinkage __visible __trap_section void do_trap_insn_illegal(struct pt_regs *regs)
>  {
>  	bool handled;
>  
> +	if(!mark_trap_entry(EXC_INST_ILLEGAL, regs))
> +		return;
> +
>  	if (user_mode(regs)) {
>  		irqentry_enter_from_user_mode(regs);
>  
> @@ -237,10 +276,11 @@ asmlinkage __visible __trap_section void do_trap_insn_illegal(struct pt_regs *re
>  
>  		irqentry_nmi_exit(regs, state);
>  	}
> +	mark_trap_exit(EXC_INST_ILLEGAL, regs);
>  }
>  
>  DO_ERROR_INFO(do_trap_load_fault,
> -	SIGSEGV, SEGV_ACCERR, "load access fault");
> +	SIGSEGV, SEGV_ACCERR, "load access fault", EXC_LOAD_ACCESS);
>  
>  enum misaligned_access_type {
>  	MISALIGNED_STORE,
> @@ -264,6 +304,9 @@ static void do_trap_misaligned(struct pt_regs *regs, enum misaligned_access_type
>  {
>  	irqentry_state_t state;
>  
> +	if(!mark_trap_entry(EXC_LOAD_MISALIGNED, regs))
> +		return;
> +
>  	if (user_mode(regs)) {
>  		irqentry_enter_from_user_mode(regs);
>  		local_irq_enable();
> @@ -281,6 +324,8 @@ static void do_trap_misaligned(struct pt_regs *regs, enum misaligned_access_type
>  	} else {
>  		irqentry_nmi_exit(regs, state);
>  	}
> +
> +	mark_trap_exit(EXC_LOAD_MISALIGNED, regs);
>  }
>  
>  asmlinkage __visible __trap_section void do_trap_load_misaligned(struct pt_regs *regs)
> @@ -290,15 +335,36 @@ asmlinkage __visible __trap_section void do_trap_load_misaligned(struct pt_regs
>  
>  asmlinkage __visible __trap_section void do_trap_store_misaligned(struct pt_regs *regs)
>  {
> -	do_trap_misaligned(regs, MISALIGNED_STORE);
> +	if(!mark_trap_entry(EXC_STORE_MISALIGNED, regs))
> +		return;
> +
> +	if (user_mode(regs)) {
> +		irqentry_enter_from_user_mode(regs);
> +
> +		if (handle_misaligned_store(regs))
> +			do_trap_error(regs, SIGBUS, BUS_ADRALN, regs->epc,
> +				"Oops - store (or AMO) address misaligned");
> +
> +		irqentry_exit_to_user_mode(regs);
> +	} else {
> +		irqentry_state_t state = irqentry_nmi_enter(regs);
> +
> +		if (handle_misaligned_store(regs))
> +			do_trap_error(regs, SIGBUS, BUS_ADRALN, regs->epc,
> +				"Oops - store (or AMO) address misaligned");
> +
> +		irqentry_nmi_exit(regs, state);
> +	}
> +
> +	mark_trap_exit(EXC_STORE_MISALIGNED, regs);
>  }
>  
>  DO_ERROR_INFO(do_trap_store_fault,
> -	SIGSEGV, SEGV_ACCERR, "store (or AMO) access fault");
> +	SIGSEGV, SEGV_ACCERR, "store (or AMO) access fault", EXC_STORE_ACCESS);
>  DO_ERROR_INFO(do_trap_ecall_s,
> -	SIGILL, ILL_ILLTRP, "environment call from S-mode");
> +	SIGILL, ILL_ILLTRP, "environment call from S-mode", EXC_SYSCALL);
>  DO_ERROR_INFO(do_trap_ecall_m,
> -	SIGILL, ILL_ILLTRP, "environment call from M-mode");
> +	SIGILL, ILL_ILLTRP, "environment call from M-mode", EXC_SUPERVISOR_SYSCALL);
>  
>  static inline unsigned long get_break_insn_length(unsigned long pc)
>  {
> @@ -350,6 +416,9 @@ void handle_break(struct pt_regs *regs)
>  
>  asmlinkage __visible __trap_section void do_trap_break(struct pt_regs *regs)
>  {
> +	if(!mark_trap_entry_raw(EXC_BREAKPOINT, regs))
> +		return;
> +
>  	if (user_mode(regs)) {
>  		irqentry_enter_from_user_mode(regs);
>  		local_irq_enable();
> @@ -365,6 +434,8 @@ asmlinkage __visible __trap_section void do_trap_break(struct pt_regs *regs)
>  
>  		irqentry_nmi_exit(regs, state);
>  	}
> +
> +	mark_trap_exit_raw(EXC_BREAKPOINT, regs);
>  }
>  
>  asmlinkage __visible __trap_section  __no_stack_protector
> @@ -381,6 +452,15 @@ void do_trap_ecall_u(struct pt_regs *regs)
>  
>  		syscall = syscall_enter_from_user_mode(regs, syscall);
>  
> +		if(dovetailing()) {
> +			if (syscall == EXIT_SYSCALL_OOB) {
> +				hard_local_irq_disable();
> +				return;
> +			}
> +			if (syscall == EXIT_SYSCALL_TAIL)
> +				goto done_inband;
> +		}
> +
>  		add_random_kstack_offset();
>  
>  		if (syscall >= 0 && syscall < NR_syscalls)
> @@ -398,6 +478,7 @@ void do_trap_ecall_u(struct pt_regs *regs)
>  		 */
>  		choose_random_kstack_offset(get_random_u16());
>  
> +done_inband:
>  		syscall_exit_to_user_mode(regs);
>  	} else {
>  		irqentry_state_t state = irqentry_nmi_enter(regs);
> @@ -407,7 +488,6 @@ void do_trap_ecall_u(struct pt_regs *regs)
>  
>  		irqentry_nmi_exit(regs, state);
>  	}
> -
>  }
>  
>  #ifdef CONFIG_MMU
> diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
> index d64d01874a79..42450c47ee12 100644
> --- a/arch/riscv/mm/fault.c
> +++ b/arch/riscv/mm/fault.c
> @@ -385,6 +385,10 @@ void handle_page_fault(struct pt_regs *regs)
>  		die_kernel_fault("access to user memory without uaccess routines", addr, regs);
>  	}
>  
> +	oob_trap_notify(EXC_INST_PAGE_FAULT, regs);
> +	if (!running_inband())
> +		goto out;
> +
>  	perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr);
>  
>  	if (cause == EXC_STORE_PAGE_FAULT)
> @@ -403,7 +407,7 @@ void handle_page_fault(struct pt_regs *regs)
>  		count_vm_vma_lock_event(VMA_LOCK_SUCCESS);
>  		tsk->thread.bad_cause = cause;
>  		bad_area_nosemaphore(regs, SEGV_ACCERR, addr);
> -		return;
> +		goto out;
>  	}
>  
>  	fault = handle_mm_fault(vma, addr, flags | FAULT_FLAG_VMA_LOCK, regs);
> @@ -421,7 +425,7 @@ void handle_page_fault(struct pt_regs *regs)
>  	if (fault_signal_pending(fault, regs)) {
>  		if (!user_mode(regs))
>  			no_context(regs, addr);
> -		return;
> +		goto out;
>  	}
>  lock_mmap:
>  
> @@ -430,7 +434,7 @@ void handle_page_fault(struct pt_regs *regs)
>  	if (unlikely(!vma)) {
>  		tsk->thread.bad_cause = cause;
>  		bad_area_nosemaphore(regs, code, addr);
> -		return;
> +		goto out;
>  	}
>  
>  	/*
> @@ -442,7 +446,7 @@ void handle_page_fault(struct pt_regs *regs)
>  	if (unlikely(access_error(cause, vma))) {
>  		tsk->thread.bad_cause = cause;
>  		bad_area(regs, mm, code, addr);
> -		return;
> +		goto out;
>  	}
>  
>  	/*
> @@ -460,12 +464,12 @@ void handle_page_fault(struct pt_regs *regs)
>  	if (fault_signal_pending(fault, regs)) {
>  		if (!user_mode(regs))
>  			no_context(regs, addr);
> -		return;
> +		goto out;
>  	}
>  
>  	/* The fault is fully completed (including releasing mmap lock) */
>  	if (fault & VM_FAULT_COMPLETED)
> -		return;
> +		goto out;
>  
>  	if (unlikely(fault & VM_FAULT_RETRY)) {
>  		flags |= FAULT_FLAG_TRIED;
> @@ -484,7 +488,8 @@ void handle_page_fault(struct pt_regs *regs)
>  	if (unlikely(fault & VM_FAULT_ERROR)) {
>  		tsk->thread.bad_cause = cause;
>  		mm_fault_error(regs, addr, fault);
> -		return;
>  	}
> -	return;
> +
> +out:
> +	oob_trap_unwind(EXC_INST_PAGE_FAULT, regs);
>  }

-- 
Philippe.

[PATCH dovetail v4 12/12] riscv: add dovetail-aware memory management

[Tobias Schaffner]

From: shannmu <shanmu1901@gmail.com>

This patch provides the foundation for safely managing memory contexts
for both in-band and out-of-band tasks in the co-kernel.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 arch/riscv/include/asm/mmu_context.h |  6 ++----
 arch/riscv/mm/cacheflush.c           |  5 +++--
 arch/riscv/mm/context.c              | 18 +++++++++++++++++-
 arch/riscv/mm/fault.c                |  3 ++-
 arch/riscv/mm/tlbflush.c             |  5 +++--
 5 files changed, 27 insertions(+), 10 deletions(-)

diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/mmu_context.h
index 444136d7b253..da5d88d270c3 100644
--- a/arch/riscv/include/asm/mmu_context.h
+++ b/arch/riscv/include/asm/mmu_context.h
@@ -15,6 +15,8 @@
 
 void switch_mm(struct mm_struct *prev, struct mm_struct *next,
 	struct task_struct *task);
+void switch_oob_mm(struct mm_struct *prev, struct mm_struct *next,
+	struct task_struct *task);
 
 #define activate_mm activate_mm
 static inline void activate_mm(struct mm_struct *prev,
@@ -38,10 +40,6 @@ static inline int init_new_context(struct task_struct *tsk,
 	return 0;
 }
 
-static inline void
-switch_oob_mm(struct mm_struct *prev, struct mm_struct *next,
-	      struct task_struct *tsk) { }
-
 DECLARE_STATIC_KEY_FALSE(use_asid_allocator);
 
 #ifdef CONFIG_RISCV_ISA_SUPM
diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c
index d83a612464f6..762139b0ebb7 100644
--- a/arch/riscv/mm/cacheflush.c
+++ b/arch/riscv/mm/cacheflush.c
@@ -58,8 +58,9 @@ void flush_icache_mm(struct mm_struct *mm, bool local)
 {
 	unsigned int cpu;
 	cpumask_t others, *mask;
+	unsigned long flags;
 
-	preempt_disable();
+	flags = hard_preempt_disable();
 
 	/* Mark every hart's icache as needing a flush for this MM. */
 	mask = &mm->context.icache_stale_mask;
@@ -91,7 +92,7 @@ void flush_icache_mm(struct mm_struct *mm, bool local)
 		on_each_cpu_mask(&others, ipi_remote_fence_i, NULL, 1);
 	}
 
-	preempt_enable();
+	hard_preempt_enable(flags);
 }
 
 #endif /* CONFIG_SMP */
diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c
index 55c20ad1f744..f5f50b3e8407 100644
--- a/arch/riscv/mm/context.c
+++ b/arch/riscv/mm/context.c
@@ -315,7 +315,7 @@ static inline void flush_icache_deferred(struct mm_struct *mm, unsigned int cpu,
 #endif
 }
 
-void switch_mm(struct mm_struct *prev, struct mm_struct *next,
+static void do_switch_mm(struct mm_struct *prev, struct mm_struct *next,
 	struct task_struct *task)
 {
 	unsigned int cpu;
@@ -336,3 +336,19 @@ void switch_mm(struct mm_struct *prev, struct mm_struct *next,
 
 	flush_icache_deferred(next, cpu, task);
 }
+
+void switch_mm(struct mm_struct *prev, struct mm_struct *next,
+	struct task_struct *task)
+{
+	unsigned flags;
+
+	protect_inband_mm(flags);
+	do_switch_mm(prev, next, task);
+	unprotect_inband_mm(flags);
+}
+
+void switch_oob_mm(struct mm_struct *prev, struct mm_struct *next,
+	struct task_struct *task)
+{
+	do_switch_mm(prev, next, task);
+}
\ No newline at end of file
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 42450c47ee12..0e2269e33cb9 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -7,6 +7,7 @@
  */
 
 
+#include <linux/preempt.h>
 #include <linux/mm.h>
 #include <linux/kernel.h>
 #include <linux/interrupt.h>
@@ -369,7 +370,7 @@ void handle_page_fault(struct pt_regs *regs)
 	 * If we're in an interrupt, have no user context, or are running
 	 * in an atomic region, then we must not take the fault.
 	 */
-	if (unlikely(faulthandler_disabled() || !mm)) {
+	if (unlikely(running_inband() && (faulthandler_disabled() || !mm))) {
 		tsk->thread.bad_cause = cause;
 		no_context(regs, addr);
 		return;
diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c
index 8404530ec00f..294f0af63d6a 100644
--- a/arch/riscv/mm/tlbflush.c
+++ b/arch/riscv/mm/tlbflush.c
@@ -122,11 +122,12 @@ static void __flush_tlb_range(struct mm_struct *mm,
 {
 	unsigned long asid = get_mm_asid(mm);
 	unsigned int cpu;
+	unsigned long flags;
 
 	if (cpumask_empty(cmask))
 		return;
 
-	cpu = get_cpu();
+	cpu = hard_get_cpu(flags);
 
 	/* Check if the TLB flush needs to be sent to other CPUs. */
 	if (cpumask_any_but(cmask, cpu) >= nr_cpu_ids) {
@@ -143,7 +144,7 @@ static void __flush_tlb_range(struct mm_struct *mm,
 		on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1);
 	}
 
-	put_cpu();
+	hard_put_cpu(flags);
 
 	if (mm)
 		mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, start + size);
-- 
2.43.0

Re: [PATCH dovetail v4 00/12] riscv: Add dovetail support

[Florian Bezdeka]

On Thu, 2026-01-22 at 11:22 +0100, Tobias Schaffner wrote:
> Hi all,
> 
> this series introduces initial support for Dovetail on RISC-V.
> 
> Feedback, suggestions, and especially reviews are very welcome.
> 
> Changes since v1:
>  * Disable independent irq/softirq stack usage when pipelining
>  * Drop KVM changes as long as they can not be properly tested
> 
> Changes since v2:
>  * Rework trap handling
> 
> Changes since v3:
>  * Included the IRQ pipelining patches to have all riscv specific
>    patches included
>  * Rebased to dovetail/v6.18.y-dovetail-rebase
>  * Merged most of the fixes
>  * Fixed IRQ Stack handling and enabled IRQ_STACKS again
> 
> Best,
> Tobias
> 

Some questions: 

- What is the stability state of this series right now? Any known and
  open issues?

- Which platform(s) was/were used for testing?

- Any stress and over-night tests that were executed?
  I'm not necessarily talking about the CI tests that we do. I'm more
  interested in io/cpu/memory/swapping/suspend stress tests.

- State of CI integration: I see that some risc-v support was added to
  xenomai images. Is there real HW behind already? If so: What kind of
  board?


I guess it's too late for 6.18 as we still have some problems for the
existing archs, but 6.19 (once we were able to catch up again) looks
doable.

Florian

Re: [PATCH dovetail v4 00/12] riscv: Add dovetail support

[Tobias Schaffner]

Hi Florian,

On 1/22/26 12:05, Florian Bezdeka wrote:
> On Thu, 2026-01-22 at 11:22 +0100, Tobias Schaffner wrote:
>> Hi all,
>>
>> this series introduces initial support for Dovetail on RISC-V.
>>
>> Feedback, suggestions, and especially reviews are very welcome.
>>
>> Changes since v1:
>>   * Disable independent irq/softirq stack usage when pipelining
>>   * Drop KVM changes as long as they can not be properly tested
>>
>> Changes since v2:
>>   * Rework trap handling
>>
>> Changes since v3:
>>   * Included the IRQ pipelining patches to have all riscv specific
>>     patches included
>>   * Rebased to dovetail/v6.18.y-dovetail-rebase
>>   * Merged most of the fixes
>>   * Fixed IRQ Stack handling and enabled IRQ_STACKS again
>>
>> Best,
>> Tobias
>>
> 
> Some questions:
> 
> - What is the stability state of this series right now? Any known and
>    open issues?

At the moment I am mainly testing against the EVL-testsuite [1].
This includes running hectic.

There is still the "riscv: irq_pipeline: synchronize IRQs on exit to 
user mode" patch in place which is a ugly hack. I want to sort this out 
soon.

Except from that I am not aware of other issues.

> 
> - Which platform(s) was/were used for testing?

This is not running on a physical target yet, but this is the next step. 
I will run it on a StarFive VisionFive 2 board soon to get some first 
numbers.

> - Any stress and over-night tests that were executed?
>    I'm not necessarily talking about the CI tests that we do. I'm more
>    interested in io/cpu/memory/swapping/suspend stress tests.

I ran hectic for a longer period of time but far from over night.

I stress the system with stress-ng (stress-ng --vm 2 --vm-bytes 1G 
--mmap 2 --mmap-bytes 1G --page-in --matrix 0 --matrix-size 64) while 
running the evl test suite.

> - State of CI integration: I see that some risc-v support was added to
>    xenomai images. Is there real HW behind already? If so: What kind of
>    board?
RISC-V support is not yet merged into xenomai-images. I assume you are 
referring to my internal pipelines. These are running some custom 
patches that allow executing EVL tests with background stress, as well 
as building and testing the RISC-V custom branches.

I will send these patches to the mailing list step by step.

At the moment, we do not yet have a RISC-V board in the CI. I planned to 
integrate a BeagleV-Fire board, which will also be used by the CIP 
project, but I observed large latencies on PREEMPT_RT that I cannot yet 
fully explain. See the discussion on the CIP mailing list [2]. It is 
possible that I will stick with the VisionFive 2 board for now, as I 
know what to expect from it. In any case, I will integrate one of the 
boards into our CI as soon as I manage to do so.


Best,
Tobias


[1] https://gitlab.com/Xenomai/xenomai4/libevl/-/tree/master/tests
[2] 
https://lists.cip-project.org/g/cip-dev/message/21503?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3Acreated%2C%2CTobias%2C20%2C2%2C0%2C117149606

> 
> I guess it's too late for 6.18 as we still have some problems for the
> existing archs, but 6.19 (once we were able to catch up again) looks
> doable.
> 
> Florian