[PATCH RT 0/3] riscv: add PREEMPT_RT support

[PATCH RT 0/3] riscv: add PREEMPT_RT support

[Jisheng Zhang]

This series is to add PREEMPT_RT support to riscv. Compared with last
try[1], there are two major changes:

1. riscv has been converted to Generic Entry. And riscv uses
asm-generic/preeempt.h, so we need to patch asm-generic's preeempt to
enable lazy preempt support for riscv. This is what patch1 does.
However, it duplicates the preempt_lazy_count() defintion, I'm sure
there must be an elegant solution. Neverless, it doesn't impact the
riscv PREEMPT_RT support itself.

2. three preparation patches(patch1/2/3 in [1]) has been merged in
mainline.

I back-ported the lastest linux-6.3.y-rt patches to the lastest Linus tree,
then cook this series.

Link: https://lore.kernel.org/linux-riscv/20220831175920.2806-1-jszhang@kernel.org/

Jisheng Zhang (3):
  asm-generic/preempt: also check preempt_lazy_count for
    should_resched() etc.
  riscv: add lazy preempt support
  riscv: Allow to enable RT

 arch/riscv/Kconfig                   | 2 ++
 arch/riscv/include/asm/thread_info.h | 5 ++++-
 arch/riscv/kernel/asm-offsets.c      | 1 +
 include/asm-generic/preempt.h        | 8 +++++++-
 4 files changed, 14 insertions(+), 2 deletions(-)

-- 
2.40.1

[PATCH RT 1/3] asm-generic/preempt: also check preempt_lazy_count for should_resched() etc.

[Jisheng Zhang]

lazy preempt count is great mechanism to help ordinary SCHED_OTHER
tasks' throughput Under PREEMPT_RT. But current implementation relies
on each arch-specific code to check the preempt_lazy_count in
should_resched() and __preempt_count_dec_and_test(), if the arch, e.g
riscv use the asm-generic preempt implementation, it losts the great
lazy preempt mechanism.

Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
---
 include/asm-generic/preempt.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/include/asm-generic/preempt.h b/include/asm-generic/preempt.h
index b4d43a4af5f7..b583e7c38ccf 100644
--- a/include/asm-generic/preempt.h
+++ b/include/asm-generic/preempt.h
@@ -59,6 +59,11 @@ static __always_inline void __preempt_count_sub(int val)
 	*preempt_count_ptr() -= val;
 }
 
+#ifdef CONFIG_PREEMPT_LAZY
+#define preempt_lazy_count()		(current_thread_info()->preempt_lazy_count)
+#else
+#define preempt_lazy_count()		(0)
+#endif
 static __always_inline bool __preempt_count_dec_and_test(void)
 {
 	/*
@@ -66,7 +71,7 @@ static __always_inline bool __preempt_count_dec_and_test(void)
 	 * operations; we cannot use PREEMPT_NEED_RESCHED because it might get
 	 * lost.
 	 */
-	return !--*preempt_count_ptr() && tif_need_resched();
+	return !--*preempt_count_ptr() && !preempt_lazy_count() && tif_need_resched();
 }
 
 /*
@@ -75,6 +80,7 @@ static __always_inline bool __preempt_count_dec_and_test(void)
 static __always_inline bool should_resched(int preempt_offset)
 {
 	return unlikely(preempt_count() == preempt_offset &&
+			!preempt_lazy_count() &&
 			tif_need_resched());
 }
 
-- 
2.40.1

[PATCH RT 2/3] riscv: add lazy preempt support

[Jisheng Zhang]

Implement the lazy preempt for riscv.

Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
---
 arch/riscv/Kconfig                   | 1 +
 arch/riscv/include/asm/thread_info.h | 5 ++++-
 arch/riscv/kernel/asm-offsets.c      | 1 +
 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 348c0fa1fc8c..89e9d9fb35c4 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -118,6 +118,7 @@ config RISCV
 	select HAVE_PERF_REGS
 	select HAVE_PERF_USER_STACK_DUMP
 	select HAVE_POSIX_CPU_TIMERS_TASK_WORK
+	select HAVE_PREEMPT_LAZY
 	select HAVE_REGS_AND_STACK_ACCESS_API
 	select HAVE_RSEQ
 	select HAVE_STACKPROTECTOR
diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h
index e0d202134b44..c5e9223e9b91 100644
--- a/arch/riscv/include/asm/thread_info.h
+++ b/arch/riscv/include/asm/thread_info.h
@@ -59,6 +59,7 @@ extern unsigned long spin_shadow_stack;
 struct thread_info {
 	unsigned long		flags;		/* low level flags */
 	int                     preempt_count;  /* 0=>preemptible, <0=>BUG */
+	int			preempt_lazy_count;	/* 0=>preemptible, <0=>BUG */
 	/*
 	 * These stack pointers are overwritten on every system call or
 	 * exception.  SP is also saved to the stack it can be recovered when
@@ -90,6 +91,7 @@ struct thread_info {
  * - pending work-to-be-done flags are in lowest half-word
  * - other flags in upper half-word(s)
  */
+#define TIF_NEED_RESCHED_LAZY	0	/* lazy rescheduling */
 #define TIF_NOTIFY_RESUME	1	/* callback before returning to user */
 #define TIF_SIGPENDING		2	/* signal pending */
 #define TIF_NEED_RESCHED	3	/* rescheduling necessary */
@@ -104,9 +106,10 @@ struct thread_info {
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
 #define _TIF_NOTIFY_SIGNAL	(1 << TIF_NOTIFY_SIGNAL)
 #define _TIF_UPROBE		(1 << TIF_UPROBE)
+#define _TIF_NEED_RESCHED_LAZY	(1 << TIF_NEED_RESCHED_LAZY)
 
 #define _TIF_WORK_MASK \
 	(_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \
-	 _TIF_NOTIFY_SIGNAL | _TIF_UPROBE)
+	 _TIF_NEED_RESCHED_LAZY | _TIF_NOTIFY_SIGNAL | _TIF_UPROBE)
 
 #endif /* _ASM_RISCV_THREAD_INFO_H */
diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c
index d6a75aac1d27..1f8cccacb44e 100644
--- a/arch/riscv/kernel/asm-offsets.c
+++ b/arch/riscv/kernel/asm-offsets.c
@@ -36,6 +36,7 @@ void asm_offsets(void)
 	OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]);
 	OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags);
 	OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count);
+	OFFSET(TASK_TI_PREEMPT_LAZY_COUNT, task_struct, thread_info.preempt_lazy_count);
 	OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp);
 	OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp);
 
-- 
2.40.1

[PATCH RT 3/3] riscv: Allow to enable RT

[Jisheng Zhang]

Now, it's ready to enable RT on riscv.

Signed-off-by: Jisheng Zhang <jszhang@kernel.org>
---
 arch/riscv/Kconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 89e9d9fb35c4..622561d1e388 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -42,6 +42,7 @@ config RISCV
 	select ARCH_SUPPORTS_DEBUG_PAGEALLOC if MMU
 	select ARCH_SUPPORTS_HUGETLBFS if MMU
 	select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU
+	select ARCH_SUPPORTS_RT
 	select ARCH_USE_MEMTEST
 	select ARCH_USE_QUEUED_RWLOCKS
 	select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
-- 
2.40.1

Re: [PATCH RT 0/3] riscv: add PREEMPT_RT support

[Schaffner, Tobias]

[-- Attachment #1: Type: text/plain, Size: 1388 bytes --]

On 10.05.23 18:24, Jisheng Zhang wrote:
> This series is to add PREEMPT_RT support to riscv. Compared with last
> try[1], there are two major changes:
> 
> 1. riscv has been converted to Generic Entry. And riscv uses
> asm-generic/preeempt.h, so we need to patch asm-generic's preeempt to
> enable lazy preempt support for riscv. This is what patch1 does.
> However, it duplicates the preempt_lazy_count() defintion, I'm sure
> there must be an elegant solution. Neverless, it doesn't impact the
> riscv PREEMPT_RT support itself.
> 
> 2. three preparation patches(patch1/2/3 in [1]) has been merged in
> mainline.
> 
> I back-ported the lastest linux-6.3.y-rt patches to the lastest Linus tree,
> then cook this series.
> 
> Link: https://lore.kernel.org/linux-riscv/20220831175920.2806-1-jszhang@kernel.org/

Any news on this series? Are there any open tasks blocking this?
I am willing to help, but do not see what's missing to get this merged.

> Jisheng Zhang (3):
>    asm-generic/preempt: also check preempt_lazy_count for
>      should_resched() etc.
>    riscv: add lazy preempt support
>    riscv: Allow to enable RT
> 
>   arch/riscv/Kconfig                   | 2 ++
>   arch/riscv/include/asm/thread_info.h | 5 ++++-
>   arch/riscv/kernel/asm-offsets.c      | 1 +
>   include/asm-generic/preempt.h        | 8 +++++++-
>   4 files changed, 14 insertions(+), 2 deletions(-)
> 


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 10139 bytes --]

Re: [PATCH RT 0/3] riscv: add PREEMPT_RT support

[Jisheng Zhang]

On Mon, Oct 23, 2023 at 04:33:13PM +0000, Schaffner, Tobias wrote:
> On 10.05.23 18:24, Jisheng Zhang wrote:
> > This series is to add PREEMPT_RT support to riscv. Compared with last
> > try[1], there are two major changes:
> > 
> > 1. riscv has been converted to Generic Entry. And riscv uses
> > asm-generic/preeempt.h, so we need to patch asm-generic's preeempt to
> > enable lazy preempt support for riscv. This is what patch1 does.
> > However, it duplicates the preempt_lazy_count() defintion, I'm sure
> > there must be an elegant solution. Neverless, it doesn't impact the
> > riscv PREEMPT_RT support itself.
> > 
> > 2. three preparation patches(patch1/2/3 in [1]) has been merged in
> > mainline.
> > 
> > I back-ported the lastest linux-6.3.y-rt patches to the lastest Linus tree,
> > then cook this series.
> > 
> > Link: https://lore.kernel.org/linux-riscv/20220831175920.2806-1-jszhang@kernel.org/
> 
> Any news on this series? Are there any open tasks blocking this?
> I am willing to help, but do not see what's missing to get this merged.

Hi Thomas, Sebastian

could you please review? Any comments are appreciated. or do you want a
rebase on linux-6.5.y-rt?

Thanks

> 
> > Jisheng Zhang (3):
> >    asm-generic/preempt: also check preempt_lazy_count for
> >      should_resched() etc.
> >    riscv: add lazy preempt support
> >    riscv: Allow to enable RT
> > 
> >   arch/riscv/Kconfig                   | 2 ++
> >   arch/riscv/include/asm/thread_info.h | 5 ++++-
> >   arch/riscv/kernel/asm-offsets.c      | 1 +
> >   include/asm-generic/preempt.h        | 8 +++++++-
> >   4 files changed, 14 insertions(+), 2 deletions(-)
> > 
> 

Re: [PATCH RT 0/3] riscv: add PREEMPT_RT support

[Sebastian Andrzej Siewior]

On 2023-10-24 00:40:20 [+0800], Jisheng Zhang wrote:
> Hi Thomas, Sebastian
> 
> could you please review? Any comments are appreciated. or do you want a
> rebase on linux-6.5.y-rt?

Please resend on top of latest v6.6-RT. Lazy preempt is gone so only
PREEMPT_RT config switch remains from your three patch series. If you
have generic-entry then you could use the new PREEMPT_AUTO.

Are there any reports of this booting without warnings with LOCKDEP and
CONFIG_DEBUG_ATOMIC_SLEEP enabled? I remember there was something
earlier.

> Thanks

Sebastian

Re: [PATCH RT 0/3] riscv: add PREEMPT_RT support

[Jisheng Zhang]

On Tue, Oct 24, 2023 at 08:18:52AM +0200, Sebastian Andrzej Siewior wrote:
> On 2023-10-24 00:40:20 [+0800], Jisheng Zhang wrote:
> > Hi Thomas, Sebastian
> > 
> > could you please review? Any comments are appreciated. or do you want a
> > rebase on linux-6.5.y-rt?
> 
> Please resend on top of latest v6.6-RT. Lazy preempt is gone so only
> PREEMPT_RT config switch remains from your three patch series. If you
> have generic-entry then you could use the new PREEMPT_AUTO.

Hi Sebastian,

Thank you so much for pointing out PREEMPT_AUTO, I read the discussions
last weekend, glad to know PREEMPT_AUTO. And riscv has switched to
generic entry, so it's easy to support PREEMPT_AUTO for riscv. V2 was
sent out a few mintues ago. Could you please review?

> 
> Are there any reports of this booting without warnings with LOCKDEP and
> CONFIG_DEBUG_ATOMIC_SLEEP enabled? I remember there was something
> earlier.

IIRC, Conor reported the warning and stack trace is ext4 related. But
I didn't reproduce the warning. And Schaffner also tried the series
but it seems he didn't see the warning either. So I'm asking for Conor's
help to retry v2.

Thanks
> 
> > Thanks
> 
> Sebastian

Re: [PATCH RT 0/3] riscv: add PREEMPT_RT support

[Sebastian Andrzej Siewior]

On 2023-10-31 22:43:52 [+0800], Jisheng Zhang wrote:
> Hi Sebastian,
Hi Jisheng,

> Thank you so much for pointing out PREEMPT_AUTO, I read the discussions
> last weekend, glad to know PREEMPT_AUTO. And riscv has switched to
> generic entry, so it's easy to support PREEMPT_AUTO for riscv. V2 was
> sent out a few mintues ago. Could you please review?

Sure. I have no idea what the upstream status about PREEMPT_AUTO but I
think we want this.

> > Are there any reports of this booting without warnings with LOCKDEP and
> > CONFIG_DEBUG_ATOMIC_SLEEP enabled? I remember there was something
> > earlier.
> 
> IIRC, Conor reported the warning and stack trace is ext4 related. But
> I didn't reproduce the warning. And Schaffner also tried the series
> but it seems he didn't see the warning either. So I'm asking for Conor's
> help to retry v2.

oki.

> Thanks

Sebastian