[patch 00/15] timers: Provide timer_shutdown[_sync]()

[patch 00/15] timers: Provide timer_shutdown[_sync]()

[Thomas Gleixner]

Tearing down timers can be tedious when there are circular dependencies to
other things which need to be torn down. A prime example is timer and
workqueue where the timer schedules work and the work arms the timer.

Steven and the Google Chromebook team ran into such an issue in the
Bluetooth HCI code.

Steven suggested to create a new function del_timer_free() which marks the
timer as shutdown. Rearm attempts of shutdown timers are discarded and he
wanted to emit a warning for that case:

   https://lore.kernel.org/all/20220407161745.7d6754b3@gandalf.local.home

This resulted in a lengthy discussion and suggestions how this should be
implemented. The patch series went through several iterations and during
the review of the last version it turned out that this approach is
suboptimal:

   https://lore.kernel.org/all/20221110064101.429013735@goodmis.org

The warning is not really helpful because it's entirely unclear how it
should be acted upon. The only way to address such a case is to add 'if
(in_shutdown)' conditionals all over the place. This is error prone and in
most cases of teardown like the HCI one which started this discussion not
required all.

What needs to prevented is that pending work which is drained via
destroy_workqueue() does not rearm the previously shutdown timer. Nothing
in that shutdown sequence relies on the timer being functional.

The conclusion was that the semantics of timer_shutdown_sync() should be:

    - timer is not enqueued
    - timer callback is not running
    - timer cannot be rearmed

Preventing the rearming of shutdown timers is done by discarding rearm
attempts silently.

As Steven is short of cycles, I made some spare cycles available and
reworked the patch series to follow the new semantics and plugged the races
which were discovered during review.

The patches have been split up into small pieces to make review easier and
I took the liberty to throw a bunch of overdue cleanups into the picture
instead of proliferating the existing state further.

The last patch in the series addresses the HCI teardown issue for real.

The series is also available from git:

   git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git timers

Thanks,

	tglx
---
 Documentation/RCU/Design/Requirements/Requirements.rst |    2 
 Documentation/core-api/local_ops.rst                   |    2 
 Documentation/kernel-hacking/locking.rst               |   13 
 arch/arm/mach-spear/time.c                             |    8 
 drivers/bluetooth/hci_qca.c                            |   10 
 drivers/char/tpm/tpm-dev-common.c                      |    4 
 drivers/clocksource/arm_arch_timer.c                   |   12 
 drivers/clocksource/timer-sp804.c                      |    6 
 drivers/staging/wlan-ng/hfa384x_usb.c                  |    4 
 drivers/staging/wlan-ng/prism2usb.c                    |    6 
 include/linux/timer.h                                  |   35 +
 kernel/time/timer.c                                    |  409 +++++++++++++----
 net/sunrpc/xprt.c                                      |    2 
 13 files changed, 383 insertions(+), 130 deletions(-)

[patch 01/15] ARM: spear: Do not use timer namespace for timer_shutdown() function

[Thomas Gleixner]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

A new "shutdown" timer state is being added to the generic timer code. One
of the functions to change the timer into the state is called
"timer_shutdown()". This means that there can not be other functions called
"timer_shutdown()" as the timer code owns the "timer_*" name space.

Rename timer_shutdown() to spear_timer_shutdown() to avoid this conflict.

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Reviewed-by: Guenter Roeck <linux@roeck-us.net>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
Link: https://lkml.kernel.org/r/20221106212701.822440504@goodmis.org
Link: https://lore.kernel.org/all/20221105060155.228348078@goodmis.org/
Link: https://lore.kernel.org/r/20221110064146.810953418@goodmis.org
---
 arch/arm/mach-spear/time.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/arm/mach-spear/time.c b/arch/arm/mach-spear/time.c
index e979e2197f8e..5371c824786d 100644
--- a/arch/arm/mach-spear/time.c
+++ b/arch/arm/mach-spear/time.c
@@ -90,7 +90,7 @@ static void __init spear_clocksource_init(void)
 		200, 16, clocksource_mmio_readw_up);
 }
 
-static inline void timer_shutdown(struct clock_event_device *evt)
+static inline void spear_timer_shutdown(struct clock_event_device *evt)
 {
 	u16 val = readw(gpt_base + CR(CLKEVT));
 
@@ -101,7 +101,7 @@ static inline void timer_shutdown(struct clock_event_device *evt)
 
 static int spear_shutdown(struct clock_event_device *evt)
 {
-	timer_shutdown(evt);
+	spear_timer_shutdown(evt);
 
 	return 0;
 }
@@ -111,7 +111,7 @@ static int spear_set_oneshot(struct clock_event_device *evt)
 	u16 val;
 
 	/* stop the timer */
-	timer_shutdown(evt);
+	spear_timer_shutdown(evt);
 
 	val = readw(gpt_base + CR(CLKEVT));
 	val |= CTRL_ONE_SHOT;
@@ -126,7 +126,7 @@ static int spear_set_periodic(struct clock_event_device *evt)
 	u16 val;
 
 	/* stop the timer */
-	timer_shutdown(evt);
+	spear_timer_shutdown(evt);
 
 	period = clk_get_rate(gpt_clk) / HZ;
 	period >>= CTRL_PRESCALER16;
-- 
2.35.1

[patch 02/15] clocksource/drivers/arm_arch_timer: Do not use timer namespace for timer_shutdown() function

[Thomas Gleixner]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

A new "shutdown" timer state is being added to the generic timer code. One
of the functions to change the timer into the state is called
"timer_shutdown()". This means that there can not be other functions
called "timer_shutdown()" as the timer code owns the "timer_*" name space.

Rename timer_shutdown() to arch_timer_shutdown() to avoid this conflict.

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Reviewed-by: Guenter Roeck <linux@roeck-us.net>
Acked-by: Marc Zyngier <maz@kernel.org>
Link: https://lkml.kernel.org/r/20221106212702.002251651@goodmis.org
Link: https://lore.kernel.org/all/20221105060155.409832154@goodmis.org/
Link: https://lore.kernel.org/r/20221110064146.981725531@goodmis.org


---
 drivers/clocksource/arm_arch_timer.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/clocksource/arm_arch_timer.c b/drivers/clocksource/arm_arch_timer.c
index a7ff77550e17..9c3420a0d19d 100644
--- a/drivers/clocksource/arm_arch_timer.c
+++ b/drivers/clocksource/arm_arch_timer.c
@@ -687,8 +687,8 @@ static irqreturn_t arch_timer_handler_virt_mem(int irq, void *dev_id)
 	return timer_handler(ARCH_TIMER_MEM_VIRT_ACCESS, evt);
 }
 
-static __always_inline int timer_shutdown(const int access,
-					  struct clock_event_device *clk)
+static __always_inline int arch_timer_shutdown(const int access,
+					       struct clock_event_device *clk)
 {
 	unsigned long ctrl;
 
@@ -701,22 +701,22 @@ static __always_inline int timer_shutdown(const int access,
 
 static int arch_timer_shutdown_virt(struct clock_event_device *clk)
 {
-	return timer_shutdown(ARCH_TIMER_VIRT_ACCESS, clk);
+	return arch_timer_shutdown(ARCH_TIMER_VIRT_ACCESS, clk);
 }
 
 static int arch_timer_shutdown_phys(struct clock_event_device *clk)
 {
-	return timer_shutdown(ARCH_TIMER_PHYS_ACCESS, clk);
+	return arch_timer_shutdown(ARCH_TIMER_PHYS_ACCESS, clk);
 }
 
 static int arch_timer_shutdown_virt_mem(struct clock_event_device *clk)
 {
-	return timer_shutdown(ARCH_TIMER_MEM_VIRT_ACCESS, clk);
+	return arch_timer_shutdown(ARCH_TIMER_MEM_VIRT_ACCESS, clk);
 }
 
 static int arch_timer_shutdown_phys_mem(struct clock_event_device *clk)
 {
-	return timer_shutdown(ARCH_TIMER_MEM_PHYS_ACCESS, clk);
+	return arch_timer_shutdown(ARCH_TIMER_MEM_PHYS_ACCESS, clk);
 }
 
 static __always_inline void set_next_event(const int access, unsigned long evt,
-- 
2.35.1

[patch 03/15] clocksource/drivers/sp804: Do not use timer namespace for timer_shutdown() function

[Thomas Gleixner]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

A new "shutdown" timer state is being added to the generic timer code. One
of the functions to change the timer into the state is called
"timer_shutdown()". This means that there can not be other functions
called "timer_shutdown()" as the timer code owns the "timer_*" name space.

Rename timer_shutdown() to evt_timer_shutdown() to avoid this conflict.

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Reviewed-by: Guenter Roeck <linux@roeck-us.net>
Link: https://lkml.kernel.org/r/20221106212702.182883323@goodmis.org
Link: https://lore.kernel.org/all/20221105060155.592778858@goodmis.org/
Link: https://lore.kernel.org/r/20221110064147.158230501@goodmis.org


---
 drivers/clocksource/timer-sp804.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/clocksource/timer-sp804.c b/drivers/clocksource/timer-sp804.c
index e6a87f4af2b5..cd1916c05325 100644
--- a/drivers/clocksource/timer-sp804.c
+++ b/drivers/clocksource/timer-sp804.c
@@ -155,14 +155,14 @@ static irqreturn_t sp804_timer_interrupt(int irq, void *dev_id)
 	return IRQ_HANDLED;
 }
 
-static inline void timer_shutdown(struct clock_event_device *evt)
+static inline void evt_timer_shutdown(struct clock_event_device *evt)
 {
 	writel(0, common_clkevt->ctrl);
 }
 
 static int sp804_shutdown(struct clock_event_device *evt)
 {
-	timer_shutdown(evt);
+	evt_timer_shutdown(evt);
 	return 0;
 }
 
@@ -171,7 +171,7 @@ static int sp804_set_periodic(struct clock_event_device *evt)
 	unsigned long ctrl = TIMER_CTRL_32BIT | TIMER_CTRL_IE |
 			     TIMER_CTRL_PERIODIC | TIMER_CTRL_ENABLE;
 
-	timer_shutdown(evt);
+	evt_timer_shutdown(evt);
 	writel(common_clkevt->reload, common_clkevt->load);
 	writel(ctrl, common_clkevt->ctrl);
 	return 0;
-- 
2.35.1

[patch 04/15] timers: Get rid of del_singleshot_timer_sync()

[Thomas Gleixner]

del_singleshot_timer_sync() used to be an optimization for deleting timers
which are not rearmed from the timer callback function.

This optimization turned out to be broken and got mapped to
del_timer_sync() about 17 years ago.

Get rid of the undocumented indirection and use del_timer_sync() directly.

No functional change.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 drivers/char/tpm/tpm-dev-common.c     |    4 ++--
 drivers/staging/wlan-ng/hfa384x_usb.c |    4 ++--
 drivers/staging/wlan-ng/prism2usb.c   |    6 +++---
 include/linux/timer.h                 |    2 --
 kernel/time/timer.c                   |    2 +-
 net/sunrpc/xprt.c                     |    2 +-
 6 files changed, 9 insertions(+), 11 deletions(-)

--- a/drivers/char/tpm/tpm-dev-common.c
+++ b/drivers/char/tpm/tpm-dev-common.c
@@ -155,7 +155,7 @@ ssize_t tpm_common_read(struct file *fil
 out:
 	if (!priv->response_length) {
 		*off = 0;
-		del_singleshot_timer_sync(&priv->user_read_timer);
+		del_timer_sync(&priv->user_read_timer);
 		flush_work(&priv->timeout_work);
 	}
 	mutex_unlock(&priv->buffer_mutex);
@@ -262,7 +262,7 @@ ssize_t tpm_common_write(struct file *fi
 void tpm_common_release(struct file *file, struct file_priv *priv)
 {
 	flush_work(&priv->async_work);
-	del_singleshot_timer_sync(&priv->user_read_timer);
+	del_timer_sync(&priv->user_read_timer);
 	flush_work(&priv->timeout_work);
 	file->private_data = NULL;
 	priv->response_length = 0;
--- a/drivers/staging/wlan-ng/hfa384x_usb.c
+++ b/drivers/staging/wlan-ng/hfa384x_usb.c
@@ -1116,8 +1116,8 @@ static int hfa384x_usbctlx_complete_sync
 		if (ctlx == get_active_ctlx(hw)) {
 			spin_unlock_irqrestore(&hw->ctlxq.lock, flags);
 
-			del_singleshot_timer_sync(&hw->reqtimer);
-			del_singleshot_timer_sync(&hw->resptimer);
+			del_timer_sync(&hw->reqtimer);
+			del_timer_sync(&hw->resptimer);
 			hw->req_timer_done = 1;
 			hw->resp_timer_done = 1;
 			usb_kill_urb(&hw->ctlx_urb);
--- a/drivers/staging/wlan-ng/prism2usb.c
+++ b/drivers/staging/wlan-ng/prism2usb.c
@@ -170,9 +170,9 @@ static void prism2sta_disconnect_usb(str
 		 */
 		prism2sta_ifstate(wlandev, P80211ENUM_ifstate_disable);
 
-		del_singleshot_timer_sync(&hw->throttle);
-		del_singleshot_timer_sync(&hw->reqtimer);
-		del_singleshot_timer_sync(&hw->resptimer);
+		del_timer_sync(&hw->throttle);
+		del_timer_sync(&hw->reqtimer);
+		del_timer_sync(&hw->resptimer);
 
 		/* Unlink all the URBs. This "removes the wheels"
 		 * from the entire CTLX handling mechanism.
--- a/include/linux/timer.h
+++ b/include/linux/timer.h
@@ -190,8 +190,6 @@ extern int try_to_del_timer_sync(struct
 # define del_timer_sync(t)		del_timer(t)
 #endif
 
-#define del_singleshot_timer_sync(t) del_timer_sync(t)
-
 extern void init_timers(void);
 struct hrtimer;
 extern enum hrtimer_restart it_real_fn(struct hrtimer *);
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1933,7 +1933,7 @@ signed long __sched schedule_timeout(sig
 	timer_setup_on_stack(&timer.timer, process_timeout, 0);
 	__mod_timer(&timer.timer, expire, MOD_TIMER_NOTPENDING);
 	schedule();
-	del_singleshot_timer_sync(&timer.timer);
+	del_timer_sync(&timer.timer);
 
 	/* Remove the timer from the object tracker */
 	destroy_timer_on_stack(&timer.timer);
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -1164,7 +1164,7 @@ xprt_request_enqueue_receive(struct rpc_
 	spin_unlock(&xprt->queue_lock);
 
 	/* Turn off autodisconnect */
-	del_singleshot_timer_sync(&xprt->timer);
+	del_timer_sync(&xprt->timer);
 	return 0;
 }
 

Re: [patch 04/15] timers: Get rid of del_singleshot_timer_sync()

[Steven Rostedt]

On Tue, 15 Nov 2022 21:28:39 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> del_singleshot_timer_sync() used to be an optimization for deleting timers
> which are not rearmed from the timer callback function.
> 
> This optimization turned out to be broken and got mapped to
> del_timer_sync() about 17 years ago.
> 
> Get rid of the undocumented indirection and use del_timer_sync() directly.
> 
> No functional change.
> 
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
>


> --- a/net/sunrpc/xprt.c
> +++ b/net/sunrpc/xprt.c
> @@ -1164,7 +1164,7 @@ xprt_request_enqueue_receive(struct rpc_
>  	spin_unlock(&xprt->queue_lock);
>  
>  	/* Turn off autodisconnect */
> -	del_singleshot_timer_sync(&xprt->timer);
> +	del_timer_sync(&xprt->timer);

And this was not even a single shot timer. It used the
del_singleshot_timer_sync() function because of incorrect assumptions.

Link: https://lore.kernel.org/all/20221105060155.047357452@goodmis.org/

  0f9dc2b16884b ("RPC: Clean up socket autodisconnect")
  55c888d6d09a0 ("timers fixes/improvements")

Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>

-- Steve


>  	return 0;
>  }
>  

[patch 05/15] timers: Replace BUG_ON()s

[Thomas Gleixner]

The timer code still has a few BUG_ON()s left which are crashing the kernel
in situations where it still can recover or simply refuse to take an
action.

Remove the one in the hotplug callback which checks for the CPU being
offline. If that happens then the whole hotplug machinery will explode in
colourful ways.

Replace the rest with WARN_ON_ONCE() and conditional returns where
appropriate.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 kernel/time/timer.c |   11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1193,7 +1193,8 @@ EXPORT_SYMBOL(timer_reduce);
  */
 void add_timer(struct timer_list *timer)
 {
-	BUG_ON(timer_pending(timer));
+	if (WARN_ON_ONCE(timer_pending(timer)))
+		return;
 	__mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING);
 }
 EXPORT_SYMBOL(add_timer);
@@ -1210,7 +1211,8 @@ void add_timer_on(struct timer_list *tim
 	struct timer_base *new_base, *base;
 	unsigned long flags;
 
-	BUG_ON(timer_pending(timer) || !timer->function);
+	if (WARN_ON_ONCE(timer_pending(timer) || !timer->function))
+		return;
 
 	new_base = get_timer_cpu_base(timer->flags, cpu);
 
@@ -2017,8 +2019,6 @@ int timers_dead_cpu(unsigned int cpu)
 	struct timer_base *new_base;
 	int b, i;
 
-	BUG_ON(cpu_online(cpu));
-
 	for (b = 0; b < NR_BASES; b++) {
 		old_base = per_cpu_ptr(&timer_bases[b], cpu);
 		new_base = get_cpu_ptr(&timer_bases[b]);
@@ -2035,7 +2035,8 @@ int timers_dead_cpu(unsigned int cpu)
 		 */
 		forward_timer_base(new_base);
 
-		BUG_ON(old_base->running_timer);
+		WARN_ON_ONCE(old_base->running_timer);
+		old_base->running_timer = NULL;
 
 		for (i = 0; i < WHEEL_SIZE; i++)
 			migrate_timer_list(new_base, old_base->vectors + i);

Re: [patch 05/15] timers: Replace BUG_ON()s

[Steven Rostedt]

On Tue, 15 Nov 2022 21:28:41 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> The timer code still has a few BUG_ON()s left which are crashing the kernel
> in situations where it still can recover or simply refuse to take an
> action.
> 
> Remove the one in the hotplug callback which checks for the CPU being
> offline. If that happens then the whole hotplug machinery will explode in
> colourful ways.
> 
> Replace the rest with WARN_ON_ONCE() and conditional returns where
> appropriate.
> 
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> ---
>  kernel/time/timer.c |   11 ++++++-----
>  1 file changed, 6 insertions(+), 5 deletions(-)
> 
> --- a/kernel/time/timer.c
> +++ b/kernel/time/timer.c
> @@ -1193,7 +1193,8 @@ EXPORT_SYMBOL(timer_reduce);
>   */
>  void add_timer(struct timer_list *timer)
>  {
> -	BUG_ON(timer_pending(timer));
> +	if (WARN_ON_ONCE(timer_pending(timer)))
> +		return;
>  	__mod_timer(timer, timer->expires, MOD_TIMER_NOTPENDING);
>  }
>  EXPORT_SYMBOL(add_timer);
> @@ -1210,7 +1211,8 @@ void add_timer_on(struct timer_list *tim
>  	struct timer_base *new_base, *base;
>  	unsigned long flags;
>  
> -	BUG_ON(timer_pending(timer) || !timer->function);
> +	if (WARN_ON_ONCE(timer_pending(timer) || !timer->function))
> +		return;
>  
>  	new_base = get_timer_cpu_base(timer->flags, cpu);
>  

I wonder if this patch should be split up into two patches, as the above is
trivial and the below is a bit more "complex". Although it's probably moot
as none of these should ever trigger.

> @@ -2017,8 +2019,6 @@ int timers_dead_cpu(unsigned int cpu)
>  	struct timer_base *new_base;
>  	int b, i;
>  
> -	BUG_ON(cpu_online(cpu));
> -
>  	for (b = 0; b < NR_BASES; b++) {
>  		old_base = per_cpu_ptr(&timer_bases[b], cpu);
>  		new_base = get_cpu_ptr(&timer_bases[b]);
> @@ -2035,7 +2035,8 @@ int timers_dead_cpu(unsigned int cpu)
>  		 */
>  		forward_timer_base(new_base);
>  
> -		BUG_ON(old_base->running_timer);
> +		WARN_ON_ONCE(old_base->running_timer);
> +		old_base->running_timer = NULL;

I guess interesting things could happen if running_timer was not NULL, but
again, WARN_ON_ONCE() should never be triggered in a correctly running
kernel.

Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>

-- Steve

>  
>  		for (i = 0; i < WHEEL_SIZE; i++)
>  			migrate_timer_list(new_base, old_base->vectors + i);

[patch 06/15] timers: Update kernel-doc for various functions

[Thomas Gleixner]

The kernel-doc of timer related functions is partially uncomprehensible
word salad. Rewrite it to make it useful.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 kernel/time/timer.c |  131 ++++++++++++++++++++++++++++++----------------------
 1 file changed, 77 insertions(+), 54 deletions(-)

--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1121,14 +1121,16 @@ static inline int
 }
 
 /**
- * mod_timer_pending - modify a pending timer's timeout
- * @timer: the pending timer to be modified
- * @expires: new timeout in jiffies
+ * mod_timer_pending - Modify a pending timer's timeout
+ * @timer:	The pending timer to be modified
+ * @expires:	New timeout in jiffies
  *
- * mod_timer_pending() is the same for pending timers as mod_timer(),
- * but will not re-activate and modify already deleted timers.
+ * mod_timer_pending() is the same for pending timers as mod_timer(), but
+ * will not activate inactive timers.
  *
- * It is useful for unserialized use of timers.
+ * Return:
+ * * %0 - The timer was inactive and not modified
+ * * %1 - The timer was active and requeued to expire at @expires
  */
 int mod_timer_pending(struct timer_list *timer, unsigned long expires)
 {
@@ -1137,9 +1139,9 @@ int mod_timer_pending(struct timer_list
 EXPORT_SYMBOL(mod_timer_pending);
 
 /**
- * mod_timer - modify a timer's timeout
- * @timer: the timer to be modified
- * @expires: new timeout in jiffies
+ * mod_timer - Modify a timer's timeout
+ * @timer:	The timer to be modified
+ * @expires:	New timeout in jiffies
  *
  * mod_timer() is a more efficient way to update the expire field of an
  * active timer (if the timer is inactive it will be activated)
@@ -1152,9 +1154,11 @@ EXPORT_SYMBOL(mod_timer_pending);
  * same timer, then mod_timer() is the only safe way to modify the timeout,
  * since add_timer() cannot modify an already running timer.
  *
- * The function returns whether it has modified a pending timer or not.
- * (ie. mod_timer() of an inactive timer returns 0, mod_timer() of an
- * active timer returns 1.)
+ * Return:
+ * * %0 - The timer was inactive and started
+ * * %1 - The timer was active and requeued to expire at @expires or
+ *	  the timer was active and not modified because @expires did
+ *	  not change the effective expiry time
  */
 int mod_timer(struct timer_list *timer, unsigned long expires)
 {
@@ -1168,8 +1172,15 @@ EXPORT_SYMBOL(mod_timer);
  * @expires:	New timeout in jiffies
  *
  * timer_reduce() is very similar to mod_timer(), except that it will only
- * modify a running timer if that would reduce the expiration time (it will
- * start a timer that isn't running).
+ * modify an enqueued timer if that would reduce the expiration time. If
+ * @timer is not enqueued it starts the timer.
+ *
+ * Return:
+ * * %0 - The timer was inactive and started
+ * * %1 - The timer was active and requeued to expire at @expires or
+ *	  the timer was active and not modified because @expires
+ *	  did not change the effective expiry time such that the
+ *	  timer would expire earlier than already scheduled
  */
 int timer_reduce(struct timer_list *timer, unsigned long expires)
 {
@@ -1178,18 +1189,18 @@ int timer_reduce(struct timer_list *time
 EXPORT_SYMBOL(timer_reduce);
 
 /**
- * add_timer - start a timer
- * @timer: the timer to be added
+ * add_timer - Start a timer
+ * @timer:	The timer to be started
  *
- * The kernel will do a ->function(@timer) callback from the
- * timer interrupt at the ->expires point in the future. The
- * current time is 'jiffies'.
+ * Start @timer to expire at @timer->expires in the future. @timer->expires
+ * is the absolute expiry time measured in 'jiffies'. When the timer expires
+ * timer->function(timer) will be invoked from soft interrupt context.
  *
- * The timer's ->expires, ->function fields must be set prior calling this
- * function.
+ * The @timer->expires and @timer->function fields must be set prior
+ * calling this function.
  *
- * Timers with an ->expires field in the past will be executed in the next
- * timer tick.
+ * If @timer->expires is already in the past @timer will be queued to
+ * expire at the next timer tick.
  */
 void add_timer(struct timer_list *timer)
 {
@@ -1200,11 +1211,12 @@ void add_timer(struct timer_list *timer)
 EXPORT_SYMBOL(add_timer);
 
 /**
- * add_timer_on - start a timer on a particular CPU
- * @timer: the timer to be added
- * @cpu: the CPU to start it on
+ * add_timer_on - Start a timer on a particular CPU
+ * @timer:	The timer to be started
+ * @cpu:	The CPU to start it on
  *
- * This is not very scalable on SMP. Double adds are not possible.
+ * This can only operate on an inactive timer. Attempts to invoke this on
+ * an active timer are rejected with a warning.
  */
 void add_timer_on(struct timer_list *timer, int cpu)
 {
@@ -1240,15 +1252,17 @@ void add_timer_on(struct timer_list *tim
 EXPORT_SYMBOL_GPL(add_timer_on);
 
 /**
- * del_timer - deactivate a timer.
- * @timer: the timer to be deactivated
- *
- * del_timer() deactivates a timer - this works on both active and inactive
- * timers.
+ * del_timer - Deactivate a timer.
+ * @timer:	The timer to be deactivated
  *
- * The function returns whether it has deactivated a pending timer or not.
- * (ie. del_timer() of an inactive timer returns 0, del_timer() of an
- * active timer returns 1.)
+ * Contrary to del_timer_sync() this function does not wait for an
+ * eventually running timer callback on a different CPU and it neither
+ * prevents rearming of the timer.  If @timer can be rearmed concurrently
+ * then the return value of this function is meaningless.
+ *
+ * Return:
+ * * %0 - The timer was not pending
+ * * %1 - The timer was pending and deactivated
  */
 int del_timer(struct timer_list *timer)
 {
@@ -1270,10 +1284,16 @@ EXPORT_SYMBOL(del_timer);
 
 /**
  * try_to_del_timer_sync - Try to deactivate a timer
- * @timer: timer to delete
+ * @timer:	Timer to deactivate
  *
- * This function tries to deactivate a timer. Upon successful (ret >= 0)
- * exit the timer is not queued and the handler is not running on any CPU.
+ * This function cannot guarantee that the timer cannot be rearmed right
+ * after dropping the base lock. That needs to be prevented by the calling
+ * code if necessary.
+ *
+ * Return:
+ * * %0  - The timer was not pending
+ * * %1  - The timer was pending and deactivated
+ * * %-1 - The timer callback function is running on a different CPU
  */
 int try_to_del_timer_sync(struct timer_list *timer)
 {
@@ -1369,23 +1389,19 @@ static inline void del_timer_wait_runnin
 
 #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT_RT)
 /**
- * del_timer_sync - deactivate a timer and wait for the handler to finish.
- * @timer: the timer to be deactivated
- *
- * This function only differs from del_timer() on SMP: besides deactivating
- * the timer it also makes sure the handler has finished executing on other
- * CPUs.
+ * del_timer_sync - Deactivate a timer and wait for the handler to finish.
+ * @timer:	The timer to be deactivated
  *
  * Synchronization rules: Callers must prevent restarting of the timer,
  * otherwise this function is meaningless. It must not be called from
  * interrupt contexts unless the timer is an irqsafe one. The caller must
- * not hold locks which would prevent completion of the timer's
- * handler. The timer's handler must not call add_timer_on(). Upon exit the
- * timer is not queued and the handler is not running on any CPU.
- *
- * Note: For !irqsafe timers, you must not hold locks that are held in
- *   interrupt context while calling this function. Even if the lock has
- *   nothing to do with the timer in question.  Here's why::
+ * not hold locks which would prevent completion of the timer's callback
+ * function. The timer's handler must not call add_timer_on(). Upon exit
+ * the timer is not queued and the handler is not running on any CPU.
+ *
+ * For !irqsafe timers, the caller must not hold locks that are held in
+ * interrupt context. Even if the lock has nothing to do with the timer in
+ * question.  Here's why::
  *
  *    CPU0                             CPU1
  *    ----                             ----
@@ -1399,10 +1415,17 @@ static inline void del_timer_wait_runnin
  *    while (base->running_timer == mytimer);
  *
  * Now del_timer_sync() will never return and never release somelock.
- * The interrupt on the other CPU is waiting to grab somelock but
- * it has interrupted the softirq that CPU0 is waiting to finish.
+ * The interrupt on the other CPU is waiting to grab somelock but it has
+ * interrupted the softirq that CPU0 is waiting to finish.
  *
- * The function returns whether it has deactivated a pending timer or not.
+ * This function cannot guarantee that the timer is not rearmed again by
+ * some concurrent or preempting code, right after it dropped the base
+ * lock. If there is the possibility of a concurrent rearm then the return
+ * value of the function is meaningless.
+ *
+ * Return:
+ * * %0	- The timer was not pending
+ * * %1	- The timer was pending and deactivated
  */
 int del_timer_sync(struct timer_list *timer)
 {

Re: [patch 06/15] timers: Update kernel-doc for various functions

[Steven Rostedt]

On Tue, 15 Nov 2022 21:28:43 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> The kernel-doc of timer related functions is partially uncomprehensible
> word salad. Rewrite it to make it useful.
> 
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> ---
>  kernel/time/timer.c |  131 ++++++++++++++++++++++++++++++----------------------
>  1 file changed, 77 insertions(+), 54 deletions(-)
> 
> --- a/kernel/time/timer.c
> +++ b/kernel/time/timer.c
> @@ -1121,14 +1121,16 @@ static inline int
>  }
>  
>  /**
> - * mod_timer_pending - modify a pending timer's timeout
> - * @timer: the pending timer to be modified
> - * @expires: new timeout in jiffies
> + * mod_timer_pending - Modify a pending timer's timeout
> + * @timer:	The pending timer to be modified
> + * @expires:	New timeout in jiffies
>   *
> - * mod_timer_pending() is the same for pending timers as mod_timer(),
> - * but will not re-activate and modify already deleted timers.
> + * mod_timer_pending() is the same for pending timers as mod_timer(), but
> + * will not activate inactive timers.
>   *
> - * It is useful for unserialized use of timers.
> + * Return:
> + * * %0 - The timer was inactive and not modified
> + * * %1 - The timer was active and requeued to expire at @expires

I didn't know of the '%' option in kernel-doc.

Looking it up, I see it's for constants. Although it's missing in the
examples for return values:

  Documentation/doc-guide/kernel-doc.rst:

```
Return values
~~~~~~~~~~~~~

The return value, if any, should be described in a dedicated section
named ``Return``.

.. note::

  #) The multi-line descriptive text you provide does *not* recognize
     line breaks, so if you try to format some text nicely, as in::

        * Return:
        * 0 - OK
        * -EINVAL - invalid argument
        * -ENOMEM - out of memory

     this will all run together and produce::

        Return: 0 - OK -EINVAL - invalid argument -ENOMEM - out of memory

     So, in order to produce the desired line breaks, you need to use a
     ReST list, e. g.::

      * Return:
      * * 0             - OK to runtime suspend the device
      * * -EBUSY        - Device should not be runtime suspended
```

Should this be updated?


>   */
>  int mod_timer_pending(struct timer_list *timer, unsigned long expires)
>  {
> @@ -1137,9 +1139,9 @@ int mod_timer_pending(struct timer_list
>  EXPORT_SYMBOL(mod_timer_pending);
>  
>  /**
> - * mod_timer - modify a timer's timeout
> - * @timer: the timer to be modified
> - * @expires: new timeout in jiffies
> + * mod_timer - Modify a timer's timeout
> + * @timer:	The timer to be modified
> + * @expires:	New timeout in jiffies
>   *
>   * mod_timer() is a more efficient way to update the expire field of an

BTW, one can ask, "more efficient" than what?

If you are updating this, perhaps swap it around a little.

 * mod_timer(timer, expires) is equivalent to:
 *
 *     del_timer(timer); timer->expires = expires; add_timer(timer);
 *
 * mod_timer() is a more efficient way to update the expire field of an
 * active timer (if the timer is inactive it will be activated)
 *

As seeing the equivalent first and then seeing "more efficient" makes a bit
more sense.

>   * active timer (if the timer is inactive it will be activated)
> @@ -1152,9 +1154,11 @@ EXPORT_SYMBOL(mod_timer_pending);
>   * same timer, then mod_timer() is the only safe way to modify the timeout,
>   * since add_timer() cannot modify an already running timer.
>   *
> - * The function returns whether it has modified a pending timer or not.
> - * (ie. mod_timer() of an inactive timer returns 0, mod_timer() of an
> - * active timer returns 1.)
> + * Return:
> + * * %0 - The timer was inactive and started
> + * * %1 - The timer was active and requeued to expire at @expires or
> + *	  the timer was active and not modified because @expires did
> + *	  not change the effective expiry time
>   */
>  int mod_timer(struct timer_list *timer, unsigned long expires)
>  {
> @@ -1168,8 +1172,15 @@ EXPORT_SYMBOL(mod_timer);
>   * @expires:	New timeout in jiffies
>   *
>   * timer_reduce() is very similar to mod_timer(), except that it will only
> - * modify a running timer if that would reduce the expiration time (it will
> - * start a timer that isn't running).
> + * modify an enqueued timer if that would reduce the expiration time. If
> + * @timer is not enqueued it starts the timer.
> + *
> + * Return:
> + * * %0 - The timer was inactive and started
> + * * %1 - The timer was active and requeued to expire at @expires or
> + *	  the timer was active and not modified because @expires
> + *	  did not change the effective expiry time such that the
> + *	  timer would expire earlier than already scheduled
>   */
>  int timer_reduce(struct timer_list *timer, unsigned long expires)
>  {
> @@ -1178,18 +1189,18 @@ int timer_reduce(struct timer_list *time
>  EXPORT_SYMBOL(timer_reduce);
>  
>  /**
> - * add_timer - start a timer
> - * @timer: the timer to be added
> + * add_timer - Start a timer
> + * @timer:	The timer to be started
>   *
> - * The kernel will do a ->function(@timer) callback from the
> - * timer interrupt at the ->expires point in the future. The
> - * current time is 'jiffies'.
> + * Start @timer to expire at @timer->expires in the future. @timer->expires
> + * is the absolute expiry time measured in 'jiffies'. When the timer expires
> + * timer->function(timer) will be invoked from soft interrupt context.
>   *
> - * The timer's ->expires, ->function fields must be set prior calling this
> - * function.
> + * The @timer->expires and @timer->function fields must be set prior
> + * calling this function.

 "set prior to calling this function"

>   *
> - * Timers with an ->expires field in the past will be executed in the next
> - * timer tick.
> + * If @timer->expires is already in the past @timer will be queued to
> + * expire at the next timer tick.
>   */
>  void add_timer(struct timer_list *timer)
>  {
> @@ -1200,11 +1211,12 @@ void add_timer(struct timer_list *timer)
>  EXPORT_SYMBOL(add_timer);
>  
>  /**
> - * add_timer_on - start a timer on a particular CPU
> - * @timer: the timer to be added
> - * @cpu: the CPU to start it on
> + * add_timer_on - Start a timer on a particular CPU
> + * @timer:	The timer to be started
> + * @cpu:	The CPU to start it on
>   *
> - * This is not very scalable on SMP. Double adds are not possible.
> + * This can only operate on an inactive timer. Attempts to invoke this on
> + * an active timer are rejected with a warning.
>   */
>  void add_timer_on(struct timer_list *timer, int cpu)
>  {
> @@ -1240,15 +1252,17 @@ void add_timer_on(struct timer_list *tim
>  EXPORT_SYMBOL_GPL(add_timer_on);
>  
>  /**
> - * del_timer - deactivate a timer.
> - * @timer: the timer to be deactivated
> - *
> - * del_timer() deactivates a timer - this works on both active and inactive
> - * timers.
> + * del_timer - Deactivate a timer.
> + * @timer:	The timer to be deactivated
>   *
> - * The function returns whether it has deactivated a pending timer or not.
> - * (ie. del_timer() of an inactive timer returns 0, del_timer() of an
> - * active timer returns 1.)
> + * Contrary to del_timer_sync() this function does not wait for an
> + * eventually running timer callback on a different CPU and it neither

I'm a little confused with the "eventually running timer". Does that simply
mean one that is about to run next (that is, it doesn't handle race
conditions and the timer is in the process of starting), but will still
deactivate one that has not been started and the timer code for that CPU
hasn't triggered yet?

> + * prevents rearming of the timer.  If @timer can be rearmed concurrently
> + * then the return value of this function is meaningless.
> + *
> + * Return:
> + * * %0 - The timer was not pending
> + * * %1 - The timer was pending and deactivated
>   */
>  int del_timer(struct timer_list *timer)
>  {
> @@ -1270,10 +1284,16 @@ EXPORT_SYMBOL(del_timer);
>  
>  /**
>   * try_to_del_timer_sync - Try to deactivate a timer
> - * @timer: timer to delete
> + * @timer:	Timer to deactivate
>   *
> - * This function tries to deactivate a timer. Upon successful (ret >= 0)
> - * exit the timer is not queued and the handler is not running on any CPU.
> + * This function cannot guarantee that the timer cannot be rearmed right
> + * after dropping the base lock. That needs to be prevented by the calling
> + * code if necessary.


Hmm, you seemed to have deleted the description of what the function does
and replaced it with only what it cannot do.

The rest looks good.

-- Steve

> + *
> + * Return:
> + * * %0  - The timer was not pending
> + * * %1  - The timer was pending and deactivated
> + * * %-1 - The timer callback function is running on a different CPU
>   */
>  int try_to_del_timer_sync(struct timer_list *timer)
>  {
> @@ -1369,23 +1389,19 @@ static inline void del_timer_wait_runnin
>  
>  #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT_RT)
>  /**
> - * del_timer_sync - deactivate a timer and wait for the handler to finish.
> - * @timer: the timer to be deactivated
> - *
> - * This function only differs from del_timer() on SMP: besides deactivating
> - * the timer it also makes sure the handler has finished executing on other
> - * CPUs.
> + * del_timer_sync - Deactivate a timer and wait for the handler to finish.
> + * @timer:	The timer to be deactivated
>   *
>   * Synchronization rules: Callers must prevent restarting of the timer,
>   * otherwise this function is meaningless. It must not be called from
>   * interrupt contexts unless the timer is an irqsafe one. The caller must
> - * not hold locks which would prevent completion of the timer's
> - * handler. The timer's handler must not call add_timer_on(). Upon exit the
> - * timer is not queued and the handler is not running on any CPU.
> - *
> - * Note: For !irqsafe timers, you must not hold locks that are held in
> - *   interrupt context while calling this function. Even if the lock has
> - *   nothing to do with the timer in question.  Here's why::
> + * not hold locks which would prevent completion of the timer's callback
> + * function. The timer's handler must not call add_timer_on(). Upon exit
> + * the timer is not queued and the handler is not running on any CPU.
> + *
> + * For !irqsafe timers, the caller must not hold locks that are held in
> + * interrupt context. Even if the lock has nothing to do with the timer in
> + * question.  Here's why::
>   *
>   *    CPU0                             CPU1
>   *    ----                             ----
> @@ -1399,10 +1415,17 @@ static inline void del_timer_wait_runnin
>   *    while (base->running_timer == mytimer);
>   *
>   * Now del_timer_sync() will never return and never release somelock.
> - * The interrupt on the other CPU is waiting to grab somelock but
> - * it has interrupted the softirq that CPU0 is waiting to finish.
> + * The interrupt on the other CPU is waiting to grab somelock but it has
> + * interrupted the softirq that CPU0 is waiting to finish.
>   *
> - * The function returns whether it has deactivated a pending timer or not.
> + * This function cannot guarantee that the timer is not rearmed again by
> + * some concurrent or preempting code, right after it dropped the base
> + * lock. If there is the possibility of a concurrent rearm then the return
> + * value of the function is meaningless.
> + *
> + * Return:
> + * * %0	- The timer was not pending
> + * * %1	- The timer was pending and deactivated
>   */
>  int del_timer_sync(struct timer_list *timer)
>  {

Re: [patch 06/15] timers: Update kernel-doc for various functions

[Randy Dunlap]

On 11/21/22 12:43, Steven Rostedt wrote:
> On Tue, 15 Nov 2022 21:28:43 +0100 (CET)
> Thomas Gleixner <tglx@linutronix.de> wrote:
> 
>> The kernel-doc of timer related functions is partially uncomprehensible
>> word salad. Rewrite it to make it useful.
>>
>> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
>> ---
>>  kernel/time/timer.c |  131 ++++++++++++++++++++++++++++++----------------------
>>  1 file changed, 77 insertions(+), 54 deletions(-)
>>
>> --- a/kernel/time/timer.c
>> +++ b/kernel/time/timer.c
>> @@ -1121,14 +1121,16 @@ static inline int
>>  }
>>  
>>  /**
>> - * mod_timer_pending - modify a pending timer's timeout
>> - * @timer: the pending timer to be modified
>> - * @expires: new timeout in jiffies
>> + * mod_timer_pending - Modify a pending timer's timeout
>> + * @timer:	The pending timer to be modified
>> + * @expires:	New timeout in jiffies
>>   *
>> - * mod_timer_pending() is the same for pending timers as mod_timer(),
>> - * but will not re-activate and modify already deleted timers.
>> + * mod_timer_pending() is the same for pending timers as mod_timer(), but
>> + * will not activate inactive timers.
>>   *
>> - * It is useful for unserialized use of timers.
>> + * Return:
>> + * * %0 - The timer was inactive and not modified
>> + * * %1 - The timer was active and requeued to expire at @expires
> 
> I didn't know of the '%' option in kernel-doc.
> 
> Looking it up, I see it's for constants. Although it's missing in the
> examples for return values:
> 
>   Documentation/doc-guide/kernel-doc.rst:
> 
> ```
> Return values
> ~~~~~~~~~~~~~
> 
> The return value, if any, should be described in a dedicated section
> named ``Return``.
> 
> .. note::
> 
>   #) The multi-line descriptive text you provide does *not* recognize
>      line breaks, so if you try to format some text nicely, as in::
> 
>         * Return:
>         * 0 - OK
>         * -EINVAL - invalid argument
>         * -ENOMEM - out of memory
> 
>      this will all run together and produce::
> 
>         Return: 0 - OK -EINVAL - invalid argument -ENOMEM - out of memory
> 
>      So, in order to produce the desired line breaks, you need to use a
>      ReST list, e. g.::
> 
>       * Return:
>       * * 0             - OK to runtime suspend the device
>       * * -EBUSY        - Device should not be runtime suspended
> ```
> 
> Should this be updated?

Sure. Do you want to do it?

-- 
~Randy

Re: [patch 06/15] timers: Update kernel-doc for various functions

[Steven Rostedt]

On Mon, 21 Nov 2022 16:09:43 -0800
Randy Dunlap <rdunlap@infradead.org> wrote:

> > Should this be updated?  
> 
> Sure. Do you want to do it?

Nah, you can. ;-)

-- Steve

Re: [patch 06/15] timers: Update kernel-doc for various functions

[Thomas Gleixner]

On Mon, Nov 21 2022 at 15:43, Steven Rostedt wrote:
> On Tue, 15 Nov 2022 21:28:43 +0100 (CET)
> Thomas Gleixner <tglx@linutronix.de> wrote:
>>  EXPORT_SYMBOL(mod_timer_pending);
>>  
>>  /**
>> - * mod_timer - modify a timer's timeout
>> - * @timer: the timer to be modified
>> - * @expires: new timeout in jiffies
>> + * mod_timer - Modify a timer's timeout
>> + * @timer:	The timer to be modified
>> + * @expires:	New timeout in jiffies
>>   *
>>   * mod_timer() is a more efficient way to update the expire field of an
>
> BTW, one can ask, "more efficient" than what?
>
> If you are updating this, perhaps swap it around a little.
>
>  * mod_timer(timer, expires) is equivalent to:
>  *
>  *     del_timer(timer); timer->expires = expires; add_timer(timer);
>  *
>  * mod_timer() is a more efficient way to update the expire field of an
>  * active timer (if the timer is inactive it will be activated)
>  *
>
> As seeing the equivalent first and then seeing "more efficient" makes a bit
> more sense.

Point taken.

>>   *
>> - * The timer's ->expires, ->function fields must be set prior calling this
>> - * function.
>> + * The @timer->expires and @timer->function fields must be set prior
>> + * calling this function.
>
>  "set prior to calling this function"

Fixed.

>>   *
>> - * The function returns whether it has deactivated a pending timer or not.
>> - * (ie. del_timer() of an inactive timer returns 0, del_timer() of an
>> - * active timer returns 1.)
>> + * Contrary to del_timer_sync() this function does not wait for an
>> + * eventually running timer callback on a different CPU and it neither
>
> I'm a little confused with the "eventually running timer". Does that simply
> mean one that is about to run next (that is, it doesn't handle race
> conditions and the timer is in the process of starting), but will still
> deactivate one that has not been started and the timer code for that CPU
> hasn't triggered yet?

Let me try again.

  The function only deactivates a pending timer, but contrary to
  del_timer_sync() it does not take into account whether the timers
  callback function is concurrently executed on a different CPU or not.

Does that make more sense?

>> + * prevents rearming of the timer.  If @timer can be rearmed concurrently
>> + * then the return value of this function is meaningless.
>> + *
>> + * Return:
>> + * * %0 - The timer was not pending
>> + * * %1 - The timer was pending and deactivated
>>   */
>>  int del_timer(struct timer_list *timer)
>>  {
>> @@ -1270,10 +1284,16 @@ EXPORT_SYMBOL(del_timer);
>>  
>>  /**
>>   * try_to_del_timer_sync - Try to deactivate a timer
>> - * @timer: timer to delete
>> + * @timer:	Timer to deactivate
>>   *
>> - * This function tries to deactivate a timer. Upon successful (ret >= 0)
>> - * exit the timer is not queued and the handler is not running on any CPU.
>> + * This function cannot guarantee that the timer cannot be rearmed right
>> + * after dropping the base lock. That needs to be prevented by the calling
>> + * code if necessary.
>
>
> Hmm, you seemed to have deleted the description of what the function does
> and replaced it with only what it cannot do.

Ooops.

Re: [patch 06/15] timers: Update kernel-doc for various functions

[Steven Rostedt]

On Tue, 22 Nov 2022 16:18:37 +0100
Thomas Gleixner <tglx@linutronix.de> wrote:

> >>   *
> >> - * The function returns whether it has deactivated a pending timer or not.
> >> - * (ie. del_timer() of an inactive timer returns 0, del_timer() of an
> >> - * active timer returns 1.)
> >> + * Contrary to del_timer_sync() this function does not wait for an
> >> + * eventually running timer callback on a different CPU and it neither  
> >
> > I'm a little confused with the "eventually running timer". Does that simply
> > mean one that is about to run next (that is, it doesn't handle race
> > conditions and the timer is in the process of starting), but will still
> > deactivate one that has not been started and the timer code for that CPU
> > hasn't triggered yet?  
> 
> Let me try again.
> 
>   The function only deactivates a pending timer, but contrary to
>   del_timer_sync() it does not take into account whether the timers
>   callback function is concurrently executed on a different CPU or not.
> 
> Does that make more sense?

Yes, much better. Thanks!

-- Steve

Re: [patch 06/15] timers: Update kernel-doc for various functions

[Steven Rostedt]

On Tue, 22 Nov 2022 16:18:37 +0100
Thomas Gleixner <tglx@linutronix.de> wrote:

> >> + * This function cannot guarantee that the timer cannot be rearmed right
> >> + * after dropping the base lock. That needs to be prevented by the calling
> >> + * code if necessary.  
> >

Also, re-reading it again, I wounder if we can avoid the double use of
"cannot", in "cannot guarantee that the timer cannot".

What about:

    This function does not prevent the timer from being rearmed right after
    dropping the base lock.

?

-- Steve

Re: [patch 06/15] timers: Update kernel-doc for various functions

[Thomas Gleixner]

On Tue, Nov 22 2022 at 10:41, Steven Rostedt wrote:

> On Tue, 22 Nov 2022 16:18:37 +0100
> Thomas Gleixner <tglx@linutronix.de> wrote:
>
>> >> + * This function cannot guarantee that the timer cannot be rearmed right
>> >> + * after dropping the base lock. That needs to be prevented by the calling
>> >> + * code if necessary.  
>> >
>
> Also, re-reading it again, I wounder if we can avoid the double use of
> "cannot", in "cannot guarantee that the timer cannot".
>
> What about:
>
>     This function does not prevent the timer from being rearmed right after
>     dropping the base lock.

Funny enough I noticed myself when I copied this sentence into the code
and did exactly the same change :)

[patch 07/15] timers: Use del_timer_sync() even on UP

[Thomas Gleixner]

del_timer_sync() is assumed to be pointless on uniprocessor systems and can
be mapped to del_timer() because in theory del_timer() can never be invoked
while the timer callback function is executed.

This is not entirely true because del_timer() can be invoked from interrupt
context and therefore hit in the middle of a running timer callback.

Contrary to that del_timer_sync() is not allowed to be invoked from
interrupt context unless the affected timer is marked with TIMER_IRQSAFE.
del_timer_sync() has proper checks in place to detect such a situation.

Give up on the UP optimization and make del_timer_sync() unconditionally
available.

Co-developed-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/all/20220407161745.7d6754b3@gandalf.local.home
Link: https://lore.kernel.org/all/20221110064101.429013735@goodmis.org
---
 include/linux/timer.h |    7 +------
 kernel/time/timer.c   |    2 --
 2 files changed, 1 insertion(+), 8 deletions(-)

--- a/include/linux/timer.h
+++ b/include/linux/timer.h
@@ -183,12 +183,7 @@ extern int timer_reduce(struct timer_lis
 extern void add_timer(struct timer_list *timer);
 
 extern int try_to_del_timer_sync(struct timer_list *timer);
-
-#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT_RT)
-  extern int del_timer_sync(struct timer_list *timer);
-#else
-# define del_timer_sync(t)		del_timer(t)
-#endif
+extern int del_timer_sync(struct timer_list *timer);
 
 extern void init_timers(void);
 struct hrtimer;
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1387,7 +1387,6 @@ static inline void timer_sync_wait_runni
 static inline void del_timer_wait_running(struct timer_list *timer) { }
 #endif
 
-#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT_RT)
 /**
  * del_timer_sync - Deactivate a timer and wait for the handler to finish.
  * @timer:	The timer to be deactivated
@@ -1468,7 +1467,6 @@ int del_timer_sync(struct timer_list *ti
 	return ret;
 }
 EXPORT_SYMBOL(del_timer_sync);
-#endif
 
 static void call_timer_fn(struct timer_list *timer,
 			  void (*fn)(struct timer_list *),

[patch 08/15] timers: Rename del_timer_sync() to timer_delete_sync()

[Thomas Gleixner]

The timer related functions do not have a strict timer_ prefixed namespace
which is really annoying.

Rename del_timer_sync() to timer_delete_sync() and provide del_timer_sync()
as a wrapper. Document that del_timer_sync() is not for new code.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 include/linux/timer.h |   15 ++++++++++++++-
 kernel/time/timer.c   |   18 +++++++++---------
 2 files changed, 23 insertions(+), 10 deletions(-)

--- a/include/linux/timer.h
+++ b/include/linux/timer.h
@@ -183,7 +183,20 @@ extern int timer_reduce(struct timer_lis
 extern void add_timer(struct timer_list *timer);
 
 extern int try_to_del_timer_sync(struct timer_list *timer);
-extern int del_timer_sync(struct timer_list *timer);
+extern int timer_delete_sync(struct timer_list *timer);
+
+/**
+ * del_timer_sync - Delete a pending timer and wait for a running callback
+ * @timer:	The timer to be deleted
+ *
+ * See timer_delete_sync() for detailed explanation.
+ *
+ * Do not use in new code. Use timer_delete_sync() instead.
+ */
+static inline int del_timer_sync(struct timer_list *timer)
+{
+	return timer_delete_sync(timer);
+}
 
 extern void init_timers(void);
 struct hrtimer;
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1083,7 +1083,7 @@ static inline int
 		/*
 		 * We are trying to schedule the timer on the new base.
 		 * However we can't change timer's base while it is running,
-		 * otherwise del_timer_sync() can't detect that the timer's
+		 * otherwise timer_delete_sync() can't detect that the timer's
 		 * handler yet has not finished. This also guarantees that the
 		 * timer is serialized wrt itself.
 		 */
@@ -1255,7 +1255,7 @@ EXPORT_SYMBOL_GPL(add_timer_on);
  * del_timer - Deactivate a timer.
  * @timer:	The timer to be deactivated
  *
- * Contrary to del_timer_sync() this function does not wait for an
+ * Contrary to timer_delete_sync() this function does not wait for an
  * eventually running timer callback on a different CPU and it neither
  * prevents rearming of the timer.  If @timer can be rearmed concurrently
  * then the return value of this function is meaningless.
@@ -1388,7 +1388,7 @@ static inline void del_timer_wait_runnin
 #endif
 
 /**
- * del_timer_sync - Deactivate a timer and wait for the handler to finish.
+ * timer_delete_sync - Deactivate a timer and wait for the handler to finish.
  * @timer:	The timer to be deactivated
  *
  * Synchronization rules: Callers must prevent restarting of the timer,
@@ -1410,10 +1410,10 @@ static inline void del_timer_wait_runnin
  *    spin_lock_irq(somelock);
  *                                     <IRQ>
  *                                        spin_lock(somelock);
- *    del_timer_sync(mytimer);
+ *    timer_delete_sync(mytimer);
  *    while (base->running_timer == mytimer);
  *
- * Now del_timer_sync() will never return and never release somelock.
+ * Now timer_delete_sync() will never return and never release somelock.
  * The interrupt on the other CPU is waiting to grab somelock but it has
  * interrupted the softirq that CPU0 is waiting to finish.
  *
@@ -1426,7 +1426,7 @@ static inline void del_timer_wait_runnin
  * * %0	- The timer was not pending
  * * %1	- The timer was pending and deactivated
  */
-int del_timer_sync(struct timer_list *timer)
+int timer_delete_sync(struct timer_list *timer)
 {
 	int ret;
 
@@ -1466,7 +1466,7 @@ int del_timer_sync(struct timer_list *ti
 
 	return ret;
 }
-EXPORT_SYMBOL(del_timer_sync);
+EXPORT_SYMBOL(timer_delete_sync);
 
 static void call_timer_fn(struct timer_list *timer,
 			  void (*fn)(struct timer_list *),
@@ -1488,8 +1488,8 @@ static void call_timer_fn(struct timer_l
 #endif
 	/*
 	 * Couple the lock chain with the lock chain at
-	 * del_timer_sync() by acquiring the lock_map around the fn()
-	 * call here and in del_timer_sync().
+	 * timer_delete_sync() by acquiring the lock_map around the fn()
+	 * call here and in timer_delete_sync().
 	 */
 	lock_map_acquire(&lockdep_map);
 

Re: [patch 08/15] timers: Rename del_timer_sync() to timer_delete_sync()

[Steven Rostedt]

On Tue, 15 Nov 2022 21:28:46 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> The timer related functions do not have a strict timer_ prefixed namespace
> which is really annoying.
> 
> Rename del_timer_sync() to timer_delete_sync() and provide del_timer_sync()
> as a wrapper. Document that del_timer_sync() is not for new code.
> 
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> ---
>  include/linux/timer.h |   15 ++++++++++++++-
>  kernel/time/timer.c   |   18 +++++++++---------
>  2 files changed, 23 insertions(+), 10 deletions(-)
> 

Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>

-- Steve

[patch 09/15] timers: Rename del_timer() to timer_delete()

[Thomas Gleixner]

The timer related functions do not have a strict timer_ prefixed namespace
which is really annoying.

Rename del_timer() to timer_delete() and provide del_timer()
as a wrapper. Document that del_timer() is not for new code.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 include/linux/timer.h |   15 ++++++++++++++-
 kernel/time/timer.c   |    6 +++---
 2 files changed, 17 insertions(+), 4 deletions(-)

--- a/include/linux/timer.h
+++ b/include/linux/timer.h
@@ -169,7 +169,6 @@ static inline int timer_pending(const st
 }
 
 extern void add_timer_on(struct timer_list *timer, int cpu);
-extern int del_timer(struct timer_list * timer);
 extern int mod_timer(struct timer_list *timer, unsigned long expires);
 extern int mod_timer_pending(struct timer_list *timer, unsigned long expires);
 extern int timer_reduce(struct timer_list *timer, unsigned long expires);
@@ -184,6 +183,7 @@ extern void add_timer(struct timer_list
 
 extern int try_to_del_timer_sync(struct timer_list *timer);
 extern int timer_delete_sync(struct timer_list *timer);
+extern int timer_delete(struct timer_list *timer);
 
 /**
  * del_timer_sync - Delete a pending timer and wait for a running callback
@@ -198,6 +198,19 @@ static inline int del_timer_sync(struct
 	return timer_delete_sync(timer);
 }
 
+/**
+ * del_timer - Delete a pending timer
+ * @timer:	The timer to be deleted
+ *
+ * See timer_delete() for detailed explanation.
+ *
+ * Do not use in new code. Use timer_delete() instead.
+ */
+static inline int del_timer(struct timer_list *timer)
+{
+	return timer_delete(timer);
+}
+
 extern void init_timers(void);
 struct hrtimer;
 extern enum hrtimer_restart it_real_fn(struct hrtimer *);
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1252,7 +1252,7 @@ void add_timer_on(struct timer_list *tim
 EXPORT_SYMBOL_GPL(add_timer_on);
 
 /**
- * del_timer - Deactivate a timer.
+ * timer_delete - Deactivate a timer.
  * @timer:	The timer to be deactivated
  *
  * Contrary to timer_delete_sync() this function does not wait for an
@@ -1264,7 +1264,7 @@ EXPORT_SYMBOL_GPL(add_timer_on);
  * * %0 - The timer was not pending
  * * %1 - The timer was pending and deactivated
  */
-int del_timer(struct timer_list *timer)
+int timer_delete(struct timer_list *timer)
 {
 	struct timer_base *base;
 	unsigned long flags;
@@ -1280,7 +1280,7 @@ int del_timer(struct timer_list *timer)
 
 	return ret;
 }
-EXPORT_SYMBOL(del_timer);
+EXPORT_SYMBOL(timer_delete);
 
 /**
  * try_to_del_timer_sync - Try to deactivate a timer

Re: [patch 09/15] timers: Rename del_timer() to timer_delete()

[Steven Rostedt]

On Tue, 15 Nov 2022 21:28:48 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> The timer related functions do not have a strict timer_ prefixed namespace
> which is really annoying.
> 
> Rename del_timer() to timer_delete() and provide del_timer()
> as a wrapper. Document that del_timer() is not for new code.
> 
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> ---

Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>

BTW, I did a: git "grep del_timer Documentation" and there's a lot of
references to the old name. Perhaps you want to update those too (for both
del_timer() and del_timer_sync()).

-- Steve

[patch 10/15] timers: Silently ignore timers with a NULL function

[Thomas Gleixner]

Tearing down timers which have circular dependencies to other
functionality, e.g. workqueues, where the timer can schedule work and work
can arm timers is not trivial.

In those cases it is desired to shutdown the timer in a way which prevents
rearming of the timer. The mechanism to do so it to set timer->function to
NULL and use this as an indicator for the timer arming functions to ignore
the (re)arm request.

In preparation for that replace the warnings in the relevant code pathes
with checks for timer->function == NULL and discard the rearm request
silently.

Add debug_assert_init() instead of the WARN_ON_ONCE(!timer->function)
checks so that debug objects can warn about non-initialized timers.

If developers fail to enable debug objects and then waste lots of time to
figure out why their non-initialized timer is not firing, they deserve it.

Co-developed-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/all/20220407161745.7d6754b3@gandalf.local.home
Link: https://lore.kernel.org/all/20221110064101.429013735@goodmis.org
---
 kernel/time/timer.c |   51 +++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 49 insertions(+), 2 deletions(-)

--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1017,7 +1017,7 @@ static inline int
 	unsigned int idx = UINT_MAX;
 	int ret = 0;
 
-	BUG_ON(!timer->function);
+	debug_assert_init(timer);
 
 	/*
 	 * This is a common optimization triggered by the networking code - if
@@ -1044,6 +1044,14 @@ static inline int
 		 * dequeue/enqueue dance.
 		 */
 		base = lock_timer_base(timer, &flags);
+		/*
+		 * Has @timer been shutdown? This needs to be evaluated
+		 * while holding base lock to prevent a race against the
+		 * shutdown code.
+		 */
+		if (!timer->function)
+			goto out_unlock;
+
 		forward_timer_base(base);
 
 		if (timer_pending(timer) && (options & MOD_TIMER_REDUCE) &&
@@ -1070,6 +1078,14 @@ static inline int
 		}
 	} else {
 		base = lock_timer_base(timer, &flags);
+		/*
+		 * Has @timer been shutdown? This needs to be evaluated
+		 * while holding base lock to prevent a race against the
+		 * shutdown code.
+		 */
+		if (!timer->function)
+			goto out_unlock;
+
 		forward_timer_base(base);
 	}
 
@@ -1128,6 +1144,9 @@ static inline int
  * mod_timer_pending() is the same for pending timers as mod_timer(), but
  * will not activate inactive timers.
  *
+ * If @timer->function == NULL then the start operation is silently
+ * discarded.
+ *
  * Return:
  * * %0 - The timer was inactive and not modified
  * * %1 - The timer was active and requeued to expire at @expires
@@ -1154,6 +1173,9 @@ EXPORT_SYMBOL(mod_timer_pending);
  * same timer, then mod_timer() is the only safe way to modify the timeout,
  * since add_timer() cannot modify an already running timer.
  *
+ * If @timer->function == NULL then the start operation is silently
+ * discarded, the return value is 0 and meaningless.
+ *
  * Return:
  * * %0 - The timer was inactive and started
  * * %1 - The timer was active and requeued to expire at @expires or
@@ -1175,6 +1197,9 @@ EXPORT_SYMBOL(mod_timer);
  * modify an enqueued timer if that would reduce the expiration time. If
  * @timer is not enqueued it starts the timer.
  *
+ * If @timer->function == NULL then the start operation is silently
+ * discarded.
+ *
  * Return:
  * * %0 - The timer was inactive and started
  * * %1 - The timer was active and requeued to expire at @expires or
@@ -1201,6 +1226,9 @@ EXPORT_SYMBOL(timer_reduce);
  *
  * If @timer->expires is already in the past @timer will be queued to
  * expire at the next timer tick.
+ *
+ * If @timer->function == NULL then the start operation is silently
+ * discarded.
  */
 void add_timer(struct timer_list *timer)
 {
@@ -1217,13 +1245,18 @@ EXPORT_SYMBOL(add_timer);
  *
  * This can only operate on an inactive timer. Attempts to invoke this on
  * an active timer are rejected with a warning.
+ *
+ * If @timer->function == NULL then the start operation is silently
+ * discarded.
  */
 void add_timer_on(struct timer_list *timer, int cpu)
 {
 	struct timer_base *new_base, *base;
 	unsigned long flags;
 
-	if (WARN_ON_ONCE(timer_pending(timer) || !timer->function))
+	debug_assert_init(timer);
+
+	if (WARN_ON_ONCE(timer_pending(timer)))
 		return;
 
 	new_base = get_timer_cpu_base(timer->flags, cpu);
@@ -1234,6 +1267,13 @@ void add_timer_on(struct timer_list *tim
 	 * wrong base locked.  See lock_timer_base().
 	 */
 	base = lock_timer_base(timer, &flags);
+	/*
+	 * Has @timer been shutdown? This needs to be evaluated while
+	 * holding base lock to prevent a race against the shutdown code.
+	 */
+	if (!timer->function)
+		goto out_unlock;
+
 	if (base != new_base) {
 		timer->flags |= TIMER_MIGRATING;
 
@@ -1247,6 +1287,7 @@ void add_timer_on(struct timer_list *tim
 
 	debug_timer_activate(timer);
 	internal_add_timer(base, timer);
+out_unlock:
 	raw_spin_unlock_irqrestore(&base->lock, flags);
 }
 EXPORT_SYMBOL_GPL(add_timer_on);
@@ -1532,6 +1573,12 @@ static void expire_timers(struct timer_b
 
 		fn = timer->function;
 
+		if (WARN_ON_ONCE(!fn)) {
+			/* Should never happen. Emphasis on should! */
+			base->running_timer = NULL;
+			return;
+		}
+
 		if (timer->flags & TIMER_IRQSAFE) {
 			raw_spin_unlock(&base->lock);
 			call_timer_fn(timer, fn, baseclk);

Re: [patch 10/15] timers: Silently ignore timers with a NULL function

[Steven Rostedt]

On Tue, 15 Nov 2022 21:28:49 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> @@ -1128,6 +1144,9 @@ static inline int
>   * mod_timer_pending() is the same for pending timers as mod_timer(), but
>   * will not activate inactive timers.
>   *
> + * If @timer->function == NULL then the start operation is silently
> + * discarded.
> + *
>   * Return:
>   * * %0 - The timer was inactive and not modified
>   * * %1 - The timer was active and requeued to expire at @expires
> @@ -1154,6 +1173,9 @@ EXPORT_SYMBOL(mod_timer_pending);
>   * same timer, then mod_timer() is the only safe way to modify the timeout,
>   * since add_timer() cannot modify an already running timer.
>   *
> + * If @timer->function == NULL then the start operation is silently
> + * discarded, the return value is 0 and meaningless.
> + *
>   * Return:
>   * * %0 - The timer was inactive and started

For those that only read the "Return" portion of kernel-doc, perhaps add
here:
             "or the timer is in the shutdown state and was not started".


>   * * %1 - The timer was active and requeued to expire at @expires or
> @@ -1175,6 +1197,9 @@ EXPORT_SYMBOL(mod_timer);
>   * modify an enqueued timer if that would reduce the expiration time. If
>   * @timer is not enqueued it starts the timer.
>   *
> + * If @timer->function == NULL then the start operation is silently
> + * discarded.
> + *
>   * Return:
>   * * %0 - The timer was inactive and started
>   * * %1 - The timer was active and requeued to expire at @expires or
> @@ -1201,6 +1226,9 @@ EXPORT_SYMBOL(timer_reduce);
>   *
>   * If @timer->expires is already in the past @timer will be queued to
>   * expire at the next timer tick.
> + *
> + * If @timer->function == NULL then the start operation is silently
> + * discarded.
>   */
>  void add_timer(struct timer_list *timer)
>  {
> @@ -1217,13 +1245,18 @@ EXPORT_SYMBOL(add_timer);
>   *
>   * This can only operate on an inactive timer. Attempts to invoke this on
>   * an active timer are rejected with a warning.
> + *
> + * If @timer->function == NULL then the start operation is silently
> + * discarded.
>   */
>  void add_timer_on(struct timer_list *timer, int cpu)
>  {
>  	struct timer_base *new_base, *base;
>  	unsigned long flags;
>  
> -	if (WARN_ON_ONCE(timer_pending(timer) || !timer->function))
> +	debug_assert_init(timer);
> +
> +	if (WARN_ON_ONCE(timer_pending(timer)))
>  		return;
>  
>  	new_base = get_timer_cpu_base(timer->flags, cpu);
> @@ -1234,6 +1267,13 @@ void add_timer_on(struct timer_list *tim
>  	 * wrong base locked.  See lock_timer_base().
>  	 */
>  	base = lock_timer_base(timer, &flags);
> +	/*
> +	 * Has @timer been shutdown? This needs to be evaluated while
> +	 * holding base lock to prevent a race against the shutdown code.
> +	 */
> +	if (!timer->function)
> +		goto out_unlock;
> +
>  	if (base != new_base) {
>  		timer->flags |= TIMER_MIGRATING;
>  
> @@ -1247,6 +1287,7 @@ void add_timer_on(struct timer_list *tim
>  
>  	debug_timer_activate(timer);
>  	internal_add_timer(base, timer);
> +out_unlock:
>  	raw_spin_unlock_irqrestore(&base->lock, flags);
>  }
>  EXPORT_SYMBOL_GPL(add_timer_on);
> @@ -1532,6 +1573,12 @@ static void expire_timers(struct timer_b
>  
>  		fn = timer->function;
>  
> +		if (WARN_ON_ONCE(!fn)) {
> +			/* Should never happen. Emphasis on should! */
> +			base->running_timer = NULL;
> +			return;

Why return and not continue?

Wont this drop the other timers in the queue?

-- Steve


> +		}
> +
>  		if (timer->flags & TIMER_IRQSAFE) {
>  			raw_spin_unlock(&base->lock);
>  			call_timer_fn(timer, fn, baseclk);

Re: [patch 10/15] timers: Silently ignore timers with a NULL function

[Thomas Gleixner]

On Mon, Nov 21 2022 at 16:35, Steven Rostedt wrote:
> On Tue, 15 Nov 2022 21:28:49 +0100 (CET)
> Thomas Gleixner <tglx@linutronix.de> wrote:
>> @@ -1532,6 +1573,12 @@ static void expire_timers(struct timer_b
>>  
>>  		fn = timer->function;
>>  
>> +		if (WARN_ON_ONCE(!fn)) {
>> +			/* Should never happen. Emphasis on should! */
>> +			base->running_timer = NULL;
>> +			return;
>
> Why return and not continue?
>
> Wont this drop the other timers in the queue?

Duh. Yes. Thanks for catching that!

[patch 11/15] timers: Split [try_to_]del_timer[_sync]() to prepare for shutdown mode

[Thomas Gleixner]

Tearing down timers which have circular dependencies to other
functionality, e.g. workqueues, where the timer can schedule work and work
can arm timers is not trivial.

In those cases it is desired to shutdown the timer in a way which prevents
rearming of the timer. The mechanism to do so it to set timer->function to
NULL and use this as an indicator for the timer arming functions to ignore
the (re)arm request.

Split the inner workings of try_do_del_timer_sync(), del_timer_sync() and
del_timer() into helper functions to prepare for implementing the shutdown
functionality.

No functional change.

Co-developed-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/all/20220407161745.7d6754b3@gandalf.local.home
Link: https://lore.kernel.org/all/20221110064101.429013735@goodmis.org
---
 kernel/time/timer.c |  136 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 89 insertions(+), 47 deletions(-)

--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1293,19 +1293,14 @@ void add_timer_on(struct timer_list *tim
 EXPORT_SYMBOL_GPL(add_timer_on);
 
 /**
- * timer_delete - Deactivate a timer.
+ * __timer_delete - Internal function: Deactivate a timer.
  * @timer:	The timer to be deactivated
  *
- * Contrary to timer_delete_sync() this function does not wait for an
- * eventually running timer callback on a different CPU and it neither
- * prevents rearming of the timer.  If @timer can be rearmed concurrently
- * then the return value of this function is meaningless.
- *
  * Return:
  * * %0 - The timer was not pending
  * * %1 - The timer was pending and deactivated
  */
-int timer_delete(struct timer_list *timer)
+static int __timer_delete(struct timer_list *timer)
 {
 	struct timer_base *base;
 	unsigned long flags;
@@ -1321,22 +1316,36 @@ int timer_delete(struct timer_list *time
 
 	return ret;
 }
+
+/**
+ * timer_delete - Deactivate a timer.
+ * @timer:	The timer to be deactivated
+ *
+ * Contrary to timer_delete_sync() this function does not wait for an
+ * eventually running timer callback on a different CPU and it neither
+ * prevents rearming of the timer.  If @timer can be rearmed concurrently
+ * then the return value of this function is meaningless.
+ *
+ * Return:
+ * * %0 - The timer was not pending
+ * * %1 - The timer was pending and deactivated
+ */
+int timer_delete(struct timer_list *timer)
+{
+	return __timer_delete(timer);
+}
 EXPORT_SYMBOL(timer_delete);
 
 /**
- * try_to_del_timer_sync - Try to deactivate a timer
+ * __try_to_del_timer_sync - Internal function: Try to deactivate a timer
  * @timer:	Timer to deactivate
  *
- * This function cannot guarantee that the timer cannot be rearmed right
- * after dropping the base lock. That needs to be prevented by the calling
- * code if necessary.
- *
  * Return:
  * * %0  - The timer was not pending
  * * %1  - The timer was pending and deactivated
  * * %-1 - The timer callback function is running on a different CPU
  */
-int try_to_del_timer_sync(struct timer_list *timer)
+static int __try_to_del_timer_sync(struct timer_list *timer)
 {
 	struct timer_base *base;
 	unsigned long flags;
@@ -1353,6 +1362,25 @@ int try_to_del_timer_sync(struct timer_l
 
 	return ret;
 }
+
+/**
+ * try_to_del_timer_sync - Try to deactivate a timer
+ * @timer:	Timer to deactivate
+ *
+ * This function cannot guarantee that the timer cannot be rearmed
+ * right after dropping the base lock. That needs to be prevented
+ * by the calling code if necessary. If the timer can be rearmed
+ * concurrently then the return value is meaningless.
+ *
+ * Return:
+ * * %0  - The timer was not pending
+ * * %1  - The timer was pending and deactivated
+ * * %-1 - The timer callback function is running on a different CPU
+ */
+int try_to_del_timer_sync(struct timer_list *timer)
+{
+	return __try_to_del_timer_sync(timer);
+}
 EXPORT_SYMBOL(try_to_del_timer_sync);
 
 #ifdef CONFIG_PREEMPT_RT
@@ -1429,45 +1457,15 @@ static inline void del_timer_wait_runnin
 #endif
 
 /**
- * timer_delete_sync - Deactivate a timer and wait for the handler to finish.
+ * __timer_delete_sync - Internal function: Deactivate a timer and wait
+ *			 for the handler to finish.
  * @timer:	The timer to be deactivated
  *
- * Synchronization rules: Callers must prevent restarting of the timer,
- * otherwise this function is meaningless. It must not be called from
- * interrupt contexts unless the timer is an irqsafe one. The caller must
- * not hold locks which would prevent completion of the timer's callback
- * function. The timer's handler must not call add_timer_on(). Upon exit
- * the timer is not queued and the handler is not running on any CPU.
- *
- * For !irqsafe timers, the caller must not hold locks that are held in
- * interrupt context. Even if the lock has nothing to do with the timer in
- * question.  Here's why::
- *
- *    CPU0                             CPU1
- *    ----                             ----
- *                                     <SOFTIRQ>
- *                                       call_timer_fn();
- *                                       base->running_timer = mytimer;
- *    spin_lock_irq(somelock);
- *                                     <IRQ>
- *                                        spin_lock(somelock);
- *    timer_delete_sync(mytimer);
- *    while (base->running_timer == mytimer);
- *
- * Now timer_delete_sync() will never return and never release somelock.
- * The interrupt on the other CPU is waiting to grab somelock but it has
- * interrupted the softirq that CPU0 is waiting to finish.
- *
- * This function cannot guarantee that the timer is not rearmed again by
- * some concurrent or preempting code, right after it dropped the base
- * lock. If there is the possibility of a concurrent rearm then the return
- * value of the function is meaningless.
- *
  * Return:
  * * %0	- The timer was not pending
  * * %1	- The timer was pending and deactivated
  */
-int timer_delete_sync(struct timer_list *timer)
+static int __timer_delete_sync(struct timer_list *timer)
 {
 	int ret;
 
@@ -1497,7 +1495,7 @@ int timer_delete_sync(struct timer_list
 		lockdep_assert_preemption_enabled();
 
 	do {
-		ret = try_to_del_timer_sync(timer);
+		ret = __try_to_del_timer_sync(timer);
 
 		if (unlikely(ret < 0)) {
 			del_timer_wait_running(timer);
@@ -1507,6 +1505,50 @@ int timer_delete_sync(struct timer_list
 
 	return ret;
 }
+
+/**
+ * timer_delete_sync - Deactivate a timer and wait for the handler to finish.
+ * @timer:	The timer to be deactivated
+ *
+ * Synchronization rules: Callers must prevent restarting of the timer,
+ * otherwise this function is meaningless. It must not be called from
+ * interrupt contexts unless the timer is an irqsafe one. The caller must
+ * not hold locks which would prevent completion of the timer's callback
+ * function. The timer's handler must not call add_timer_on(). Upon exit
+ * the timer is not queued and the handler is not running on any CPU.
+ *
+ * For !irqsafe timers, the caller must not hold locks that are held in
+ * interrupt context. Even if the lock has nothing to do with the timer in
+ * question.  Here's why::
+ *
+ *    CPU0                             CPU1
+ *    ----                             ----
+ *                                     <SOFTIRQ>
+ *                                       call_timer_fn();
+ *                                       base->running_timer = mytimer;
+ *    spin_lock_irq(somelock);
+ *                                     <IRQ>
+ *                                        spin_lock(somelock);
+ *    timer_delete_sync(mytimer);
+ *    while (base->running_timer == mytimer);
+ *
+ * Now timer_delete_sync() will never return and never release somelock.
+ * The interrupt on the other CPU is waiting to grab somelock but it has
+ * interrupted the softirq that CPU0 is waiting to finish.
+ *
+ * This function cannot guarantee that the timer is not rearmed again by
+ * some concurrent or preempting code, right after it dropped the base
+ * lock. If there is the possibility of a concurrent rearm then the return
+ * value of the function is meaningless.
+ *
+ * Return:
+ * * %0	- The timer was not pending
+ * * %1	- The timer was pending and deactivated
+ */
+int timer_delete_sync(struct timer_list *timer)
+{
+	return __timer_delete_sync(timer);
+}
 EXPORT_SYMBOL(timer_delete_sync);
 
 static void call_timer_fn(struct timer_list *timer,

[patch 12/15] timers: Add shutdown mechanism to the internal functions

[Thomas Gleixner]

Tearing down timers which have circular dependencies to other
functionality, e.g. workqueues, where the timer can schedule work and work
can arm timers is not trivial.

In those cases it is desired to shutdown the timer in a way which prevents
rearming of the timer. The mechanism to do so it to set timer->function to
NULL and use this as an indicator for the timer arming functions to ignore
the (re)arm request.

Add a shutdown argument to the relevant internal functions which makes the
actual deactivation code set timer->function to NULL which in turn prevents
rearming of the timer.

Co-developed-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/all/20220407161745.7d6754b3@gandalf.local.home
Link: https://lore.kernel.org/all/20221110064101.429013735@goodmis.org
---
 kernel/time/timer.c |   64 ++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 55 insertions(+), 9 deletions(-)

--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1293,14 +1293,21 @@ void add_timer_on(struct timer_list *tim
 EXPORT_SYMBOL_GPL(add_timer_on);
 
 /**
- * __timer_delete - Internal function: Deactivate a timer.
+ * __timer_delete - Internal function: Deactivate a timer
  * @timer:	The timer to be deactivated
+ * @shutdown:	If true this indicates that the timer is about to be
+ *		shutdown permanently.
+ *
+ * If @shutdown is true then @timer->function is set to NULL under the
+ * timer base lock which prevents further rearming of the time. In that
+ * case any attempt to rearm @timer after this function returns will be
+ * silently ignored.
  *
  * Return:
  * * %0 - The timer was not pending
  * * %1 - The timer was pending and deactivated
  */
-static int __timer_delete(struct timer_list *timer)
+static int __timer_delete(struct timer_list *timer, bool shutdown)
 {
 	struct timer_base *base;
 	unsigned long flags;
@@ -1308,9 +1315,22 @@ static int __timer_delete(struct timer_l
 
 	debug_assert_init(timer);
 
-	if (timer_pending(timer)) {
+	/*
+	 * If @shutdown is set then the lock has to be taken whether the
+	 * timer is pending or not to protect against a concurrent rearm
+	 * which might hit between the lockless pending check and the lock
+	 * aquisition. By taking the lock it is ensured that such a newly
+	 * enqueued timer is dequeued and cannot end up with
+	 * timer->function == NULL in the expiry code.
+	 *
+	 * If timer->function is currently executed, then this makes sure
+	 * that the callback cannot requeue the timer.
+	 */
+	if (timer_pending(timer) || shutdown) {
 		base = lock_timer_base(timer, &flags);
 		ret = detach_if_pending(timer, base, true);
+		if (shutdown)
+			timer->function = NULL;
 		raw_spin_unlock_irqrestore(&base->lock, flags);
 	}
 
@@ -1332,20 +1352,31 @@ static int __timer_delete(struct timer_l
  */
 int timer_delete(struct timer_list *timer)
 {
-	return __timer_delete(timer);
+	return __timer_delete(timer, false);
 }
 EXPORT_SYMBOL(timer_delete);
 
 /**
  * __try_to_del_timer_sync - Internal function: Try to deactivate a timer
  * @timer:	Timer to deactivate
+ * @shutdown:	If true this indicates that the timer is about to be
+ *		shutdown permanently.
+ *
+ * If @shutdown is true then @timer->function is set to NULL under the
+ * timer base lock which prevents further rearming of the timer. Any
+ * attempt to rearm @timer after this function returns will be silently
+ * ignored.
+ *
+ * This function cannot guarantee that the timer cannot be rearmed
+ * right after dropping the base lock if @shutdown is false. That
+ * needs to be prevented by the calling code if necessary.
  *
  * Return:
  * * %0  - The timer was not pending
  * * %1  - The timer was pending and deactivated
  * * %-1 - The timer callback function is running on a different CPU
  */
-static int __try_to_del_timer_sync(struct timer_list *timer)
+static int __try_to_del_timer_sync(struct timer_list *timer, bool shutdown)
 {
 	struct timer_base *base;
 	unsigned long flags;
@@ -1357,6 +1388,8 @@ static int __try_to_del_timer_sync(struc
 
 	if (base->running_timer != timer)
 		ret = detach_if_pending(timer, base, true);
+	if (shutdown)
+		timer->function = NULL;
 
 	raw_spin_unlock_irqrestore(&base->lock, flags);
 
@@ -1379,7 +1412,7 @@ static int __try_to_del_timer_sync(struc
  */
 int try_to_del_timer_sync(struct timer_list *timer)
 {
-	return __try_to_del_timer_sync(timer);
+	return __try_to_del_timer_sync(timer, false);
 }
 EXPORT_SYMBOL(try_to_del_timer_sync);
 
@@ -1460,12 +1493,25 @@ static inline void del_timer_wait_runnin
  * __timer_delete_sync - Internal function: Deactivate a timer and wait
  *			 for the handler to finish.
  * @timer:	The timer to be deactivated
+ * @shutdown:	If true @timer->function will be set to NULL under the
+ *		timer base lock which prevents rearming of @timer
+ *
+ * If @shutdown is not set the timer can be rearmed later. If the timer can
+ * be rearmed concurrently, i.e. after dropping the base lock then the
+ * return value is meaningless.
+ *
+ * If @shutdown is set then @timer->function is set to NULL under timer
+ * base lock which prevents rearming of the timer. Any attempt to rearm
+ * a shutdown timer is silently ignored.
+ *
+ * If the timer should be reused after shutdown it has to be initialized
+ * again.
  *
  * Return:
  * * %0	- The timer was not pending
  * * %1	- The timer was pending and deactivated
  */
-static int __timer_delete_sync(struct timer_list *timer)
+static int __timer_delete_sync(struct timer_list *timer, bool shutdown)
 {
 	int ret;
 
@@ -1495,7 +1541,7 @@ static int __timer_delete_sync(struct ti
 		lockdep_assert_preemption_enabled();
 
 	do {
-		ret = __try_to_del_timer_sync(timer);
+		ret = __try_to_del_timer_sync(timer, shutdown);
 
 		if (unlikely(ret < 0)) {
 			del_timer_wait_running(timer);
@@ -1547,7 +1593,7 @@ static int __timer_delete_sync(struct ti
  */
 int timer_delete_sync(struct timer_list *timer)
 {
-	return __timer_delete_sync(timer);
+	return __timer_delete_sync(timer, false);
 }
 EXPORT_SYMBOL(timer_delete_sync);
 

Re: [patch 12/15] timers: Add shutdown mechanism to the internal functions

[Steven Rostedt]

On Tue, 15 Nov 2022 21:28:52 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> Tearing down timers which have circular dependencies to other
> functionality, e.g. workqueues, where the timer can schedule work and work
> can arm timers is not trivial.
> 
> In those cases it is desired to shutdown the timer in a way which prevents
> rearming of the timer. The mechanism to do so it to set timer->function to
> NULL and use this as an indicator for the timer arming functions to ignore
> the (re)arm request.
> 
> Add a shutdown argument to the relevant internal functions which makes the
> actual deactivation code set timer->function to NULL which in turn prevents
> rearming of the timer.
> 
> Co-developed-by: Steven Rostedt <rostedt@goodmis.org>
> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Link: https://lore.kernel.org/all/20220407161745.7d6754b3@gandalf.local.home
> Link: https://lore.kernel.org/all/20221110064101.429013735@goodmis.org
> ---
>  kernel/time/timer.c |   64 ++++++++++++++++++++++++++++++++++++++++++++--------
>  1 file changed, 55 insertions(+), 9 deletions(-)
> 
> --- a/kernel/time/timer.c
> +++ b/kernel/time/timer.c
> @@ -1293,14 +1293,21 @@ void add_timer_on(struct timer_list *tim
>  EXPORT_SYMBOL_GPL(add_timer_on);
>  
>  /**
> - * __timer_delete - Internal function: Deactivate a timer.
> + * __timer_delete - Internal function: Deactivate a timer
>   * @timer:	The timer to be deactivated
> + * @shutdown:	If true this indicates that the timer is about to be

Nit, needs a common (or "then") after true.


> + *		shutdown permanently.
> + *
> + * If @shutdown is true then @timer->function is set to NULL under the
> + * timer base lock which prevents further rearming of the time. In that
> + * case any attempt to rearm @timer after this function returns will be
> + * silently ignored.
>   *
>   * Return:
>   * * %0 - The timer was not pending
>   * * %1 - The timer was pending and deactivated
>   */
> -static int __timer_delete(struct timer_list *timer)
> +static int __timer_delete(struct timer_list *timer, bool shutdown)
>  {
>  	struct timer_base *base;
>  	unsigned long flags;
> @@ -1308,9 +1315,22 @@ static int __timer_delete(struct timer_l
>  
>  	debug_assert_init(timer);
>  
> -	if (timer_pending(timer)) {
> +	/*
> +	 * If @shutdown is set then the lock has to be taken whether the
> +	 * timer is pending or not to protect against a concurrent rearm
> +	 * which might hit between the lockless pending check and the lock
> +	 * aquisition. By taking the lock it is ensured that such a newly
> +	 * enqueued timer is dequeued and cannot end up with
> +	 * timer->function == NULL in the expiry code.
> +	 *
> +	 * If timer->function is currently executed, then this makes sure
> +	 * that the callback cannot requeue the timer.
> +	 */
> +	if (timer_pending(timer) || shutdown) {
>  		base = lock_timer_base(timer, &flags);
>  		ret = detach_if_pending(timer, base, true);
> +		if (shutdown)
> +			timer->function = NULL;
>  		raw_spin_unlock_irqrestore(&base->lock, flags);
>  	}
>  
> @@ -1332,20 +1352,31 @@ static int __timer_delete(struct timer_l
>   */
>  int timer_delete(struct timer_list *timer)
>  {
> -	return __timer_delete(timer);
> +	return __timer_delete(timer, false);
>  }
>  EXPORT_SYMBOL(timer_delete);
>  
>  /**
>   * __try_to_del_timer_sync - Internal function: Try to deactivate a timer
>   * @timer:	Timer to deactivate
> + * @shutdown:	If true this indicates that the timer is about to be

Same here.

> + *		shutdown permanently.
> + *
> + * If @shutdown is true then @timer->function is set to NULL under the
> + * timer base lock which prevents further rearming of the timer. Any
> + * attempt to rearm @timer after this function returns will be silently
> + * ignored.
> + *
> + * This function cannot guarantee that the timer cannot be rearmed
> + * right after dropping the base lock if @shutdown is false. That
> + * needs to be prevented by the calling code if necessary.
>   *
>   * Return:
>   * * %0  - The timer was not pending
>   * * %1  - The timer was pending and deactivated
>   * * %-1 - The timer callback function is running on a different CPU
>   */
> -static int __try_to_del_timer_sync(struct timer_list *timer)
> +static int __try_to_del_timer_sync(struct timer_list *timer, bool shutdown)
>  {
>  	struct timer_base *base;
>  	unsigned long flags;
> @@ -1357,6 +1388,8 @@ static int __try_to_del_timer_sync(struc
>  
>  	if (base->running_timer != timer)
>  		ret = detach_if_pending(timer, base, true);
> +	if (shutdown)
> +		timer->function = NULL;
>  
>  	raw_spin_unlock_irqrestore(&base->lock, flags);
>  
> @@ -1379,7 +1412,7 @@ static int __try_to_del_timer_sync(struc
>   */
>  int try_to_del_timer_sync(struct timer_list *timer)
>  {
> -	return __try_to_del_timer_sync(timer);
> +	return __try_to_del_timer_sync(timer, false);
>  }
>  EXPORT_SYMBOL(try_to_del_timer_sync);
>  
> @@ -1460,12 +1493,25 @@ static inline void del_timer_wait_runnin
>   * __timer_delete_sync - Internal function: Deactivate a timer and wait
>   *			 for the handler to finish.
>   * @timer:	The timer to be deactivated
> + * @shutdown:	If true @timer->function will be set to NULL under the

Here too.

-- Steve

> + *		timer base lock which prevents rearming of @timer
> + *
> + * If @shutdown is not set the timer can be rearmed later. If the timer can
> + * be rearmed concurrently, i.e. after dropping the base lock then the
> + * return value is meaningless.
> + *
> + * If @shutdown is set then @timer->function is set to NULL under timer
> + * base lock which prevents rearming of the timer. Any attempt to rearm
> + * a shutdown timer is silently ignored.
> + *
> + * If the timer should be reused after shutdown it has to be initialized
> + * again.
>   *
>   * Return:
>   * * %0	- The timer was not pending
>   * * %1	- The timer was pending and deactivated
>   */
> -static int __timer_delete_sync(struct timer_list *timer)
> +static int __timer_delete_sync(struct timer_list *timer, bool shutdown)
>  {
>  	int ret;
>  
> @@ -1495,7 +1541,7 @@ static int __timer_delete_sync(struct ti
>  		lockdep_assert_preemption_enabled();
>  
>  	do {
> -		ret = __try_to_del_timer_sync(timer);
> +		ret = __try_to_del_timer_sync(timer, shutdown);
>  
>  		if (unlikely(ret < 0)) {
>  			del_timer_wait_running(timer);
> @@ -1547,7 +1593,7 @@ static int __timer_delete_sync(struct ti
>   */
>  int timer_delete_sync(struct timer_list *timer)
>  {
> -	return __timer_delete_sync(timer);
> +	return __timer_delete_sync(timer, false);
>  }
>  EXPORT_SYMBOL(timer_delete_sync);
>  

[patch 13/15] timers: Provide timer_shutdown[_sync]()

[Thomas Gleixner]

Tearing down timers which have circular dependencies to other
functionality, e.g. workqueues, where the timer can schedule work and work
can arm timers is not trivial.

In those cases it is desired to shutdown the timer in a way which prevents
rearming of the timer. The mechanism to do so it to set timer->function to
NULL and use this as an indicator for the timer arming functions to ignore
the (re)arm request.

Expose new interfaces for this: timer_shutdown_sync() and timer_shutdown().

timer_shutdown_sync() has the same functionality as timer_delete_sync()
plus the NULL-ification of the timer function.

timer_shutdown() has the same functionality as timer_delete() plus the
NULL-ification of the timer function.

In both cases the rearming of the timer is prevented by silently discarding
rearm attempts due to timer->function being NULL.

Co-developed-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/all/20220407161745.7d6754b3@gandalf.local.home
Link: https://lore.kernel.org/all/20221110064101.429013735@goodmis.org
---
 include/linux/timer.h |    2 +
 kernel/time/timer.c   |   66 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+)

--- a/include/linux/timer.h
+++ b/include/linux/timer.h
@@ -184,6 +184,8 @@ extern void add_timer(struct timer_list
 extern int try_to_del_timer_sync(struct timer_list *timer);
 extern int timer_delete_sync(struct timer_list *timer);
 extern int timer_delete(struct timer_list *timer);
+extern int timer_shutdown_sync(struct timer_list *timer);
+extern int timer_shutdown(struct timer_list *timer);
 
 /**
  * del_timer_sync - Delete a pending timer and wait for a running callback
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1357,6 +1357,27 @@ int timer_delete(struct timer_list *time
 EXPORT_SYMBOL(timer_delete);
 
 /**
+ * timer_shutdown - Deactivate a timer and prevent rearming
+ * @timer:	The timer to be deactivated
+ *
+ * The function does not wait for an eventually running timer callback on a
+ * different CPU but it prevents rearming of the timer. Any attempt to arm
+ * @timer after this function returns will be silently ignored.
+ *
+ * This function is useful for teardown code and should only be used when
+ * timer_shutdown_sync() cannot be invoked due to locking or context constraints.
+ *
+ * Return:
+ * * %0 - The timer was not pending
+ * * %1 - The timer was pending
+ */
+int timer_shutdown(struct timer_list *timer)
+{
+	return __timer_delete(timer, true);
+}
+EXPORT_SYMBOL_GPL(timer_shutdown);
+
+/**
  * __try_to_del_timer_sync - Internal function: Try to deactivate a timer
  * @timer:	Timer to deactivate
  * @shutdown:	If true this indicates that the timer is about to be
@@ -1587,6 +1608,9 @@ static int __timer_delete_sync(struct ti
  * lock. If there is the possibility of a concurrent rearm then the return
  * value of the function is meaningless.
  *
+ * If such a guarantee is needed, e.g. for teardown situations then use
+ * timer_shutdown_sync() instead.
+ *
  * Return:
  * * %0	- The timer was not pending
  * * %1	- The timer was pending and deactivated
@@ -1597,6 +1621,48 @@ int timer_delete_sync(struct timer_list
 }
 EXPORT_SYMBOL(timer_delete_sync);
 
+/**
+ * timer_shutdown_sync - Shutdown a timer and prevent rearming
+ * @timer: The timer to be shutdown
+ *
+ * When the function returns it is guaranteed that:
+ *   - @timer is not queued
+ *   - The callback function of @timer is not running
+ *   - @timer cannot be enqueued again. Any attempt to rearm
+ *     @timer is silently ignored.
+ *
+ * See timer_delete_sync() for synchronization rules.
+ *
+ * This function is useful for final teardown of an infrastructure where
+ * the timer is subject to a circular dependency problem.
+ *
+ * A common pattern for this is a timer and a workqueue where the timer can
+ * schedule work and work can arm the timer. On shutdown the workqueue must
+ * be destroyed and the timer must be prevented from rearming. Unless the
+ * code has conditionals like 'if (mything->in_shutdown)' to prevent that
+ * there is no way to get this correct with timer_delete_sync().
+ *
+ * timer_shutdown_sync() is solving the problem. The correct ordering of
+ * calls in this case is:
+ *
+ *	timer_shutdown_sync(&mything->timer);
+ *	workqueue_destroy(&mything->workqueue);
+ *
+ * After this 'mything' can be safely freed.
+ *
+ * This obviously requires that the timer is not required to be functional
+ * for the rest of the shutdown operation.
+ *
+ * Return:
+ * * %0 - The timer was not pending
+ * * %1 - The timer was pending
+ */
+int timer_shutdown_sync(struct timer_list *timer)
+{
+	return __timer_delete_sync(timer, true);
+}
+EXPORT_SYMBOL_GPL(timer_shutdown_sync);
+
 static void call_timer_fn(struct timer_list *timer,
 			  void (*fn)(struct timer_list *),
 			  unsigned long baseclk)

Re: [patch 13/15] timers: Provide timer_shutdown[_sync]()

[Steven Rostedt]

On Tue, 15 Nov 2022 21:28:54 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> +/**
> + * timer_shutdown_sync - Shutdown a timer and prevent rearming
> + * @timer: The timer to be shutdown
> + *
> + * When the function returns it is guaranteed that:
> + *   - @timer is not queued
> + *   - The callback function of @timer is not running
> + *   - @timer cannot be enqueued again. Any attempt to rearm
> + *     @timer is silently ignored.
> + *
> + * See timer_delete_sync() for synchronization rules.

 "See timer_delete_sync() for synchronization and context rules."

As where it can be executed is as important as the synchronization that is
needed.

-- Steve


> + *
> + * This function is useful for final teardown of an infrastructure where
> + * the timer is subject to a circular dependency problem.
> + *
> + * A common pattern for this is a timer and a workqueue where the timer can
> + * schedule work and work can arm the timer. On shutdown the workqueue must
> + * be destroyed and the timer must be prevented from rearming. Unless the
> + * code has conditionals like 'if (mything->in_shutdown)' to prevent that
> + * there is no way to get this correct with timer_delete_sync().
> + *
> + * timer_shutdown_sync() is solving the problem. The correct ordering of
> + * calls in this case is:
> + *
> + *	timer_shutdown_sync(&mything->timer);
> + *	workqueue_destroy(&mything->workqueue);
> + *
> + * After this 'mything' can be safely freed.
> + *
> + * This obviously requires that the timer is not required to be functional
> + * for the rest of the shutdown operation.
> + *
> + * Return:
> + * * %0 - The timer was not pending
> + * * %1 - The timer was pending
> + */
> +int timer_shutdown_sync(struct timer_list *timer)
> +{
> +	return __timer_delete_sync(timer, true);
> +}
> +EXPORT_SYMBOL_GPL(timer_shutdown_sync);
> +

[patch 14/15] timers: Update the documentation to reflect on the new timer_shutdown() API

[Thomas Gleixner]

From: "Steven Rostedt (Google)" <rostedt@goodmis.org>

In order to make sure that a timer is not re-armed after it is stopped
before freeing, a new shutdown state is added to the timer code. The API
timer_shutdown_sync() and timer_shutdown() must be called before the
object that holds the timer can be freed.

Update the documentation to reflect this new workflow.

[ tglx: Updated to the new semantics and removed the bogus claim that
  	del_timer_sync() returns the number of removal attempts ]

Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Link: https://lore.kernel.org/r/20221110064147.712934793@goodmis.org
---
 Documentation/RCU/Design/Requirements/Requirements.rst |    2 +-
 Documentation/core-api/local_ops.rst                   |    2 +-
 Documentation/kernel-hacking/locking.rst               |   13 ++++++++-----
 3 files changed, 10 insertions(+), 7 deletions(-)

--- a/Documentation/RCU/Design/Requirements/Requirements.rst
+++ b/Documentation/RCU/Design/Requirements/Requirements.rst
@@ -1858,7 +1858,7 @@ unloaded. After a given module has been
 one of its functions results in a segmentation fault. The module-unload
 functions must therefore cancel any delayed calls to loadable-module
 functions, for example, any outstanding mod_timer() must be dealt
-with via del_timer_sync() or similar.
+with via timer_shutdown_sync().
 
 Unfortunately, there is no way to cancel an RCU callback; once you
 invoke call_rcu(), the callback function is eventually going to be
--- a/Documentation/core-api/local_ops.rst
+++ b/Documentation/core-api/local_ops.rst
@@ -191,7 +191,7 @@ Here is a sample module which implements
 
     static void __exit test_exit(void)
     {
-            del_timer_sync(&test_timer);
+            timer_shutdown_sync(&test_timer);
     }
 
     module_init(test_init);
--- a/Documentation/kernel-hacking/locking.rst
+++ b/Documentation/kernel-hacking/locking.rst
@@ -1003,11 +1003,14 @@ If 0, it means (in this case) that it is
 
 
 Another common problem is deleting timers which restart themselves (by
-calling add_timer() at the end of their timer function).
-Because this is a fairly common case which is prone to races, you should
-use del_timer_sync() (``include/linux/timer.h``) to
-handle this case. It returns the number of times the timer had to be
-deleted before we finally stopped it from adding itself back in.
+calling add_timer() at the end of their timer function).  Because this is a
+fairly common case which is prone to races, you should use del_timer_sync()
+(``include/linux/timer.h``) to handle this case.
+
+Before freeing a timer, timer_shutdown() or timer_shutdown_sync() should be
+called which will keep it from being rearmed. Any subsequent attempt to
+rearm the timer will be silently ignored by the core code.
+
 
 Locking Speed
 =============

[patch 15/15] Bluetooth: hci_qca: Fix the teardown problem for real

[Thomas Gleixner]

While discussing solutions for the teardown problem which results from
circular dependencies between timers and workqueues, where timers schedule
work from their timer callback and workqueues arm the timers from work
items, it was discovered that the recent fix to the QCA code is incorrect.

That commit fixes the obvious problem of using del_timer() instead of
del_timer_sync() and reorders the teardown calls to

   destroy_workqueue(wq);
   del_timer_sync(t);

This makes it less likely to explode, but it's still broken:

   destroy_workqueue(wq);
   /* After this point @wq cannot be touched anymore */

   ---> timer expires
         queue_work(wq) <---- Results in a NULl pointer dereference
			      deep in the work queue core code.
   del_timer_sync(t);

Use the new timer_shutdown_sync() function to ensure that the timers are
disarmed, no timer callbacks are running and the timers cannot be armed
again. This restores the original teardown sequence:

   timer_shutdown_sync(t);
   destroy_workqueue(wq);

which is now correct because the timer core silently ignores potential
rearming attempts which can happen when destroy_workqueue() drains pending
work before mopping up the workqueue.

Fixes: 72ef98445aca ("Bluetooth: hci_qca: Use del_timer_sync() before freeing")
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Marcel Holtmann <marcel@holtmann.org>
Cc: Johan Hedberg <johan.hedberg@gmail.com>
Cc: Luiz Augusto von Dentz <luiz.dentz@gmail.com>
Cc: linux-bluetooth@vger.kernel.org
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Jakub Kicinski <kuba@kernel.org>
Cc: Paolo Abeni <pabeni@redhat.com>
Cc: netdev@vger.kernel.org
Link: https://lore.kernel.org/all/87iljhsftt.ffs@tglx
---
 drivers/bluetooth/hci_qca.c |   10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

--- a/drivers/bluetooth/hci_qca.c
+++ b/drivers/bluetooth/hci_qca.c
@@ -696,9 +696,15 @@ static int qca_close(struct hci_uart *hu
 	skb_queue_purge(&qca->tx_wait_q);
 	skb_queue_purge(&qca->txq);
 	skb_queue_purge(&qca->rx_memdump_q);
+	/*
+	 * Shut the timers down so they can't be rearmed when
+	 * destroy_workqueue() drains pending work which in turn might try
+	 * to arm a timer.  After shutdown rearm attempts are silently
+	 * ignored by the timer core code.
+	 */
+	timer_shutdown_sync(&qca->tx_idle_timer);
+	timer_shutdown_sync(&qca->wake_retrans_timer);
 	destroy_workqueue(qca->workqueue);
-	del_timer_sync(&qca->tx_idle_timer);
-	del_timer_sync(&qca->wake_retrans_timer);
 	qca->hu = NULL;
 
 	kfree_skb(qca->rx_skb);

Re: [patch 15/15] Bluetooth: hci_qca: Fix the teardown problem for real

[Luiz Augusto von Dentz]

Hi Thomas,

On Tue, Nov 15, 2022 at 12:28 PM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> While discussing solutions for the teardown problem which results from
> circular dependencies between timers and workqueues, where timers schedule
> work from their timer callback and workqueues arm the timers from work
> items, it was discovered that the recent fix to the QCA code is incorrect.
>
> That commit fixes the obvious problem of using del_timer() instead of
> del_timer_sync() and reorders the teardown calls to
>
>    destroy_workqueue(wq);
>    del_timer_sync(t);
>
> This makes it less likely to explode, but it's still broken:
>
>    destroy_workqueue(wq);
>    /* After this point @wq cannot be touched anymore */
>
>    ---> timer expires
>          queue_work(wq) <---- Results in a NULl pointer dereference
>                               deep in the work queue core code.
>    del_timer_sync(t);
>
> Use the new timer_shutdown_sync() function to ensure that the timers are
> disarmed, no timer callbacks are running and the timers cannot be armed
> again. This restores the original teardown sequence:
>
>    timer_shutdown_sync(t);
>    destroy_workqueue(wq);
>
> which is now correct because the timer core silently ignores potential
> rearming attempts which can happen when destroy_workqueue() drains pending
> work before mopping up the workqueue.
>
> Fixes: 72ef98445aca ("Bluetooth: hci_qca: Use del_timer_sync() before freeing")
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Cc: Marcel Holtmann <marcel@holtmann.org>
> Cc: Johan Hedberg <johan.hedberg@gmail.com>
> Cc: Luiz Augusto von Dentz <luiz.dentz@gmail.com>
> Cc: linux-bluetooth@vger.kernel.org
> Cc: "David S. Miller" <davem@davemloft.net>
> Cc: Eric Dumazet <edumazet@google.com>
> Cc: Jakub Kicinski <kuba@kernel.org>
> Cc: Paolo Abeni <pabeni@redhat.com>
> Cc: netdev@vger.kernel.org
> Link: https://lore.kernel.org/all/87iljhsftt.ffs@tglx
> ---
>  drivers/bluetooth/hci_qca.c |   10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
>
> --- a/drivers/bluetooth/hci_qca.c
> +++ b/drivers/bluetooth/hci_qca.c
> @@ -696,9 +696,15 @@ static int qca_close(struct hci_uart *hu
>         skb_queue_purge(&qca->tx_wait_q);
>         skb_queue_purge(&qca->txq);
>         skb_queue_purge(&qca->rx_memdump_q);
> +       /*
> +        * Shut the timers down so they can't be rearmed when
> +        * destroy_workqueue() drains pending work which in turn might try
> +        * to arm a timer.  After shutdown rearm attempts are silently
> +        * ignored by the timer core code.
> +        */
> +       timer_shutdown_sync(&qca->tx_idle_timer);
> +       timer_shutdown_sync(&qca->wake_retrans_timer);
>         destroy_workqueue(qca->workqueue);
> -       del_timer_sync(&qca->tx_idle_timer);
> -       del_timer_sync(&qca->wake_retrans_timer);
>         qca->hu = NULL;
>
>         kfree_skb(qca->rx_skb);
>

Acked-by: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>

-- 
Luiz Augusto von Dentz

Re: [patch 00/15] timers: Provide timer_shutdown[_sync]()

[Guenter Roeck]

On Tue, Nov 15, 2022 at 09:28:32PM +0100, Thomas Gleixner wrote:
> Tearing down timers can be tedious when there are circular dependencies to
> other things which need to be torn down. A prime example is timer and
> workqueue where the timer schedules work and the work arms the timer.
> 
> Steven and the Google Chromebook team ran into such an issue in the
> Bluetooth HCI code.
> 
> Steven suggested to create a new function del_timer_free() which marks the
> timer as shutdown. Rearm attempts of shutdown timers are discarded and he
> wanted to emit a warning for that case:
> 
>    https://lore.kernel.org/all/20220407161745.7d6754b3@gandalf.local.home
> 
> This resulted in a lengthy discussion and suggestions how this should be
> implemented. The patch series went through several iterations and during
> the review of the last version it turned out that this approach is
> suboptimal:
> 
>    https://lore.kernel.org/all/20221110064101.429013735@goodmis.org
> 
> The warning is not really helpful because it's entirely unclear how it
> should be acted upon. The only way to address such a case is to add 'if
> (in_shutdown)' conditionals all over the place. This is error prone and in
> most cases of teardown like the HCI one which started this discussion not
> required all.
> 
> What needs to prevented is that pending work which is drained via
> destroy_workqueue() does not rearm the previously shutdown timer. Nothing
> in that shutdown sequence relies on the timer being functional.
> 
> The conclusion was that the semantics of timer_shutdown_sync() should be:
> 
>     - timer is not enqueued
>     - timer callback is not running
>     - timer cannot be rearmed
> 
> Preventing the rearming of shutdown timers is done by discarding rearm
> attempts silently.
> 
> As Steven is short of cycles, I made some spare cycles available and
> reworked the patch series to follow the new semantics and plugged the races
> which were discovered during review.
> 
> The patches have been split up into small pieces to make review easier and
> I took the liberty to throw a bunch of overdue cleanups into the picture
> instead of proliferating the existing state further.
> 
> The last patch in the series addresses the HCI teardown issue for real.
> 

I applied the series to the top of v6.1-rc5, and also applied the result of
running the coccinelle script to auto-convert simple cases. Running this
set of patches through my testbed showed no build errors, runtime
failures, or warnings. I also backported the series to chromeos-5.15,
again applied the coccinelle generated patches, and ran it through a
regression test. No failures either.

With that, for the series,

Tested-by: Guenter Roeck <linux@roeck-us.net>

Let me know if I should send individual tags for each patch in the series.

Thanks,
Guenter

Re: [patch 00/15] timers: Provide timer_shutdown[_sync]()

[Thomas Gleixner]

On Tue, Nov 15 2022 at 21:28, Thomas Gleixner wrote:
>
> As Steven is short of cycles, I made some spare cycles available and
> reworked the patch series to follow the new semantics and plugged the races
> which were discovered during review.

Any opinions on this pile or should I just declare it's perfect and
queue it for 6.2?

Thanks,

        tglx

Re: [patch 00/15] timers: Provide timer_shutdown[_sync]()

[Steven Rostedt]

On Mon, 21 Nov 2022 16:15:00 +0100
Thomas Gleixner <tglx@linutronix.de> wrote:

> > As Steven is short of cycles, I made some spare cycles available and
> > reworked the patch series to follow the new semantics and plugged the races
> > which were discovered during review.  
> 
> Any opinions on this pile or should I just declare it's perfect and
> queue it for 6.2?

I have time cut out of today to go over it. Thanks Thomas,

-- Steve

Re: [patch 00/15] timers: Provide timer_shutdown[_sync]()

[Steven Rostedt]

On Tue, 15 Nov 2022 21:28:32 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> The patches have been split up into small pieces to make review easier and
> I took the liberty to throw a bunch of overdue cleanups into the picture
> instead of proliferating the existing state further.

After applying all these patches, and then my updates to the rest of
the kernel, as well as my update to the debug objects to require
shutdown. It reported this was needed:

-- Steve

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 0fbb71950ca2..3e84a2621913 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -2188,7 +2188,7 @@ signed long __sched schedule_timeout(signed long timeout)
 	timer_setup_on_stack(&timer.timer, process_timeout, 0);
 	__mod_timer(&timer.timer, expire, MOD_TIMER_NOTPENDING);
 	schedule();
-	del_timer_sync(&timer.timer);
+	timer_shutdown_sync(&timer.timer);
 
 	/* Remove the timer from the object tracker */
 	destroy_timer_on_stack(&timer.timer);