[PATCH] docs: power: Adjust freezing-of-tasks.rst due to the freezer logic changes

[PATCH] docs: power: Adjust freezing-of-tasks.rst due to the freezer logic changes

[Kevin Hao]

Peter has rewritten core freezer logic in commit f5d39b020809
("freezer,sched: Rewrite core freezer logic"), adjust the
freezing-of-tasks.rst according to this commit. The main changes
include:
- Drop the mention of PF_FROZEN and PF_FREEZER_SKIP
- Introduce TASK_FROZEN, TASK_FREEZABLE and __TASK_FREEZABLE_UNSAFE
- Replace system_freezing_cnt with freezer_active
- Use another example for the loop of a freezable kernel thread since
  the old codes are already gone

Signed-off-by: Kevin Hao <haokexin@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: "Rafael J. Wysocki" <rafael@kernel.org>
Cc: Pavel Machek <pavel@ucw.cz>
Cc: Len Brown <len.brown@intel.com>
---
 Documentation/power/freezing-of-tasks.rst | 83 +++++++++++++----------
 1 file changed, 47 insertions(+), 36 deletions(-)

diff --git a/Documentation/power/freezing-of-tasks.rst b/Documentation/power/freezing-of-tasks.rst
index 53b6a56c4635..8969ed244b20 100644
--- a/Documentation/power/freezing-of-tasks.rst
+++ b/Documentation/power/freezing-of-tasks.rst
@@ -14,27 +14,28 @@ architectures).
 II. How does it work?
 =====================
 
-There are three per-task flags used for that, PF_NOFREEZE, PF_FROZEN
-and PF_FREEZER_SKIP (the last one is auxiliary).  The tasks that have
-PF_NOFREEZE unset (all user space processes and some kernel threads) are
-regarded as 'freezable' and treated in a special way before the system enters a
-suspend state as well as before a hibernation image is created (in what follows
-we only consider hibernation, but the description also applies to suspend).
+There are one per-task flag (PF_NOFREEZE) and three per-task states
+(TASK_FROZEN, TASK_FREEZABLE and __TASK_FREEZABLE_UNSAFE) used for that.
+The tasks that have PF_NOFREEZE unset (all user space processes and some kernel
+threads) are regarded as 'freezable' and treated in a special way before the
+system enters a suspend state as well as before a hibernation image is created
+(in what follows we only consider hibernation, but the description also applies
+to suspend).
 
 Namely, as the first step of the hibernation procedure the function
 freeze_processes() (defined in kernel/power/process.c) is called.  A system-wide
-variable system_freezing_cnt (as opposed to a per-task flag) is used to indicate
-whether the system is to undergo a freezing operation. And freeze_processes()
-sets this variable.  After this, it executes try_to_freeze_tasks() that sends a
-fake signal to all user space processes, and wakes up all the kernel threads.
-All freezable tasks must react to that by calling try_to_freeze(), which
-results in a call to __refrigerator() (defined in kernel/freezer.c), which sets
-the task's PF_FROZEN flag, changes its state to TASK_UNINTERRUPTIBLE and makes
-it loop until PF_FROZEN is cleared for it. Then, we say that the task is
-'frozen' and therefore the set of functions handling this mechanism is referred
-to as 'the freezer' (these functions are defined in kernel/power/process.c,
-kernel/freezer.c & include/linux/freezer.h). User space processes are generally
-frozen before kernel threads.
+static key freezer_active (as opposed to a per-task flag or state) is used to
+indicate whether the system is to undergo a freezing operation. And
+freeze_processes() sets this variable.  After this, it executes
+try_to_freeze_tasks() that sends a fake signal to all user space processes, and
+wakes up all the kernel threads. All freezable tasks must react to that by
+calling try_to_freeze(), which results in a call to __refrigerator() (defined
+in kernel/freezer.c), which sets the task's state to TASK_FROZEN, and makes it
+loop until it is woken by an explicit TASK_FROZEN wakeup. Then, we say that the
+task is 'frozen' and therefore the set of functions handling this mechanism is
+referred to as 'the freezer' (these functions are defined in
+kernel/power/process.c, kernel/freezer.c & include/linux/freezer.h). User space
+processes are generally frozen before kernel threads.
 
 __refrigerator() must not be called directly.  Instead, use the
 try_to_freeze() function (defined in include/linux/freezer.h), that checks
@@ -43,31 +44,40 @@ if the task is to be frozen and makes the task enter __refrigerator().
 For user space processes try_to_freeze() is called automatically from the
 signal-handling code, but the freezable kernel threads need to call it
 explicitly in suitable places or use the wait_event_freezable() or
-wait_event_freezable_timeout() macros (defined in include/linux/freezer.h)
-that combine interruptible sleep with checking if the task is to be frozen and
-calling try_to_freeze().  The main loop of a freezable kernel thread may look
-like the following one::
+wait_event_freezable_timeout() macros (defined in include/linux/wait.h)
+that puts the task to sleep (TASK_INTERRUPTIBLE) or frozen (TASK_FROZEN) if the
+task is freezing. The main loop of a freezable kernel thread may look like the
+following one::
 
 	set_freezable();
-	do {
-		hub_events();
-		wait_event_freezable(khubd_wait,
-				!list_empty(&hub_event_list) ||
-				kthread_should_stop());
-	} while (!kthread_should_stop() || !list_empty(&hub_event_list));
 
-(from drivers/usb/core/hub.c::hub_thread()).
+	while (true) {
+		struct task_struct *tsk = NULL;
 
-If a freezable kernel thread fails to call try_to_freeze() after the freezer has
-initiated a freezing operation, the freezing of tasks will fail and the entire
-hibernation operation will be cancelled.  For this reason, freezable kernel
-threads must call try_to_freeze() somewhere or use one of the
+		wait_event_freezable(oom_reaper_wait, oom_reaper_list != NULL);
+		spin_lock_irq(&oom_reaper_lock);
+		if (oom_reaper_list != NULL) {
+			tsk = oom_reaper_list;
+			oom_reaper_list = tsk->oom_reaper_list;
+		}
+		spin_unlock_irq(&oom_reaper_lock);
+
+		if (tsk)
+			oom_reap_task(tsk);
+	}
+
+(from mm/oom_kill.c::oom_reaper()).
+
+If a freezable kernel thread fails to be put to frozen state after the freezer
+has initiated a freezing operation, the freezing of tasks will fail and the
+entire hibernation operation will be cancelled.  For this reason, freezable
+kernel threads must call try_to_freeze() somewhere or use one of the
 wait_event_freezable() and wait_event_freezable_timeout() macros.
 
 After the system memory state has been restored from a hibernation image and
 devices have been reinitialized, the function thaw_processes() is called in
-order to clear the PF_FROZEN flag for each frozen task.  Then, the tasks that
-have been frozen leave __refrigerator() and continue running.
+order to wake up each frozen task.  Then, the tasks that have been frozen leave
+__refrigerator() and continue running.
 
 
 Rationale behind the functions dealing with freezing and thawing of tasks
@@ -96,7 +106,8 @@ III. Which kernel threads are freezable?
 Kernel threads are not freezable by default.  However, a kernel thread may clear
 PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE
 directly is not allowed).  From this point it is regarded as freezable
-and must call try_to_freeze() in a suitable place.
+and must call try_to_freeze() or variants of wait_event_freezable() in a
+suitable place.
 
 IV. Why do we do that?
 ======================
-- 
2.39.2

Re: [PATCH] docs: power: Adjust freezing-of-tasks.rst due to the freezer logic changes

[Rafael J. Wysocki]

On Tue, Dec 12, 2023 at 3:01 PM Kevin Hao <haokexin@gmail.com> wrote:
>
> Peter has rewritten core freezer logic in commit f5d39b020809
> ("freezer,sched: Rewrite core freezer logic"), adjust the
> freezing-of-tasks.rst according to this commit. The main changes
> include:
> - Drop the mention of PF_FROZEN and PF_FREEZER_SKIP
> - Introduce TASK_FROZEN, TASK_FREEZABLE and __TASK_FREEZABLE_UNSAFE
> - Replace system_freezing_cnt with freezer_active
> - Use another example for the loop of a freezable kernel thread since
>   the old codes are already gone
>
> Signed-off-by: Kevin Hao <haokexin@gmail.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: "Rafael J. Wysocki" <rafael@kernel.org>
> Cc: Pavel Machek <pavel@ucw.cz>
> Cc: Len Brown <len.brown@intel.com>
> ---
>  Documentation/power/freezing-of-tasks.rst | 83 +++++++++++++----------
>  1 file changed, 47 insertions(+), 36 deletions(-)
>
> diff --git a/Documentation/power/freezing-of-tasks.rst b/Documentation/power/freezing-of-tasks.rst
> index 53b6a56c4635..8969ed244b20 100644
> --- a/Documentation/power/freezing-of-tasks.rst
> +++ b/Documentation/power/freezing-of-tasks.rst
> @@ -14,27 +14,28 @@ architectures).
>  II. How does it work?
>  =====================
>
> -There are three per-task flags used for that, PF_NOFREEZE, PF_FROZEN
> -and PF_FREEZER_SKIP (the last one is auxiliary).  The tasks that have
> -PF_NOFREEZE unset (all user space processes and some kernel threads) are
> -regarded as 'freezable' and treated in a special way before the system enters a
> -suspend state as well as before a hibernation image is created (in what follows
> -we only consider hibernation, but the description also applies to suspend).
> +There are one per-task flag (PF_NOFREEZE) and three per-task states
> +(TASK_FROZEN, TASK_FREEZABLE and __TASK_FREEZABLE_UNSAFE) used for that.
> +The tasks that have PF_NOFREEZE unset (all user space processes and some kernel
> +threads) are regarded as 'freezable' and treated in a special way before the
> +system enters a suspend state as well as before a hibernation image is created
> +(in what follows we only consider hibernation, but the description also applies
> +to suspend).
>
>  Namely, as the first step of the hibernation procedure the function
>  freeze_processes() (defined in kernel/power/process.c) is called.  A system-wide
> -variable system_freezing_cnt (as opposed to a per-task flag) is used to indicate
> -whether the system is to undergo a freezing operation. And freeze_processes()
> -sets this variable.  After this, it executes try_to_freeze_tasks() that sends a
> -fake signal to all user space processes, and wakes up all the kernel threads.
> -All freezable tasks must react to that by calling try_to_freeze(), which
> -results in a call to __refrigerator() (defined in kernel/freezer.c), which sets
> -the task's PF_FROZEN flag, changes its state to TASK_UNINTERRUPTIBLE and makes
> -it loop until PF_FROZEN is cleared for it. Then, we say that the task is
> -'frozen' and therefore the set of functions handling this mechanism is referred
> -to as 'the freezer' (these functions are defined in kernel/power/process.c,
> -kernel/freezer.c & include/linux/freezer.h). User space processes are generally
> -frozen before kernel threads.
> +static key freezer_active (as opposed to a per-task flag or state) is used to
> +indicate whether the system is to undergo a freezing operation. And
> +freeze_processes() sets this variable.  After this, it executes
> +try_to_freeze_tasks() that sends a fake signal to all user space processes, and
> +wakes up all the kernel threads. All freezable tasks must react to that by
> +calling try_to_freeze(), which results in a call to __refrigerator() (defined
> +in kernel/freezer.c), which sets the task's state to TASK_FROZEN, and makes it
> +loop until it is woken by an explicit TASK_FROZEN wakeup. Then, we say that the
> +task is 'frozen' and therefore the set of functions handling this mechanism is
> +referred to as 'the freezer' (these functions are defined in
> +kernel/power/process.c, kernel/freezer.c & include/linux/freezer.h). User space
> +processes are generally frozen before kernel threads.
>
>  __refrigerator() must not be called directly.  Instead, use the
>  try_to_freeze() function (defined in include/linux/freezer.h), that checks
> @@ -43,31 +44,40 @@ if the task is to be frozen and makes the task enter __refrigerator().
>  For user space processes try_to_freeze() is called automatically from the
>  signal-handling code, but the freezable kernel threads need to call it
>  explicitly in suitable places or use the wait_event_freezable() or
> -wait_event_freezable_timeout() macros (defined in include/linux/freezer.h)
> -that combine interruptible sleep with checking if the task is to be frozen and
> -calling try_to_freeze().  The main loop of a freezable kernel thread may look
> -like the following one::
> +wait_event_freezable_timeout() macros (defined in include/linux/wait.h)
> +that puts the task to sleep (TASK_INTERRUPTIBLE) or frozen (TASK_FROZEN) if the
> +task is freezing. The main loop of a freezable kernel thread may look like the
> +following one::
>
>         set_freezable();
> -       do {
> -               hub_events();
> -               wait_event_freezable(khubd_wait,
> -                               !list_empty(&hub_event_list) ||
> -                               kthread_should_stop());
> -       } while (!kthread_should_stop() || !list_empty(&hub_event_list));
>
> -(from drivers/usb/core/hub.c::hub_thread()).
> +       while (true) {
> +               struct task_struct *tsk = NULL;
>
> -If a freezable kernel thread fails to call try_to_freeze() after the freezer has
> -initiated a freezing operation, the freezing of tasks will fail and the entire
> -hibernation operation will be cancelled.  For this reason, freezable kernel
> -threads must call try_to_freeze() somewhere or use one of the
> +               wait_event_freezable(oom_reaper_wait, oom_reaper_list != NULL);
> +               spin_lock_irq(&oom_reaper_lock);
> +               if (oom_reaper_list != NULL) {
> +                       tsk = oom_reaper_list;
> +                       oom_reaper_list = tsk->oom_reaper_list;
> +               }
> +               spin_unlock_irq(&oom_reaper_lock);
> +
> +               if (tsk)
> +                       oom_reap_task(tsk);
> +       }
> +
> +(from mm/oom_kill.c::oom_reaper()).
> +
> +If a freezable kernel thread fails to be put to frozen state after the freezer
> +has initiated a freezing operation, the freezing of tasks will fail and the
> +entire hibernation operation will be cancelled.  For this reason, freezable
> +kernel threads must call try_to_freeze() somewhere or use one of the
>  wait_event_freezable() and wait_event_freezable_timeout() macros.
>
>  After the system memory state has been restored from a hibernation image and
>  devices have been reinitialized, the function thaw_processes() is called in
> -order to clear the PF_FROZEN flag for each frozen task.  Then, the tasks that
> -have been frozen leave __refrigerator() and continue running.
> +order to wake up each frozen task.  Then, the tasks that have been frozen leave
> +__refrigerator() and continue running.
>
>
>  Rationale behind the functions dealing with freezing and thawing of tasks
> @@ -96,7 +106,8 @@ III. Which kernel threads are freezable?
>  Kernel threads are not freezable by default.  However, a kernel thread may clear
>  PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE
>  directly is not allowed).  From this point it is regarded as freezable
> -and must call try_to_freeze() in a suitable place.
> +and must call try_to_freeze() or variants of wait_event_freezable() in a
> +suitable place.
>
>  IV. Why do we do that?
>  ======================
> --

Applied as 6.8 material with some edits in the subject and changelog.

I have also edited some new documentation language added by the patch
for clarity etc.

Thanks!

Re: [PATCH] docs: power: Adjust freezing-of-tasks.rst due to the freezer logic changes

[Kevin Hao]

[-- Attachment #1: Type: text/plain, Size: 265 bytes --]

On Tue, Dec 19, 2023 at 09:16:34PM +0100, Rafael J. Wysocki wrote:
> Applied as 6.8 material with some edits in the subject and changelog.
> 
> I have also edited some new documentation language added by the patch
> for clarity etc.

Thanks Rafael.

Kevin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]