[PATCH 0/5] arm64: hibernate: Fix DEBUG_PAGEALLOC and hibernate on non-boot cpu

[PATCH 0/5] arm64: hibernate: Fix DEBUG_PAGEALLOC and hibernate on non-boot cpu

[James Morse]

Hi all,

These patches improve the hibernate support for v4.8.

Patches 1&2 fix the hibernate problem with DEBUG_PAGEALLOC reported by Will.
This also addresses the outstanding comment from Catalin [0] regarding cleaning
of the whole kernel to the PoC. Now we clean just the sections grouped by
'TEXT_TEXT'.

Patch one adds sections.h to avoid duplicate definitions of
__hyp_idmap_text_start. (it looks like kprobes will duplicate this too [1)

The rest of the patches fix hibernate/resume on any CPU as discussed on the
list[2]. Suzuki's maxcpus improvements mean the CPU we hibernated on may be
offline during boot, and kexec will allow the CPUs logical order to be shuffled.
Log output below[3], suspend happens on cpu0, with mpidr 0x103. For
resume, cpu5 is brought up as it has mpidr 0x103, and cpu0 is taken down.
After resume, /proc/cpuinfo shows only the original cpu0.


[0] http://www.spinics.net/lists/arm-kernel/msg499305.html
[1] http://www.spinics.net/lists/arm-kernel/msg506451.html
[2] http://www.spinics.net/lists/arm-kernel/msg499036.html
[3] hotplug cpu0, kexec, hibernate, resume
-------------------------%<-------------------------
root@ubuntu:~# echo disk > /sys/power/state
PM: Syncing filesystems ... done.
Freezing user space processes ... (elapsed 0.007 seconds) done.
PM: Preallocating image memory... done (allocated 10112 pages)
PM: Allocated 647168 kbytes in 2.69 seconds (240.58 MB/s)
Freezing remaining freezable tasks ... (elapsed 0.005 seconds) done.
PM: freeze of devices complete after 27.013 msecs
PM: late freeze of devices complete after 20.624 msecs
PM: noirq freeze of devices complete after 22.211 msecs
Disabling non-boot CPUs ...
PM: Creating hibernation image:
PM: Need to copy 10103 pages
PM: Hibernation image created (10103 pages copied)
PM: noirq thaw of devices complete after 22.350 msecs
PM: early thaw of devices complete after 23.680 msecs
PM: thaw of devices complete after 98.331 msecs
hibernate: Suspending on cpu 0 [mpidr:0x103]
PM: Using 1 thread(s) for compression.
PM: Compressing and saving image data (10105 pages)...
PM: Image saving progress:   0%
atkbd serio0: keyboard reset failed on 1c060000.kmi
atkbd serio1: keyboard reset failed on 1c070000.kmi
PM: Image saving progress:  10%
PM: Image saving progress:  20%
PM: Image saving progress:  30%
PM: Image saving progress:  40%
PM: Image saving progress:  50%
PM: Image saving progress:  60%
PM: Image saving progress:  70%
PM: Image saving progress:  80%
PM: Image saving progress:  90%
PM: Image saving progress: 100%
PM: Image saving done.
PM: Wrote 646720 kbytes in 93.08 seconds (6.94 MB/s)
PM: S|
reboot: Power down

[ ... ]

Freezing user space processes ... (elapsed 0.000 seconds) done.
PM: Using 1 thread(s) for decompression.
PM: Loading and decompressing image data (10105 pages)...
hibernate: Suspended on cpu 5 [mpidr:0x103]
hibernate: Suspended on a CPU that is offline! Brining CPU up.
Detected VIPT I-cache on CPU5
CPU5: Booted secondary processor [410fd033]
random: nonblocking pool is initialized
PM: Image loading progress:   0%
PM: Image loading progress:  10%
PM: Image loading progress:  20%
PM: Image loading progress:  30%
PM: Image loading progress:  40%
PM: Image loading progress:  50%
PM: Image loading progress:  60%
PM: Image loading progress:  70%
PM: Image loading progress:  80%
PM: Image loading progress:  90%
PM: Image loading progress: 100%
PM: Image loading done.
PM: Read 646720 kbytes in 30.47 seconds (21.22 MB/s)
PM: quiesce of devices complete after 32.958 msecs
PM: late quiesce of devices complete after 11.574 msecs
PM: noirq quiesce of devices complete after 24.918 msecs
hibernate: Disabling secondary CPUs ...
IRQ1 no longer affine to CPU0
IRQ6 no longer affine to CPU0
IRQ28 no longer affine to CPU0
IRQ29 no longer affine to CPU0
IRQ32 no longer affine to CPU0
IRQ34 no longer affine to CPU0
IRQ35 no longer affine to CPU0
IRQ37 no longer affine to CPU0
IRQ41 no longer affine to CPU0
IRQ48 no longer affine to CPU0
CPU0: shutdown

psci: CPU0 killed.
PM: noirq restore of devices complete after 27.419 msecs
PM: early restore of devices complete after 23.554 msecs
PM: restore of devices complete after 113.188 msecs
Restarting tasks ... done.
root@ubuntu:~# atkbd serio0: keyboard reset failed on 1c060000.kmi
atkbd serio1: keyboard reset failed on 1c070000.kmi
root@ubuntu:~# cat /proc/cpuinfo
processor       : 0
BogoMIPS        : 100.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 3

root@ubuntu:~#
-------------------------%<-------------------------

James Morse (5):
  arm64: Add markers around .text section and create sections.h
  arm64: hibernate: Support DEBUG_PAGEALLOC
  PM / Hibernate: Allow architectures to specify the hibernate/resume
    CPU
  arm64: hibernate: Identify the CPU to resume on by its MPIDR
  Revert "arm64: hibernate: Refuse to hibernate if the boot cpu is
    offline"

 arch/arm64/Kconfig                |   1 -
 arch/arm64/include/asm/Kbuild     |   1 -
 arch/arm64/include/asm/sections.h |  29 ++++++++
 arch/arm64/include/asm/suspend.h  |   1 +
 arch/arm64/include/asm/traps.h    |   3 +-
 arch/arm64/include/asm/virt.h     |   5 +-
 arch/arm64/kernel/alternative.c   |   7 +-
 arch/arm64/kernel/hibernate.c     | 137 ++++++++++++++++++++++++++++----------
 arch/arm64/kernel/vmlinux.lds.S   |   2 +
 arch/arm64/kvm/reset.c            |   3 +-
 arch/arm64/mm/pageattr.c          |  40 ++++++++++-
 kernel/power/hibernate.c          |  16 ++++-
 12 files changed, 191 insertions(+), 54 deletions(-)
 create mode 100644 arch/arm64/include/asm/sections.h

-- 
2.8.0.rc3

[PATCH 3/5] PM / Hibernate: Allow architectures to specify the hibernate/resume CPU

[James Morse]

On arm64 the cpu with logical id 0 is assumed to be the boot CPU. If a
user hotplugs this CPU out, then uses kexec to boot a new kernel, the new
kernel will assign logical id 0 to a different physical CPU.
This breaks hibernate as hibernate and resume will be attempted on different
CPUs.

Define a weak symbol arch_hibernation_disable_cpus(), which defaults to
calling disable_nonboot_cpus). Architectures that allow CPU 0 to be
hotplugged can use this to control which CPU is used for hibernate/resume.

Signed-off-by: James Morse <james.morse@arm.com>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Pavel Machek <pavel@ucw.cz>
---
I would have preferred a macro, but there is no hibernate-relevant header
file that all arch's have. arm, arm64 and x86 have a suspend.h, but powerpc
doesn't.

If this approach is acceptable, this patch should go with 4&5 via the arm64
tree.

 kernel/power/hibernate.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/kernel/power/hibernate.c b/kernel/power/hibernate.c
index fca9254280ee..0e668a3207ec 100644
--- a/kernel/power/hibernate.c
+++ b/kernel/power/hibernate.c
@@ -70,6 +70,16 @@ bool hibernation_available(void)
 }
 
 /**
+ * arch_hibernation_disable_cpus - Allow architectures to direct which CPU
+ * is used for suspend or resume.
+ * @suspend: True during hibernate, false for resume.
+ */
+int __weak arch_hibernation_disable_cpus(__maybe_unused bool suspend)
+{
+	return disable_nonboot_cpus();
+}
+
+/**
  * hibernation_set_ops - Set the global hibernate operations.
  * @ops: Hibernation operations to use in subsequent hibernation transitions.
  */
@@ -279,7 +289,7 @@ static int create_image(int platform_mode)
 	if (error || hibernation_test(TEST_PLATFORM))
 		goto Platform_finish;
 
-	error = disable_nonboot_cpus();
+	error = arch_hibernation_disable_cpus(true);
 	if (error || hibernation_test(TEST_CPUS))
 		goto Enable_cpus;
 
@@ -433,7 +443,7 @@ static int resume_target_kernel(bool platform_mode)
 	if (error)
 		goto Cleanup;
 
-	error = disable_nonboot_cpus();
+	error = arch_hibernation_disable_cpus(false);
 	if (error)
 		goto Enable_cpus;
 
@@ -551,7 +561,7 @@ int hibernation_platform_enter(void)
 	if (error)
 		goto Platform_finish;
 
-	error = disable_nonboot_cpus();
+	error = arch_hibernation_disable_cpus(true);
 	if (error)
 		goto Enable_cpus;
 
-- 
2.8.0.rc3

Re: [PATCH 3/5] PM / Hibernate: Allow architectures to specify the hibernate/resume CPU

[Rafael J. Wysocki]

On Thursday, June 02, 2016 03:10:55 PM James Morse wrote:
> On arm64 the cpu with logical id 0 is assumed to be the boot CPU. If a
> user hotplugs this CPU out, then uses kexec to boot a new kernel, the new
> kernel will assign logical id 0 to a different physical CPU.
> This breaks hibernate as hibernate and resume will be attempted on different
> CPUs.
> 
> Define a weak symbol arch_hibernation_disable_cpus(), which defaults to
> calling disable_nonboot_cpus). Architectures that allow CPU 0 to be
> hotplugged can use this to control which CPU is used for hibernate/resume.
> 
> Signed-off-by: James Morse <james.morse@arm.com>
> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> Cc: Pavel Machek <pavel@ucw.cz>
> ---
> I would have preferred a macro, but there is no hibernate-relevant header
> file that all arch's have. arm, arm64 and x86 have a suspend.h, but powerpc
> doesn't.

What about include/linux/suspend.h?

There are arch_ things declared in there.

Thanks,
Rafael

Re: [PATCH 3/5] PM / Hibernate: Allow architectures to specify the hibernate/resume CPU

[Rafael J. Wysocki]

On Tuesday, June 14, 2016 12:05:28 AM Rafael J. Wysocki wrote:
> On Thursday, June 02, 2016 03:10:55 PM James Morse wrote:
> > On arm64 the cpu with logical id 0 is assumed to be the boot CPU. If a
> > user hotplugs this CPU out, then uses kexec to boot a new kernel, the new
> > kernel will assign logical id 0 to a different physical CPU.
> > This breaks hibernate as hibernate and resume will be attempted on different
> > CPUs.
> > 
> > Define a weak symbol arch_hibernation_disable_cpus(), which defaults to
> > calling disable_nonboot_cpus). Architectures that allow CPU 0 to be
> > hotplugged can use this to control which CPU is used for hibernate/resume.
> > 
> > Signed-off-by: James Morse <james.morse@arm.com>
> > Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> > Cc: Pavel Machek <pavel@ucw.cz>
> > ---
> > I would have preferred a macro, but there is no hibernate-relevant header
> > file that all arch's have. arm, arm64 and x86 have a suspend.h, but powerpc
> > doesn't.
> 
> What about include/linux/suspend.h?
> 
> There are arch_ things declared in there.

I mean, what about using something like

#ifndef arch_hibernation_disable_cpus
#define arch_hibernation_disable_cpus disable_nonboot_cpus
#endif

in there or in another header file?

Thanks,
Rafael

Re: [PATCH 3/5] PM / Hibernate: Allow architectures to specify the hibernate/resume CPU

[James Morse]

Hi Rafael,

On 13/06/16 23:10, Rafael J. Wysocki wrote:
> On Tuesday, June 14, 2016 12:05:28 AM Rafael J. Wysocki wrote:
>> On Thursday, June 02, 2016 03:10:55 PM James Morse wrote:
>>> On arm64 the cpu with logical id 0 is assumed to be the boot CPU. If a
>>> user hotplugs this CPU out, then uses kexec to boot a new kernel, the new
>>> kernel will assign logical id 0 to a different physical CPU.
>>> This breaks hibernate as hibernate and resume will be attempted on different
>>> CPUs.
>>>
>>> Define a weak symbol arch_hibernation_disable_cpus(), which defaults to
>>> calling disable_nonboot_cpus). Architectures that allow CPU 0 to be
>>> hotplugged can use this to control which CPU is used for hibernate/resume.
>>>
>>> Signed-off-by: James Morse <james.morse@arm.com>
>>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
>>> Cc: Pavel Machek <pavel@ucw.cz>
>>> ---
>>> I would have preferred a macro, but there is no hibernate-relevant header
>>> file that all arch's have. arm, arm64 and x86 have a suspend.h, but powerpc
>>> doesn't.
>>
>> What about include/linux/suspend.h?
>>
>> There are arch_ things declared in there.
> 
> I mean, what about using something like
> 
> #ifndef arch_hibernation_disable_cpus
> #define arch_hibernation_disable_cpus disable_nonboot_cpus
> #endif
> 
> in there or in another header file?

This would work fine for an architecture that doesn't need to use
arch_hibernation_disable_cpus(), but which header file should we use for an
architecture that does?

As a macro it needs to be in an arch-specific header included before the #ifndef
above.

Of those:
include/linux/suspend.h only includes <asm/errno.h>
kernel/power/power.h doesn't include any 'asm' headers.
snapshot.c includes:
> #include <asm/uaccess.h>
> #include <asm/mmu_context.h>
> #include <asm/pgtable.h>
> #include <asm/tlbflush.h>
> #include <asm/io.h>

... but none of those are appropriate.


I initially added an include for <asm/suspend.h> to kernel/power/power.h, but it
broke powerpc, as there is no suspend.h there. I took this as a dead-end for the
macro approach, unless someone has a clever trick?!



Thanks,

James

Re: [PATCH 3/5] PM / Hibernate: Allow architectures to specify the hibernate/resume CPU

[Rafael J. Wysocki]

On Tuesday, June 14, 2016 10:44:47 AM James Morse wrote:
> Hi Rafael,
> 
> On 13/06/16 23:10, Rafael J. Wysocki wrote:
> > On Tuesday, June 14, 2016 12:05:28 AM Rafael J. Wysocki wrote:
> >> On Thursday, June 02, 2016 03:10:55 PM James Morse wrote:
> >>> On arm64 the cpu with logical id 0 is assumed to be the boot CPU. If a
> >>> user hotplugs this CPU out, then uses kexec to boot a new kernel, the new
> >>> kernel will assign logical id 0 to a different physical CPU.
> >>> This breaks hibernate as hibernate and resume will be attempted on different
> >>> CPUs.
> >>>
> >>> Define a weak symbol arch_hibernation_disable_cpus(), which defaults to
> >>> calling disable_nonboot_cpus). Architectures that allow CPU 0 to be
> >>> hotplugged can use this to control which CPU is used for hibernate/resume.
> >>>
> >>> Signed-off-by: James Morse <james.morse@arm.com>
> >>> Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
> >>> Cc: Pavel Machek <pavel@ucw.cz>
> >>> ---
> >>> I would have preferred a macro, but there is no hibernate-relevant header
> >>> file that all arch's have. arm, arm64 and x86 have a suspend.h, but powerpc
> >>> doesn't.
> >>
> >> What about include/linux/suspend.h?
> >>
> >> There are arch_ things declared in there.
> > 
> > I mean, what about using something like
> > 
> > #ifndef arch_hibernation_disable_cpus
> > #define arch_hibernation_disable_cpus disable_nonboot_cpus
> > #endif
> > 
> > in there or in another header file?
> 
> This would work fine for an architecture that doesn't need to use
> arch_hibernation_disable_cpus(), but which header file should we use for an
> architecture that does?
> 
> As a macro it needs to be in an arch-specific header included before the #ifndef
> above.
> 
> Of those:
> include/linux/suspend.h only includes <asm/errno.h>
> kernel/power/power.h doesn't include any 'asm' headers.
> snapshot.c includes:
> > #include <asm/uaccess.h>
> > #include <asm/mmu_context.h>
> > #include <asm/pgtable.h>
> > #include <asm/tlbflush.h>
> > #include <asm/io.h>
> 
> ... but none of those are appropriate.
> 
> 
> I initially added an include for <asm/suspend.h> to kernel/power/power.h, but it
> broke powerpc, as there is no suspend.h there. I took this as a dead-end for the
> macro approach, unless someone has a clever trick?!

OK, let's do the __weak function, then.

After all, it can be changed later if someone has a better idea.

Thanks,
Rafael