[PATCH v4 00/13] arm64: kernel: Add support for hibernate/suspend-to-disk

[PATCH v4 00/13] arm64: kernel: Add support for hibernate/suspend-to-disk

[James Morse]

Hi all,

This version of hibernate is rebased onto v4.5-rc1, including updated patches
shared with kexec v13 [0] (1-5, 10).

I've also tested it with Ard's KASLR series [1].
The most significant change is the use of the arch-header to store the
location of the page tables and resume code when it has been moved by KASLR.

This implicitly allows resuming with a different kernel version, (and the work
needed to do this is the same). Patch 12 restricts resume to kernels with the
same MMU configuration, (page size, number of page levels etc).
Patch 13 forbids restoring with a different kernel version altogether - do
we want to support this?

Due to the arch-header changes, I dropped Pavel's Ack on patch 12.

Parts of patch 5 were reworked from Akashi Takahiro's original version, to
avoid another round of changes to hyp-entry.S:el1_sync.

This series can be retrieved from:
git://linux-arm.org/linux-jm.git -b hibernate/v4

Changes since v3:
 * To work with kaslr:
   * hibernate now uses the arch-header to store the address of the page
     tables, and the point to re-enter the resumed kernel.
   * The el2 vectors are reloaded to point to the 'safe' page, then back to the
     resumed kernel.
   * PoC cleaning is done after the jump to the resumed kernel, as we don't
     know the restored kernel's boundaries in advance.
   * Some variables are accessed via aliases in the linear map, as the kernel
     text is not mapped during resume. restore_pblist is one example.
   * Execute the safe page from the bottom of memory, not the top, so that we
     can restore the resumed kernel's page tables directly.

 * Rebased the common patches onto v13 of kexec
 * Changed hibernate-asm.S to use the new copy_page macro.
 * Changed copy_p?d()s to use the do { } while(); pattern.
 * Added some missing barriers. (dsb after ic ialluis).

Changes from v2:
 * Rewrote restore in-place patch - we can't clean pages in copy_page(), we
   need to publish a list for the architecture to clean
 * Added missing pgprot_val() in hibernate.c, spotted by STRICT_MM_TYPECHECKS
 * Removed 'tcr_set_idmap_t0sz' from proc.S - I missed this when rebase-ing
 * Re-imported the first four patches from kexec v12
 * Rebased onto v4.4-rc2
 * Changes from Pavel Machek's comments

Changes from v1:
 * Removed for_each_process(){ for_each_vma() { } }; cache cleaning, replaced
   with icache_flush_range() call in core hibernate code
 * Rebased onto conflicting tcr_ek1.t0sz bug-fix patch


[v3] http://www.spinics.net/lists/arm-kernel/msg463590.html
[v2] http://lists.infradead.org/pipermail/linux-arm-kernel/2015-October/376450.html
[v1] http://lists.infradead.org/pipermail/linux-arm-kernel/2015-October/376450.html

[0] http://www.spinics.net/lists/arm-kernel/msg474889.html
[1] http://permalink.gmane.org/gmane.linux.kernel/2116531

AKASHI Takahiro (1):
  arm64: kvm: allows kvm cpu hotplug

Geoff Levand (5):
  arm64: Fold proc-macros.S into assembler.h
  arm64: Cleanup SCTLR flags
  arm64: Convert hcalls to use HVC immediate value
  arm64: Add new hcall HVC_CALL_FUNC
  arm64: Add new asm macro copy_page

James Morse (7):
  arm64: kernel: Rework finisher callback out of __cpu_suspend_enter().
  arm64: Change cpu_resume() to enable mmu early then access sleep_sp by
    va
  arm64: kernel: Include _AC definition in page.h
  arm64: Promote KERNEL_START/KERNEL_END definitions to a header file
  PM / Hibernate: Call flush_icache_range() on pages restored in-place
  arm64: kernel: Add support for hibernate/suspend-to-disk
  arm64: hibernate: Prevent resume from a different kernel version

 arch/arm/include/asm/kvm_host.h    |  10 +-
 arch/arm/include/asm/kvm_mmu.h     |   1 +
 arch/arm/kvm/arm.c                 |  98 +++++---
 arch/arm/kvm/mmu.c                 |   5 +
 arch/arm64/Kconfig                 |   7 +
 arch/arm64/include/asm/assembler.h |  89 ++++++-
 arch/arm64/include/asm/kvm_arm.h   |  11 -
 arch/arm64/include/asm/kvm_host.h  |   1 -
 arch/arm64/include/asm/kvm_mmu.h   |  19 ++
 arch/arm64/include/asm/memory.h    |   3 +
 arch/arm64/include/asm/page.h      |   2 +
 arch/arm64/include/asm/suspend.h   |  30 ++-
 arch/arm64/include/asm/sysreg.h    |  19 +-
 arch/arm64/include/asm/virt.h      |  40 ++++
 arch/arm64/kernel/Makefile         |   1 +
 arch/arm64/kernel/asm-offsets.c    |  10 +-
 arch/arm64/kernel/head.S           |   5 +-
 arch/arm64/kernel/hibernate-asm.S  | 137 +++++++++++
 arch/arm64/kernel/hibernate.c      | 472 +++++++++++++++++++++++++++++++++++++
 arch/arm64/kernel/hyp-stub.S       |  43 +++-
 arch/arm64/kernel/setup.c          |   1 -
 arch/arm64/kernel/sleep.S          | 145 ++++--------
 arch/arm64/kernel/suspend.c        | 108 ++++-----
 arch/arm64/kernel/vmlinux.lds.S    |  15 ++
 arch/arm64/kvm/hyp-init.S          |  47 +++-
 arch/arm64/kvm/hyp.S               |   3 +-
 arch/arm64/kvm/hyp/hyp-entry.S     |   9 +-
 arch/arm64/mm/cache.S              |   2 -
 arch/arm64/mm/proc-macros.S        |  86 -------
 arch/arm64/mm/proc.S               |  38 +--
 kernel/power/swap.c                |  18 ++
 31 files changed, 1114 insertions(+), 361 deletions(-)
 create mode 100644 arch/arm64/kernel/hibernate-asm.S
 create mode 100644 arch/arm64/kernel/hibernate.c
 delete mode 100644 arch/arm64/mm/proc-macros.S

-- 
2.6.2

[PATCH v4 11/13] PM / Hibernate: Call flush_icache_range() on pages restored in-place

[James Morse]

Some architectures require code written to memory as if it were data to be
'cleaned' from any data caches before the processor can fetch them as new
instructions.

During resume from hibernate, the snapshot code copies some pages directly,
meaning these architectures do not get a chance to perform their cache
maintenance. Modify the read and decompress code to call
flush_icache_range() on all pages that are restored, so that the restored
in-place pages are guaranteed to be executable on these architectures.

Signed-off-by: James Morse <james.morse@arm.com>
---
 kernel/power/swap.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/kernel/power/swap.c b/kernel/power/swap.c
index 12cd989dadf6..a30645d2e93f 100644
--- a/kernel/power/swap.c
+++ b/kernel/power/swap.c
@@ -37,6 +37,14 @@
 #define HIBERNATE_SIG	"S1SUSPEND"
 
 /*
+ * When reading an {un,}compressed image, we may restore pages in place,
+ * in which case some architectures need these pages cleaning before they
+ * can be executed. We don't know which pages these may be, so clean the lot.
+ */
+bool clean_pages_on_read = false;
+bool clean_pages_on_decompress = false;
+
+/*
  *	The swap map is a data structure used for keeping track of each page
  *	written to a swap partition.  It consists of many swap_map_page
  *	structures that contain each an array of MAP_PAGE_ENTRIES swap entries.
@@ -241,6 +249,9 @@ static void hib_end_io(struct bio *bio)
 
 	if (bio_data_dir(bio) == WRITE)
 		put_page(page);
+	else if (clean_pages_on_read)
+		flush_icache_range((unsigned long)page_address(page),
+				   (unsigned long)page_address(page) + PAGE_SIZE);
 
 	if (bio->bi_error && !hb->error)
 		hb->error = bio->bi_error;
@@ -1049,6 +1060,7 @@ static int load_image(struct swap_map_handle *handle,
 
 	hib_init_batch(&hb);
 
+	clean_pages_on_read = true;
 	printk(KERN_INFO "PM: Loading image data pages (%u pages)...\n",
 		nr_to_read);
 	m = nr_to_read / 10;
@@ -1124,6 +1136,10 @@ static int lzo_decompress_threadfn(void *data)
 		d->unc_len = LZO_UNC_SIZE;
 		d->ret = lzo1x_decompress_safe(d->cmp + LZO_HEADER, d->cmp_len,
 		                               d->unc, &d->unc_len);
+		if (clean_pages_on_decompress)
+			flush_icache_range((unsigned long)d->unc,
+					   (unsigned long)d->unc + d->unc_len);
+
 		atomic_set(&d->stop, 1);
 		wake_up(&d->done);
 	}
@@ -1189,6 +1205,8 @@ static int load_image_lzo(struct swap_map_handle *handle,
 	}
 	memset(crc, 0, offsetof(struct crc_data, go));
 
+	clean_pages_on_decompress = true;
+
 	/*
 	 * Start the decompression threads.
 	 */
-- 
2.6.2

Re: [PATCH v4 00/13] arm64: kernel: Add support for hibernate/suspend-to-disk

[Kevin Hilman]

Hi James,

James Morse <james.morse@arm.com> writes:

> This version of hibernate is rebased onto v4.5-rc1, including updated patches
> shared with kexec v13 [0] (1-5, 10).

Thanks for this series!

I'd like to help in testing this so I'm just curious which platforms
you've been testing this on.  I'm assuming a Juno (r2?), anything else?

Are you testing the resume from cold boot, or just from kexec?

For cold boot on Juno, I'm assuming there would be some
booloader/firmware changes needed to find and boot from the hibernation
image?  Is that being worked on?  If not Juno, are you aware of any
other platforms that could be tested with resume from cold boot?

Not knowing the answers to the above, I tested your branch using arm64
defconfig + CONFIG_HIBERNATION=y on my Juno and noticed that it didn't
stay suspended (full suspend log below) so I'm looking for
ideas/recommenations on how to test this.

FWIW, my Juno is running the latest ATF + u-boot firmware from the
Linaro ARM landing team.

Kevin

/ # echo disk > /sys/power/state
[   29.563430] PM: Syncing filesystems ... done.
[   29.567901] Freezing user space processes ... (elapsed 0.001 seconds) done.
[   29.576989] PM: Preallocating image memory... done (allocated 65512 pages)
[   31.969977] PM: Allocated 262048 kbytes in 2.38 seconds (110.10 MB/s)
[   31.976362] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[   31.985303] Suspending console(s) (use no_console_suspend to debug)
[   31.992900] PM: freeze of devices complete after 1.370 msecs
[   31.993877] PM: late freeze of devices complete after 0.960 msecs
[   31.995109] PM: noirq freeze of devices complete after 1.225 msecs
[   31.995112] Disabling non-boot CPUs ...
[   32.012751] CPU1: shutdown
[   32.028584] psci: Retrying again to check for CPU kill
[   32.028587] psci: CPU1 killed.
[   32.060689] CPU2: shutdown
[   32.076579] psci: Retrying again to check for CPU kill
[   32.076583] psci: CPU2 killed.
[   32.108734] CPU3: shutdown
[   32.124579] psci: Retrying again to check for CPU kill
[   32.124582] psci: CPU3 killed.
[   32.160690] CPU4: shutdown
[   32.176578] psci: Retrying again to check for CPU kill
[   32.176581] psci: CPU4 killed.
[   32.212685] CPU5: shutdown
[   32.228580] psci: Retrying again to check for CPU kill
[   32.228584] psci: CPU5 killed.
[   32.241393] PM: Creating hibernation image:
[   32.241393] PM: Need to copy 63772 pages
[   32.241393] PM: Hibernation image created (63772 pages copied)
[   32.241440] Enabling non-boot CPUs ...
[   32.274634] Detected PIPT I-cache on CPU1
[   32.274680] CPU1: Booted secondary processor [410fd080]
[   32.274896]  cache: parent cpu1 should not be sleeping
[   32.275095] CPU1 is up
[   32.306712] Detected PIPT I-cache on CPU2
[   32.306738] CPU2: Booted secondary processor [410fd080]
[   32.306922]  cache: parent cpu2 should not be sleeping
[   32.307120] CPU2 is up
[   32.338913] Detected VIPT I-cache on CPU3
[   32.338959] CPU3: Booted secondary processor [410fd033]
[   32.339184]  cache: parent cpu3 should not be sleeping
[   32.339378] CPU3 is up
[   32.371098] Detected VIPT I-cache on CPU4
[   32.371125] CPU4: Booted secondary processor [410fd033]
[   32.371341]  cache: parent cpu4 should not be sleeping
[   32.371536] CPU4 is up
[   32.403281] Detected VIPT I-cache on CPU5
[   32.403309] CPU5: Booted secondary processor [410fd033]
[   32.403531]  cache: parent cpu5 should not be sleeping
[   32.403741] CPU5 is up
[   32.404202] PM: noirq thaw of devices complete after 0.454 msecs
[   32.404979] PM: early thaw of devices complete after 0.723 msecs
[...]

Re: [PATCH v4 11/13] PM / Hibernate: Call flush_icache_range() on pages restored in-place

[Pavel Machek]

On Thu 2016-01-28 10:42:44, James Morse wrote:
> Some architectures require code written to memory as if it were data to be
> 'cleaned' from any data caches before the processor can fetch them as new
> instructions.
> 
> During resume from hibernate, the snapshot code copies some pages directly,
> meaning these architectures do not get a chance to perform their cache
> maintenance. Modify the read and decompress code to call
> flush_icache_range() on all pages that are restored, so that the restored
> in-place pages are guaranteed to be executable on these architectures.
> 
> Signed-off-by: James Morse <james.morse@arm.com>

Acked-by: Pavel Machek <pavel@ucw.cz>

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [PATCH v4 00/13] arm64: kernel: Add support for hibernate/suspend-to-disk

[James Morse]

Hi Kevin,

On 29/01/16 22:34, Kevin Hilman wrote:
> James Morse <james.morse@arm.com> writes:
> I'd like to help in testing this so I'm just curious which platforms
> you've been testing this on.  I'm assuming a Juno (r2?), anything else?

That would be great - thanks!

I've done most of the testing on a Juno r1, but also gave it a spin on a
stray Seattle.


> Are you testing the resume from cold boot, or just from kexec?

>From cold boot. I haven't tried with kexec, but I doubt that's a use-case anyone
wants as you would resume immediately. (might be interesting for testing though)


> For cold boot on Juno, I'm assuming there would be some
> booloader/firmware changes needed to find and boot from the hibernation
> image?

Not at all! Your firmware only needs to support some mechanism to
turning CPUs off.

If you add 'resume=/dev/sda2' (or wherever your swap partition is located), the
kernel will check this partition for the hibernate-image signature, if found, it
will resume from that partition. Otherwise booting is as normal. (there is one
hoop to jump through to ensure your rootfs hasn't been mounted before the kernel
starts resume, as you could corrupt it - an initramfs in the kernel is the best
fix for this).

No firmware changes needed.

> Is that being worked on?  If not Juno, are you aware of any
> other platforms that could be tested with resume from cold boot?

Any arm64 platform with persistent storage should work. I've been using a swap
partition on a usb drive.


> Not knowing the answers to the above, I tested your branch using arm64
> defconfig + CONFIG_HIBERNATION=y on my Juno and noticed that it didn't
> stay suspended (full suspend log below) so I'm looking for
> ideas/recommenations on how to test this.

That trace looks quite normal, (one of mine below[0] for comparison). Any
failure would have happened after the point you stopped ... did you have a swap
partition 'on'?

'syscore' will freeze all processes and stop all devices, then create a
copy of the minimum amount of memory it needs to save. Then it starts
all the devices again, as it needs to write this image out to swap. This is what
you are seeing.

Once it has done this it calls poweroff or reboot.


Thanks,

James



[0] kernel output for hibernate/resume on Juno-r1:
root@localhost:/sys/power# swapon /dev/sda2
Adding 236540k swap on /dev/sda2.  Priority:-1 extents:1 across:236540k
root@localhost:/sys/power# ls
disk        pm_async           reserved_size  state
image_size  pm_freeze_timeout  resume         wakeup_count
root@localhost:/sys/power# cat disk
[shutdown] reboot suspend
root@localhost:/sys/power# echo disk > state
PM: Syncing filesystems ... done.
Freezing user space processes ... (elapsed 0.001 seconds) done.
PM: Preallocating image memory... done (allocated 93967 pages)
PM: Allocated 375868 kbytes in 3.74 seconds (100.49 MB/s)
Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
PM: freeze of devices complete after 17.794 msecs
PM: late freeze of devices complete after 1.358 msecs
PM: noirq freeze of devices complete after 1.771 msecs
Disabling non-boot CPUs ...
PM: Creating hibernation image:
PM: Need to copy 93868 pages
PM: Hibernation image created (93868 pages copied)
PM: noirq thaw of devices complete after 1.319 msecs
PM: early thaw of devices complete after 0.890 msecs
serio serio0: device_attach() failed for 1c060000.kmi (1c060000.kmi), error: -51
7
serio serio1: device_attach() failed for 1c070000.kmi (1c070000.kmi), error: -51
7
PM: thaw of devices complete after 92.523 msecs
PM: Using 1 thread(s) for compression.
PM: Compressing and saving image data (94052 pages)...
PM: Image saving progress:   0%
atkbd serio0: keyboard reset failed on 1c060000.kmi
atkbd serio1: keyboard reset failed on 1c070000.kmi
PM: Image saving progress:  10%
PM: Image saving progress:  20%
PM: Image saving progress:  30%
PM: Image saving progress:  40%
PM: Image saving progress:  50%
PM: Image saving progress:  60%
PM: Image saving progress:  70%
PM: Image saving progress:  80%
PM: Image saving progress:  90%
PM: Image saving progress: 100%
PM: Image saving done.
PM: Wrote 376208 kbytes in 21.76 seconds (17.28 MB/s)
PM: S|
kvm: exiting hardware virtualization
reboot: Power down

Board powered down, use REBOOT to restart.

Cmd>

ARM V2M-Juno Boot loader v1.0.0
HBI0262 build 1635

ARM V2M_Juno r1 Firmware v1.3.3
Build Date: Mar 31 2015

Time :  08:25:55
Date :  01:02:2016

Press Enter to stop auto boot...
Powering up system...

Switching on ATXPSU...
PMIC RAM configuration (pms_v104.bin)...
MBtemp   : 28 degC

Configuring motherboard (rev C, var A)...
IOFPGA image \MB\HBI0262C\io_b117.bit
IOFPGA  config: PASSED
OSC CLK config: PASSED

Configuring SCC registers...
Writing SCC 0x00000054 with 0x0007FFFE
Writing SCC 0x0000005C with 0x00FE001E
Writing SCC 0x00000100 with 0x003F1000
Writing SCC 0x00000104 with 0x0001F300
Writing SCC 0x00000108 with 0x00331000
Writing SCC 0x0000010C with 0x00019300
Writing SCC 0x00000118 with 0x003F1000
Writing SCC 0x0000011C with 0x0001F100
Writing SCC 0x000000F4 with 0x00000018
Writing SCC 0x000000F8 with 0x0BEC0000
Writing SCC 0x000000FC with 0xABE40000
Writing SCC 0x00000A14 with 0x00000000
Writing SCC 0x0000000C with 0x000000C2
Writing SCC 0x00000010 with 0x000000C2

Peripheral ID0:0x000000AD
Peripheral ID1:0x000000B0
Peripheral ID2:0x0000001B
Peripheral ID3:0x00000000
Peripheral ID4:0x0000000D
Peripheral ID5:0x000000F0
Peripheral ID6:0x00000005
Peripheral ID7:0x000000B1

Programming NOR Flash
PCIE clock configured...

Testing motherboard interfaces (FPGA build 117)...
SRAM 32MB test: PASSED
LAN9118   test: PASSED
KMI1/2    test: PASSED
MMC       test: PASSED
PB/LEDs   test: PASSED
FPGA UART test: PASSED
PCIe init test: PASSED
MAC addrs test: PASSED

The default boot selection will start in   1 seconds
Downloading the file <Image> from the TFTP server
[=======================================>]    9656 Kb
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
Booting Linux on physical CPU 0x100
Linux version 4.5.0-rc1+ (morse@melchizedek) (gcc version 4.9.3 20141031 (prerel
ease) (Linaro GCC 2014.11) ) #1839 SMP PREEMPT Fri Jan 29 15:05:57 GMT 2016
Boot CPU: AArch64 Processor [410fd033]
earlycon: Early serial console at MMIO 0x7ff80000 (options '')
bootconsole [uart0] enabled
efi: Getting EFI parameters from FDT:
EFI v2.50 by ARM Juno EFI Nov 24 2015 12:36:35
efi:  ACPI=0xf95b0000  ACPI 2.0=0xf95b0014  PROP=0xfe8db4d8
cma: Reserved 16 MiB at 0x00000000fd800000
psci: probing for conduit method from DT.
psci: PSCIv1.0 detected in firmware.
psci: Using standard PSCI v0.2 function IDs
psci: Trusted OS migration not required
PERCPU: Embedded 20 pages/cpu @ffff80097ff4f000 s42240 r8192 d31488 u81920
Detected VIPT I-cache on CPU0
CPU features: enabling workaround for ARM erratum 845719
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 2060048
Kernel command line: console=ttyAMA0,115200 earlycon=pl011,0x7ff80000 root=/dev/
nfs nfsroot=10.xx.xx.xx:xx/aarch64-current resume=/dev/sda2 no_console_s
uspend ip=dhcp rw init=/bin/bash crashkernel=256M maxcpus=1
log_buf_len individual max cpu contribution: 4096 bytes
log_buf_len total cpu_extra contributions: 20480 bytes
log_buf_len min size: 16384 bytes
log_buf_len: 65536 bytes
early log buf free: 14304(87%)
PID hash table entries: 4096 (order: 3, 32768 bytes)
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
software IO TLB [mem 0xf54a0000-0xf94a0000] (64MB) mapped at [ffff8000754a0000-f
fff80007949ffff]
Memory: 8130452K/8371264K available (6007K kernel code, 524K rwdata, 2524K rodat
a, 592K init, 229K bss, 224428K reserved, 16384K cma-reserved)
Virtual kernel memory layout:
    vmalloc : 0xffff000000000000 - 0xffff7bffbfff0000   (126974 GB)
    vmemmap : 0xffff7bffc0000000 - 0xffff7fffc0000000   (  4096 GB maximum)
              0xffff7bffc2000000 - 0xffff7bffe8000000   (   608 MB actual)
    fixed   : 0xffff7ffffa7fd000 - 0xffff7ffffac00000   (  4108 KB)
    PCI I/O : 0xffff7ffffae00000 - 0xffff7ffffbe00000   (    16 MB)
    modules : 0xffff7ffffc000000 - 0xffff800000000000   (    64 MB)
    memory  : 0xffff800000000000 - 0xffff800980000000   ( 38912 MB)
      .init : 0xffff8000008d7000 - 0xffff80000096b000   (   592 KB)
      .text : 0xffff800000080000 - 0xffff8000008d6c64   (  8540 KB)
      .data : 0xffff80000096b000 - 0xffff8000009ee200   (   525 KB)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=6, Nodes=1
Preemptible hierarchical RCU implementation.
        Build-time adjustment of leaf fanout to 64.
        RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=6.
RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=6
NR_IRQS:64 nr_irqs:64 0
GIC: Using split EOI/Deactivate mode
GICv2m: range[mem 0x2c1c0000-0x2c1c0fff], SPI[224:255]
Architected cp15 and mmio timer(s) running at 50.00MHz (phys/phys).
clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, m
ax_idle_ns: 440795202655 ns
sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
Console: colour dummy device 80x25
Calibrating delay loop (skipped), value calculated using timer frequency.. 100.0
0 BogoMIPS (lpj=200000)
pid_max: default: 32768 minimum: 301
Security Framework initialized
Mount-cache hash table entries: 16384 (order: 5, 131072 bytes)
Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes)
ASID allocator initialised with 65536 entries
Remapping and enabling EFI services.
 EFI remap 0x0000000008000000 => 0000000020000000
  EFI remap 0x000000001c170000 => 0000000024000000
  EFI remap 0x00000000f94a0000 => 0000000024010000
  EFI remap 0x00000000f9520000 => 0000000024020000
  EFI remap 0x00000000f9530000 => 0000000024030000
  EFI remap 0x00000000f9540000 => 0000000024040000
  EFI remap 0x00000000f9570000 => 0000000024070000
  EFI remap 0x00000000f9580000 => 0000000024080000
  EFI remap 0x00000000f95c0000 => 00000000240a0000
  EFI remap 0x00000000f9640000 => 0000000024120000
  EFI remap 0x00000000f9650000 => 0000000024130000
  EFI remap 0x00000000f9720000 => 0000000024200000
  EFI remap 0x00000000f9730000 => 0000000024210000
  EFI remap 0x00000000f9740000 => 0000000024220000
  EFI remap 0x00000000f9780000 => 0000000024260000
  EFI remap 0x00000000f9790000 => 0000000024270000
  EFI remap 0x00000000f9800000 => 00000000242e0000
  EFI remap 0x00000000f9810000 => 00000000242f0000
  EFI remap 0x00000000f9820000 => 0000000024300000
  EFI remap 0x00000000fe820000 => 0000000024330000
  EFI remap 0x00000000fe830000 => 0000000024340000
  EFI remap 0x00000000fe840000 => 0000000024350000
  EFI remap 0x00000000fe870000 => 0000000024370000
Brought up 1 CPUs
SMP: Total of 1 processors activated.
CPU: All CPU(s) started at EL2
alternatives: patching kernel code
devtmpfs: initialized
DMI not present or invalid.
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645
041785100000 ns
pinctrl core: initialized pinctrl subsystem
NET: Registered protocol family 16
cpuidle: using governor ladder
cpuidle: using governor menu
vdso: 2 pages (1 code @ ffff800000972000, 1 data @ ffff800000971000)
hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
DMA: preallocated 256 KiB pool for atomic allocations
Serial: AMBA PL011 UART driver
7ff80000.uart: ttyAMA0 at MMIO 0x7ff80000 (irq = 25, base_baud = 0) is a PL011 r
ev3
console [ttyAMA0] enabled
console [ttyAMA0] enabled
bootconsole [uart0] disabled
bootconsole [uart0] disabled
HugeTLB registered 2 MB page size, pre-allocated 0 pages
ACPI: Interpreter disabled.
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
dmi: Firmware registration failed.
clocksource: Switched to clocksource arch_sys_counter
VFS: Disk quotas dquot_6.6.0
VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
pnp: PnP ACPI: disabled
NET: Registered protocol family 2
TCP established hash table entries: 65536 (order: 7, 524288 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 65536 bind 65536)
UDP hash table entries: 4096 (order: 5, 131072 bytes)
UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
hw perfevents: enabled with armv8_pmuv3 PMU driver, 7 counters available
kvm [1]: interrupt-controller@2c04f000 IRQ14
kvm [1]: timer IRQ4
kvm [1]: 8-bit VMID
kvm [1]: Hyp mode initialized successfully
futex hash table entries: 2048 (order: 6, 262144 bytes)
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(0.432:1): initialized
NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered
fuse init (API version 7.24)
9p: Installing v9fs 9p2000 file system support
io scheduler noop registered
io scheduler cfq registered (default)
PCI host bridge /pcie-controller@30000000 ranges:
   IO 0x5f800000..0x5fffffff -> 0x5f800000
  MEM 0x50000000..0x57ffffff -> 0x50000000
  MEM 0x4000000000..0x40ffffffff -> 0x4000000000
pci-host-generic 40000000.pcie-controller: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [bus 00-ff]
pci_bus 0000:00: root bus resource [io  0x0000-0x7fffff] (bus address [0x5f80000
0-0x5fffffff])
pci_bus 0000:00: root bus resource [mem 0x50000000-0x57ffffff]
pci_bus 0000:00: root bus resource [mem 0x4000000000-0x40ffffffff pref]
pci 0000:03:00.0: reg 0x20: initial BAR value 0x00000000 invalid
pci 0000:03:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with
 'pcie_aspm=force'
pci 0000:08:00.0: reg 0x18: initial BAR value 0x00000000 invalid
pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x501fffff]
pci 0000:00:00.0: BAR 0: assigned [mem 0x4000000000-0x4000003fff 64bit pref]
pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x2fff]
pci 0000:01:00.0: BAR 8: assigned [mem 0x50000000-0x501fffff]
pci 0000:01:00.0: BAR 7: assigned [io  0x1000-0x2fff]
pci 0000:02:01.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
pci 0000:02:1f.0: BAR 8: assigned [mem 0x50100000-0x501fffff]
pci 0000:02:01.0: BAR 7: assigned [io  0x1000-0x1fff]
pci 0000:02:1f.0: BAR 7: assigned [io  0x2000-0x2fff]
pci 0000:03:00.0: BAR 6: assigned [mem 0x50000000-0x5007ffff pref]
pci 0000:03:00.0: BAR 2: assigned [mem 0x50080000-0x50083fff 64bit]
pci 0000:03:00.0: BAR 0: assigned [mem 0x50084000-0x5008407f 64bit]
pci 0000:03:00.0: BAR 4: assigned [io  0x1000-0x107f]
pci 0000:02:01.0: PCI bridge to [bus 03]
pci 0000:02:01.0:   bridge window [io  0x1000-0x1fff]
pci 0000:02:01.0:   bridge window [mem 0x50000000-0x500fffff]
pci 0000:02:02.0: PCI bridge to [bus 04]
pci 0000:02:03.0: PCI bridge to [bus 05]
pci 0000:02:0c.0: PCI bridge to [bus 06]
pci 0000:02:10.0: PCI bridge to [bus 07]
pci 0000:08:00.0: BAR 0: assigned [mem 0x50100000-0x50103fff 64bit]
pci 0000:08:00.0: BAR 2: assigned [io  0x2000-0x20ff]
pci 0000:02:1f.0: PCI bridge to [bus 08]
pci 0000:02:1f.0:   bridge window [io  0x2000-0x2fff]
pci 0000:02:1f.0:   bridge window [mem 0x50100000-0x501fffff]
pci 0000:01:00.0: PCI bridge to [bus 02-08]
pci 0000:01:00.0:   bridge window [io  0x1000-0x2fff]
pci 0000:01:00.0:   bridge window [mem 0x50000000-0x501fffff]
pci 0000:00:00.0: PCI bridge to [bus 01-08]
pci 0000:00:00.0:   bridge window [io  0x1000-0x2fff]
pci 0000:00:00.0:   bridge window [mem 0x50000000-0x501fffff]
pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
pci 0000:02:01.0: Signaling PME through PCIe PME interrupt
pci 0000:03:00.0: Signaling PME through PCIe PME interrupt
pci 0000:02:02.0: Signaling PME through PCIe PME interrupt
pci 0000:02:03.0: Signaling PME through PCIe PME interrupt
pci 0000:02:0c.0: Signaling PME through PCIe PME interrupt
pci 0000:02:10.0: Signaling PME through PCIe PME interrupt
pci 0000:02:1f.0: Signaling PME through PCIe PME interrupt
pci 0000:08:00.0: Signaling PME through PCIe PME interrupt
xenfs: not registering filesystem on non-xen platform
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
msm_serial: driver initialized
loop: module loaded
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
sky2: driver version 1.30
sky2 0000:08:00.0: enabling device (0000 -> 0003)
sky2 0000:08:00.0: Yukon-2 UL 2 chip revision 0
sky2 0000:08:00.0 (unnamed net_device) (uninitialized): Invalid MAC address, def
aulting to random
sky2 0000:08:00.0 eth0: addr c6:94:53:3d:70:00
libphy: smsc911x-mdio: probed
Generic PHY 18000000.etherne:01: attached PHY driver [Generic PHY] (mii_bus:phy_
addr=18000000.etherne:01, irq=-1)
smsc911x 18000000.ethernet eth1: MAC Address: 00:02:f7:00:60:c5
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
ehci-platform: EHCI generic platform driver
ehci-platform 7ffc0000.ehci: EHCI Host Controller
ehci-platform 7ffc0000.ehci: new USB bus registered, assigned bus number 1
ehci-platform 7ffc0000.ehci: irq 28, io mem 0x7ffc0000
ehci-platform 7ffc0000.ehci: USB 2.0 started, EHCI 1.00
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
ohci-pci: OHCI PCI platform driver
ohci-platform: OHCI generic platform driver
ohci-platform 7ffb0000.ohci: Generic Platform OHCI controller
ohci-platform 7ffb0000.ohci: new USB bus registered, assigned bus number 2
ohci-platform 7ffb0000.ohci: irq 27, io mem 0x7ffb0000
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 1 port detected
usbcore: registered new interface driver usb-storage
mousedev: PS/2 mouse device common for all mice
rtc-efi rtc-efi: rtc core: registered rtc-efi as rtc0
mmci-pl18x 1c050000.mmci: mmc0: PL180 manf 41 rev0 at 0x1c050000 irq 33,0 (pio)
mmci-pl18x 1c050000.mmci: DMA channels RX none, TX none
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
Synopsys Designware Multimedia Card Interface Driver
sdhci-pltfm: SDHCI platform and OF driver helper
ledtrig-cpu: registered to indicate activity on CPUs
EFI Variables Facility v0.08 2004-May-17
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
NET: Registered protocol family 17
9pnet: Installing 9P2000 support
Key type dns_resolver registered
registered taskstats version 1
rtc-efi rtc-efi: setting system clock to 2016-02-01 08:26:43 UTC (1454315203)
sky2 0000:08:00.0 eth0: enabling interface
smsc911x 18000000.ethernet eth1: SMSC911x/921x identified at 0xffff000000120000,
 IRQ: 31
usb 1-1: new high-speed USB device number 2 using ehci-platform
hub 1-1:1.0: USB hub found
hub 1-1:1.0: 4 ports detected
usb 1-1.2: new high-speed USB device number 3 using ehci-platform
usb-storage 1-1.2:1.0: USB Mass Storage device detected
scsi host0: usb-storage 1-1.2:1.0
atkbd serio0: keyboard reset failed on 1c060000.kmi
scsi 0:0:0:0: Direct-Access     TOSHIBA  TransMemory      1.00 PQ: 0 ANSI: 4
sd 0:0:0:0: [sda] 15155200 512-byte logical blocks: (7.76 GB/7.23 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DP
O or FUA
 sda: sda1 sda2
sd 0:0:0:0: [sda] Attached SCSI removable disk
atkbd serio1: keyboard reset failed on 1c070000.kmi
Sending DHCP requests ., OK
IP-Config: Got DHCP answer from 10.xx.xx.xx, my address is 10.xx.xx.xx
sky2 0000:08:00.0 eth0: disabling interface
IP-Config: Complete:
     device=eth1, hwaddr=00:02:f7:xx:xx:xx, ipaddr=10.xx.xx.xx, mask=255.255.255
.0, gw=10.xx.xx.xx
     host=10.xx.xx.xx, domain=cambridge.arm.com, nis-domain=(none)
     bootserver=0.0.0.0, rootserver=10.xx.xx.xx, rootpath=     nameserver0=10.xx
.xx.xx, nameserver1=10.xx.xx.xx
Freezing user space processes ... (elapsed 0.000 seconds) done.
PM: Using 1 thread(s) for decompression.
PM: Loading and decompressing image data (94052 pages)...
PM: Image loading progress:   0%
PM: Image loading progress:  10%
PM: Image loading progress:  20%
PM: Image loading progress:  30%
PM: Image loading progress:  40%
PM: Image loading progress:  50%
PM: Image loading progress:  60%
PM: Image loading progress:  70%
PM: Image loading progress:  80%
PM: Image loading progress:  90%
PM: Image loading progress: 100%
PM: Image loading done.
PM: Read 376208 kbytes in 7.09 seconds (53.06 MB/s)
PM: quiesce of devices complete after 16.499 msecs
PM: late quiesce of devices complete after 0.938 msecs
PM: noirq quiesce of devices complete after 11.044 msecs
Disabling non-boot CPUs ...
PM: noirq restore of devices complete after 2.397 msecs
PM: early restore of devices complete after 0.917 msecs
serio serio0: device_attach() failed for 1c060000.kmi (1c060000.kmi), error: -51
7
serio serio1: device_attach() failed for 1c070000.kmi (1c070000.kmi), error: -51
7
PM: restore of devices complete after 93.241 msecs
serio serio0: device_attach() failed for 1c060000.kmi (1c060000.kmi), error: -51
7
Restarting tasks ... done.
atkbd serio0: keyboard reset failed on 1c060000.kmi
atkbd serio1: keyboard reset failed on 1c070000.kmi
atkbd serio1: keyboard reset failed on 1c070000.kmi
root@localhost:/sys/power#
root@localhost:/sys/power# uptime
 08:27:00 up 2 days, 16:57,  2 users,  load average: 0.37, 0.14, 0.09
root@localhost:/sys/power#

Re: [PATCH v4 00/13] arm64: kernel: Add support for hibernate/suspend-to-disk

[Kevin Hilman]

James Morse <james.morse@arm.com> writes:

> On 29/01/16 22:34, Kevin Hilman wrote:
>> James Morse <james.morse@arm.com> writes:
>> I'd like to help in testing this so I'm just curious which platforms
>> you've been testing this on.  I'm assuming a Juno (r2?), anything else?
>
> That would be great - thanks!
>
> I've done most of the testing on a Juno r1, but also gave it a spin on a
> stray Seattle.

OK, I'm using a very recently arrived Juno R2, and I did get it
working.  I have a few other arm64 boards around that ar part of
kernelci.org and will try those as well.

>> Are you testing the resume from cold boot, or just from kexec?
>
> From cold boot. I haven't tried with kexec, but I doubt that's a use-case anyone
> wants as you would resume immediately. (might be interesting for testing though)
>
>
>> For cold boot on Juno, I'm assuming there would be some
>> booloader/firmware changes needed to find and boot from the hibernation
>> image?
>
> Not at all! Your firmware only needs to support some mechanism to
> turning CPUs off.
>
> If you add 'resume=/dev/sda2' (or wherever your swap partition is located), the
> kernel will check this partition for the hibernate-image signature, if found, it
> will resume from that partition. Otherwise booting is as normal. (there is one
> hoop to jump through to ensure your rootfs hasn't been mounted before the kernel
> starts resume, as you could corrupt it - an initramfs in the kernel is the best
> fix for this).
>
> No firmware changes needed.
>
>> Is that being worked on?  If not Juno, are you aware of any
>> other platforms that could be tested with resume from cold boot?
>
> Any arm64 platform with persistent storage should work. I've been using a swap
> partition on a usb drive.

>> Not knowing the answers to the above, I tested your branch using arm64
>> defconfig + CONFIG_HIBERNATION=y on my Juno and noticed that it didn't
>> stay suspended (full suspend log below) so I'm looking for
>> ideas/recommenations on how to test this.
>
> That trace looks quite normal, (one of mine below[0] for comparison). Any
> failure would have happened after the point you stopped ... did you have a swap
> partition 'on'?

I think I must have not setup swap correctly since after testing again
it's working fine.  Maybe I forgot the 'swapon'?  In any case, being a
little more careful, testing again and things are working fine on Juno.

I'm also using swap on USB storage for now.

> 'syscore' will freeze all processes and stop all devices, then create a
> copy of the minimum amount of memory it needs to save. Then it starts
> all the devices again, as it needs to write this image out to swap. This is what
> you are seeing.
>
> Once it has done this it calls poweroff or reboot.

Yeah, I wasn't seeing the call to poweroff, but most likely because it
was failing to fully write the image due to my missing/incorrect swap
setup.

Thanks for the help,

Kevin

Re: [PATCH v4 00/13] arm64: kernel: Add support for hibernate/suspend-to-disk

[James Morse]

Hi Kevin,

On 03/02/16 00:42, Kevin Hilman wrote:
> James Morse <james.morse@arm.com> writes:
>> On 29/01/16 22:34, Kevin Hilman wrote:
>>> James Morse <james.morse@arm.com> writes:
>>> I'd like to help in testing this so I'm just curious which platforms
>>> you've been testing this on.  I'm assuming a Juno (r2?), anything else?
>>
>> That would be great - thanks!
>>
>> I've done most of the testing on a Juno r1, but also gave it a spin on a
>> stray Seattle.
> 
> OK, I'm using a very recently arrived Juno R2, and I did get it
> working.  I have a few other arm64 boards around that ar part of
> kernelci.org and will try those as well.

Great,
It would be good to have a Tested-by if it works to your satisfaction...


Thanks,

James

Re: [PATCH v4 00/13] arm64: kernel: Add support for hibernate/suspend-to-disk

[Kevin Hilman]

James Morse <james.morse@arm.com> writes:

> Hi Kevin,
>
> On 03/02/16 00:42, Kevin Hilman wrote:
>> James Morse <james.morse@arm.com> writes:
>>> On 29/01/16 22:34, Kevin Hilman wrote:
>>>> James Morse <james.morse@arm.com> writes:
>>>> I'd like to help in testing this so I'm just curious which platforms
>>>> you've been testing this on.  I'm assuming a Juno (r2?), anything else?
>>>
>>> That would be great - thanks!
>>>
>>> I've done most of the testing on a Juno r1, but also gave it a spin on a
>>> stray Seattle.
>> 
>> OK, I'm using a very recently arrived Juno R2, and I did get it
>> working.  I have a few other arm64 boards around that ar part of
>> kernelci.org and will try those as well.
>
> Great,
> It would be good to have a Tested-by if it works to your satisfaction...

Yes, feel free to add

Tested-by: Kevin Hilman <khilman@baylibre.com> # Tested on Juno R2

I tried to also test on the qcom 96boards, but that platform doesn't
support suspend/resume yet. :(

Kevin