Memory consumption difference between in-kernel and userspace hibernation

Memory consumption difference between in-kernel and userspace hibernation

[Thomas Petazzoni]

Hi,

Let me first introduce my question, and then give details about the
context.

Question: is there any difference in terms of memory requirements for
the in-kernel hibernation (echo disk > /sys/power/state) and the
userspace hibernation interface (through /dev/snapshot) ? With exactly
the same userspace workload and applications running, the in-kernel
hibernation works, but the hibernating using the userspace hibernation
interface fails because not enough memory can be freed.

Now, the context.

I'm implementing hibernation on an embedded device, which has no swap
since the only storage available is NAND flash.

I started by using the in-kernel hibernation mechanism, which saved the
resume image directly into an MTD partition, declared as a swap just
before starting the hibernation process (swapon /dev/mtdblockX; echo
disk > /sys/power/state). This worked like a charm.

But writing the resume image directly to the MTD partition is not
satisfying since it doesn't handle bad erase blocks and wear leveling.
Therefore, I wanted to save the resume image into a file, inside a
JFFS2 or YAFFS2 filesystem. For this, I used the /dev/snapshot
userspace interface to swsusp. With a light workload, it works
perfectly (both suspend and resume). But with a similar workload than
the one tested with the in-kernel hibernation, things fail at the
SNAPSHOT_ATOMIC_SNAPSHOT ioctl() step, which returns ENOMEM.

To get some details about the issue, I've added a few printk()s in
swsusp_shrink_memory(). Here is the patch:

==================================================================
--- foo.orig/kernel/power/swsusp.c
+++ foo/kernel/power/swsusp.c
@@ -226,15 +226,20 @@
                highmem_size = count_highmem_pages();
                size = count_data_pages() + PAGES_FOR_IO;
                tmp = size;
+               printk("size=%d\n", size);
                size += highmem_size;
                for_each_zone (zone)
                        if (populated_zone(zone)) {
                                if (is_highmem(zone)) {
                                        highmem_size -= zone->free_pages;
                                } else {
+                                       printk("1 tmp=%d\n", tmp);
                                        tmp -= zone->free_pages;
+                                       printk("2 tmp=%d\n", tmp);
                                        tmp += zone->lowmem_reserve[ZONE_NORMAL];
+                                       printk("3 tmp=%d\n", tmp);
                                        tmp += snapshot_additional_pages(zone);
+                                       printk("4 tmp=%d\n", tmp);
                                }
                        }
 
@@ -243,9 +248,12 @@
 
                tmp += highmem_size;
                if (tmp > 0) {
+                       printk("trying to free %d pages\n", tmp);
                        tmp = __shrink_memory(tmp);
-                       if (!tmp)
+                       if (!tmp) {
+                               printk("\bfailed, ENOMEM\n");
                                return -ENOMEM;
+                       }
                        pages += tmp;
                } else if (size > image_size / PAGE_SIZE) {
                        tmp = __shrink_memory(size - (image_size / PAGE_SIZE));
==================================================================

I get the following output:

==================================================================
Stopping tasks ... done.
Shrinking memory...  size=6967
1 tmp=6967
2 tmp=6639
3 tmp=6639
4 tmp=6643
trying to free 6643 pages
-size=4036
1 tmp=4036
2 tmp=777
3 tmp=777
4 tmp=781
trying to free 781 pages
failed, ENOMEM
Restarting tasks ... done.
==================================================================

Note 1: I've already reduced PAGES_FOR_IO from 1024 to 128.

Note 2: As usual in the embedded space, I'm stuck with an old 2.6.25
        kernel.

Any idea on why it works with the in-kernel solution and not the
userspace one ?

Thanks a lot for your inputs,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

Re: Memory consumption difference between in-kernel and userspace hibernation

[Rafael J. Wysocki]

On Thursday 12 November 2009, Thomas Petazzoni wrote:
> Hi,
> 
> Let me first introduce my question, and then give details about the
> context.
> 
> Question: is there any difference in terms of memory requirements for
> the in-kernel hibernation (echo disk > /sys/power/state) and the
> userspace hibernation interface (through /dev/snapshot) ? With exactly
> the same userspace workload and applications running, the in-kernel
> hibernation works, but the hibernating using the userspace hibernation
> interface fails because not enough memory can be freed.
> 
> Now, the context.
> 
> I'm implementing hibernation on an embedded device, which has no swap
> since the only storage available is NAND flash.
> 
> I started by using the in-kernel hibernation mechanism, which saved the
> resume image directly into an MTD partition, declared as a swap just
> before starting the hibernation process (swapon /dev/mtdblockX; echo
> disk > /sys/power/state). This worked like a charm.
> 
> But writing the resume image directly to the MTD partition is not
> satisfying since it doesn't handle bad erase blocks and wear leveling.
> Therefore, I wanted to save the resume image into a file, inside a
> JFFS2 or YAFFS2 filesystem.

The userspace interface doesn't really allow you to write to a file.  You can
write into the area the file occupies on the partition, but you can't use the
filesystem code for the actual writing.  At least you shouldn't do that.

> For this, I used the /dev/snapshot
> userspace interface to swsusp. With a light workload, it works
> perfectly (both suspend and resume). But with a similar workload than
> the one tested with the in-kernel hibernation, things fail at the
> SNAPSHOT_ATOMIC_SNAPSHOT ioctl() step, which returns ENOMEM.
> 
> To get some details about the issue, I've added a few printk()s in
> swsusp_shrink_memory(). Here is the patch:
> 
> ==================================================================
> --- foo.orig/kernel/power/swsusp.c
> +++ foo/kernel/power/swsusp.c
> @@ -226,15 +226,20 @@
>                 highmem_size = count_highmem_pages();
>                 size = count_data_pages() + PAGES_FOR_IO;
>                 tmp = size;
> +               printk("size=%d\n", size);
>                 size += highmem_size;
>                 for_each_zone (zone)
>                         if (populated_zone(zone)) {
>                                 if (is_highmem(zone)) {
>                                         highmem_size -= zone->free_pages;
>                                 } else {
> +                                       printk("1 tmp=%d\n", tmp);
>                                         tmp -= zone->free_pages;
> +                                       printk("2 tmp=%d\n", tmp);
>                                         tmp += zone->lowmem_reserve[ZONE_NORMAL];
> +                                       printk("3 tmp=%d\n", tmp);
>                                         tmp += snapshot_additional_pages(zone);
> +                                       printk("4 tmp=%d\n", tmp);
>                                 }
>                         }
>  
> @@ -243,9 +248,12 @@
>  
>                 tmp += highmem_size;
>                 if (tmp > 0) {
> +                       printk("trying to free %d pages\n", tmp);
>                         tmp = __shrink_memory(tmp);
> -                       if (!tmp)
> +                       if (!tmp) {
> +                               printk("\bfailed, ENOMEM\n");
>                                 return -ENOMEM;
> +                       }
>                         pages += tmp;
>                 } else if (size > image_size / PAGE_SIZE) {
>                         tmp = __shrink_memory(size - (image_size / PAGE_SIZE));
> ==================================================================
> 
> I get the following output:
> 
> ==================================================================
> Stopping tasks ... done.
> Shrinking memory...  size=6967
> 1 tmp=6967
> 2 tmp=6639
> 3 tmp=6639
> 4 tmp=6643
> trying to free 6643 pages
> -size=4036
> 1 tmp=4036
> 2 tmp=777
> 3 tmp=777
> 4 tmp=781
> trying to free 781 pages
> failed, ENOMEM
> Restarting tasks ... done.
> ==================================================================

swsusp_shrink_memory() is used by both the in-kernel code and the userspace
code more-or-less in the same way, so it looks strange.  How much memory is
there in the system?
 
> Note 1: I've already reduced PAGES_FOR_IO from 1024 to 128.
> 
> Note 2: As usual in the embedded space, I'm stuck with an old 2.6.25
>         kernel.
> 
> Any idea on why it works with the in-kernel solution and not the
> userspace one ?

s2disk allocates a few buffers for itself, but I'm not sure if that matters
at all.

Which version of s2disk do you use, do you have encryption enabled in s2disk?

Rafael

Re: Memory consumption difference between in-kernel and userspace hibernation

[Thomas Petazzoni]

Hello,

Thanks for your feedback.

Le Thu, 12 Nov 2009 21:52:02 +0100,
"Rafael J. Wysocki" <rjw@sisk.pl> a écrit :

> The userspace interface doesn't really allow you to write to a file.
> You can write into the area the file occupies on the partition, but
> you can't use the filesystem code for the actual writing.  At least
> you shouldn't do that.

Ah. But it seems to work fairly nicely. Why can't the filesystem code
could be used to store the resume image ? Note that my file is stored
in a separate partition, fully dedicated to storing the resume file and
mounted only at very specific points in the system lifetime.

I'm doing the following things upon suspend :

 * SNAPSHOT_FREEZE

 * SNAPSHOT_ATOMIC_SNAPSHOT, which according to my understanding is
   making a snapshot of the memory inside a copy, so that once the
   snapshot is made, the kernel can be used as usual.

 * SNAPSHOT_UNFREEZE

 * Mount my YAFFS2 filesystem that will contain the resume image

 * Read /dev/snapshot and write the result to a file in the YAFFS2
   filesystem

 * Unmount the YAFFS2 filesystem

 * Shutdown

On resume, I'm doing the following operations in a /init application in
the kernel initramfs :

 * Mount the YAFFS2 filesystem

 * Open the resume image and the /dev/snapshot device

 * Read the image from the file and write it to /dev/snapshot

 * Unmount the YAFFS2 filesystem

 * SNAPSHOT_FREEZE

 * SNAPSHOT_ATOMIC_RESTORE

And done. So my YAFFS2 filesystem is *never* mounted before taking the
snapshot or before beginning to restore the snapshot.

Isn't this safe ?

> swsusp_shrink_memory() is used by both the in-kernel code and the
> userspace code more-or-less in the same way, so it looks strange.
> How much memory is there in the system?

Not that much:

# free
              total         used         free       shared      buffers
  Mem:        26116        11264        14852            0            0
 Swap:            0            0            0
Total:        26116        11264        14852

(This is the figure without the heavy applications loaded)

> s2disk allocates a few buffers for itself, but I'm not sure if that
> matters at all.
> 
> Which version of s2disk do you use, do you have encryption enabled in
> s2disk?

I'm not using s2disk at all, so no encryption, nothing. I implemented my
own suspend/resume applications (I can show the source code if needed,
even though the code is pretty ugly).

BTW, I've done a little test: compress with LZO each page that needs to
be saved as part of the snapshot. Instead of 11419648 bytes in my case,
only 5725951 bytes (half!) would be needed to store the snapshot in
memory, reducing significantly the memory pressure caused by the
allocation of the pages needed to store the snapshot. Has this already
been considered ?

Thanks again for the feedback,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com
_______________________________________________
linux-pm mailing list
linux-pm@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/linux-pm

Re: Memory consumption difference between in-kernel and userspace hibernation

[Rafael J. Wysocki]

On Thursday 12 November 2009, Thomas Petazzoni wrote:
> Hello,
> 
> Thanks for your feedback.
> 
> Le Thu, 12 Nov 2009 21:52:02 +0100,
> "Rafael J. Wysocki" <rjw@sisk.pl> a écrit :
> 
> > The userspace interface doesn't really allow you to write to a file.
> > You can write into the area the file occupies on the partition, but
> > you can't use the filesystem code for the actual writing.  At least
> > you shouldn't do that.
> 
> Ah. But it seems to work fairly nicely. Why can't the filesystem code
> could be used to store the resume image ? Note that my file is stored
> in a separate partition, fully dedicated to storing the resume file and
> mounted only at very specific points in the system lifetime.

That doesn't really matter.

The problem is that the image is likely to contain filesystem data (eg.
superblocks etc.) that correspond to the state before the image has been
created.  Now, your using the filesystem code for writing the image modifies
the on-disk metadata which become inconsistent with the filesystem data in
the image.  This inconsistencies may very well result in an unfixable
corruption of the file system after the resume (that actually happened to
a number of people, so it's not just pure theory).

That really depends on the fileystem used, though.

Thanks,
Rafael
_______________________________________________
linux-pm mailing list
linux-pm@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/linux-pm

Re: Memory consumption difference between in-kernel and userspace hibernation

[Pavel Machek]

On Fri 2009-11-13 21:05:07, Rafael J. Wysocki wrote:
> On Thursday 12 November 2009, Thomas Petazzoni wrote:
> > Hello,
> > 
> > Thanks for your feedback.
> > 
> > Le Thu, 12 Nov 2009 21:52:02 +0100,
> > "Rafael J. Wysocki" <rjw@sisk.pl> a ??crit :
> > 
> > > The userspace interface doesn't really allow you to write to a file.
> > > You can write into the area the file occupies on the partition, but
> > > you can't use the filesystem code for the actual writing.  At least
> > > you shouldn't do that.
> > 
> > Ah. But it seems to work fairly nicely. Why can't the filesystem code
> > could be used to store the resume image ? Note that my file is stored
> > in a separate partition, fully dedicated to storing the resume file and
> > mounted only at very specific points in the system lifetime.
> 
> That doesn't really matter.
> 
> The problem is that the image is likely to contain filesystem data (eg.
> superblocks etc.) that correspond to the state before the image has been
> created.  Now, your using the filesystem code for writing the image modifies
> the on-disk metadata which become inconsistent with the filesystem data in
> the image.  This inconsistencies may very well result in an unfixable
> corruption of the file system after the resume (that actually happened to
> a number of people, so it's not just pure theory).
> 
> That really depends on the fileystem used, though.

But based on  his description... if fs is only mounted after atomic
snapshot... it actually should be safe.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Memory consumption difference between in-kernel and userspace hibernation

[Thomas Petazzoni]

Hello,

Le Thu, 12 Nov 2009 21:01:55 +0100,
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> a écrit :

> Question: is there any difference in terms of memory requirements for
> the in-kernel hibernation (echo disk > /sys/power/state) and the
> userspace hibernation interface (through /dev/snapshot) ? With exactly
> the same userspace workload and applications running, the in-kernel
> hibernation works, but the hibernating using the userspace hibernation
> interface fails because not enough memory can be freed.

The difference is that when using the in-kernel solution, a swap is
enabled, and it seems that it allows the kernel to free some memory in
order to create the snapshot.

But I don't understand why : won't the swap contents be completely
erased when the resume image will be written to it, and therefore all
the things migrated to the swap when freing the memory would be lost ?

Thanks for your input,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com
_______________________________________________
linux-pm mailing list
linux-pm@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/linux-pm

Re: Memory consumption difference between in-kernel and userspace hibernation

[Rafael J. Wysocki]

On Friday 13 November 2009, Thomas Petazzoni wrote:
> Hello,
> 
> Le Thu, 12 Nov 2009 21:01:55 +0100,
> Thomas Petazzoni <thomas.petazzoni@free-electrons.com> a écrit :
> 
> > Question: is there any difference in terms of memory requirements for
> > the in-kernel hibernation (echo disk > /sys/power/state) and the
> > userspace hibernation interface (through /dev/snapshot) ? With exactly
> > the same userspace workload and applications running, the in-kernel
> > hibernation works, but the hibernating using the userspace hibernation
> > interface fails because not enough memory can be freed.
> 
> The difference is that when using the in-kernel solution, a swap is
> enabled, and it seems that it allows the kernel to free some memory in
> order to create the snapshot.

Yes, it does.

> But I don't understand why : won't the swap contents be completely
> erased when the resume image will be written to it, and therefore all
> the things migrated to the swap when freing the memory would be lost ?

No, the kernel doesn't use the swap space already in use for saving the
image.  Also, s2disk uses the kernel for allocating swap space in which to
save the image, so it avoids overwriting the swap space in use too.

Thanks,
Rafael
_______________________________________________
linux-pm mailing list
linux-pm@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/linux-pm