From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pb0-f53.google.com (mail-pb0-f53.google.com [209.85.160.53]) by kanga.kvack.org (Postfix) with ESMTP id C857C6B00DC for ; Wed, 23 Oct 2013 17:51:38 -0400 (EDT) Received: by mail-pb0-f53.google.com with SMTP id up7so1289898pbc.40 for ; Wed, 23 Oct 2013 14:51:38 -0700 (PDT) Received: from psmtp.com ([74.125.245.162]) by mx.google.com with SMTP id js8si3130642pbc.284.2013.10.23.14.51.37 for ; Wed, 23 Oct 2013 14:51:37 -0700 (PDT) Message-ID: <526844E6.1080307@codeaurora.org> Date: Wed, 23 Oct 2013 14:51:34 -0700 From: Olav Haugan MIME-Version: 1.0 Subject: zram/zsmalloc issues in very low memory conditions Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: minchan@kernel.org, sjenning@linux.vnet.ibm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org I am trying to use zram in very low memory conditions and I am having some issues. zram is in the reclaim path. So if the system is very low on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to get a page from zsmalloc and thus zram fails to store the page. We get into a cycle where the system is low on memory so it tries to swap out to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out because the swap device is essentially "full" (since there is no more memory available for zram pages). Has anyone thought about this issue already and have ideas how to solve this or am I missing something and I should not be seeing this issue? I am also seeing a couple other issues that I was wondering whether folks have already thought about: 1) The size of a swap device is statically computed when the swap device is turned on (nr_swap_pages). The size of zram swap device is dynamic since we are compressing the pages and thus the swap subsystem thinks that the zram swap device is full when it is not really full. Any plans/thoughts about the possibility of being able to update the size and/or the # of available pages in a swap device on the fly? 2) zsmalloc fails when the page allocated is at physical address 0 (pfn = 0) since the handle returned from zsmalloc is encoded as (, ) and thus the resulting handle will be 0 (since obj_idx starts at 0). zs_malloc returns the handle but does not distinguish between a valid handle of 0 and a failure to allocate. A possible solution to this would be to start the obj_idx at 1. Is this feasible? Thanks, Olav Haugan -- The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pb0-f46.google.com (mail-pb0-f46.google.com [209.85.160.46]) by kanga.kvack.org (Postfix) with ESMTP id 587856B00DC for ; Wed, 23 Oct 2013 18:14:12 -0400 (EDT) Received: by mail-pb0-f46.google.com with SMTP id un1so1597413pbc.5 for ; Wed, 23 Oct 2013 15:14:12 -0700 (PDT) Received: from psmtp.com ([74.125.245.129]) by mx.google.com with SMTP id ar2si3169428pbc.292.2013.10.23.15.14.10 for ; Wed, 23 Oct 2013 15:14:11 -0700 (PDT) Received: by mail-ie0-f169.google.com with SMTP id ar20so2470994iec.14 for ; Wed, 23 Oct 2013 15:14:09 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <526844E6.1080307@codeaurora.org> References: <526844E6.1080307@codeaurora.org> Date: Wed, 23 Oct 2013 15:14:09 -0700 Message-ID: Subject: Re: zram/zsmalloc issues in very low memory conditions From: Luigi Semenzato Content-Type: multipart/alternative; boundary=089e013cc02815efb004e96fd6d6 Sender: owner-linux-mm@kvack.org List-ID: To: Olav Haugan Cc: Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org --089e013cc02815efb004e96fd6d6 Content-Type: text/plain; charset=ISO-8859-1 On Wed, Oct 23, 2013 at 2:51 PM, Olav Haugan wrote: > I am trying to use zram in very low memory conditions and I am having > some issues. zram is in the reclaim path. So if the system is very low > on memory the system is trying to reclaim pages by swapping out (in this > case to zram). However, since we are very low on memory zram fails to > get a page from zsmalloc and thus zram fails to store the page. We get > into a cycle where the system is low on memory so it tries to swap out > to get more memory but swap out fails because there is not enough memory > in the system! The major problem I am seeing is that there does not seem > to be a way for zram to tell the upper layers to stop swapping out > because the swap device is essentially "full" (since there is no more > memory available for zram pages). Has anyone thought about this issue > already and have ideas how to solve this or am I missing something and I > should not be seeing this issue? What do you want the system to do at this point? OOM kill? Also, if you are that low on memory, how are you preventing thrashing on the code pages? I am asking because we also use zram but haven't run into this problem---however we had to deal with other problems that motivate these questions. > > I am also seeing a couple other issues that I was wondering whether > folks have already thought about: > > 1) The size of a swap device is statically computed when the swap device > is turned on (nr_swap_pages). The size of zram swap device is dynamic > since we are compressing the pages and thus the swap subsystem thinks > that the zram swap device is full when it is not really full. Any > plans/thoughts about the possibility of being able to update the size > and/or the # of available pages in a swap device on the fly? > That is a known limitation of zram. If can predict your compression ratio and your working set size, it's not a big problem: allocate a swap device which, based on the expected compression ratio, will use up RAM until what's left is just enough for the working set. > > 2) zsmalloc fails when the page allocated is at physical address 0 (pfn > = 0) since the handle returned from zsmalloc is encoded as (, > ) and thus the resulting handle will be 0 (since obj_idx starts > at 0). zs_malloc returns the handle but does not distinguish between a > valid handle of 0 and a failure to allocate. A possible solution to this > would be to start the obj_idx at 1. Is this feasible? > Sorry, no idea on this. Probably Minchan can reply. > > Thanks, > > Olav Haugan > > -- > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, > hosted by The Linux Foundation > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: email@kvack.org > --089e013cc02815efb004e96fd6d6 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable



On Wed, Oct 23, 2013 at 2:51 PM, Olav Haugan <= ohaugan@codeaur= ora.org> wrote:
I am trying to use zram in very low memory c= onditions and I am having
some issues. zram is in the reclaim path. So if the system is very low
on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to
get a page from zsmalloc and thus zram fails to store the page. We get
into a cycle where the system is low on memory so it tries to swap out
to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out
because the swap device is essentially "full" (since there is no = more
memory available for zram pages). Has anyone thought about this issue
already and have ideas how to solve this or am I missing something and I should not be seeing this issue?

What do yo= u want the system to do at this point? =A0OOM kill? =A0Also, if you are tha= t low on memory, how are you preventing thrashing on the code pages?

I am asking because we also use zram but haven't ru= n into this problem---however we had to deal with other problems that motiv= ate these questions.=A0
=A0

I am also seeing a couple other issues that I was wondering whether
folks have already thought about:

1) The size of a swap device is statically computed when the swap device is turned on (nr_swap_pages). The size of zram swap device is dynamic
since we are compressing the pages and thus the swap subsystem thinks
that the zram swap device is full when it is not really full. Any
plans/thoughts about the possibility of being able to update the size
and/or the # of available pages in a swap device on the fly?

That is a known limitation of zram. =A0If can predict= your compression ratio and your working set size, it's not a big probl= em: allocate a swap device which, based on the expected compression ratio, = will use up RAM until what's left is just enough for the working set.
=A0

2) zsmalloc fails when the page allocated is at physical address 0 (pfn
=3D 0) since the handle returned from zsmalloc is encoded as (<PFN>,<= br> <obj_idx>) and thus the resulting handle will be 0 (since obj_idx sta= rts
at 0). zs_malloc returns the handle but does not distinguish between a
valid handle of 0 and a failure to allocate. A possible solution to this would be to start the obj_idx at 1. Is this feasible?
=
Sorry, no idea on this. =A0Probably Minchan can reply.
=
=A0

Thanks,

Olav Haugan

--
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.= =A0For more info on Linux MM,
see: http://www.linu= x-mm.org/ .
Don't email: <a href=3Dmailto:"dont@kvack.org"> email@kva= ck.org </a>

--089e013cc02815efb004e96fd6d6-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pd0-f174.google.com (mail-pd0-f174.google.com [209.85.192.174]) by kanga.kvack.org (Postfix) with ESMTP id E85D96B00DC for ; Wed, 23 Oct 2013 18:17:28 -0400 (EDT) Received: by mail-pd0-f174.google.com with SMTP id y13so1890069pdi.33 for ; Wed, 23 Oct 2013 15:17:28 -0700 (PDT) Received: from psmtp.com ([74.125.245.123]) by mx.google.com with SMTP id a10si77258pac.163.2013.10.23.15.17.27 for ; Wed, 23 Oct 2013 15:17:27 -0700 (PDT) Received: by mail-ie0-f173.google.com with SMTP id u16so2526936iet.32 for ; Wed, 23 Oct 2013 15:17:26 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <526844E6.1080307@codeaurora.org> References: <526844E6.1080307@codeaurora.org> Date: Wed, 23 Oct 2013 15:17:26 -0700 Message-ID: Subject: Re: zram/zsmalloc issues in very low memory conditions From: Luigi Semenzato Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Olav Haugan Cc: Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org (sorry about the HTML in the previous message) On Wed, Oct 23, 2013 at 2:51 PM, Olav Haugan wrote: > I am trying to use zram in very low memory conditions and I am having > some issues. zram is in the reclaim path. So if the system is very low > on memory the system is trying to reclaim pages by swapping out (in this > case to zram). However, since we are very low on memory zram fails to > get a page from zsmalloc and thus zram fails to store the page. We get > into a cycle where the system is low on memory so it tries to swap out > to get more memory but swap out fails because there is not enough memory > in the system! The major problem I am seeing is that there does not seem > to be a way for zram to tell the upper layers to stop swapping out > because the swap device is essentially "full" (since there is no more > memory available for zram pages). Has anyone thought about this issue > already and have ideas how to solve this or am I missing something and I > should not be seeing this issue? What do you want the system to do at this point? OOM kill? Also, if you are that low on memory, how are you preventing thrashing on the code pages? I am asking because we also use zram but haven't run into this problem---however we had to deal with other problems that motivate these questions. > > I am also seeing a couple other issues that I was wondering whether > folks have already thought about: > > 1) The size of a swap device is statically computed when the swap device > is turned on (nr_swap_pages). The size of zram swap device is dynamic > since we are compressing the pages and thus the swap subsystem thinks > that the zram swap device is full when it is not really full. Any > plans/thoughts about the possibility of being able to update the size > and/or the # of available pages in a swap device on the fly? That is a known limitation of zram. If you can predict your compression ratio and your working set size, it's not a big problem: allocate a swap device which, based on the expected compression ratio, will use up RAM until what's left is just enough for the working set. > 2) zsmalloc fails when the page allocated is at physical address 0 (pfn > = 0) since the handle returned from zsmalloc is encoded as (, > ) and thus the resulting handle will be 0 (since obj_idx starts > at 0). zs_malloc returns the handle but does not distinguish between a > valid handle of 0 and a failure to allocate. A possible solution to this > would be to start the obj_idx at 1. Is this feasible? > > Thanks, > > Olav Haugan > > -- > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, > hosted by The Linux Foundation > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: email@kvack.org -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pb0-f52.google.com (mail-pb0-f52.google.com [209.85.160.52]) by kanga.kvack.org (Postfix) with ESMTP id E41E06B00DC for ; Wed, 23 Oct 2013 20:55:30 -0400 (EDT) Received: by mail-pb0-f52.google.com with SMTP id wy17so1760740pbc.25 for ; Wed, 23 Oct 2013 17:55:30 -0700 (PDT) Received: from psmtp.com ([74.125.245.130]) by mx.google.com with SMTP id if1si338215pad.146.2013.10.23.17.55.29 for ; Wed, 23 Oct 2013 17:55:30 -0700 (PDT) Message-ID: <52686FF4.5000303@oracle.com> Date: Thu, 24 Oct 2013 08:55:16 +0800 From: Bob Liu MIME-Version: 1.0 Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> In-Reply-To: <526844E6.1080307@codeaurora.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Olav Haugan Cc: minchan@kernel.org, sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org On 10/24/2013 05:51 AM, Olav Haugan wrote: > I am trying to use zram in very low memory conditions and I am having > some issues. zram is in the reclaim path. So if the system is very low > on memory the system is trying to reclaim pages by swapping out (in this > case to zram). However, since we are very low on memory zram fails to > get a page from zsmalloc and thus zram fails to store the page. We get > into a cycle where the system is low on memory so it tries to swap out > to get more memory but swap out fails because there is not enough memory > in the system! The major problem I am seeing is that there does not seem > to be a way for zram to tell the upper layers to stop swapping out > because the swap device is essentially "full" (since there is no more > memory available for zram pages). Has anyone thought about this issue > already and have ideas how to solve this or am I missing something and I > should not be seeing this issue? > The same question as Luigi "What do you want the system to do at this point?" If swap fails then OOM killer will be triggered, I don't think this will be a issue. By the way, could you take a try with zswap? Which can write pages to real swap device if compressed pool is full. > I am also seeing a couple other issues that I was wondering whether > folks have already thought about: > > 1) The size of a swap device is statically computed when the swap device > is turned on (nr_swap_pages). The size of zram swap device is dynamic > since we are compressing the pages and thus the swap subsystem thinks > that the zram swap device is full when it is not really full. Any > plans/thoughts about the possibility of being able to update the size > and/or the # of available pages in a swap device on the fly? > > 2) zsmalloc fails when the page allocated is at physical address 0 (pfn AFAIK, this will never happen. > = 0) since the handle returned from zsmalloc is encoded as (, > ) and thus the resulting handle will be 0 (since obj_idx starts > at 0). zs_malloc returns the handle but does not distinguish between a > valid handle of 0 and a failure to allocate. A possible solution to this > would be to start the obj_idx at 1. Is this feasible? > > Thanks, > > Olav Haugan > -- Regards, -Bob -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pd0-f175.google.com (mail-pd0-f175.google.com [209.85.192.175]) by kanga.kvack.org (Postfix) with ESMTP id 789DA6B00DC for ; Thu, 24 Oct 2013 06:42:51 -0400 (EDT) Received: by mail-pd0-f175.google.com with SMTP id g10so2229828pdj.20 for ; Thu, 24 Oct 2013 03:42:51 -0700 (PDT) Received: from psmtp.com ([74.125.245.102]) by mx.google.com with SMTP id hb3si1565627pac.239.2013.10.24.03.42.49 for ; Thu, 24 Oct 2013 03:42:50 -0700 (PDT) Received: by mail-ie0-f178.google.com with SMTP id x13so3628441ief.9 for ; Thu, 24 Oct 2013 03:42:48 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <526844E6.1080307@codeaurora.org> References: <526844E6.1080307@codeaurora.org> Date: Thu, 24 Oct 2013 18:42:48 +0800 Message-ID: Subject: Re: zram/zsmalloc issues in very low memory conditions From: Weijie Yang Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Olav Haugan Cc: minchan@kernel.org, sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, semenzato@google.com, bob.liu@oracle.com On Thu, Oct 24, 2013 at 5:51 AM, Olav Haugan wrote: > I am trying to use zram in very low memory conditions and I am having > some issues. zram is in the reclaim path. So if the system is very low > on memory the system is trying to reclaim pages by swapping out (in this > case to zram). However, since we are very low on memory zram fails to > get a page from zsmalloc and thus zram fails to store the page. We get > into a cycle where the system is low on memory so it tries to swap out > to get more memory but swap out fails because there is not enough memory > in the system! The major problem I am seeing is that there does not seem > to be a way for zram to tell the upper layers to stop swapping out > because the swap device is essentially "full" (since there is no more > memory available for zram pages). Has anyone thought about this issue > already and have ideas how to solve this or am I missing something and I > should not be seeing this issue? I agree with Luigi and Bob. zram's size is based on how many free memory you expect to use for zram. In my test, the compression ratio is about 1:4, of course the working sets may be different with yours. Further more, may be you can modify vm_swap_full() to let kernel free swap_entry aggressively. > I am also seeing a couple other issues that I was wondering whether > folks have already thought about: > > 1) The size of a swap device is statically computed when the swap device > is turned on (nr_swap_pages). The size of zram swap device is dynamic > since we are compressing the pages and thus the swap subsystem thinks > that the zram swap device is full when it is not really full. Any > plans/thoughts about the possibility of being able to update the size > and/or the # of available pages in a swap device on the fly? > > 2) zsmalloc fails when the page allocated is at physical address 0 (pfn > = 0) since the handle returned from zsmalloc is encoded as (, > ) and thus the resulting handle will be 0 (since obj_idx starts > at 0). zs_malloc returns the handle but does not distinguish between a > valid handle of 0 and a failure to allocate. A possible solution to this > would be to start the obj_idx at 1. Is this feasible? > > Thanks, > > Olav Haugan > > -- > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, > hosted by The Linux Foundation > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pb0-f41.google.com (mail-pb0-f41.google.com [209.85.160.41]) by kanga.kvack.org (Postfix) with ESMTP id E5A006B00DD for ; Thu, 24 Oct 2013 20:35:28 -0400 (EDT) Received: by mail-pb0-f41.google.com with SMTP id rp16so3442431pbb.14 for ; Thu, 24 Oct 2013 17:35:28 -0700 (PDT) Received: from psmtp.com ([74.125.245.168]) by mx.google.com with SMTP id ud7si3291661pac.323.2013.10.24.17.35.27 for ; Thu, 24 Oct 2013 17:35:28 -0700 (PDT) Message-ID: <5269BCCC.6090509@codeaurora.org> Date: Thu, 24 Oct 2013 17:35:24 -0700 From: Olav Haugan MIME-Version: 1.0 Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> In-Reply-To: <52686FF4.5000303@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Bob Liu Cc: minchan@kernel.org, sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, semenzato@google.com Hi Bob, Luigi, On 10/23/2013 5:55 PM, Bob Liu wrote: > > On 10/24/2013 05:51 AM, Olav Haugan wrote: >> I am trying to use zram in very low memory conditions and I am having >> some issues. zram is in the reclaim path. So if the system is very low >> on memory the system is trying to reclaim pages by swapping out (in this >> case to zram). However, since we are very low on memory zram fails to >> get a page from zsmalloc and thus zram fails to store the page. We get >> into a cycle where the system is low on memory so it tries to swap out >> to get more memory but swap out fails because there is not enough memory >> in the system! The major problem I am seeing is that there does not seem >> to be a way for zram to tell the upper layers to stop swapping out >> because the swap device is essentially "full" (since there is no more >> memory available for zram pages). Has anyone thought about this issue >> already and have ideas how to solve this or am I missing something and I >> should not be seeing this issue? >> > > The same question as Luigi "What do you want the system to do at this > point?" > > If swap fails then OOM killer will be triggered, I don't think this will > be a issue. I definitely don't want OOM killer to run since OOM killer can kill critical processes (this is on Android so we have Android LMK to handle the killing in a more "safe" way). However, what I am seeing is that when I run low on memory zram fails to swap out and returns error but the swap subsystem just continues to try to swap out even when this error occurs (it tries over and over again very rapidly causing the kernel to be filled with error messages [at least two error messages per failure btw]). What I expected to happen is for the swap subsystem to stop trying to swap out until memory is available to swap out. I guess this could be handled several ways. Either 1) the swap subsystem, upon encountering an error to swap out, backs off from trying to swap out for some time or 2) zram informs the swap subsystem that the swap device is full. Could this be handled by congestion control? However, I found the following comment in the code in vmscan.c: * If the page is swapcache, write it back even if that would * block, for some throttling. This happens by accident, because * swap_backing_dev_info is bust: it doesn't reflect the * congestion state of the swapdevs. Easy to fix, if needed. However, how would one update the congested state of zram when it becomes un-congested? > By the way, could you take a try with zswap? Which can write pages to > real swap device if compressed pool is full. zswap might not be feasible in all cases if you only have flash as backing storage. >> I am also seeing a couple other issues that I was wondering whether >> folks have already thought about: >> >> 2) zsmalloc fails when the page allocated is at physical address 0 (pfn > > AFAIK, this will never happen. I can easily get this to happen since I have memory starting at physical address 0. Thanks, Olav Haugan -- The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f42.google.com (mail-pa0-f42.google.com [209.85.220.42]) by kanga.kvack.org (Postfix) with ESMTP id 291C56B00DD for ; Thu, 24 Oct 2013 21:12:33 -0400 (EDT) Received: by mail-pa0-f42.google.com with SMTP id kp14so1455436pab.1 for ; Thu, 24 Oct 2013 18:12:32 -0700 (PDT) Received: from psmtp.com ([74.125.245.197]) by mx.google.com with SMTP id hb3si3387069pac.181.2013.10.24.18.12.29 for ; Thu, 24 Oct 2013 18:12:31 -0700 (PDT) Received: by mail-ie0-f179.google.com with SMTP id aq17so5408363iec.10 for ; Thu, 24 Oct 2013 18:12:28 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <5269BCCC.6090509@codeaurora.org> References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> Date: Thu, 24 Oct 2013 18:12:28 -0700 Message-ID: Subject: Re: zram/zsmalloc issues in very low memory conditions From: Luigi Semenzato Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Olav Haugan Cc: Bob Liu , Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan wrote: > Hi Bob, Luigi, > > On 10/23/2013 5:55 PM, Bob Liu wrote: >> >> On 10/24/2013 05:51 AM, Olav Haugan wrote: >>> I am trying to use zram in very low memory conditions and I am having >>> some issues. zram is in the reclaim path. So if the system is very low >>> on memory the system is trying to reclaim pages by swapping out (in this >>> case to zram). However, since we are very low on memory zram fails to >>> get a page from zsmalloc and thus zram fails to store the page. We get >>> into a cycle where the system is low on memory so it tries to swap out >>> to get more memory but swap out fails because there is not enough memory >>> in the system! The major problem I am seeing is that there does not seem >>> to be a way for zram to tell the upper layers to stop swapping out >>> because the swap device is essentially "full" (since there is no more >>> memory available for zram pages). Has anyone thought about this issue >>> already and have ideas how to solve this or am I missing something and I >>> should not be seeing this issue? >>> >> >> The same question as Luigi "What do you want the system to do at this >> point?" >> >> If swap fails then OOM killer will be triggered, I don't think this will >> be a issue. > > I definitely don't want OOM killer to run since OOM killer can kill > critical processes (this is on Android so we have Android LMK to handle > the killing in a more "safe" way). However, what I am seeing is that > when I run low on memory zram fails to swap out and returns error but > the swap subsystem just continues to try to swap out even when this > error occurs (it tries over and over again very rapidly causing the > kernel to be filled with error messages [at least two error messages per > failure btw]). > > What I expected to happen is for the swap subsystem to stop trying to > swap out until memory is available to swap out. I guess this could be > handled several ways. Either 1) the swap subsystem, upon encountering an > error to swap out, backs off from trying to swap out for some time or 2) > zram informs the swap subsystem that the swap device is full. There is a lot I don't know, both about the specifics of your case and the MM subsystem in general, so I'll make some guesses. Don't trust anything I say here (as if you would anyway :-). As the system gets low on memory, the MM tries to reclaim it in various ways. The biggest (I think) sources of reclaim come from swapping out anonymous pages (process data), and discarding file-backed pages (code, for instance). The "swappiness" parameter decides how much attention to give to each of these types of memory. It's possible that you get in a situation in which attempts to swap out anonymous pages with zram fail because you're out of memory at that point, but then some memory is reclaimed by discarding file pages, and that's why you don't see OOM kills or kernel panic. Either way you should be really close to that moment, unless something funny is going on. You could try to snapshot the memory usage when those message are produced. You can, for instance, use SysRQ-M to dump a bunch of data in the syslog. You may also want to monitor the zram device utlization from the sysfs. It's possible that by the time you see those messages you're already thrashing badly and that things slowed down so much that you aren't quite getting to the OOM killer. You could try to reduce the size of your swap device, and/or change the swappiness. By the way, I am under the impression that Android uses the OOM killer as part of their memory management strategy. > Could this be handled by congestion control? However, I found the > following comment in the code in vmscan.c: > > * If the page is swapcache, write it back even if that would > * block, for some throttling. This happens by accident, because > * swap_backing_dev_info is bust: it doesn't reflect the > * congestion state of the swapdevs. Easy to fix, if needed. > > However, how would one update the congested state of zram when it > becomes un-congested? > >> By the way, could you take a try with zswap? Which can write pages to >> real swap device if compressed pool is full. > > zswap might not be feasible in all cases if you only have flash as > backing storage. Zswap can be configured to run without a backing storage. > >>> I am also seeing a couple other issues that I was wondering whether >>> folks have already thought about: >>> >>> 2) zsmalloc fails when the page allocated is at physical address 0 (pfn >> >> AFAIK, this will never happen. > > I can easily get this to happen since I have memory starting at physical > address 0. > > Thanks, > > Olav Haugan > > -- > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, > hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f48.google.com (mail-pa0-f48.google.com [209.85.220.48]) by kanga.kvack.org (Postfix) with ESMTP id 448BF6B00DD for ; Thu, 24 Oct 2013 22:59:20 -0400 (EDT) Received: by mail-pa0-f48.google.com with SMTP id bj1so4299111pad.21 for ; Thu, 24 Oct 2013 19:59:19 -0700 (PDT) Received: from psmtp.com ([74.125.245.175]) by mx.google.com with SMTP id sg3si2659025pbb.193.2013.10.24.19.59.18 for ; Thu, 24 Oct 2013 19:59:19 -0700 (PDT) Message-ID: <5269DE7B.6060106@oracle.com> Date: Fri, 25 Oct 2013 10:59:07 +0800 From: Bob Liu MIME-Version: 1.0 Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> In-Reply-To: <5269BCCC.6090509@codeaurora.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Olav Haugan Cc: minchan@kernel.org, sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, semenzato@google.com On 10/25/2013 08:35 AM, Olav Haugan wrote: > Hi Bob, Luigi, > > On 10/23/2013 5:55 PM, Bob Liu wrote: >> >> On 10/24/2013 05:51 AM, Olav Haugan wrote: >>> I am trying to use zram in very low memory conditions and I am having >>> some issues. zram is in the reclaim path. So if the system is very low >>> on memory the system is trying to reclaim pages by swapping out (in this >>> case to zram). However, since we are very low on memory zram fails to >>> get a page from zsmalloc and thus zram fails to store the page. We get >>> into a cycle where the system is low on memory so it tries to swap out >>> to get more memory but swap out fails because there is not enough memory >>> in the system! The major problem I am seeing is that there does not seem >>> to be a way for zram to tell the upper layers to stop swapping out >>> because the swap device is essentially "full" (since there is no more >>> memory available for zram pages). Has anyone thought about this issue >>> already and have ideas how to solve this or am I missing something and I >>> should not be seeing this issue? >>> >> >> The same question as Luigi "What do you want the system to do at this >> point?" >> >> If swap fails then OOM killer will be triggered, I don't think this will >> be a issue. > > I definitely don't want OOM killer to run since OOM killer can kill > critical processes (this is on Android so we have Android LMK to handle > the killing in a more "safe" way). However, what I am seeing is that > when I run low on memory zram fails to swap out and returns error but > the swap subsystem just continues to try to swap out even when this > error occurs (it tries over and over again very rapidly causing the > kernel to be filled with error messages [at least two error messages per > failure btw]). > A simple way to disable the error messages is delete the printk line in zram source code. > What I expected to happen is for the swap subsystem to stop trying to > swap out until memory is available to swap out. I guess this could be In my opinion, the system already entered heavy memory pressure state when zram fails to allocate a page. In this case, the OOM killer or Low Memory Killer should be waked up and kill some processes in order to free some memory pages. After this happen, the system free memory may above water mark and no swap will happen. Even swap happens again, it's unlikely that zram will fail to alloc page. If it fails again, OOM killer or LMK should be triggered once more. > handled several ways. Either 1) the swap subsystem, upon encountering an > error to swap out, backs off from trying to swap out for some time or 2) > zram informs the swap subsystem that the swap device is full. > > Could this be handled by congestion control? However, I found the > following comment in the code in vmscan.c: > > * If the page is swapcache, write it back even if that would > * block, for some throttling. This happens by accident, because > * swap_backing_dev_info is bust: it doesn't reflect the > * congestion state of the swapdevs. Easy to fix, if needed. > > However, how would one update the congested state of zram when it > becomes un-congested? > >> By the way, could you take a try with zswap? Which can write pages to >> real swap device if compressed pool is full. > > zswap might not be feasible in all cases if you only have flash as > backing storage. > Yes, that's still a problem of zswap. Perhaps you can create a swap file on the backing storage. I'll try to add a fake swap device for zswap, so that users can have one more choice besides zram. >>> I am also seeing a couple other issues that I was wondering whether >>> folks have already thought about: >>> >>> 2) zsmalloc fails when the page allocated is at physical address 0 (pfn >> >> AFAIK, this will never happen. > > I can easily get this to happen since I have memory starting at physical > address 0. > Could you confirm this? AFAIR, physical memory start from 0 usually reserved for some special usage. I used 'cat /proc/zoneinfo' on x86 and arm, neither of the 'start_pfn' was 0. -- Regards, -Bob -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f48.google.com (mail-pa0-f48.google.com [209.85.220.48]) by kanga.kvack.org (Postfix) with ESMTP id 34B9E6B00E0 for ; Fri, 25 Oct 2013 05:19:56 -0400 (EDT) Received: by mail-pa0-f48.google.com with SMTP id bj1so4797305pad.21 for ; Fri, 25 Oct 2013 02:19:55 -0700 (PDT) Received: from psmtp.com ([74.125.245.120]) by mx.google.com with SMTP id vs7si3549199pbc.115.2013.10.25.02.19.54 for ; Fri, 25 Oct 2013 02:19:55 -0700 (PDT) Received: by mail-we0-f170.google.com with SMTP id u57so3558710wes.1 for ; Fri, 25 Oct 2013 02:19:52 -0700 (PDT) Date: Fri, 25 Oct 2013 18:19:44 +0900 From: Minchan Kim Subject: Re: zram/zsmalloc issues in very low memory conditions Message-ID: <20131025091924.GA4970@gmail.com> References: <526844E6.1080307@codeaurora.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <526844E6.1080307@codeaurora.org> Sender: owner-linux-mm@kvack.org List-ID: To: Olav Haugan Cc: sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Hello, I had no enough time to think over your great questions since I should enjoy in Edinburgh so if I miss something, Sorry! On Wed, Oct 23, 2013 at 02:51:34PM -0700, Olav Haugan wrote: > I am trying to use zram in very low memory conditions and I am having > some issues. zram is in the reclaim path. So if the system is very low > on memory the system is trying to reclaim pages by swapping out (in this > case to zram). However, since we are very low on memory zram fails to > get a page from zsmalloc and thus zram fails to store the page. We get > into a cycle where the system is low on memory so it tries to swap out > to get more memory but swap out fails because there is not enough memory > in the system! The major problem I am seeing is that there does not seem > to be a way for zram to tell the upper layers to stop swapping out True. The zram is block device so at a moment, I don't want to make zram swap-specific if it's possible. > because the swap device is essentially "full" (since there is no more > memory available for zram pages). Has anyone thought about this issue > already and have ideas how to solve this or am I missing something and I > should not be seeing this issue? It's true. We might need feedback loop and it shoudn't be specific for zram-swap. One think I can imagine is that we could move failed victim pages into LRU active list when the swapout failed so VM will have more weight for file pages than anon ones. For detail, you could see AOP_WRITEPAGE_ACTIVATE and get_scan_count for detail. The problem is it's on fs layer while zram is on block layer so what I can think at a moment is follwing as diff --git a/mm/vmscan.c b/mm/vmscan.c index 8ed1b77..c80b0b4 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -502,6 +502,8 @@ static pageout_t pageout(struct page *page, struct address_space *mapping, if (!PageWriteback(page)) { /* synchronous write or broken a_ops? */ ClearPageReclaim(page); + if (PageError(page)) + return PAGE_ACTIVATE; } trace_mm_vmscan_writepage(page, trace_reclaim_flags(page)); inc_zone_page_state(page, NR_VMSCAN_WRITE); It doesn't prevent swapout at all but it should throttle pick up anonymous pages for reclaiming so file-backed pages will be preferred by VM so sometime, zsmalloc succeed to allocate a free page and swapout will resume again. > > I am also seeing a couple other issues that I was wondering whether > folks have already thought about: > > 1) The size of a swap device is statically computed when the swap device > is turned on (nr_swap_pages). The size of zram swap device is dynamic > since we are compressing the pages and thus the swap subsystem thinks > that the zram swap device is full when it is not really full. Any > plans/thoughts about the possibility of being able to update the size > and/or the # of available pages in a swap device on the fly? It's really good question. We could make zram's size bigger to prevent such problem when you set zram's disksize from the beginning but in this case, zram's meta(ie, struct table) size will be increased a bit. Is such memory overhead is critical for you? > > 2) zsmalloc fails when the page allocated is at physical address 0 (pfn > = 0) since the handle returned from zsmalloc is encoded as (, > ) and thus the resulting handle will be 0 (since obj_idx starts > at 0). zs_malloc returns the handle but does not distinguish between a > valid handle of 0 and a failure to allocate. A possible solution to this > would be to start the obj_idx at 1. Is this feasible? I think it's doable. > > Thanks, > > Olav Haugan > > -- > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, > hosted by The Linux Foundation -- Kind regards, Minchan Kim -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pd0-f180.google.com (mail-pd0-f180.google.com [209.85.192.180]) by kanga.kvack.org (Postfix) with ESMTP id 0F30E6B0036 for ; Thu, 31 Oct 2013 19:34:07 -0400 (EDT) Received: by mail-pd0-f180.google.com with SMTP id p10so3074245pdj.11 for ; Thu, 31 Oct 2013 16:34:07 -0700 (PDT) Received: from psmtp.com ([74.125.245.190]) by mx.google.com with SMTP id gu5si3430184pac.217.2013.10.31.16.34.06 for ; Thu, 31 Oct 2013 16:34:07 -0700 (PDT) Message-ID: <5272E8EB.5030900@codeaurora.org> Date: Thu, 31 Oct 2013 16:34:03 -0700 From: Olav Haugan MIME-Version: 1.0 Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Luigi Semenzato Cc: Bob Liu , Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org Hi Luigi, On 10/24/2013 6:12 PM, Luigi Semenzato wrote: > On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan wrote: >> Hi Bob, Luigi, >> >> On 10/23/2013 5:55 PM, Bob Liu wrote: >>> >>> On 10/24/2013 05:51 AM, Olav Haugan wrote: >> >>> By the way, could you take a try with zswap? Which can write pages to >>> real swap device if compressed pool is full. >> >> zswap might not be feasible in all cases if you only have flash as >> backing storage. > > Zswap can be configured to run without a backing storage. > I was under the impression that zswap requires a backing storage. Can you elaborate on how to configure zswap to not need a backing storage? Olav Haugan -- The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f42.google.com (mail-pa0-f42.google.com [209.85.220.42]) by kanga.kvack.org (Postfix) with ESMTP id 2CE3F6B0036 for ; Thu, 31 Oct 2013 20:25:28 -0400 (EDT) Received: by mail-pa0-f42.google.com with SMTP id kp14so3316219pab.1 for ; Thu, 31 Oct 2013 17:25:27 -0700 (PDT) Received: from psmtp.com ([74.125.245.138]) by mx.google.com with SMTP id mj9si3478562pab.335.2013.10.31.17.25.26 for ; Thu, 31 Oct 2013 17:25:27 -0700 (PDT) Received: by mail-ie0-f177.google.com with SMTP id e14so6178582iej.22 for ; Thu, 31 Oct 2013 17:25:25 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <5272E8EB.5030900@codeaurora.org> References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> <5272E8EB.5030900@codeaurora.org> Date: Thu, 31 Oct 2013 17:25:25 -0700 Message-ID: Subject: Re: zram/zsmalloc issues in very low memory conditions From: Luigi Semenzato Content-Type: multipart/alternative; boundary=047d7bdc10ec44951604ea129aa3 Sender: owner-linux-mm@kvack.org List-ID: To: Olav Haugan , Stephen Barber Cc: Bob Liu , Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org --047d7bdc10ec44951604ea129aa3 Content-Type: text/plain; charset=ISO-8859-1 Hi Olav, I haven't personally done it. Seth outlines the configuration in this thread: http://thread.gmane.org/gmane.linux.kernel.mm/105378/focus=105543 Stephen, can you add more detail from your experience? Thanks! Luigi On Thu, Oct 31, 2013 at 4:34 PM, Olav Haugan wrote: > Hi Luigi, > > On 10/24/2013 6:12 PM, Luigi Semenzato wrote: > > On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan > wrote: > >> Hi Bob, Luigi, > >> > >> On 10/23/2013 5:55 PM, Bob Liu wrote: > >>> > >>> On 10/24/2013 05:51 AM, Olav Haugan wrote: > >> > >>> By the way, could you take a try with zswap? Which can write pages to > >>> real swap device if compressed pool is full. > >> > >> zswap might not be feasible in all cases if you only have flash as > >> backing storage. > > > > Zswap can be configured to run without a backing storage. > > > > I was under the impression that zswap requires a backing storage. Can > you elaborate on how to configure zswap to not need a backing storage? > > > Olav Haugan > > -- > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, > hosted by The Linux Foundation > --047d7bdc10ec44951604ea129aa3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Olav,

I haven't perso= nally done it. =A0Seth outlines the configuration in this thread:

http://thread.gmane.org/gmane.linux.kernel.mm/105378/focus= =3D105543

Stephen, can you add more detail from your experience?<= /div>

Thanks!
Luigi

<= br>


On Thu, Oct 31, 2013 at 4:34 PM, Olav Haugan <ohaugan@codeaurora.org= > wrote:
Hi Luigi,

On 10/24/2013 6:12 PM, Luigi Semenzato wrote:
> On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan <ohaugan@codeaurora.org> wrote:
>> Hi Bob, Luigi,
>>
>> On 10/23/2013 5:55 PM, Bob Liu wrote:
>>>
>>> On 10/24/2013 05:51 AM, Olav Haugan wrote:
>>
>>> By the way, could you take a try with = zswap? Which can write pages to
>>> real swap device if compressed pool is full.
>>
>> zswap might not be feasible in all cases if you only have flash as=
>> backing storage.
>
> Zswap can be configured to run without a backing storage.
>

I was under the impression that zswap requires a backing storage. Can=
you elaborate on how to configure zswap to not need a backing storage?


Olav Haugan

--
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by The Linux Foundation

--047d7bdc10ec44951604ea129aa3-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f54.google.com (mail-pa0-f54.google.com [209.85.220.54]) by kanga.kvack.org (Postfix) with ESMTP id 250346B0036 for ; Thu, 31 Oct 2013 20:27:19 -0400 (EDT) Received: by mail-pa0-f54.google.com with SMTP id fa1so3264851pad.41 for ; Thu, 31 Oct 2013 17:27:18 -0700 (PDT) Received: from psmtp.com ([74.125.245.204]) by mx.google.com with SMTP id cj2si3180332pbc.297.2013.10.31.17.27.17 for ; Thu, 31 Oct 2013 17:27:18 -0700 (PDT) Received: by mail-ie0-f182.google.com with SMTP id as1so6382900iec.27 for ; Thu, 31 Oct 2013 17:27:16 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <5272E8EB.5030900@codeaurora.org> References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> <5272E8EB.5030900@codeaurora.org> Date: Thu, 31 Oct 2013 17:27:16 -0700 Message-ID: Subject: Re: zram/zsmalloc issues in very low memory conditions From: Luigi Semenzato Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Olav Haugan , Stephen Barber Cc: Bob Liu , Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org [apologies for the previous HTML email] Hi Olav, I haven't personally done it. Seth outlines the configuration in this thread: http://thread.gmane.org/gmane.linux.kernel.mm/105378/focus=105543 Stephen, can you add more detail from your experience? Thanks! On Thu, Oct 31, 2013 at 4:34 PM, Olav Haugan wrote: > Hi Luigi, > > On 10/24/2013 6:12 PM, Luigi Semenzato wrote: >> On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan wrote: >>> Hi Bob, Luigi, >>> >>> On 10/23/2013 5:55 PM, Bob Liu wrote: >>>> >>>> On 10/24/2013 05:51 AM, Olav Haugan wrote: >>> >>>> By the way, could you take a try with zswap? Which can write pages to >>>> real swap device if compressed pool is full. >>> >>> zswap might not be feasible in all cases if you only have flash as >>> backing storage. >> >> Zswap can be configured to run without a backing storage. >> > > I was under the impression that zswap requires a backing storage. Can > you elaborate on how to configure zswap to not need a backing storage? > > > Olav Haugan > > -- > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, > hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pb0-f53.google.com (mail-pb0-f53.google.com [209.85.160.53]) by kanga.kvack.org (Postfix) with ESMTP id B99506B0036 for ; Thu, 31 Oct 2013 20:36:06 -0400 (EDT) Received: by mail-pb0-f53.google.com with SMTP id up7so3530199pbc.40 for ; Thu, 31 Oct 2013 17:36:06 -0700 (PDT) Received: from psmtp.com ([74.125.245.202]) by mx.google.com with SMTP id it5si3229651pbc.65.2013.10.31.17.36.05 for ; Thu, 31 Oct 2013 17:36:05 -0700 (PDT) Message-ID: <5272F766.8000804@oracle.com> Date: Fri, 01 Nov 2013 08:35:50 +0800 From: Bob Liu MIME-Version: 1.0 Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> <5272E8EB.5030900@codeaurora.org> In-Reply-To: <5272E8EB.5030900@codeaurora.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Olav Haugan Cc: Luigi Semenzato , Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org Hi Olav, On 11/01/2013 07:34 AM, Olav Haugan wrote: > Hi Luigi, > > On 10/24/2013 6:12 PM, Luigi Semenzato wrote: >> On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan wrote: >>> Hi Bob, Luigi, >>> >>> On 10/23/2013 5:55 PM, Bob Liu wrote: >>>> >>>> On 10/24/2013 05:51 AM, Olav Haugan wrote: >>> >>>> By the way, could you take a try with zswap? Which can write pages to >>>> real swap device if compressed pool is full. >>> >>> zswap might not be feasible in all cases if you only have flash as >>> backing storage. >> >> Zswap can be configured to run without a backing storage. >> > > I was under the impression that zswap requires a backing storage. Can > you elaborate on how to configure zswap to not need a backing storage? > AFAIK, currently zswap can't be used without a backing storage. Perhaps you can take a try by creating a swap device backed by a file on storage. -- Regards, -Bob -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f41.google.com (mail-pa0-f41.google.com [209.85.220.41]) by kanga.kvack.org (Postfix) with ESMTP id BAC856B0035 for ; Fri, 1 Nov 2013 21:00:03 -0400 (EDT) Received: by mail-pa0-f41.google.com with SMTP id rd3so4737919pab.14 for ; Fri, 01 Nov 2013 18:00:03 -0700 (PDT) Received: from psmtp.com ([74.125.245.181]) by mx.google.com with SMTP id ba2si5090151pbc.358.2013.11.01.18.00.02 for ; Fri, 01 Nov 2013 18:00:02 -0700 (PDT) Message-ID: <52744E8F.3040405@codeaurora.org> Date: Fri, 01 Nov 2013 17:59:59 -0700 From: Olav Haugan MIME-Version: 1.0 Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> <20131025091924.GA4970@gmail.com> In-Reply-To: <20131025091924.GA4970@gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Minchan Kim Cc: sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org On 10/25/2013 2:19 AM, Minchan Kim wrote: > Hello, > > I had no enough time to think over your great questions since I should enjoy > in Edinburgh so if I miss something, Sorry! > > On Wed, Oct 23, 2013 at 02:51:34PM -0700, Olav Haugan wrote: >> I am trying to use zram in very low memory conditions and I am having >> some issues. zram is in the reclaim path. So if the system is very low >> on memory the system is trying to reclaim pages by swapping out (in this >> case to zram). However, since we are very low on memory zram fails to >> get a page from zsmalloc and thus zram fails to store the page. We get >> into a cycle where the system is low on memory so it tries to swap out >> to get more memory but swap out fails because there is not enough memory >> in the system! The major problem I am seeing is that there does not seem >> to be a way for zram to tell the upper layers to stop swapping out > > True. The zram is block device so at a moment, I don't want to make zram > swap-specific if it's possible. > >> because the swap device is essentially "full" (since there is no more >> memory available for zram pages). Has anyone thought about this issue >> already and have ideas how to solve this or am I missing something and I >> should not be seeing this issue? > > It's true. We might need feedback loop and it shoudn't be specific for > zram-swap. One think I can imagine is that we could move failed victim > pages into LRU active list when the swapout failed so VM will have more > weight for file pages than anon ones. For detail, you could see > AOP_WRITEPAGE_ACTIVATE and get_scan_count for detail. > > The problem is it's on fs layer while zram is on block layer so what I > can think at a moment is follwing as > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 8ed1b77..c80b0b4 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -502,6 +502,8 @@ static pageout_t pageout(struct page *page, struct address_space *mapping, > if (!PageWriteback(page)) { > /* synchronous write or broken a_ops? */ > ClearPageReclaim(page); > + if (PageError(page)) > + return PAGE_ACTIVATE; > } > trace_mm_vmscan_writepage(page, trace_reclaim_flags(page)); > inc_zone_page_state(page, NR_VMSCAN_WRITE); > > > It doesn't prevent swapout at all but it should throttle pick up anonymous > pages for reclaiming so file-backed pages will be preferred by VM so sometime, > zsmalloc succeed to allocate a free page and swapout will resume again. I tried the above suggestion but it does not seem to have any noticeable impact. The system is still trying to swap out at a very high rate after zram reported failure to swap out. The error logging is actually so much that my system crashed due to excessive logging (we have a watchdog that is not getting pet because the kernel is busy logging kernel messages). There isn't anything that can be set to tell the fs layer to back off completely for a while (congestion control)? Olav Haugan -- The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pd0-f179.google.com (mail-pd0-f179.google.com [209.85.192.179]) by kanga.kvack.org (Postfix) with ESMTP id 231D06B0035 for ; Fri, 1 Nov 2013 22:51:09 -0400 (EDT) Received: by mail-pd0-f179.google.com with SMTP id y10so4557214pdj.24 for ; Fri, 01 Nov 2013 19:51:08 -0700 (PDT) Received: from psmtp.com ([74.125.245.147]) by mx.google.com with SMTP id gl1si6219978pac.53.2013.11.01.19.51.07 for ; Fri, 01 Nov 2013 19:51:08 -0700 (PDT) Message-ID: <5274688D.7050902@oracle.com> Date: Sat, 02 Nov 2013 10:50:53 +0800 From: Bob Liu MIME-Version: 1.0 Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> <20131025091924.GA4970@gmail.com> <52744E8F.3040405@codeaurora.org> In-Reply-To: <52744E8F.3040405@codeaurora.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Olav Haugan Cc: Minchan Kim , sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Hi Olav, On 11/02/2013 08:59 AM, Olav Haugan wrote: > > I tried the above suggestion but it does not seem to have any noticeable > impact. The system is still trying to swap out at a very high rate after > zram reported failure to swap out. The error logging is actually so much > that my system crashed due to excessive logging (we have a watchdog that > is not getting pet because the kernel is busy logging kernel messages). > I have a question that why the low memory killer didn't get triggered in this situation? Is it possible to set the LMK a bit more aggressive? > There isn't anything that can be set to tell the fs layer to back off > completely for a while (congestion control)? > The other way I think might fix your issue is the same as your mentioned in your previous email. Set the congested bit for swap device also. Like: diff --git a/drivers/staging/zram/zram_drv.c b/drivers/staging/zram/zram_drv.c index 91d94b5..c4fc63e 100644 --- a/drivers/staging/zram/zram_drv.c +++ b/drivers/staging/zram/zram_drv.c @@ -474,6 +474,7 @@ static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, u32 index, if (!handle) { pr_info("Error allocating memory for compressed page: %u, size=%zu\n", index, clen); + blk_set_queue_congested(zram->disk->queue, BLK_RW_ASYNC); ret = -ENOMEM; goto out; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 8ed1b77..1c790ee 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -394,8 +394,6 @@ static inline int is_page_cache_freeable(struct page *page) static int may_write_to_queue(struct backing_dev_info *bdi, struct scan_control *sc) { - if (current->flags & PF_SWAPWRITE) - return 1; -------------------------------------------------------------- For the update of the congested state of zram, I think you can clear it from use space eg. after LMK triggered and reclaimed some memory. Of course this depends on zram driver to export a sysfs node like "/sys/block/zram0/clear_congested". -- Regards, -Bob -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pd0-f179.google.com (mail-pd0-f179.google.com [209.85.192.179]) by kanga.kvack.org (Postfix) with ESMTP id D680D6B0035 for ; Sat, 2 Nov 2013 03:40:51 -0400 (EDT) Received: by mail-pd0-f179.google.com with SMTP id y10so4730230pdj.24 for ; Sat, 02 Nov 2013 00:40:51 -0700 (PDT) Received: from psmtp.com ([74.125.245.166]) by mx.google.com with SMTP id j10si6593557pac.170.2013.11.02.00.40.50 for ; Sat, 02 Nov 2013 00:40:50 -0700 (PDT) Received: from smtp.stanford.edu (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 1ED8622663 for ; Sat, 2 Nov 2013 00:40:49 -0700 (PDT) Received: from mail-pa0-f49.google.com (mail-pa0-f49.google.com [209.85.220.49]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by smtp.stanford.edu (Postfix) with ESMTPS id BFC4A2260D for ; Sat, 2 Nov 2013 00:40:48 -0700 (PDT) Received: by mail-pa0-f49.google.com with SMTP id lj1so4963375pab.22 for ; Sat, 02 Nov 2013 00:40:48 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> <5272E8EB.5030900@codeaurora.org> Date: Sat, 2 Nov 2013 00:40:48 -0700 Message-ID: Subject: Re: zram/zsmalloc issues in very low memory conditions From: Stephen Barber Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Luigi Semenzato Cc: Olav Haugan , Bob Liu , Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org I did get zswap working without a backing store, using Bob's patches from http://thread.gmane.org/gmane.linux.kernel.mm/105627/focus=105642 applied to 3.11-rc6, The major issue we had with it was that i915 graphics buffers were getting corrupted somehow, which wasn't a problem with zram on the same kernel version. I'm sure with a little more investigation that problem could be fixed, though, and I don't think there were any other major problems with it in our (admittedly not very exhaustive) testing of those patches. Hope that helps, Steve Barber On Thu, Oct 31, 2013 at 5:27 PM, Luigi Semenzato wrote: > [apologies for the previous HTML email] > > Hi Olav, > > I haven't personally done it. Seth outlines the configuration in this thread: > > http://thread.gmane.org/gmane.linux.kernel.mm/105378/focus=105543 > > Stephen, can you add more detail from your experience? > > Thanks! > > > On Thu, Oct 31, 2013 at 4:34 PM, Olav Haugan wrote: >> Hi Luigi, >> >> On 10/24/2013 6:12 PM, Luigi Semenzato wrote: >>> On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan wrote: >>>> Hi Bob, Luigi, >>>> >>>> On 10/23/2013 5:55 PM, Bob Liu wrote: >>>>> >>>>> On 10/24/2013 05:51 AM, Olav Haugan wrote: >>>> >>>>> By the way, could you take a try with zswap? Which can write pages to >>>>> real swap device if compressed pool is full. >>>> >>>> zswap might not be feasible in all cases if you only have flash as >>>> backing storage. >>> >>> Zswap can be configured to run without a backing storage. >>> >> >> I was under the impression that zswap requires a backing storage. Can >> you elaborate on how to configure zswap to not need a backing storage? >> >> >> Olav Haugan >> >> -- >> The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, >> hosted by The Linux Foundation -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753200Ab3JWVvh (ORCPT ); Wed, 23 Oct 2013 17:51:37 -0400 Received: from smtp.codeaurora.org ([198.145.11.231]:50425 "EHLO smtp.codeaurora.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753000Ab3JWVvg (ORCPT ); Wed, 23 Oct 2013 17:51:36 -0400 Message-ID: <526844E6.1080307@codeaurora.org> Date: Wed, 23 Oct 2013 14:51:34 -0700 From: Olav Haugan Organization: Qualcomm Innovation Center, Inc. User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: minchan@kernel.org, sjenning@linux.vnet.ibm.com CC: linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: zram/zsmalloc issues in very low memory conditions Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org I am trying to use zram in very low memory conditions and I am having some issues. zram is in the reclaim path. So if the system is very low on memory the system is trying to reclaim pages by swapping out (in this case to zram). However, since we are very low on memory zram fails to get a page from zsmalloc and thus zram fails to store the page. We get into a cycle where the system is low on memory so it tries to swap out to get more memory but swap out fails because there is not enough memory in the system! The major problem I am seeing is that there does not seem to be a way for zram to tell the upper layers to stop swapping out because the swap device is essentially "full" (since there is no more memory available for zram pages). Has anyone thought about this issue already and have ideas how to solve this or am I missing something and I should not be seeing this issue? I am also seeing a couple other issues that I was wondering whether folks have already thought about: 1) The size of a swap device is statically computed when the swap device is turned on (nr_swap_pages). The size of zram swap device is dynamic since we are compressing the pages and thus the swap subsystem thinks that the zram swap device is full when it is not really full. Any plans/thoughts about the possibility of being able to update the size and/or the # of available pages in a swap device on the fly? 2) zsmalloc fails when the page allocated is at physical address 0 (pfn = 0) since the handle returned from zsmalloc is encoded as (, ) and thus the resulting handle will be 0 (since obj_idx starts at 0). zs_malloc returns the handle but does not distinguish between a valid handle of 0 and a failure to allocate. A possible solution to this would be to start the obj_idx at 1. Is this feasible? Thanks, Olav Haugan -- The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752798Ab3JWWR2 (ORCPT ); Wed, 23 Oct 2013 18:17:28 -0400 Received: from mail-ie0-f176.google.com ([209.85.223.176]:46196 "EHLO mail-ie0-f176.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751674Ab3JWWR0 (ORCPT ); Wed, 23 Oct 2013 18:17:26 -0400 MIME-Version: 1.0 In-Reply-To: <526844E6.1080307@codeaurora.org> References: <526844E6.1080307@codeaurora.org> Date: Wed, 23 Oct 2013 15:17:26 -0700 Message-ID: Subject: Re: zram/zsmalloc issues in very low memory conditions From: Luigi Semenzato To: Olav Haugan Cc: Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org (sorry about the HTML in the previous message) On Wed, Oct 23, 2013 at 2:51 PM, Olav Haugan wrote: > I am trying to use zram in very low memory conditions and I am having > some issues. zram is in the reclaim path. So if the system is very low > on memory the system is trying to reclaim pages by swapping out (in this > case to zram). However, since we are very low on memory zram fails to > get a page from zsmalloc and thus zram fails to store the page. We get > into a cycle where the system is low on memory so it tries to swap out > to get more memory but swap out fails because there is not enough memory > in the system! The major problem I am seeing is that there does not seem > to be a way for zram to tell the upper layers to stop swapping out > because the swap device is essentially "full" (since there is no more > memory available for zram pages). Has anyone thought about this issue > already and have ideas how to solve this or am I missing something and I > should not be seeing this issue? What do you want the system to do at this point? OOM kill? Also, if you are that low on memory, how are you preventing thrashing on the code pages? I am asking because we also use zram but haven't run into this problem---however we had to deal with other problems that motivate these questions. > > I am also seeing a couple other issues that I was wondering whether > folks have already thought about: > > 1) The size of a swap device is statically computed when the swap device > is turned on (nr_swap_pages). The size of zram swap device is dynamic > since we are compressing the pages and thus the swap subsystem thinks > that the zram swap device is full when it is not really full. Any > plans/thoughts about the possibility of being able to update the size > and/or the # of available pages in a swap device on the fly? That is a known limitation of zram. If you can predict your compression ratio and your working set size, it's not a big problem: allocate a swap device which, based on the expected compression ratio, will use up RAM until what's left is just enough for the working set. > 2) zsmalloc fails when the page allocated is at physical address 0 (pfn > = 0) since the handle returned from zsmalloc is encoded as (, > ) and thus the resulting handle will be 0 (since obj_idx starts > at 0). zs_malloc returns the handle but does not distinguish between a > valid handle of 0 and a failure to allocate. A possible solution to this > would be to start the obj_idx at 1. Is this feasible? > > Thanks, > > Olav Haugan > > -- > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, > hosted by The Linux Foundation > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753176Ab3JXAzd (ORCPT ); Wed, 23 Oct 2013 20:55:33 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:17766 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751790Ab3JXAzc (ORCPT ); Wed, 23 Oct 2013 20:55:32 -0400 Message-ID: <52686FF4.5000303@oracle.com> Date: Thu, 24 Oct 2013 08:55:16 +0800 From: Bob Liu User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130308 Thunderbird/17.0.4 MIME-Version: 1.0 To: Olav Haugan CC: minchan@kernel.org, sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> In-Reply-To: <526844E6.1080307@codeaurora.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Source-IP: ucsinet21.oracle.com [156.151.31.93] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/24/2013 05:51 AM, Olav Haugan wrote: > I am trying to use zram in very low memory conditions and I am having > some issues. zram is in the reclaim path. So if the system is very low > on memory the system is trying to reclaim pages by swapping out (in this > case to zram). However, since we are very low on memory zram fails to > get a page from zsmalloc and thus zram fails to store the page. We get > into a cycle where the system is low on memory so it tries to swap out > to get more memory but swap out fails because there is not enough memory > in the system! The major problem I am seeing is that there does not seem > to be a way for zram to tell the upper layers to stop swapping out > because the swap device is essentially "full" (since there is no more > memory available for zram pages). Has anyone thought about this issue > already and have ideas how to solve this or am I missing something and I > should not be seeing this issue? > The same question as Luigi "What do you want the system to do at this point?" If swap fails then OOM killer will be triggered, I don't think this will be a issue. By the way, could you take a try with zswap? Which can write pages to real swap device if compressed pool is full. > I am also seeing a couple other issues that I was wondering whether > folks have already thought about: > > 1) The size of a swap device is statically computed when the swap device > is turned on (nr_swap_pages). The size of zram swap device is dynamic > since we are compressing the pages and thus the swap subsystem thinks > that the zram swap device is full when it is not really full. Any > plans/thoughts about the possibility of being able to update the size > and/or the # of available pages in a swap device on the fly? > > 2) zsmalloc fails when the page allocated is at physical address 0 (pfn AFAIK, this will never happen. > = 0) since the handle returned from zsmalloc is encoded as (, > ) and thus the resulting handle will be 0 (since obj_idx starts > at 0). zs_malloc returns the handle but does not distinguish between a > valid handle of 0 and a failure to allocate. A possible solution to this > would be to start the obj_idx at 1. Is this feasible? > > Thanks, > > Olav Haugan > -- Regards, -Bob From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754931Ab3JXKmu (ORCPT ); Thu, 24 Oct 2013 06:42:50 -0400 Received: from mail-ie0-f182.google.com ([209.85.223.182]:34508 "EHLO mail-ie0-f182.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754213Ab3JXKmt (ORCPT ); Thu, 24 Oct 2013 06:42:49 -0400 MIME-Version: 1.0 In-Reply-To: <526844E6.1080307@codeaurora.org> References: <526844E6.1080307@codeaurora.org> Date: Thu, 24 Oct 2013 18:42:48 +0800 Message-ID: Subject: Re: zram/zsmalloc issues in very low memory conditions From: Weijie Yang To: Olav Haugan Cc: minchan@kernel.org, sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, semenzato@google.com, bob.liu@oracle.com Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Oct 24, 2013 at 5:51 AM, Olav Haugan wrote: > I am trying to use zram in very low memory conditions and I am having > some issues. zram is in the reclaim path. So if the system is very low > on memory the system is trying to reclaim pages by swapping out (in this > case to zram). However, since we are very low on memory zram fails to > get a page from zsmalloc and thus zram fails to store the page. We get > into a cycle where the system is low on memory so it tries to swap out > to get more memory but swap out fails because there is not enough memory > in the system! The major problem I am seeing is that there does not seem > to be a way for zram to tell the upper layers to stop swapping out > because the swap device is essentially "full" (since there is no more > memory available for zram pages). Has anyone thought about this issue > already and have ideas how to solve this or am I missing something and I > should not be seeing this issue? I agree with Luigi and Bob. zram's size is based on how many free memory you expect to use for zram. In my test, the compression ratio is about 1:4, of course the working sets may be different with yours. Further more, may be you can modify vm_swap_full() to let kernel free swap_entry aggressively. > I am also seeing a couple other issues that I was wondering whether > folks have already thought about: > > 1) The size of a swap device is statically computed when the swap device > is turned on (nr_swap_pages). The size of zram swap device is dynamic > since we are compressing the pages and thus the swap subsystem thinks > that the zram swap device is full when it is not really full. Any > plans/thoughts about the possibility of being able to update the size > and/or the # of available pages in a swap device on the fly? > > 2) zsmalloc fails when the page allocated is at physical address 0 (pfn > = 0) since the handle returned from zsmalloc is encoded as (, > ) and thus the resulting handle will be 0 (since obj_idx starts > at 0). zs_malloc returns the handle but does not distinguish between a > valid handle of 0 and a failure to allocate. A possible solution to this > would be to start the obj_idx at 1. Is this feasible? > > Thanks, > > Olav Haugan > > -- > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, > hosted by The Linux Foundation > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755129Ab3JYAf1 (ORCPT ); Thu, 24 Oct 2013 20:35:27 -0400 Received: from smtp.codeaurora.org ([198.145.11.231]:35429 "EHLO smtp.codeaurora.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753833Ab3JYAf0 (ORCPT ); Thu, 24 Oct 2013 20:35:26 -0400 Message-ID: <5269BCCC.6090509@codeaurora.org> Date: Thu, 24 Oct 2013 17:35:24 -0700 From: Olav Haugan Organization: Qualcomm Innovation Center, Inc. User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: Bob Liu CC: minchan@kernel.org, sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, semenzato@google.com Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> In-Reply-To: <52686FF4.5000303@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Bob, Luigi, On 10/23/2013 5:55 PM, Bob Liu wrote: > > On 10/24/2013 05:51 AM, Olav Haugan wrote: >> I am trying to use zram in very low memory conditions and I am having >> some issues. zram is in the reclaim path. So if the system is very low >> on memory the system is trying to reclaim pages by swapping out (in this >> case to zram). However, since we are very low on memory zram fails to >> get a page from zsmalloc and thus zram fails to store the page. We get >> into a cycle where the system is low on memory so it tries to swap out >> to get more memory but swap out fails because there is not enough memory >> in the system! The major problem I am seeing is that there does not seem >> to be a way for zram to tell the upper layers to stop swapping out >> because the swap device is essentially "full" (since there is no more >> memory available for zram pages). Has anyone thought about this issue >> already and have ideas how to solve this or am I missing something and I >> should not be seeing this issue? >> > > The same question as Luigi "What do you want the system to do at this > point?" > > If swap fails then OOM killer will be triggered, I don't think this will > be a issue. I definitely don't want OOM killer to run since OOM killer can kill critical processes (this is on Android so we have Android LMK to handle the killing in a more "safe" way). However, what I am seeing is that when I run low on memory zram fails to swap out and returns error but the swap subsystem just continues to try to swap out even when this error occurs (it tries over and over again very rapidly causing the kernel to be filled with error messages [at least two error messages per failure btw]). What I expected to happen is for the swap subsystem to stop trying to swap out until memory is available to swap out. I guess this could be handled several ways. Either 1) the swap subsystem, upon encountering an error to swap out, backs off from trying to swap out for some time or 2) zram informs the swap subsystem that the swap device is full. Could this be handled by congestion control? However, I found the following comment in the code in vmscan.c: * If the page is swapcache, write it back even if that would * block, for some throttling. This happens by accident, because * swap_backing_dev_info is bust: it doesn't reflect the * congestion state of the swapdevs. Easy to fix, if needed. However, how would one update the congested state of zram when it becomes un-congested? > By the way, could you take a try with zswap? Which can write pages to > real swap device if compressed pool is full. zswap might not be feasible in all cases if you only have flash as backing storage. >> I am also seeing a couple other issues that I was wondering whether >> folks have already thought about: >> >> 2) zsmalloc fails when the page allocated is at physical address 0 (pfn > > AFAIK, this will never happen. I can easily get this to happen since I have memory starting at physical address 0. Thanks, Olav Haugan -- The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755201Ab3JYBMa (ORCPT ); Thu, 24 Oct 2013 21:12:30 -0400 Received: from mail-ie0-f181.google.com ([209.85.223.181]:42795 "EHLO mail-ie0-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753807Ab3JYBM2 (ORCPT ); Thu, 24 Oct 2013 21:12:28 -0400 MIME-Version: 1.0 In-Reply-To: <5269BCCC.6090509@codeaurora.org> References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> Date: Thu, 24 Oct 2013 18:12:28 -0700 Message-ID: Subject: Re: zram/zsmalloc issues in very low memory conditions From: Luigi Semenzato To: Olav Haugan Cc: Bob Liu , Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan wrote: > Hi Bob, Luigi, > > On 10/23/2013 5:55 PM, Bob Liu wrote: >> >> On 10/24/2013 05:51 AM, Olav Haugan wrote: >>> I am trying to use zram in very low memory conditions and I am having >>> some issues. zram is in the reclaim path. So if the system is very low >>> on memory the system is trying to reclaim pages by swapping out (in this >>> case to zram). However, since we are very low on memory zram fails to >>> get a page from zsmalloc and thus zram fails to store the page. We get >>> into a cycle where the system is low on memory so it tries to swap out >>> to get more memory but swap out fails because there is not enough memory >>> in the system! The major problem I am seeing is that there does not seem >>> to be a way for zram to tell the upper layers to stop swapping out >>> because the swap device is essentially "full" (since there is no more >>> memory available for zram pages). Has anyone thought about this issue >>> already and have ideas how to solve this or am I missing something and I >>> should not be seeing this issue? >>> >> >> The same question as Luigi "What do you want the system to do at this >> point?" >> >> If swap fails then OOM killer will be triggered, I don't think this will >> be a issue. > > I definitely don't want OOM killer to run since OOM killer can kill > critical processes (this is on Android so we have Android LMK to handle > the killing in a more "safe" way). However, what I am seeing is that > when I run low on memory zram fails to swap out and returns error but > the swap subsystem just continues to try to swap out even when this > error occurs (it tries over and over again very rapidly causing the > kernel to be filled with error messages [at least two error messages per > failure btw]). > > What I expected to happen is for the swap subsystem to stop trying to > swap out until memory is available to swap out. I guess this could be > handled several ways. Either 1) the swap subsystem, upon encountering an > error to swap out, backs off from trying to swap out for some time or 2) > zram informs the swap subsystem that the swap device is full. There is a lot I don't know, both about the specifics of your case and the MM subsystem in general, so I'll make some guesses. Don't trust anything I say here (as if you would anyway :-). As the system gets low on memory, the MM tries to reclaim it in various ways. The biggest (I think) sources of reclaim come from swapping out anonymous pages (process data), and discarding file-backed pages (code, for instance). The "swappiness" parameter decides how much attention to give to each of these types of memory. It's possible that you get in a situation in which attempts to swap out anonymous pages with zram fail because you're out of memory at that point, but then some memory is reclaimed by discarding file pages, and that's why you don't see OOM kills or kernel panic. Either way you should be really close to that moment, unless something funny is going on. You could try to snapshot the memory usage when those message are produced. You can, for instance, use SysRQ-M to dump a bunch of data in the syslog. You may also want to monitor the zram device utlization from the sysfs. It's possible that by the time you see those messages you're already thrashing badly and that things slowed down so much that you aren't quite getting to the OOM killer. You could try to reduce the size of your swap device, and/or change the swappiness. By the way, I am under the impression that Android uses the OOM killer as part of their memory management strategy. > Could this be handled by congestion control? However, I found the > following comment in the code in vmscan.c: > > * If the page is swapcache, write it back even if that would > * block, for some throttling. This happens by accident, because > * swap_backing_dev_info is bust: it doesn't reflect the > * congestion state of the swapdevs. Easy to fix, if needed. > > However, how would one update the congested state of zram when it > becomes un-congested? > >> By the way, could you take a try with zswap? Which can write pages to >> real swap device if compressed pool is full. > > zswap might not be feasible in all cases if you only have flash as > backing storage. Zswap can be configured to run without a backing storage. > >>> I am also seeing a couple other issues that I was wondering whether >>> folks have already thought about: >>> >>> 2) zsmalloc fails when the page allocated is at physical address 0 (pfn >> >> AFAIK, this will never happen. > > I can easily get this to happen since I have memory starting at physical > address 0. > > Thanks, > > Olav Haugan > > -- > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, > hosted by The Linux Foundation From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755529Ab3JYC7W (ORCPT ); Thu, 24 Oct 2013 22:59:22 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:33781 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754056Ab3JYC7V (ORCPT ); Thu, 24 Oct 2013 22:59:21 -0400 Message-ID: <5269DE7B.6060106@oracle.com> Date: Fri, 25 Oct 2013 10:59:07 +0800 From: Bob Liu User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130308 Thunderbird/17.0.4 MIME-Version: 1.0 To: Olav Haugan CC: minchan@kernel.org, sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, semenzato@google.com Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> In-Reply-To: <5269BCCC.6090509@codeaurora.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Source-IP: acsinet21.oracle.com [141.146.126.237] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/25/2013 08:35 AM, Olav Haugan wrote: > Hi Bob, Luigi, > > On 10/23/2013 5:55 PM, Bob Liu wrote: >> >> On 10/24/2013 05:51 AM, Olav Haugan wrote: >>> I am trying to use zram in very low memory conditions and I am having >>> some issues. zram is in the reclaim path. So if the system is very low >>> on memory the system is trying to reclaim pages by swapping out (in this >>> case to zram). However, since we are very low on memory zram fails to >>> get a page from zsmalloc and thus zram fails to store the page. We get >>> into a cycle where the system is low on memory so it tries to swap out >>> to get more memory but swap out fails because there is not enough memory >>> in the system! The major problem I am seeing is that there does not seem >>> to be a way for zram to tell the upper layers to stop swapping out >>> because the swap device is essentially "full" (since there is no more >>> memory available for zram pages). Has anyone thought about this issue >>> already and have ideas how to solve this or am I missing something and I >>> should not be seeing this issue? >>> >> >> The same question as Luigi "What do you want the system to do at this >> point?" >> >> If swap fails then OOM killer will be triggered, I don't think this will >> be a issue. > > I definitely don't want OOM killer to run since OOM killer can kill > critical processes (this is on Android so we have Android LMK to handle > the killing in a more "safe" way). However, what I am seeing is that > when I run low on memory zram fails to swap out and returns error but > the swap subsystem just continues to try to swap out even when this > error occurs (it tries over and over again very rapidly causing the > kernel to be filled with error messages [at least two error messages per > failure btw]). > A simple way to disable the error messages is delete the printk line in zram source code. > What I expected to happen is for the swap subsystem to stop trying to > swap out until memory is available to swap out. I guess this could be In my opinion, the system already entered heavy memory pressure state when zram fails to allocate a page. In this case, the OOM killer or Low Memory Killer should be waked up and kill some processes in order to free some memory pages. After this happen, the system free memory may above water mark and no swap will happen. Even swap happens again, it's unlikely that zram will fail to alloc page. If it fails again, OOM killer or LMK should be triggered once more. > handled several ways. Either 1) the swap subsystem, upon encountering an > error to swap out, backs off from trying to swap out for some time or 2) > zram informs the swap subsystem that the swap device is full. > > Could this be handled by congestion control? However, I found the > following comment in the code in vmscan.c: > > * If the page is swapcache, write it back even if that would > * block, for some throttling. This happens by accident, because > * swap_backing_dev_info is bust: it doesn't reflect the > * congestion state of the swapdevs. Easy to fix, if needed. > > However, how would one update the congested state of zram when it > becomes un-congested? > >> By the way, could you take a try with zswap? Which can write pages to >> real swap device if compressed pool is full. > > zswap might not be feasible in all cases if you only have flash as > backing storage. > Yes, that's still a problem of zswap. Perhaps you can create a swap file on the backing storage. I'll try to add a fake swap device for zswap, so that users can have one more choice besides zram. >>> I am also seeing a couple other issues that I was wondering whether >>> folks have already thought about: >>> >>> 2) zsmalloc fails when the page allocated is at physical address 0 (pfn >> >> AFAIK, this will never happen. > > I can easily get this to happen since I have memory starting at physical > address 0. > Could you confirm this? AFAIR, physical memory start from 0 usually reserved for some special usage. I used 'cat /proc/zoneinfo' on x86 and arm, neither of the 'start_pfn' was 0. -- Regards, -Bob From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752377Ab3JYJTz (ORCPT ); Fri, 25 Oct 2013 05:19:55 -0400 Received: from mail-wi0-f177.google.com ([209.85.212.177]:54004 "EHLO mail-wi0-f177.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751377Ab3JYJTx (ORCPT ); Fri, 25 Oct 2013 05:19:53 -0400 Date: Fri, 25 Oct 2013 18:19:44 +0900 From: Minchan Kim To: Olav Haugan Cc: sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: zram/zsmalloc issues in very low memory conditions Message-ID: <20131025091924.GA4970@gmail.com> References: <526844E6.1080307@codeaurora.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <526844E6.1080307@codeaurora.org> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello, I had no enough time to think over your great questions since I should enjoy in Edinburgh so if I miss something, Sorry! On Wed, Oct 23, 2013 at 02:51:34PM -0700, Olav Haugan wrote: > I am trying to use zram in very low memory conditions and I am having > some issues. zram is in the reclaim path. So if the system is very low > on memory the system is trying to reclaim pages by swapping out (in this > case to zram). However, since we are very low on memory zram fails to > get a page from zsmalloc and thus zram fails to store the page. We get > into a cycle where the system is low on memory so it tries to swap out > to get more memory but swap out fails because there is not enough memory > in the system! The major problem I am seeing is that there does not seem > to be a way for zram to tell the upper layers to stop swapping out True. The zram is block device so at a moment, I don't want to make zram swap-specific if it's possible. > because the swap device is essentially "full" (since there is no more > memory available for zram pages). Has anyone thought about this issue > already and have ideas how to solve this or am I missing something and I > should not be seeing this issue? It's true. We might need feedback loop and it shoudn't be specific for zram-swap. One think I can imagine is that we could move failed victim pages into LRU active list when the swapout failed so VM will have more weight for file pages than anon ones. For detail, you could see AOP_WRITEPAGE_ACTIVATE and get_scan_count for detail. The problem is it's on fs layer while zram is on block layer so what I can think at a moment is follwing as diff --git a/mm/vmscan.c b/mm/vmscan.c index 8ed1b77..c80b0b4 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -502,6 +502,8 @@ static pageout_t pageout(struct page *page, struct address_space *mapping, if (!PageWriteback(page)) { /* synchronous write or broken a_ops? */ ClearPageReclaim(page); + if (PageError(page)) + return PAGE_ACTIVATE; } trace_mm_vmscan_writepage(page, trace_reclaim_flags(page)); inc_zone_page_state(page, NR_VMSCAN_WRITE); It doesn't prevent swapout at all but it should throttle pick up anonymous pages for reclaiming so file-backed pages will be preferred by VM so sometime, zsmalloc succeed to allocate a free page and swapout will resume again. > > I am also seeing a couple other issues that I was wondering whether > folks have already thought about: > > 1) The size of a swap device is statically computed when the swap device > is turned on (nr_swap_pages). The size of zram swap device is dynamic > since we are compressing the pages and thus the swap subsystem thinks > that the zram swap device is full when it is not really full. Any > plans/thoughts about the possibility of being able to update the size > and/or the # of available pages in a swap device on the fly? It's really good question. We could make zram's size bigger to prevent such problem when you set zram's disksize from the beginning but in this case, zram's meta(ie, struct table) size will be increased a bit. Is such memory overhead is critical for you? > > 2) zsmalloc fails when the page allocated is at physical address 0 (pfn > = 0) since the handle returned from zsmalloc is encoded as (, > ) and thus the resulting handle will be 0 (since obj_idx starts > at 0). zs_malloc returns the handle but does not distinguish between a > valid handle of 0 and a failure to allocate. A possible solution to this > would be to start the obj_idx at 1. Is this feasible? I think it's doable. > > Thanks, > > Olav Haugan > > -- > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, > hosted by The Linux Foundation -- Kind regards, Minchan Kim From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755612Ab3JaXeI (ORCPT ); Thu, 31 Oct 2013 19:34:08 -0400 Received: from smtp.codeaurora.org ([198.145.11.231]:41840 "EHLO smtp.codeaurora.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751766Ab3JaXeG (ORCPT ); Thu, 31 Oct 2013 19:34:06 -0400 Message-ID: <5272E8EB.5030900@codeaurora.org> Date: Thu, 31 Oct 2013 16:34:03 -0700 From: Olav Haugan Organization: Qualcomm Innovation Center, Inc. User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: Luigi Semenzato CC: Bob Liu , Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Luigi, On 10/24/2013 6:12 PM, Luigi Semenzato wrote: > On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan wrote: >> Hi Bob, Luigi, >> >> On 10/23/2013 5:55 PM, Bob Liu wrote: >>> >>> On 10/24/2013 05:51 AM, Olav Haugan wrote: >> >>> By the way, could you take a try with zswap? Which can write pages to >>> real swap device if compressed pool is full. >> >> zswap might not be feasible in all cases if you only have flash as >> backing storage. > > Zswap can be configured to run without a backing storage. > I was under the impression that zswap requires a backing storage. Can you elaborate on how to configure zswap to not need a backing storage? Olav Haugan -- The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753720Ab3KAA1S (ORCPT ); Thu, 31 Oct 2013 20:27:18 -0400 Received: from mail-ie0-f175.google.com ([209.85.223.175]:35332 "EHLO mail-ie0-f175.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751978Ab3KAA1Q (ORCPT ); Thu, 31 Oct 2013 20:27:16 -0400 MIME-Version: 1.0 In-Reply-To: <5272E8EB.5030900@codeaurora.org> References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> <5272E8EB.5030900@codeaurora.org> Date: Thu, 31 Oct 2013 17:27:16 -0700 Message-ID: Subject: Re: zram/zsmalloc issues in very low memory conditions From: Luigi Semenzato To: Olav Haugan , Stephen Barber Cc: Bob Liu , Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org [apologies for the previous HTML email] Hi Olav, I haven't personally done it. Seth outlines the configuration in this thread: http://thread.gmane.org/gmane.linux.kernel.mm/105378/focus=105543 Stephen, can you add more detail from your experience? Thanks! On Thu, Oct 31, 2013 at 4:34 PM, Olav Haugan wrote: > Hi Luigi, > > On 10/24/2013 6:12 PM, Luigi Semenzato wrote: >> On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan wrote: >>> Hi Bob, Luigi, >>> >>> On 10/23/2013 5:55 PM, Bob Liu wrote: >>>> >>>> On 10/24/2013 05:51 AM, Olav Haugan wrote: >>> >>>> By the way, could you take a try with zswap? Which can write pages to >>>> real swap device if compressed pool is full. >>> >>> zswap might not be feasible in all cases if you only have flash as >>> backing storage. >> >> Zswap can be configured to run without a backing storage. >> > > I was under the impression that zswap requires a backing storage. Can > you elaborate on how to configure zswap to not need a backing storage? > > > Olav Haugan > > -- > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, > hosted by The Linux Foundation From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755557Ab3KAAgJ (ORCPT ); Thu, 31 Oct 2013 20:36:09 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:49147 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752508Ab3KAAgI (ORCPT ); Thu, 31 Oct 2013 20:36:08 -0400 Message-ID: <5272F766.8000804@oracle.com> Date: Fri, 01 Nov 2013 08:35:50 +0800 From: Bob Liu User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130308 Thunderbird/17.0.4 MIME-Version: 1.0 To: Olav Haugan CC: Luigi Semenzato , Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> <5272E8EB.5030900@codeaurora.org> In-Reply-To: <5272E8EB.5030900@codeaurora.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Source-IP: acsinet22.oracle.com [141.146.126.238] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Olav, On 11/01/2013 07:34 AM, Olav Haugan wrote: > Hi Luigi, > > On 10/24/2013 6:12 PM, Luigi Semenzato wrote: >> On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan wrote: >>> Hi Bob, Luigi, >>> >>> On 10/23/2013 5:55 PM, Bob Liu wrote: >>>> >>>> On 10/24/2013 05:51 AM, Olav Haugan wrote: >>> >>>> By the way, could you take a try with zswap? Which can write pages to >>>> real swap device if compressed pool is full. >>> >>> zswap might not be feasible in all cases if you only have flash as >>> backing storage. >> >> Zswap can be configured to run without a backing storage. >> > > I was under the impression that zswap requires a backing storage. Can > you elaborate on how to configure zswap to not need a backing storage? > AFAIK, currently zswap can't be used without a backing storage. Perhaps you can take a try by creating a swap device backed by a file on storage. -- Regards, -Bob From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753243Ab3KBBAD (ORCPT ); Fri, 1 Nov 2013 21:00:03 -0400 Received: from smtp.codeaurora.org ([198.145.11.231]:56525 "EHLO smtp.codeaurora.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751894Ab3KBBAB (ORCPT ); Fri, 1 Nov 2013 21:00:01 -0400 Message-ID: <52744E8F.3040405@codeaurora.org> Date: Fri, 01 Nov 2013 17:59:59 -0700 From: Olav Haugan Organization: Qualcomm Innovation Center, Inc. User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: Minchan Kim CC: sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> <20131025091924.GA4970@gmail.com> In-Reply-To: <20131025091924.GA4970@gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/25/2013 2:19 AM, Minchan Kim wrote: > Hello, > > I had no enough time to think over your great questions since I should enjoy > in Edinburgh so if I miss something, Sorry! > > On Wed, Oct 23, 2013 at 02:51:34PM -0700, Olav Haugan wrote: >> I am trying to use zram in very low memory conditions and I am having >> some issues. zram is in the reclaim path. So if the system is very low >> on memory the system is trying to reclaim pages by swapping out (in this >> case to zram). However, since we are very low on memory zram fails to >> get a page from zsmalloc and thus zram fails to store the page. We get >> into a cycle where the system is low on memory so it tries to swap out >> to get more memory but swap out fails because there is not enough memory >> in the system! The major problem I am seeing is that there does not seem >> to be a way for zram to tell the upper layers to stop swapping out > > True. The zram is block device so at a moment, I don't want to make zram > swap-specific if it's possible. > >> because the swap device is essentially "full" (since there is no more >> memory available for zram pages). Has anyone thought about this issue >> already and have ideas how to solve this or am I missing something and I >> should not be seeing this issue? > > It's true. We might need feedback loop and it shoudn't be specific for > zram-swap. One think I can imagine is that we could move failed victim > pages into LRU active list when the swapout failed so VM will have more > weight for file pages than anon ones. For detail, you could see > AOP_WRITEPAGE_ACTIVATE and get_scan_count for detail. > > The problem is it's on fs layer while zram is on block layer so what I > can think at a moment is follwing as > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 8ed1b77..c80b0b4 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -502,6 +502,8 @@ static pageout_t pageout(struct page *page, struct address_space *mapping, > if (!PageWriteback(page)) { > /* synchronous write or broken a_ops? */ > ClearPageReclaim(page); > + if (PageError(page)) > + return PAGE_ACTIVATE; > } > trace_mm_vmscan_writepage(page, trace_reclaim_flags(page)); > inc_zone_page_state(page, NR_VMSCAN_WRITE); > > > It doesn't prevent swapout at all but it should throttle pick up anonymous > pages for reclaiming so file-backed pages will be preferred by VM so sometime, > zsmalloc succeed to allocate a free page and swapout will resume again. I tried the above suggestion but it does not seem to have any noticeable impact. The system is still trying to swap out at a very high rate after zram reported failure to swap out. The error logging is actually so much that my system crashed due to excessive logging (we have a watchdog that is not getting pet because the kernel is busy logging kernel messages). There isn't anything that can be set to tell the fs layer to back off completely for a while (congestion control)? Olav Haugan -- The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754143Ab3KBCvM (ORCPT ); Fri, 1 Nov 2013 22:51:12 -0400 Received: from aserp1040.oracle.com ([141.146.126.69]:48486 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753759Ab3KBCvK (ORCPT ); Fri, 1 Nov 2013 22:51:10 -0400 Message-ID: <5274688D.7050902@oracle.com> Date: Sat, 02 Nov 2013 10:50:53 +0800 From: Bob Liu User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130308 Thunderbird/17.0.4 MIME-Version: 1.0 To: Olav Haugan CC: Minchan Kim , sjenning@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: zram/zsmalloc issues in very low memory conditions References: <526844E6.1080307@codeaurora.org> <20131025091924.GA4970@gmail.com> <52744E8F.3040405@codeaurora.org> In-Reply-To: <52744E8F.3040405@codeaurora.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Source-IP: acsinet22.oracle.com [141.146.126.238] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Olav, On 11/02/2013 08:59 AM, Olav Haugan wrote: > > I tried the above suggestion but it does not seem to have any noticeable > impact. The system is still trying to swap out at a very high rate after > zram reported failure to swap out. The error logging is actually so much > that my system crashed due to excessive logging (we have a watchdog that > is not getting pet because the kernel is busy logging kernel messages). > I have a question that why the low memory killer didn't get triggered in this situation? Is it possible to set the LMK a bit more aggressive? > There isn't anything that can be set to tell the fs layer to back off > completely for a while (congestion control)? > The other way I think might fix your issue is the same as your mentioned in your previous email. Set the congested bit for swap device also. Like: diff --git a/drivers/staging/zram/zram_drv.c b/drivers/staging/zram/zram_drv.c index 91d94b5..c4fc63e 100644 --- a/drivers/staging/zram/zram_drv.c +++ b/drivers/staging/zram/zram_drv.c @@ -474,6 +474,7 @@ static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, u32 index, if (!handle) { pr_info("Error allocating memory for compressed page: %u, size=%zu\n", index, clen); + blk_set_queue_congested(zram->disk->queue, BLK_RW_ASYNC); ret = -ENOMEM; goto out; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 8ed1b77..1c790ee 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -394,8 +394,6 @@ static inline int is_page_cache_freeable(struct page *page) static int may_write_to_queue(struct backing_dev_info *bdi, struct scan_control *sc) { - if (current->flags & PF_SWAPWRITE) - return 1; -------------------------------------------------------------- For the update of the congested state of zram, I think you can clear it from use space eg. after LMK triggered and reclaimed some memory. Of course this depends on zram driver to export a sysfs node like "/sys/block/zram0/clear_congested". -- Regards, -Bob From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751772Ab3KBHu3 (ORCPT ); Sat, 2 Nov 2013 03:50:29 -0400 Received: from smtp3.Stanford.EDU ([171.67.219.83]:43766 "EHLO smtp.stanford.edu" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750979Ab3KBHu2 (ORCPT ); Sat, 2 Nov 2013 03:50:28 -0400 X-Greylist: delayed 579 seconds by postgrey-1.27 at vger.kernel.org; Sat, 02 Nov 2013 03:50:28 EDT MIME-Version: 1.0 In-Reply-To: References: <526844E6.1080307@codeaurora.org> <52686FF4.5000303@oracle.com> <5269BCCC.6090509@codeaurora.org> <5272E8EB.5030900@codeaurora.org> Date: Sat, 2 Nov 2013 00:40:48 -0700 Message-ID: Subject: Re: zram/zsmalloc issues in very low memory conditions From: Stephen Barber To: Luigi Semenzato Cc: Olav Haugan , Bob Liu , Minchan Kim , Seth Jennings , linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org I did get zswap working without a backing store, using Bob's patches from http://thread.gmane.org/gmane.linux.kernel.mm/105627/focus=105642 applied to 3.11-rc6, The major issue we had with it was that i915 graphics buffers were getting corrupted somehow, which wasn't a problem with zram on the same kernel version. I'm sure with a little more investigation that problem could be fixed, though, and I don't think there were any other major problems with it in our (admittedly not very exhaustive) testing of those patches. Hope that helps, Steve Barber On Thu, Oct 31, 2013 at 5:27 PM, Luigi Semenzato wrote: > [apologies for the previous HTML email] > > Hi Olav, > > I haven't personally done it. Seth outlines the configuration in this thread: > > http://thread.gmane.org/gmane.linux.kernel.mm/105378/focus=105543 > > Stephen, can you add more detail from your experience? > > Thanks! > > > On Thu, Oct 31, 2013 at 4:34 PM, Olav Haugan wrote: >> Hi Luigi, >> >> On 10/24/2013 6:12 PM, Luigi Semenzato wrote: >>> On Thu, Oct 24, 2013 at 5:35 PM, Olav Haugan wrote: >>>> Hi Bob, Luigi, >>>> >>>> On 10/23/2013 5:55 PM, Bob Liu wrote: >>>>> >>>>> On 10/24/2013 05:51 AM, Olav Haugan wrote: >>>> >>>>> By the way, could you take a try with zswap? Which can write pages to >>>>> real swap device if compressed pool is full. >>>> >>>> zswap might not be feasible in all cases if you only have flash as >>>> backing storage. >>> >>> Zswap can be configured to run without a backing storage. >>> >> >> I was under the impression that zswap requires a backing storage. Can >> you elaborate on how to configure zswap to not need a backing storage? >> >> >> Olav Haugan >> >> -- >> The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, >> hosted by The Linux Foundation