Why would some process swap in place of reclaiming free(cached) memory

Why would some process swap in place of reclaiming free(cached) memory

[Soham Chakraborty]

Hi all,

I posted this question in linux-mm list as well but it didn't spawn much
interest. So, I am putting the same question here also. Would love if
someone can put some traction.

The question is salivating and simple. When we have free and lotsa cached
memory in a system (irrespective of distro and kernel), why would some
process end up in swap space, I understand the overcommit mode of vm, the
default value of 0 and what it does. I also know that the cache might be
dirty and we don't might not want to allocate free pages thinking that we
might go under the watermark of low. But, I mainly end up getting this
question from end users - why would things swap if I have free memory. I
guess, technically we can't stop swapping even if we set vm.swappiness to 0
but well, I think I have said what I need to said. If you need data, I can
provide those.

Any reference to source where it gives the condition to transfer some pages
to swap, despite of having free memory, would be highly appreciated. I
tried to find in mm/vmscan.c but didn't get much success.

Soham
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130220/9db9c66e/attachment.html 

Why would some process swap in place of reclaiming free(cached) memory

[michi1 at michaelblizek.twilightparadox.com]

Hi!

On 20:31 Wed 20 Feb     , Soham Chakraborty wrote:
> Hi all,
> 
> I posted this question in linux-mm list as well but it didn't spawn much
> interest. So, I am putting the same question here also. Would love if
> someone can put some traction.
> 
> The question is salivating and simple. When we have free and lotsa cached
> memory in a system (irrespective of distro and kernel), why would some
> process end up in swap space,

Freeing cache memory does not need to be faster than swapping. The cached
files might be accessed more often than memory of some process. It is a
hard balancing act.

> I understand the overcommit mode of vm, the
> default value of 0 and what it does. I also know that the cache might be
> dirty and we don't might not want to allocate free pages thinking that we
> might go under the watermark of low. But, I mainly end up getting this
> question from end users - why would things swap if I have free memory.

I think this might indeed make sense in some cases. If there is lots of
memory used which is never accessed, it could be swapped out early. This
would prevent write activity if memory gets tight later. Also note that there
is a entry "SwapCached" in /proc/meminfo. It might be possible that the pages
swapped out pages are present in both swap *and* memory. Just a possibility...

> I
> guess, technically we can't stop swapping even if we set vm.swappiness to 0
> but well, I think I have said what I need to said. If you need data, I can
> provide those.

Swapping can be prevented.
1) You can disable all swap space in the system
2) Alternatively, you can call mlock() if you only care about few processes.

> ...

	-Michi
-- 
programing a layer 3+4 network protocol for mesh networks
see http://michaelblizek.twilightparadox.com

Why would some process swap in place of reclaiming free(cached) memory

[Greg Freemyer]

Soham Chakraborty <sohamwonderpiku4u@gmail.com> wrote:

>Hi all,
>
>I posted this question in linux-mm list as well but it didn't spawn
>much
>interest. So, I am putting the same question here also. Would love if
>someone can put some traction.
>
>The question is salivating and simple. When we have free and lotsa
>cached
>memory in a system (irrespective of distro and kernel), why would some
>process end up in swap space, I understand the overcommit mode of vm,
>the
>default value of 0 and what it does. I also know that the cache might
>be
>dirty and we don't might not want to allocate free pages thinking that
>we
>might go under the watermark of low. But, I mainly end up getting this
>question from end users - why would things swap if I have free memory.
>I
>guess, technically we can't stop swapping even if we set vm.swappiness
>to 0
>but well, I think I have said what I need to said. If you need data, I
>can
>provide those.
>
>Any reference to source where it gives the condition to transfer some
>pages
>to swap, despite of having free memory, would be highly appreciated. I
>tried to find in mm/vmscan.c but didn't get much success.
>
>Soham

I am not a mm expert, but let me pose a question and let someone else answer it.

What happens to the paging counts when a large program is exec'ed.

I know the pages are kept on disk initially and then swapped in as needed/referenced.  Thus if there are code pages where none of the code in the page has been executed since starting the app, then that page will remain on disk until the application logic eventually invokes it.  At that point it will be demand loaded from disk.

During that early stage, do the never used pages show as swapped out?  If so, why would anyone want all those never used pages to be brought into ram just to sit there unused?

The end result is that that swap'ed pages represent pages available to be swapped in, but not the number of pages that were actually swapped out at some point in time.

Greg
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Why would some process swap in place of reclaiming free(cached) memory

[Mandeep Sandhu]

> I know the pages are kept on disk initially and then swapped in as needed/referenced.  Thus if there are code pages where none of the code in the page has been executed since starting the app, then that page will remain on disk until the application logic eventually invokes it.  At that point it will be demand loaded from disk.
>
> During that early stage, do the never used pages show as swapped out?  If so, why would anyone want all those never used pages to be brought into ram just to sit there unused?

Are you saying that pages of code segment (of an executable) which
have not been accessed yet, are copied onto the swap space from their
location on the disk when the program is exec'ed? Or do they remain on
disk (in their original location) and loaded into memory only when
accessed (and _then_ possibly swapped out)? If they remain on disk,
then that wouldn't show up in swap space, right?

-mandeep

>
> The end result is that that swap'ed pages represent pages available to be swapped in, but not the number of pages that were actually swapped out at some point in time.
>
> Greg
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Why would some process swap in place of reclaiming free(cached) memory

[Soham Chakraborty]

On Thu, Feb 21, 2013 at 10:45 AM, Mandeep Sandhu <
mandeepsandhu.chd@gmail.com> wrote:

> > I know the pages are kept on disk initially and then swapped in as
> needed/referenced.  Thus if there are code pages where none of the code in
> the page has been executed since starting the app, then that page will
> remain on disk until the application logic eventually invokes it.  At that
> point it will be demand loaded from disk.
> >
> > During that early stage, do the never used pages show as swapped out?
>  If so, why would anyone want all those never used pages to be brought into
> ram just to sit there unused?
>
> Are you saying that pages of code segment (of an executable) which
> have not been accessed yet, are copied onto the swap space from their
> location on the disk when the program is exec'ed? Or do they remain on
> disk (in their original location) and loaded into memory only when
> accessed (and _then_ possibly swapped out)? If they remain on disk,
> then that wouldn't show up in swap space, right?
>

If the code segment of an executable which hasn't been accessed, they will
remain on disk because they have fixed filesystem backing. Only when they
are referenced, they will go to cache and then when inactive and/or while
reclaiming, they will end up in disk again. But, afaik, anything which has
a filesystem backing, a fixed storage backing, doesn't go to swap. This
part is pretty clear to me. Only anon pages go to swap and something which
is on disk, is not anon in first place.

Soham

>
> -mandeep
>
> >
> > The end result is that that swap'ed pages represent pages available to
> be swapped in, but not the number of pages that were actually swapped out
> at some point in time.
> >
> > Greg
> > --
> > Sent from my Android phone with K-9 Mail. Please excuse my brevity.
> >
> > _______________________________________________
> > Kernelnewbies mailing list
> > Kernelnewbies at kernelnewbies.org
> > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130221/7be53f43/attachment.html 

Why would some process swap in place of reclaiming free(cached) memory

[Greg Freemyer]

On Thu, Feb 21, 2013 at 12:19 AM, Soham Chakraborty
<sohamwonderpiku4u@gmail.com> wrote:
>
>
> On Thu, Feb 21, 2013 at 10:45 AM, Mandeep Sandhu
> <mandeepsandhu.chd@gmail.com> wrote:
>>
>> > I know the pages are kept on disk initially and then swapped in as
>> > needed/referenced.  Thus if there are code pages where none of the code in
>> > the page has been executed since starting the app, then that page will
>> > remain on disk until the application logic eventually invokes it.  At that
>> > point it will be demand loaded from disk.
>> >
>> > During that early stage, do the never used pages show as swapped out?
>> > If so, why would anyone want all those never used pages to be brought into
>> > ram just to sit there unused?
>>
>> Are you saying that pages of code segment (of an executable) which
>> have not been accessed yet, are copied onto the swap space from their
>> location on the disk when the program is exec'ed? Or do they remain on
>> disk (in their original location) and loaded into memory only when
>> accessed (and _then_ possibly swapped out)? If they remain on disk,
>> then that wouldn't show up in swap space, right?
>
>
> If the code segment of an executable which hasn't been accessed, they will
> remain on disk because they have fixed filesystem backing. Only when they
> are referenced, they will go to cache and then when inactive and/or while
> reclaiming, they will end up in disk again. But, afaik, anything which has a
> filesystem backing, a fixed storage backing, doesn't go to swap. This part
> is pretty clear to me. Only anon pages go to swap and something which is on
> disk, is not anon in first place.

Soham,

I agree with the details you describe.  My question was if the pages
not in RAM, but available in filesystem backing were being included in
the count of pages swapped out that you were asking about.

Let me rephrase my answer:

Total addressable pages = Allocated RAM pages + filesystem Backing
Storage pages + pages swapped out

I don't see the filesystem backing storage pages typically included in
performance tools like top.

Therefore my question was if those userspace reporting tools use the
term swap to represent both the truely swapped out pages and the
filesystem backing storage pages.

Seems like you know the MM system better than me, so I won't argue for
or against my theory. I'll just leave it as a question.

Greg

Why would some process swap in place of reclaiming free(cached) memory

[Mulyadi Santosa]

On Wed, Feb 20, 2013 at 10:01 PM, Soham Chakraborty
<sohamwonderpiku4u@gmail.com> wrote:
> The question is salivating and simple. When we have free and lotsa cached
> memory in a system (irrespective of distro and kernel), why would some
> process end up in swap space, I understand the overcommit mode of vm, the
> default value of 0 and what it does. I also know that the cache might be
> dirty and we don't might not want to allocate free pages thinking that we
> might go under the watermark of low. But, I mainly end up getting this
> question from end users - why would things swap if I have free memory.

I support michael's answer. basically because those (anonymous) pages
are inactive for quite long time. So instead of letting them sit in
RAM for nothing, they are swapped out.

IMHO, it is better to do this earlier than waiting for free RAM
getting tighter and doing swap out right at the moment some apps hit
page fault and swapping out inactive pages. If we did this way, memory
allocation will go slow (maybe really slow).


> Any reference to source where it gives the condition to transfer some pages
> to swap, despite of having free memory, would be highly appreciated.

i think it's in mm/vmscan.c

>I tried
> to find in mm/vmscan.c but didn't get much success.

really? hmm I remember there are codes there that scan inactive pages
and swap them out.

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com