release used memory

release used memory

[Andreas Leppert]

Hello,

for study reasons, I evaluate a kernel module by running a parallel version of
bzip. Input data are some senseless data files (500 x 10MB for example). While
the bzip program runs, I took a look at the output of top and noticed
something which I do not understand.

In the following, I'm refering to that two lines of top which look something
like that:

Mem:   8185716k total,  5603224k used,  2582492k free,     9104k buffers
Swap:  8388604k total,        0k used,  8388604k free,  5374400k cached

While the bzip program runs, it uses memory and thus, the value before "free"
in the Mem: line is getting smaller. As you can see above, there is also some
"cached stuff" (5374400k). What is meant by this value? Someone explained me
that it has to do something with filesystem buffers which were read or
written. Could you elaborate on this?

If I have enough input data, the memory is not enough (the "free" value gets
smaller and smaller, and the "cached" value gets bigger and bigger). Then
kswapd appears in the list of running processes. I would expect that Linux
gets rid of the "cached" stuff so that the program gets more memory. Instead,
nothing happens and kswapd is activated. What's the sense of keeping some
filesystem buffers when they are not used anymore?

The main problem is that I do not understand what by "buffers" and "cached" is
meant. It would be nice if you can help me on this.

Thanks in advance
Andreas

release used memory

[Rajat Sharma]

> If I have enough input data, the memory is not enough (the "free" value gets
> smaller and smaller, and the "cached" value gets bigger and bigger). Then
> kswapd appears in the list of running processes. I would expect that Linux
> gets rid of the "cached" stuff so that the program gets more memory. Instead,
> nothing happens and kswapd is activated. What's the sense of keeping some
> filesystem buffers when they are not used anymore?

Its filesystem's cache (more popularly known as Page Cache in Linux
(buffer cache historically)).
Writing to memory (DRAM) is always faster than writing to Disk,
similarly if you try to read back the same data, and if it is already
in cache, it would be faster to read it from DRAM. So, linux
page-cache is used for following purpose:
1. Write-back cache
2. Read-caching
3. prefetch more data in case of reading file sequentially i.e. doing
single large read from disk instead of multiple reads.

Linux implementation of page-cache is boundless it can grow as much as
available memory however there are kernel threads to do the cleanup
stuff - bdflush process (not kswapd). There are two things to be dealt
with:
1. writing dirty data to disk to make pages in memory clean. Note that
pages are still kept with file-system page-cache but are clean pages
so can be removed from cache on memory pressure.
2. reclaiming unwanted memory pages. This is done only if there is not
sufficient memory. Although there are multiple ways to clean memory
e.g. swapping out unwanted process but page-cache pages are the most
obvious target. First clean pages are removed from page-cache and if
that it not sufficient dirty pages are flushed to disk first and then
removed from page-cache.

Hope it helps.

Rajat



On Fri, Mar 11, 2011 at 11:24 PM, Andreas Leppert <wudmx@web.de> wrote:
> Hello,
>
> for study reasons, I evaluate a kernel module by running a parallel version of
> bzip. Input data are some senseless data files (500 x 10MB for example). While
> the bzip program runs, I took a look at the output of top and noticed
> something which I do not understand.
>
> In the following, I'm refering to that two lines of top which look something
> like that:
>
> Mem: ? 8185716k total, ?5603224k used, ?2582492k free, ? ? 9104k buffers
> Swap: ?8388604k total, ? ? ? ?0k used, ?8388604k free, ?5374400k cached
>
> While the bzip program runs, it uses memory and thus, the value before "free"
> in the Mem: line is getting smaller. As you can see above, there is also some
> "cached stuff" (5374400k). What is meant by this value? Someone explained me
> that it has to do something with filesystem buffers which were read or
> written. Could you elaborate on this?
>
> If I have enough input data, the memory is not enough (the "free" value gets
> smaller and smaller, and the "cached" value gets bigger and bigger). Then
> kswapd appears in the list of running processes. I would expect that Linux
> gets rid of the "cached" stuff so that the program gets more memory. Instead,
> nothing happens and kswapd is activated. What's the sense of keeping some
> filesystem buffers when they are not used anymore?
>
> The main problem is that I do not understand what by "buffers" and "cached" is
> meant. It would be nice if you can help me on this.
>
> Thanks in advance
> Andreas
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

release used memory

[Mulyadi Santosa]

Hi :)

On Sat, Mar 12, 2011 at 00:54, Andreas Leppert <wudmx@web.de> wrote:
> In the following, I'm refering to that two lines of top which look something
> like that:
>
> Mem: ? 8185716k total, ?5603224k used, ?2582492k free, ? ? 9104k buffers
> Swap: ?8388604k total, ? ? ? ?0k used, ?8388604k free, ?5374400k cached
>
> While the bzip program runs, it uses memory and thus, the value before "free"
> in the Mem: line is getting smaller. As you can see above, there is also some
> "cached stuff" (5374400k). What is meant by this value? Someone explained me
> that it has to do something with filesystem buffers which were read or
> written. Could you elaborate on this?

You probably need to read my articles (written by Mulyadi Santosa) in
these below URLs :)

http://fullcirclemagazine.org/issue-39/
http://www.linux.com/learn/tutorials/42048-uncover-the-meaning-of-tops-statistics


>Instead,
> nothing happens and kswapd is activated. What's the sense of keeping some
> filesystem buffers when they are not used anymore?

As the end user, you probably know your own usage pattern (i.e "i
won't read that file anymore"), but the kernel doesn't know that. So
the best it can do is keep them on RAM as long as it can. But to make
it somewhat more efficient (not hanging there for eternity), there's
page swapping algorithm ran that swap them out from RAM using certain
aging or replacement algorithm.

What you saw was probably they were not "old" enough to be swapped
out...thus they stayed.

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

release used memory

[Andrzej Kardas]

On 12.03.2011 02:26, Mulyadi Santosa wrote:
>
> What you saw was probably they were not "old" enough to be swapped
> out...thus they stayed.
>
There is possibility to clear the cache manually by typing:

|echo 1 > /proc/sys/vm/drop_caches|

but you don't want probably to do that, the more data in cache the better :)

--
regards
Andrzej Kardas
http://www.linux.mynotes.pl



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110312/3bb9b725/attachment.html 

release used memory

[Andreas Leppert]

On Sat, Mar 12, 2011 at 08:26:24AM +0700, Mulyadi Santosa wrote:
> You probably need to read my articles (written by Mulyadi Santosa) in
> these below URLs :)
> 
> http://fullcirclemagazine.org/issue-39/
> http://www.linux.com/learn/tutorials/42048-uncover-the-meaning-of-tops-statistics

Thank you guys for all your explanations and clarifications, the links
are also very useful, thanks Mulyadi.

Regards, 
Andreas