buffer cache question

buffer cache question

[JaniD++]

Hello, list,

I use 4 disk nodes with NBD.
All of my nodes have 2GB ram.

But the buffer cache newer rise over 830MB.

Is there some limit?
Where can i change this limit? (if it is)

Thanks,
Janos

[root@st-0001 root]# free
             total       used       free     shared    buffers     cached
Mem:       2073152     933188    1139964          0     836776      43416
-/+ buffers/cache:      52996    2020156
Swap:            0          0          0
[root@st-0001 root]# cat /proc/meminfo
MemTotal:      2073152 kB
MemFree:       1139012 kB
Buffers:        835928 kB
Cached:          43448 kB
SwapCached:          0 kB
Active:          12872 kB
Inactive:       871424 kB
HighTotal:     1179584 kB
HighFree:      1129764 kB
LowTotal:       893568 kB
LowFree:          9248 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
Mapped:           9104 kB
Slab:            30248 kB
CommitLimit:   1036576 kB
Committed_AS:    15428 kB
PageTables:        408 kB
VmallocTotal:   114680 kB
VmallocUsed:       196 kB
VmallocChunk:   114476 kB
[root@st-0001 root]#

RE: buffer cache question

[Roger Heflin]

 

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org 
> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of JaniD++
> Sent: Sunday, December 18, 2005 3:09 PM
> To: linux-kernel@vger.kernel.org
> Subject: buffer cache question
> 
> Hello, list,
> 
> I use 4 disk nodes with NBD.
> All of my nodes have 2GB ram.
> 
> But the buffer cache newer rise over 830MB.
> 
> Is there some limit?

For writes only 40% of the ram can be "dirty".

> Where can i change this limit? (if it is)

I believe there is a way to change it, I am pretty sure that it has
been discussed on the kernel list a couple of months ago, I 
don't remember exactly what the change is, and I think the change
was more complicated that was obvious.

The previous subject on a similar thing was:
"kernel 2.6.13 buffer strangeness"

                           Roger

> 
> Thanks,
> Janos
> 
> [root@st-0001 root]# free
>              total       used       free     shared    
> buffers     cached
> Mem:       2073152     933188    1139964          0     
> 836776      43416
> -/+ buffers/cache:      52996    2020156
> Swap:            0          0          0
> [root@st-0001 root]# cat /proc/meminfo
> MemTotal:      2073152 kB
> MemFree:       1139012 kB
> Buffers:        835928 kB
> Cached:          43448 kB
> SwapCached:          0 kB
> Active:          12872 kB
> Inactive:       871424 kB
> HighTotal:     1179584 kB
> HighFree:      1129764 kB
> LowTotal:       893568 kB
> LowFree:          9248 kB
> SwapTotal:           0 kB
> SwapFree:            0 kB
> Dirty:               0 kB
> Writeback:           0 kB
> Mapped:           9104 kB
> Slab:            30248 kB
> CommitLimit:   1036576 kB
> Committed_AS:    15428 kB
> PageTables:        408 kB
> VmallocTotal:   114680 kB
> VmallocUsed:       196 kB
> VmallocChunk:   114476 kB
> [root@st-0001 root]#
> 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in the body of a message to 
> majordomo@vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: buffer cache question

[JaniD++]

Hi,

I have read back in the kernel-archives, and found this messages, about the
same theme, but there is one difference!

On the old messages:
>Nate Diller wrote:
> just found the culprit.  guess i should have read the code the first
> time.  get_dirty_limits() in drivers/block/page_writeback.c has a
> hard-coded upper limit to dirty_ratio.  it's capped to half of the
> unmapped pages, so maybe 30-40% of your system's memory.  so if you are
> brave, just remove the "/ 2" parts from the 'if (dirty_ratio >
> unmapped_ratio / 2) dirty_ratio = unmapped_ratio / 2;' check, and you
> can have all the OOM goodness you want.
...
>I changed that bit of code to:
>
> if (dirty_ratio > unmapped_ratio - 10)
>  dirty_ratio = unmapped_ratio - 10;
>
>and added a couple of sanity checks so that it couldn't get below 5 or
above 95.
>
>Then set /proc/sys/vm/dirty_ratio to 95 and dirty_background_ratio to 1.

In this case, this modification is only for the *dirty* memory buffer.
I want to use more buffer *cache*! :-)
The unwritten dirty memory ratio is good enough for me.

Anybody has an idea?

Cheers,
Janos

----- Original Message ----- 
From: "Roger Heflin" <rheflin@atipa.com>
To: "'JaniD++'" <djani22@dynamicweb.hu>; <linux-kernel@vger.kernel.org>
Sent: Monday, December 19, 2005 7:15 PM
Subject: RE: buffer cache question


>
>
> > -----Original Message-----
> > From: linux-kernel-owner@vger.kernel.org
> > [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of JaniD++
> > Sent: Sunday, December 18, 2005 3:09 PM
> > To: linux-kernel@vger.kernel.org
> > Subject: buffer cache question
> >
> > Hello, list,
> >
> > I use 4 disk nodes with NBD.
> > All of my nodes have 2GB ram.
> >
> > But the buffer cache newer rise over 830MB.
> >
> > Is there some limit?
>
> For writes only 40% of the ram can be "dirty".
>
> > Where can i change this limit? (if it is)
>
> I believe there is a way to change it, I am pretty sure that it has
> been discussed on the kernel list a couple of months ago, I
> don't remember exactly what the change is, and I think the change
> was more complicated that was obvious.
>
> The previous subject on a similar thing was:
> "kernel 2.6.13 buffer strangeness"
>
>                            Roger
>
> >
> > Thanks,
> > Janos
> >
> > [root@st-0001 root]# free
> >              total       used       free     shared
> > buffers     cached
> > Mem:       2073152     933188    1139964          0
> > 836776      43416
> > -/+ buffers/cache:      52996    2020156
> > Swap:            0          0          0
> > [root@st-0001 root]# cat /proc/meminfo
> > MemTotal:      2073152 kB
> > MemFree:       1139012 kB
> > Buffers:        835928 kB
> > Cached:          43448 kB
> > SwapCached:          0 kB
> > Active:          12872 kB
> > Inactive:       871424 kB
> > HighTotal:     1179584 kB
> > HighFree:      1129764 kB
> > LowTotal:       893568 kB
> > LowFree:          9248 kB
> > SwapTotal:           0 kB
> > SwapFree:            0 kB
> > Dirty:               0 kB
> > Writeback:           0 kB
> > Mapped:           9104 kB
> > Slab:            30248 kB
> > CommitLimit:   1036576 kB
> > Committed_AS:    15428 kB
> > PageTables:        408 kB
> > VmallocTotal:   114680 kB
> > VmallocUsed:       196 kB
> > VmallocChunk:   114476 kB
> > [root@st-0001 root]#
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe
> > linux-kernel" in the body of a message to
> > majordomo@vger.kernel.org More majordomo info at
> > http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> >
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: buffer cache question

[JaniD++]

----- Original Message ----- 
From: Nate Diller
To: JaniD++
Cc: Roger Heflin ; linux-kernel@vger.kernel.org
Sent: Friday, December 23, 2005 9:58 AM
Subject: Re: buffer cache question


On 11/21/05, JaniD++ <djani22@dynamicweb.hu> wrote:
Hi,

I have read back in the kernel-archives, and found this messages, about the
same theme, but there is one difference!

On the old messages:
>Nate Diller wrote:
> just found the culprit.  guess i should have read the code the first
> time.  get_dirty_limits() in drivers/block/page_writeback.c has a
> hard-coded upper limit to dirty_ratio.  it's capped to half of the
> unmapped pages, so maybe 30-40% of your system's memory.  so if you are
> brave, just remove the "/ 2" parts from the 'if (dirty_ratio >
> unmapped_ratio / 2) dirty_ratio = unmapped_ratio / 2;' check, and you
> can have all the OOM goodness you want.
...
>I changed that bit of code to:
>
> if (dirty_ratio > unmapped_ratio - 10)
>  dirty_ratio = unmapped_ratio - 10;
>
>and added a couple of sanity checks so that it couldn't get below 5 or
above 95.
>
>Then set /proc/sys/vm/dirty_ratio to 95 and dirty_background_ratio to 1.

In this case, this modification is only for the *dirty* memory buffer.
I want to use more buffer *cache*! :-)
The unwritten dirty memory ratio is good enough for me.

Anybody has an idea?

<snip>



> > [root@st-0001 root]# free
> >              total       used       free     shared
> > buffers     cached
> > Mem:       2073152     933188    1139964          0
> > 836776      43416
> > -/+ buffers/cache:      52996    2020156
> > Swap:            0          0          0
> > [root@st-0001 root]# cat /proc/meminfo
> > MemTotal:      2073152 kB
> > MemFree:       1139012 kB
> > Buffers:        835928 kB
> > Cached:          43448 kB
> > SwapCached:          0 kB
> > Active:          12872 kB
> > Inactive:       871424 kB
> > HighTotal:     1179584 kB
> > HighFree:      1129764 kB
> > LowTotal:       893568 kB
> > LowFree:          9248 kB
> > SwapTotal:           0 kB
> > SwapFree:            0 kB
> > Dirty:               0 kB
> > Writeback:           0 kB
> > Mapped:           9104 kB
> > Slab:            30248 kB
> > CommitLimit:   1036576 kB
> > Committed_AS:    15428 kB
> > PageTables:        408 kB
> > VmallocTotal:   114680 kB
> > VmallocUsed:       196 kB
> > VmallocChunk:   114476 kB
> > [root@st-0001 root]#

looks like you're barely using any of your high memory.  maybe NBD doesn't
have highmem support.  what file system are you using?

NATE


I cannot understant this.
NBD need to support highmem for buffering?
If know right, the kernel does buffering, not NBD!
But the kernel only use ~830MB for buffer cache instead of dinamically use
all free memory like page cache.

This is one raw disk node, independent from file system.

Cheers,
Janos

Re: buffer cache question

[Nate Diller]

> ----- Original Message -----
> From: Nate Diller
> To: JaniD++
> Cc: Roger Heflin ; linux-kernel@vger.kernel.org
> Sent: Friday, December 23, 2005 9:58 AM
> Subject: Re: buffer cache question
>
> looks like you're barely using any of your high memory.  maybe NBD doesn't
> have highmem support.  what file system are you using?
>
> NATE
>
>
> I cannot understant this.
> NBD need to support highmem for buffering?
> If know right, the kernel does buffering, not NBD!
> But the kernel only use ~830MB for buffer cache instead of dinamically use
> all free memory like page cache.
>
> This is one raw disk node, independent from file system.
>

this is an NBD client, using CONFIG_BLK_DEV_NBD?  on the 2.6 series
kernel?  if so, then it, like any other kernel component using the
page cache, needs to explicity use kmap/kunmap to make use of memory
in the high memory zone.  on a 32 bit machine, any pages above the 896
meg mark are treated specially inside the linux kernel (see
http://kerneltrap.org/node/2450).

if you don't have highmem support (CONFIG_HIGHMEM4G) enabled, then
enabling that should fix it.  if you already have it enabled (it
looked like it to me, based on your /proc/meminfo) then there is a bug
somewhere.

it would seem from a brief inspection that the send/recv_bvec
functions in nbd (2.6.13) do use kmap.  I don't know the nbd code very
well, it seems that Pavel Machek wrote the code, he or block layer
maintainer Jens Axboe may know something I don't.  So if enabling
highmem in your .config doesn't help, try CC'ing them with your issue.
 in the mean time, one of the memory split patches, such as the 4G:4G
patch, should get things working.

NATE

Re: buffer cache question

[JaniD++]

Hi,

----- Original Message ----- 
From: "Nate Diller" <nate.diller@gmail.com>
To: "JaniD++" <djani22@dynamicweb.hu>
Cc: <linux-kernel@vger.kernel.org>; <pavel@suse.cz>
Sent: Thursday, December 29, 2005 5:10 AM
Subject: Re: buffer cache question


> > ----- Original Message -----
> > From: Nate Diller
> > To: JaniD++
> > Cc: Roger Heflin ; linux-kernel@vger.kernel.org
> > Sent: Friday, December 23, 2005 9:58 AM
> > Subject: Re: buffer cache question
> >
> > looks like you're barely using any of your high memory.  maybe NBD
doesn't
> > have highmem support.  what file system are you using?
> >
> > NATE
> >
> >
> > I cannot understant this.
> > NBD need to support highmem for buffering?
> > If know right, the kernel does buffering, not NBD!
> > But the kernel only use ~830MB for buffer cache instead of dinamically
use
> > all free memory like page cache.
> >
> > This is one raw disk node, independent from file system.
> >
>
> this is an NBD client, using CONFIG_BLK_DEV_NBD?  on the 2.6 series
> kernel?  if so, then it, like any other kernel component using the
> page cache, needs to explicity use kmap/kunmap to make use of memory
> in the high memory zone.  on a 32 bit machine, any pages above the 896
> meg mark are treated specially inside the linux kernel (see
> http://kerneltrap.org/node/2450).

Yes, i have read this helpful document, but there is about the memory usage,
mapping, and not exactly the buffering/caching.
Yes, i use the 2.6 series NBD.

>
> if you don't have highmem support (CONFIG_HIGHMEM4G) enabled, then
> enabling that should fix it.  if you already have it enabled (it
> looked like it to me, based on your /proc/meminfo) then there is a bug
> somewhere.

Yes, it is enabled.

>
> it would seem from a brief inspection that the send/recv_bvec
> functions in nbd (2.6.13) do use kmap.  I don't know the nbd code very
> well, it seems that Pavel Machek wrote the code, he or block layer
> maintainer Jens Axboe may know something I don't.  So if enabling
> highmem in your .config doesn't help, try CC'ing them with your issue.
>  in the mean time, one of the memory split patches, such as the 4G:4G
> patch, should get things working.

The NBD use buffer cache (i am sure), but the highest limit is about
830-840M.

Where can i get the split patch?

Thanks for helpful answers!


Cheers,
Janos


>
> NATE