why my systems never cache more than ~900 MB?

why my systems never cache more than ~900 MB?

[Tomasz Chmielewski]

On my (32 bit) systems with more than 1 GB memory it is impossible to cache more than about 900 MB. Why?

Caching never goes beyond ~900 MB (i.e. when I read a mounted drive with dd):


# free
             total       used       free     shared    buffers     cached
Mem:       2076164     966788    1109376          0     855132      68932
-/+ buffers/cache:      42724    2033440
Swap:      2097144          0    2097144


Same behaviour on 32 bit machines with 4 GB RAM.

No problems on 64 bit machines.
I have one 32 bit machine that caches beyond ~900 MB without problems.

Is it some kernel/proc/sys setting that I'm missing?


-- 
Tomasz Chmielewski
http://wpkg.org

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why my systems never cache more than ~900 MB?

[Nick Piggin]

On Tuesday 24 March 2009 19:42:08 Tomasz Chmielewski wrote:
> On my (32 bit) systems with more than 1 GB memory it is impossible to cache
> more than about 900 MB. Why?
>
> Caching never goes beyond ~900 MB (i.e. when I read a mounted drive with
> dd):

Because blockdev mappings are limited to lowmem due to sharing their
cache with filesystem metadata cache, which needs kernel mapped memory.
It will >900MB of pagecache data OK (data from regular files)

> # free
>              total       used       free     shared    buffers     cached
> Mem:       2076164     966788    1109376          0     855132      68932
> -/+ buffers/cache:      42724    2033440
> Swap:      2097144          0    2097144
>
>
> Same behaviour on 32 bit machines with 4 GB RAM.
>
> No problems on 64 bit machines.
> I have one 32 bit machine that caches beyond ~900 MB without problems.

Does it have a different user/kernel split?


> Is it some kernel/proc/sys setting that I'm missing?

No, it just can't be done without changing code.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why my systems never cache more than ~900 MB?

[Tomasz Chmielewski]

Nick Piggin schrieb:
> On Tuesday 24 March 2009 19:42:08 Tomasz Chmielewski wrote:
>> On my (32 bit) systems with more than 1 GB memory it is impossible to cache
>> more than about 900 MB. Why?
>>
>> Caching never goes beyond ~900 MB (i.e. when I read a mounted drive with
>> dd):
> 
> Because blockdev mappings are limited to lowmem due to sharing their
> cache with filesystem metadata cache, which needs kernel mapped memory.
> It will >900MB of pagecache data OK (data from regular files)

Does not help me, as what interests me here on these machines is mainly 
caching block device data; they are iSCSI targets and access block 
devices directly.

(...)

>> Same behaviour on 32 bit machines with 4 GB RAM.
>>
>> No problems on 64 bit machines.
>> I have one 32 bit machine that caches beyond ~900 MB without problems.
> 
> Does it have a different user/kernel split?

Yes it does:

CONFIG_VMSPLIT_2G_OPT=y


What split should I choose to enable blockdev mapping on the whole 
memory on 32 bit system with 3 or 4 GB RAM? Is it possible with 4 GB RAM 
at all?


>> Is it some kernel/proc/sys setting that I'm missing?
> 
> No, it just can't be done without changing code.


-- 
Tomasz Chmielewski
http://wpkg.org

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why my systems never cache more than ~900 MB?

[Nick Piggin]

On Wednesday 25 March 2009 02:35:48 Tomasz Chmielewski wrote:
> Nick Piggin schrieb:
> > On Tuesday 24 March 2009 19:42:08 Tomasz Chmielewski wrote:
> >> On my (32 bit) systems with more than 1 GB memory it is impossible to
> >> cache more than about 900 MB. Why?
> >>
> >> Caching never goes beyond ~900 MB (i.e. when I read a mounted drive with
> >> dd):
> >
> > Because blockdev mappings are limited to lowmem due to sharing their
> > cache with filesystem metadata cache, which needs kernel mapped memory.
> > It will >900MB of pagecache data OK (data from regular files)
>
> Does not help me, as what interests me here on these machines is mainly
> caching block device data; they are iSCSI targets and access block
> devices directly.

OK. Yeah, it's just due to crufty old code... I'm slowly working to
remove this limitation (can be done in 1 of 2 ways: either disconnect
filesystem metadata cache from blockdev data cache, or rework filesystems
to be able to handle highmem for metadata cache).

Actually the quickest way for you might be just to create a new type of
block device that can use highmem but not support filesystems.


> >> Same behaviour on 32 bit machines with 4 GB RAM.
> >>
> >> No problems on 64 bit machines.
> >> I have one 32 bit machine that caches beyond ~900 MB without problems.
> >
> > Does it have a different user/kernel split?
>
> Yes it does:
>
> CONFIG_VMSPLIT_2G_OPT=y
>
>
> What split should I choose to enable blockdev mapping on the whole
> memory on 32 bit system with 3 or 4 GB RAM? Is it possible with 4 GB RAM
> at all?

It's not possible to use 4GB RAM with VMSPLIT. You could use VMSPLIT_1G
to give the kernel the most amount of virtual memory. And even reduce
vmalloc space to get a few more MB with vmalloc= boot parameter.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why my systems never cache more than ~900 MB?

[Christoph Lameter]

On Tue, 24 Mar 2009, Tomasz Chmielewski wrote:

> Nick Piggin schrieb:
> Does not help me, as what interests me here on these machines is mainly
> caching block device data; they are iSCSI targets and access block devices
> directly.

You can run a 64 bit kernel on those machines. 64 bit kernels can use
32 bit userspace without a problem. Just install an additional kernel and
try booting your existing setup with it.

> What split should I choose to enable blockdev mapping on the whole memory on
> 32 bit system with 3 or 4 GB RAM? Is it possible with 4 GB RAM at all?

A 64 bit kernel will do the trick.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why my systems never cache more than ~900 MB?

[Tomasz Chmielewski]

Christoph Lameter schrieb:
> On Tue, 24 Mar 2009, Tomasz Chmielewski wrote:
> 
>> Nick Piggin schrieb:
>> Does not help me, as what interests me here on these machines is mainly
>> caching block device data; they are iSCSI targets and access block devices
>> directly.
> 
> You can run a 64 bit kernel on those machines. 64 bit kernels can use
> 32 bit userspace without a problem. Just install an additional kernel and
> try booting your existing setup with it.
> 
>> What split should I choose to enable blockdev mapping on the whole memory on
>> 32 bit system with 3 or 4 GB RAM? Is it possible with 4 GB RAM at all?
> 
> A 64 bit kernel will do the trick.

This hardware has problems booting 64 bit kernels (read: CPUs come from 
the 32-bit land).


-- 
Tomasz Chmielewski
http://wpkg.org

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why my systems never cache more than ~900 MB?

[Christoph Lameter]

On Tue, 24 Mar 2009, Tomasz Chmielewski wrote:

> Christoph Lameter schrieb:
> > On Tue, 24 Mar 2009, Tomasz Chmielewski wrote:
> >
> > > Nick Piggin schrieb:
> > > Does not help me, as what interests me here on these machines is mainly
> > > caching block device data; they are iSCSI targets and access block devices
> > > directly.
> >
> > You can run a 64 bit kernel on those machines. 64 bit kernels can use
> > 32 bit userspace without a problem. Just install an additional kernel and
> > try booting your existing setup with it.
> >
> > > What split should I choose to enable blockdev mapping on the whole memory
> > > on
> > > 32 bit system with 3 or 4 GB RAM? Is it possible with 4 GB RAM at all?
> >
> > A 64 bit kernel will do the trick.
>
> This hardware has problems booting 64 bit kernels (read: CPUs come from the
> 32-bit land).

Then the 1G/3G separation (VMSPLIT_1G) is the best you can do. Will use up
to 3G for low mem. Be aware that your I/O device must support DMA to full
32 bit addresses. If this is an old 32 bit machine with limitations on the
use of bit 31 then the box may have issues.

Put as much as you can into Highmem. Set HIGHMEM4G, HIGHPTE

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>