High Mem Options

High Mem Options

[Reed, Timothy A]

Hello all,
	Yet another quick question...is there any down side to using the
64GB option over the 4GB option if the machine only has 2GB of RAM onboard??
I would think this would be a performance issue?  Does the kernel only use
the translation table if it has to access any memory location over 4GB?

TIA
Tim Reed
Email: timothy.a.reed@lmco.com

Re: High Mem Options

[Michael Vergoz]

Hi Tim,

Every system can NOT manage more than 4GB memory on x86 processor (32 bits processor).
Because the system addressing is limited to 32Bits, well memory > 4GB is used generaly for memory spare...

Best regards,
Michael

On Wed, 05 Mar 2003 06:28:36 -0500
"Reed, Timothy A" <timothy.a.reed@lmco.com> wrote:

> Hello all,
> 	Yet another quick question...is there any down side to using the
> 64GB option over the 4GB option if the machine only has 2GB of RAM onboard??
> I would think this would be a performance issue?  Does the kernel only use
> the translation table if it has to access any memory location over 4GB?
> 
> TIA
> Tim Reed
> Email: timothy.a.reed@lmco.com
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: High Mem Options

[Alan Cox]

On Wed, 2003-03-05 at 12:11, Michael Vergoz wrote:
> Hi Tim,
> 
> Every system can NOT manage more than 4GB memory on x86 processor (32 bits processor).
> Because the system addressing is limited to 32Bits, well memory > 4GB is used generaly for memory spare...

x86 has 36bit physical addressing, its a truely bonkers implementation
but it does have the facility. The limt in reality is 3Gb per process
(1Gb is used for kernel mapping - we could do 4Gb per process but the
syscall cost would go up a lot).

Re: High Mem Options

[Michael Vergoz]

Hi Alan,

That i can't understand i when the system going to the protect mode. 
How the system can use over 4GB memory ?
On freebsd, when you have over 4GB the system say "XGB of XGB skiped..."
(i'v got a machine with 8GB running on freebsd and without memory spare)

Best regards,
Michael


On 05 Mar 2003 13:36:02 +0000
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> On Wed, 2003-03-05 at 12:11, Michael Vergoz wrote:
> > Hi Tim,
> > 
> > Every system can NOT manage more than 4GB memory on x86 processor (32 bits processor).
> > Because the system addressing is limited to 32Bits, well memory > 4GB is used generaly for memory spare...
> 
> x86 has 36bit physical addressing, its a truely bonkers implementation
> but it does have the facility. The limt in reality is 3Gb per process
> (1Gb is used for kernel mapping - we could do 4Gb per process but the
> syscall cost would go up a lot).
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: High Mem Options

[William Lee Irwin III]

On Wed, Mar 05, 2003 at 01:49:37PM +0100, Michael Vergoz wrote:
> That i can't understand i when the system going to the protect mode. 
> How the system can use over 4GB memory ?
> On freebsd, when you have over 4GB the system say "XGB of XGB skiped..."
> (i'v got a machine with 8GB running on freebsd and without memory spare)

The cpu can't look at more than 4GB at a time.

Protected mode doesn't help this, turning paging on and PAE on does.

What it can do is point pagetables at different 4GB subsets of memory.

c.f. kmap_atomic() for how to window around using what's actually a
very small set of PTE's.


-- wli

Re: High Mem Options

[Michael Vergoz]

Hi William,

Right, but if the pagetable pointing to a different 4GB subsets of memory.
The performance of the system can be disastrous, not?

Best regards,
Michael

On Wed, 5 Mar 2003 04:57:47 -0800
William Lee Irwin III <wli@holomorphy.com> wrote:

> On Wed, Mar 05, 2003 at 01:49:37PM +0100, Michael Vergoz wrote:
> > That i can't understand i when the system going to the protect mode. 
> > How the system can use over 4GB memory ?
> > On freebsd, when you have over 4GB the system say "XGB of XGB skiped..."
> > (i'v got a machine with 8GB running on freebsd and without memory spare)
> 
> The cpu can't look at more than 4GB at a time.
> 
> Protected mode doesn't help this, turning paging on and PAE on does.
> 
> What it can do is point pagetables at different 4GB subsets of memory.
> 
> c.f. kmap_atomic() for how to window around using what's actually a
> very small set of PTE's.
> 
> 
> -- wli

Re: High Mem Options

[William Lee Irwin III]

On Wed, 5 Mar 2003 04:57:47 -0800, William Lee Irwin III wrote:
>> The cpu can't look at more than 4GB at a time.
>> Protected mode doesn't help this, turning paging on and PAE on does.
>> What it can do is point pagetables at different 4GB subsets of memory.
>> c.f. kmap_atomic() for how to window around using what's actually a
>> very small set of PTE's.

On Wed, Mar 05, 2003 at 02:02:57PM +0100, Michael Vergoz wrote:
> Right, but if the pagetable pointing to a different 4GB subsets of
> memory. The performance of the system can be disastrous, not?

Well, the TLB gets blown away at the drop of a hat. Things just have
lower scaling factors with respect to memory than say, 64-bit, though
Linux doesn't do anything about TLB coverage on 64-bit yet anyway.

It might be better to take this one to kernelnewbies@nl.linux.org


-- wli

Re: High Mem Options

[Bill Davidsen]

On Wed, 5 Mar 2003, Michael Vergoz wrote:

> Right, but if the pagetable pointing to a different 4GB subsets of memory.
> The performance of the system can be disastrous, not?

No. It may be measurable, but I've seen posts from very competent people
indicating that penalties of 2-5% are usual. Swapping to disk is going to
hurt a lot more than that, so if a system has lots of processes more
memory is usually better.

Maybe some of the benchmark gurus can point you to numbers on this.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: High Mem Options

[Alan Cox]

On Wed, 2003-03-05 at 12:49, Michael Vergoz wrote:
> Hi Alan,
> 
> That i can't understand i when the system going to the protect mode. 
> How the system can use over 4GB memory ?

In PAE mode the later CPUS have 32bit virtual addresses but the page
table entries to map these allow 36bit physical addresses. It means you
can't address more than 4Gb at a time without updating MMU tables

Re: High Mem Options

[William Lee Irwin III]

On Wed, Mar 05, 2003 at 06:28:36AM -0500, Reed, Timothy A wrote:
> 	Yet another quick question...is there any down side to using the
> 64GB option over the 4GB option if the machine only has 2GB of RAM onboard??
> I would think this would be a performance issue?  Does the kernel only use
> the translation table if it has to access any memory location over 4GB?

Yes, the additional level of pagetables slows things down quite a bit.


-- wli

Re: High Mem Options

[Gerrit Huizenga]

On Wed, 05 Mar 2003 04:26:54 PST, William Lee Irwin III wrote:
> On Wed, Mar 05, 2003 at 06:28:36AM -0500, Reed, Timothy A wrote:
> > 	Yet another quick question...is there any down side to using the
> > 64GB option over the 4GB option if the machine only has 2GB of RAM onboard??
> > I would think this would be a performance issue?  Does the kernel only use
> > the translation table if it has to access any memory location over 4GB?
> 
> Yes, the additional level of pagetables slows things down quite a bit.

Bill, do you hvae measures for this?  I seem to remember PTX's impact
of PAE36 as being about 3-5% depending on workload.  Janet did one test
sometime back with DB2 that showed a net of no difference on TPC-H (PAE
slows things down, less memory pressure speeds things up) but Badari
just repeated with 2.5.62 or 2.5.63 and saw a larger degradation.

I'm wondering if some hardware is not getting correctly configured at
boot with with respect to MTRR's, perhaps...  I really wouldn't expect
a 10% impact from PAE and I don't have any consistent Linux measurements
to validate or invalidate that much impact.

gerrit

Re: High Mem Options

[William Lee Irwin III]

On Wed, 05 Mar 2003 04:26:54 PST, William Lee Irwin III wrote:
>> Yes, the additional level of pagetables slows things down quite a bit.

On Wed, Mar 05, 2003 at 12:31:03PM -0800, Gerrit Huizenga wrote:
> Bill, do you hvae measures for this?  I seem to remember PTX's impact
> of PAE36 as being about 3-5% depending on workload.  Janet did one test
> sometime back with DB2 that showed a net of no difference on TPC-H (PAE
> slows things down, less memory pressure speeds things up) but Badari
> just repeated with 2.5.62 or 2.5.63 and saw a larger degradation.
> I'm wondering if some hardware is not getting correctly configured at
> boot with with respect to MTRR's, perhaps...  I really wouldn't expect
> a 10% impact from PAE and I don't have any consistent Linux measurements
> to validate or invalidate that much impact.

Unfortunately the number of ca. 10% I got from you. I think badari's
done some $BENCHMARK_THAT_CANNOT_BE_NAMED things recently that were
consistent with the handwavy estimate but AIUI they were not intended
to measure the effect etc. etc.


-- wli

Re: High Mem Options

[Gerrit Huizenga]

On Wed, 05 Mar 2003 12:58:49 PST, William Lee Irwin III wrote:
> On Wed, 05 Mar 2003 04:26:54 PST, William Lee Irwin III wrote:
> >> Yes, the additional level of pagetables slows things down quite a bit.
> 
> On Wed, Mar 05, 2003 at 12:31:03PM -0800, Gerrit Huizenga wrote:
> > Bill, do you hvae measures for this?  I seem to remember PTX's impact
> > of PAE36 as being about 3-5% depending on workload.  Janet did one test
> > sometime back with DB2 that showed a net of no difference on TPC-H (PAE
> > slows things down, less memory pressure speeds things up) but Badari
> > just repeated with 2.5.62 or 2.5.63 and saw a larger degradation.
> > I'm wondering if some hardware is not getting correctly configured at
> > boot with with respect to MTRR's, perhaps...  I really wouldn't expect
> > a 10% impact from PAE and I don't have any consistent Linux measurements
> > to validate or invalidate that much impact.
> 
> Unfortunately the number of ca. 10% I got from you. I think badari's
> done some $BENCHMARK_THAT_CANNOT_BE_NAMED things recently that were
> consistent with the handwavy estimate but AIUI they were not intended
> to measure the effect etc. etc.

Ah, if you are referring to a number from me, that was with 2.4 and
that number seemed high to me at the time.  I don't believe that 10%
*should* be the amount of degradation.  But I don't have current numbers
(that I can share, anyway ;-) that prove anything less than that.

I expect that we'll be diving into this more over the next few months
as we can generate some large workloads and find the cause of the
degradation (and hopefully minimize it).

gerrit

Re: High Mem Options

[Martin J. Bligh]

> Ah, if you are referring to a number from me, that was with 2.4 and
> that number seemed high to me at the time.  I don't believe that 10%
> *should* be the amount of degradation.  But I don't have current numbers
> (that I can share, anyway ;-) that prove anything less than that.
> 
> I expect that we'll be diving into this more over the next few months
> as we can generate some large workloads and find the cause of the
> degradation (and hopefully minimize it).

Would also be useful to measure the overhead on a machine with < 4Gb
of RAM ... otherwise you have two effects to deal with:

1. the PAE overhead.
2. The increase in RAM - more data to manage, and potential bounce-buffers.

M.

Re: High Mem Options

[Alan Cox]

On Wed, 2003-03-05 at 11:28, Reed, Timothy A wrote:
> Hello all,
> 	Yet another quick question...is there any down side to using the
> 64GB option over the 4GB option if the machine only has 2GB of RAM onboard??
> I would think this would be a performance issue?  Does the kernel only use
> the translation table if it has to access any memory location over 4GB?

The 64Gb mode has to use different page table formats, so there is a hit
always

-- 
Alan Cox <alan@lxorguk.ukuu.org.uk>