4g/4g for 2.6.6

4g/4g for 2.6.6

[Phy Prabab]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 546 bytes --]

Hello,

Please cc me as I am not on this mailling list.

I have been researching the 4g patches for kernels. 
Seems there was a rift between people over this.  Is
there any plan to resume publishing 4g patches for
developing kernels?

I am currently trying to get 4g to work with 2.6.6-mm5
but of course running into issues,so any help on this
would be great!

Thank you for your time.
Phy


	
		
__________________________________
Do you Yahoo!?
Yahoo! Domains – Claim yours for only $14.70/year
http://smallbusiness.promotions.yahoo.com/offer 

Re: 4g/4g for 2.6.6

[Linus Torvalds]

On Sun, 23 May 2004, Phy Prabab wrote:
> 
> I have been researching the 4g patches for kernels. 
> Seems there was a rift between people over this.  Is
> there any plan to resume publishing 4g patches for
> developing kernels?

Quite frankly, a number of us are hoping that we can make them
unnecessary. The cost of the 4g/4g split is absolutely _huge_ on some
things, including basic stuff like kernel compiles.

The only valid reason for the 4g split is that the VM doesn't always 
behave well with huge amounts of highmem. The anonvma stuff in 2.6.7-pre1 
is hoped to make that much less of an issue.

Personally, if we never need to merge 4g for real, I'll be really really 
happy. I see it as a huge ugly hack.

			Linus

Re: 4g/4g for 2.6.6

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Sun, 23 May 2004, Phy Prabab wrote:
> 
>>I have been researching the 4g patches for kernels. 
>>Seems there was a rift between people over this.  Is
>>there any plan to resume publishing 4g patches for
>>developing kernels?
> 
> 
> Quite frankly, a number of us are hoping that we can make them
> unnecessary. The cost of the 4g/4g split is absolutely _huge_ on some
> things, including basic stuff like kernel compiles.


Sorta like I'm hoping that cheap and prevalent 64-bit CPUs make PAE36 
and PAE40 on ia32 largely unnecessary.  Addressing more memory than 32 
bits of memory on a 32-bit CPU always seemed like a hack to me, and a 
source of bugs and lost performance...

	Jeff

Re: 4g/4g for 2.6.6

[Phy Prabab]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 1186 bytes --]

So do I understand this correctly, in 2.6.7(+) it will
no longer be necessary to have the 4g patches?  I will
be able to get 4g/process with the going forward
kernels?

Thank you for your time.
Phy

--- Linus Torvalds <torvalds@osdl.org> wrote:
> 
> 
> On Sun, 23 May 2004, Phy Prabab wrote:
> > 
> > I have been researching the 4g patches for
> kernels. 
> > Seems there was a rift between people over this. 
> Is
> > there any plan to resume publishing 4g patches for
> > developing kernels?
> 
> Quite frankly, a number of us are hoping that we can
> make them
> unnecessary. The cost of the 4g/4g split is
> absolutely _huge_ on some
> things, including basic stuff like kernel compiles.
> 
> The only valid reason for the 4g split is that the
> VM doesn't always 
> behave well with huge amounts of highmem. The
> anonvma stuff in 2.6.7-pre1 
> is hoped to make that much less of an issue.
> 
> Personally, if we never need to merge 4g for real,
> I'll be really really 
> happy. I see it as a huge ugly hack.
> 
> 			Linus



	
		
__________________________________
Do you Yahoo!?
Yahoo! Domains – Claim yours for only $14.70/year
http://smallbusiness.promotions.yahoo.com/offer 

Re: 4g/4g for 2.6.6

[Martin J. Bligh]

> I have been researching the 4g patches for kernels. 
> Seems there was a rift between people over this.  Is
> there any plan to resume publishing 4g patches for
> developing kernels?
> 
> I am currently trying to get 4g to work with 2.6.6-mm5
> but of course running into issues,so any help on this
> would be great!

It's in -mjb tree - the update from 2.6.6-rc3 to 2.6.6 should be trivial.

M.

Re: 4g/4g for 2.6.6

[Phy Prabab]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 718 bytes --]

Okay, I will work with that patch and see if I can it
to apply.

Thanks!
Phy
--- "Martin J. Bligh" <mbligh@aracnet.com> wrote:
> > I have been researching the 4g patches for
> kernels. 
> > Seems there was a rift between people over this. 
> Is
> > there any plan to resume publishing 4g patches for
> > developing kernels?
> > 
> > I am currently trying to get 4g to work with
> 2.6.6-mm5
> > but of course running into issues,so any help on
> this
> > would be great!
> 
> It's in -mjb tree - the update from 2.6.6-rc3 to
> 2.6.6 should be trivial.
> 
> M.
> 


	
		
__________________________________
Do you Yahoo!?
Yahoo! Domains – Claim yours for only $14.70/year
http://smallbusiness.promotions.yahoo.com/offer 

Re: 4g/4g for 2.6.6

[Linus Torvalds]

On Sun, 23 May 2004, Jeff Garzik wrote:
> 
> Sorta like I'm hoping that cheap and prevalent 64-bit CPUs make PAE36 
> and PAE40 on ia32 largely unnecessary.  Addressing more memory than 32 
> bits of memory on a 32-bit CPU always seemed like a hack to me, and a 
> source of bugs and lost performance...

I agree. I held out on PAE for a longish while, in the unrealistic hope 
that people would switch to alpha's. 

Oh, well. I don't expect _everybody_ to switch to x86-64 immediately, but 
I hope we can hold out long enough without 4g that it does work out this 
time.

		Linus

Re: 4g/4g for 2.6.6

[Jeff Garzik]

Linus Torvalds wrote:
> Oh, well. I don't expect _everybody_ to switch to x86-64 immediately, but 
> I hope we can hold out long enough without 4g that it does work out this 
> time.

I think the switchover will happen fairly rapidly, since it is 
positioned "the upgrade you get next time you buy a new computer", 
similar to the current PATA->SATA switchover.

Since these CPUs run in 32-bit mode just fine, I bet you wind up with 
people running Intel or AMD 64-bit CPUs long they abandon their 32-bit 
OS.  I recall people often purchasing gigabit ethernet cards long before 
they had a gigabit switch, simply because it was the volume technology 
that was being at the time.  I think the same thing is going to happen 
here, with AMD64 and EM64T.

AMD got a lot of things right with this one...

	Jeff

Re: 4g/4g for 2.6.6

[Wim Coekaerts]

> I think the switchover will happen fairly rapidly, since it is 
> positioned "the upgrade you get next time you buy a new computer", 
> similar to the current PATA->SATA switchover.

you're thinking what people do at home, the large business side doesn't
throw stuff out.. let me know when rhat replaces every desktop with
amd64, and ll buy you a drink if that's within the next few years ;)

> Since these CPUs run in 32-bit mode just fine, I bet you wind up with 
> people running Intel or AMD 64-bit CPUs long they abandon their 32-bit 
> OS.  I recall people often purchasing gigabit ethernet cards long before 
> they had a gigabit switch, simply because it was the volume technology 
> that was being at the time.  I think the same thing is going to happen 
> here, with AMD64 and EM64T.

yeah well what we can do (and do) is advice that hey 64bit, in some form
is really a much better environment to run servers with lots of gb on.
just takes time for products to mature on the market, even today you
can't get a lot of those boxes, not in bulk, and I am not talking about
a 1 or 2 way or a desktop with 2gb. don't confuse the systems where
4/4gb matters with your home box.

on the plus side, all the changes that went in recently seem to work
mighty well on 32gb, and I still have to see the first 64gb ia32 boxes,
m sure they are out there but I doubt that really matters any more. 

anyhow, the current VM seems to be capable of 32gb systems on ia32
happily without needing ugly 4/4. which is awesome. it's also
interesting  timing. anywys, 2.6 kicks butt quite decently :)

Wim

Re: 4g/4g for 2.6.6

[William Lee Irwin III]

On Sun, May 23, 2004 at 01:32:19PM -0700, Linus Torvalds wrote:
> Quite frankly, a number of us are hoping that we can make them
> unnecessary. The cost of the 4g/4g split is absolutely _huge_ on some
> things, including basic stuff like kernel compiles.
> The only valid reason for the 4g split is that the VM doesn't always 
> behave well with huge amounts of highmem. The anonvma stuff in 2.6.7-pre1 
> is hoped to make that much less of an issue.
> Personally, if we never need to merge 4g for real, I'll be really really 
> happy. I see it as a huge ugly hack.

The performance can be improved by using a u area to store and map
things like vmas, kernel stacks, pagetables, file handles and
descriptor tables, and the like with supervisor privileges in the same
set of pagetables as the user context so that system calls may be
serviced without referencing the larger global kernel data area, which
would require the %cr3 reload. This does, however, seem at odds with
Linux' design in a number of respects, e.g. vmas etc. are on lists
containing elements belonging to different contexts. I suspect kernels
doing this would have to architect their page replacement algorithms
and truncate() semantics so as to avoid these out-of-context accesses
or otherwise suffer these operations being inefficient.

-- wli

Re: 4g/4g for 2.6.6

[Martin J. Bligh]

> Since these CPUs run in 32-bit mode just fine, I bet you wind up with 
> people running Intel or AMD 64-bit CPUs long they abandon their 32-bit OS.  
> I recall people often purchasing gigabit ethernet cards long before they 
> had a gigabit switch, simply because it was the volume technology that 
> was being at the time.  I think the same thing is going to happen here, 
> with AMD64 and EM64T.

Except some bright spark at Intel decided to fix the low end stuff first,
and not the high end stuff for ages. Marvellous - all those 64GB 2 CPU boxes
are fixed. Triples all round. 

> AMD got a lot of things right with this one...

If the rest of the world would catch up, I'd be a happy man ...

M.

Re: 4g/4g for 2.6.6

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

On Sun, 2004-05-23 at 23:55, Phy Prabab wrote:
> So do I understand this correctly, in 2.6.7(+) it will
> no longer be necessary to have the 4g patches?  I will
> be able to get 4g/process with the going forward
> kernels?

The kernel RPMs I do (http://people.redhat.com/arjanv/2.6/) pretty much
will have it always for 2.6. Not just for large memory configs, but
because several userspace applications (databases, java etc) really like
that extra Gb of virtual space too. 
As for the cost; 4:4 split seems to be hardly expensive at all, only in
some microbenchmarks.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 4g/4g for 2.6.6

[Phy Prabab]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 966 bytes --]

Is this source, or precompiled targeting just RH
setups?

Thank you for your help.
Phy

--- Arjan van de Ven <arjanv@redhat.com> wrote:
> On Sun, 2004-05-23 at 23:55, Phy Prabab wrote:
> > So do I understand this correctly, in 2.6.7(+) it
> will
> > no longer be necessary to have the 4g patches?  I
> will
> > be able to get 4g/process with the going forward
> > kernels?
> 
> The kernel RPMs I do
> (http://people.redhat.com/arjanv/2.6/) pretty much
> will have it always for 2.6. Not just for large
> memory configs, but
> because several userspace applications (databases,
> java etc) really like
> that extra Gb of virtual space too. 
> As for the cost; 4:4 split seems to be hardly
> expensive at all, only in
> some microbenchmarks.
> 

> ATTACHMENT part 2 application/pgp-signature
name=signature.asc



	
		
__________________________________
Do you Yahoo!?
Yahoo! Domains – Claim yours for only $14.70/year
http://smallbusiness.promotions.yahoo.com/offer 

Re: 4g/4g for 2.6.6

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 142 bytes --]

On Mon, May 24, 2004 at 12:11:26AM -0700, Phy Prabab wrote:
> Is this source, or precompiled targeting just RH
> setups?

it's both actually.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 4g/4g for 2.6.6

[Phy Prabab]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 539 bytes --]

Thats great!  Do you include a change file to see what
other patches have been applied?

Thank you very much for your help.
Phy

--- Arjan van de Ven <arjanv@redhat.com> wrote:
> On Mon, May 24, 2004 at 12:11:26AM -0700, Phy Prabab
> wrote:
> > Is this source, or precompiled targeting just RH
> > setups?
> 
> it's both actually.
> 

> ATTACHMENT part 2 application/pgp-signature 



	
		
__________________________________
Do you Yahoo!?
Yahoo! Domains – Claim yours for only $14.70/year
http://smallbusiness.promotions.yahoo.com/offer 

Re: 4g/4g for 2.6.6

[Ingo Molnar]

* Linus Torvalds <torvalds@osdl.org> wrote:

> Quite frankly, a number of us are hoping that we can make them
> unnecessary. The cost of the 4g/4g split is absolutely _huge_ on some
> things, including basic stuff like kernel compiles.
>
> The only valid reason for the 4g split is that the VM doesn't always
> behave well with huge amounts of highmem. The anonvma stuff in
> 2.6.7-pre1 is hoped to make that much less of an issue.
>
> Personally, if we never need to merge 4g for real, I'll be really
> really happy. I see it as a huge ugly hack.

i agree with the hack part - but the performance aspect has been blown
out of proportion. 4:4 has the same cost on kernel compiles as highpte.
There are also real workloads where it actually helps performance.

also, the 4:4 overhead is really a hardware problem - and there are
x86-compatible CPUs (amd64) where the TLB flush problem has already been
solved: on amd64 the 4:4 feature has no noticeable overhead. So as long
as people opt for a 32-bit OS (even on a 64-bit CPU, for whatever weird
compatibility reason), 4:4 can be useful. For the other workloads i as
much hope as everyone else that people switch to a 64-bit OS on x86-64
ASAP!

also, while a quick transition to x86-64 will most likely happen, the
large installed base of big x86 boxes is a matter of fact too - and they
wont vanish into thin air. Also, there will always be specific user
workloads where lowmem grows to large values. Not to mention the fact
that 4:4 is a nice debugging/security tool as you cannot dereference
user pointers ;) - it has caught countless bugs already. Plus there are
specific workloads that want 4GB of userspace (and no, 3.5 GB wont do).

So 4:4 will have its niches where it will live on. We could argue on and
on how quickly 'x86 with more than 4GB of RAM' and 'x86 with 4GB of a
userspace' will become a niche, hopefully it happens fast but we've got
to keep our options open.

	Ingo

Re: 4g/4g for 2.6.6

[Thomas Glanzmann]

Hi,

> a coworker made the old 4g patch ready to apply clean into 2.6.6 . He's
> also working on a nice model, I think he's reading lkml so maybe he'll
> answer.

forget the URL:

http://wwwcip.informatik.uni-erlangen.de/~sithglan/2.6.6-4g

	Thomas

Re: 4g/4g for 2.6.6

[Dave Jones]

On Mon, May 24, 2004 at 12:27:05AM -0700, Phy Prabab wrote:
 > Thats great!  Do you include a change file to see what
 > other patches have been applied?

It's in the kernel-2.6.spec in those RPMs.

		Dave

Re: 4g/4g for 2.6.6

[Andrea Arcangeli]

On Mon, May 24, 2004 at 10:25:22AM +0200, Ingo Molnar wrote:
> on how quickly 'x86 with more than 4GB of RAM' and 

s/4GB/32GB/

my usual x86 test box has 48G of ram (though to keep an huge margin of
safety we assume 32G is the very safe limit).

Re: 4g/4g for 2.6.6

[Rik van Riel]

On Mon, 24 May 2004, Andrea Arcangeli wrote:
> On Mon, May 24, 2004 at 10:25:22AM +0200, Ingo Molnar wrote:
> > on how quickly 'x86 with more than 4GB of RAM' and 
> 
> s/4GB/32GB/
> 
> my usual x86 test box has 48G of ram (though to keep an huge margin of
> safety we assume 32G is the very safe limit).

Just how many 3GB sized processes can you run on that
system, if each of them have a hundred or so VMAs ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: 4g/4g for 2.6.6

[Andrea Arcangeli]

On Tue, May 25, 2004 at 03:15:14PM -0400, Rik van Riel wrote:
> On Mon, 24 May 2004, Andrea Arcangeli wrote:
> > On Mon, May 24, 2004 at 10:25:22AM +0200, Ingo Molnar wrote:
> > > on how quickly 'x86 with more than 4GB of RAM' and 
> > 
> > s/4GB/32GB/
> > 
> > my usual x86 test box has 48G of ram (though to keep an huge margin of
> > safety we assume 32G is the very safe limit).
> 
> Just how many 3GB sized processes can you run on that
> system, if each of them have a hundred or so VMAs ?

with 400m of normal zone we can allocate 5000000 vmas, if each task uses
100 of them that's around 50000 processes. The 3G size doesn't matter.

I think you meant each task using some _dozen_thousand_ of vmas (not
hundreds), that's what actually happens with 32k large vmas spread over
2G of shared memory on some database (assuming no merging, with merging
it goes down to the few thousands), but remap_file_pages has been
designed exactly to avoid allocating several thousands of VMA per task,
no? So it's just 1 vma per task (plus a few more vmas for the binary
shared libs and anonymous memory).

Clearly by opening enough files or enough network sockets or enough vmas
or similar, you can still run out of normal zone, even on a 2G system,
but this is not the point or you would be shipping 4:4 on the 2G systems
too, no? We're not trying to make it impossible to run out of zone
normal, even 4:4 can't make that impossible to happen on >4G boxes.

Re: 4g/4g for 2.6.6

[Manfred Spraul]

Ingo wrote:

>also, the 4:4 overhead is really a hardware problem - and there are
>x86-compatible CPUs (amd64) where the TLB flush problem has already been
>solved: on amd64 the 4:4 feature has no noticeable overhead.
>
Do you have an idea why amd64 is better for 4g4g? Which benchmark did 
you use for testing?

--
    Manfred

Re: 4g/4g for 2.6.6

[Rik van Riel]

On Tue, 25 May 2004, Andrea Arcangeli wrote:

> Clearly by opening enough files or enough network sockets or enough vmas
> or similar, you can still run out of normal zone, even on a 2G system,
> but this is not the point or you would be shipping 4:4 on the 2G systems
> too, no?

The point is, people like to run bigger workloads on
bigger systems. Otherwise they wouldn't bother buying
those bigger systems.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: 4g/4g for 2.6.6

[Ingo Molnar]

* Manfred Spraul <manfred@colorfullife.com> wrote:

> >also, the 4:4 overhead is really a hardware problem - and there are
> >x86-compatible CPUs (amd64) where the TLB flush problem has already been
> >solved: on amd64 the 4:4 feature has no noticeable overhead.
> >
> Do you have an idea why amd64 is better for 4g4g? Which benchmark did
> you use for testing?

i used an althlon64 CPU. amd64 is better because it has a hardware
feature that 'watches' for memory updates to cached TLBs, and it tags
the TLBs by cr3. So it can avoid having to flush those TLBs that didnt
actually change. So an amd64 CPU has in excess of 1000+ TLBs ...

	Ingo

Re: 4g/4g for 2.6.6

[Rik van Riel]

On Tue, 25 May 2004, Rik van Riel wrote:

> The point is, people like to run bigger workloads on
> bigger systems. Otherwise they wouldn't bother buying
> those bigger systems.

Btw, you're right about the VMAs.  Looking through customer
stuff a bit more the more common issues are low memory being
eaten by dentry / inode cache - which you can't always reclaim
due to files being open, and don't always _want_ to reclaim
because that could well be a bigger performance hit than the
4:4 split.

The primary impact of the dentry / inode cache using memory
isn't lowmem exhaustion, btw.  It's lowmem fragmentation.

Fragmentation causes fork trouble (gone with the 4k stacks)
and trouble for the network layer and kiobuf allocation,
which still do need higher order allocations.

Sure, the 4/4 kernel could also have problems with lowmem
fragmentation, but it just seems to be nowhere near as bad.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: 4g/4g for 2.6.6

[Bill Davidsen]

Rik van Riel wrote:
> On Tue, 25 May 2004, Andrea Arcangeli wrote:
> 
> 
>>Clearly by opening enough files or enough network sockets or enough vmas
>>or similar, you can still run out of normal zone, even on a 2G system,
>>but this is not the point or you would be shipping 4:4 on the 2G systems
>>too, no?
> 
> 
> The point is, people like to run bigger workloads on
> bigger systems. Otherwise they wouldn't bother buying
> those bigger systems.
> 

Sure, and "bigger workloads" can mean a lot of small client processes 
talking to a threaded large process, like database or news, or it can be 
huge datasets, like image or multimedia processing. Unfortunately it 
seems that these all (ab)use the VM in various ways.

x86 with lots of memory is likely to remain cost effective for years, 
not only because it not only allows more memory but needs more memory in 
most cases, but because vendors will hang on to their profit margins on 
64 bit CPUs for that long. And for uses with many small client 
processes, the advantage of 64 bits is pretty small when you have 2MB 
processes. Given a bus which allows i/o to more memory, the benefits in 
performance are really hard to see.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: 4g/4g for 2.6.6

[Andrea Arcangeli]

On Tue, May 25, 2004 at 04:10:29PM -0400, Rik van Riel wrote:
> Fragmentation causes fork trouble (gone with the 4k stacks)

btw, the 4k stacks sounds not safe to me, most people only tested with
8k stacks so far, I wouldn't make that change in a production tree
without an unstable cycle of testing in between. I'd rather risk a
an allocation failure than a stack memory corruption.

x86-64 has per-irq stacks that allowed to reduce the stack size to 8k
(which is very similar to 4k for an x86, but without per-irq stack it's
too risky).

as for the dcache size, I can certainly imagine you can measure slowdown
bigger than 4:4 in some very vfs intensive workload, but the stuff that
needs 32G of ram normally uses only rawio or a few huge files as storage
anyways. as for the kiobufs they're long gone in 2.6.

Possibly some webserving could use 32G as pure pagecache for the
webserver with tons of tiny files, for such an app the 2:2 or 1:3 model
should be better than 4:4.

Re: 4g/4g for 2.6.6

[Andrew Morton]

Rik van Riel <riel@redhat.com> wrote:
>
> On Tue, 25 May 2004, Rik van Riel wrote:
> 
> > The point is, people like to run bigger workloads on
> > bigger systems. Otherwise they wouldn't bother buying
> > those bigger systems.
> 
> Btw, you're right about the VMAs.  Looking through customer
> stuff a bit more the more common issues are low memory being
> eaten by dentry / inode cache - which you can't always reclaim
> due to files being open, and don't always _want_ to reclaim
> because that could well be a bigger performance hit than the
> 4:4 split.

I did some testing a year or two back with the normal zone wound down to a
few hundred megs - filesytem benchmarks were *severely* impacted by the
increased turnover rate of fs metadata pagecache and VFS caches.  I forget
the details, but it was "wow".

> The primary impact of the dentry / inode cache using memory
> isn't lowmem exhaustion, btw.  It's lowmem fragmentation.
> 
> Fragmentation causes fork trouble (gone with the 4k stacks)
> and trouble for the network layer and kiobuf allocation,
> which still do need higher order allocations.

I'm suspecting we'll end up needing mempools (or something) of 1- and
2-order pages to support large-frame networking.  I'm surprised there isn't
more pressure to do something about this.  Maybe people are increasing
min_free_kbytes.

Re: 4g/4g for 2.6.6

[Ingo Molnar]

* Andrew Morton <akpm@osdl.org> wrote:

> > Btw, you're right about the VMAs.  Looking through customer
> > stuff a bit more the more common issues are low memory being
> > eaten by dentry / inode cache - which you can't always reclaim
> > due to files being open, and don't always _want_ to reclaim
> > because that could well be a bigger performance hit than the
> > 4:4 split.
> 
> I did some testing a year or two back with the normal zone wound down
> to a few hundred megs - filesytem benchmarks were *severely* impacted
> by the increased turnover rate of fs metadata pagecache and VFS
> caches.  I forget the details, but it was "wow".

and it's not only the normal workloads we know about - people really do
lots of weird stuff with Linux (and we are happy that they use Linux and
that Linux keeps chugging along), and people seem to prefer a 10%
slowdown to a box that locks up or -ENOMEM's. I'm not trying to insert
any unjustified fear, 3:1 can be ok with lots of RAM, but it's _clearly_
wishful thinking that 32 GB x86 will be OK with just 600 MB of lowmem.
600 MB of lowmem means a 1:80 lowmem to RAM ratio, which is insane. Yes,
it will be OK with select workloads and applications. With 4:4 it's 3.4
GB lowmem and the ratio is down to a much saner 1:9, and boxes pushed
against the wall keep up better.

(but i wont attempt to convince Andrea - and i'm not at all unhappy that
he is trying to fix 3:1 to be more usable on big boxes because those
efforts also decrease the RAM footprint of the UP kernel, which is
another sensitive area. These efforts also help sane 64-bit
architectures, so it's a win-win situation even considering our
disagreement wrt. 4:4.)

> I'm suspecting we'll end up needing mempools (or something) of 1- and
> 2-order pages to support large-frame networking.  I'm surprised there
> isn't more pressure to do something about this.  Maybe people are
> increasing min_free_kbytes.

hm, 1.5K pretty much seems to be the standard. Plus large frames can be
scatter-gathered via fragmented skbs. Seldom is there a need for a large
skb to be linear.

	Ingo

Re: 4g/4g for 2.6.6

[David S. Miller]

On Tue, 25 May 2004 23:48:17 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> hm, 1.5K pretty much seems to be the standard. Plus large frames can be
> scatter-gathered via fragmented skbs. Seldom is there a need for a large
> skb to be linear.

Unfortunately TSO with non-sendfile apps makes huge 64K SKBs get
built.

Re: 4g/4g for 2.6.6

[Ingo Molnar]

* David S. Miller <davem@redhat.com> wrote:

> > hm, 1.5K pretty much seems to be the standard. Plus large frames can be
> > scatter-gathered via fragmented skbs. Seldom is there a need for a large
> > skb to be linear.
> 
> Unfortunately TSO with non-sendfile apps makes huge 64K SKBs get
> built.

hm, shouldnt we disable TSO in this case - or is it a win even in this
case?

	Ingo

Re: 4g/4g for 2.6.6

[David S. Miller]

On Wed, 26 May 2004 00:20:32 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> > Unfortunately TSO with non-sendfile apps makes huge 64K SKBs get
> > built.
> 
> hm, shouldnt we disable TSO in this case - or is it a win even in this
> case?

It is a win even in this case.

Re: 4g/4g for 2.6.6

[Pavel Machek]

Hi!

> > I think the switchover will happen fairly rapidly, since it is 
> > positioned "the upgrade you get next time you buy a new computer", 
> > similar to the current PATA->SATA switchover.
> 
> you're thinking what people do at home, the large business side doesn't
> throw stuff out.. let me know when rhat replaces every desktop with
> amd64, and ll buy you a drink if that's within the next few years ;)
> 

Desktops do not tent to have 4GB+ memory.
-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: 4g/4g for 2.6.6

[Eric W. Biederman]

Linus Torvalds <torvalds@osdl.org> writes:

> On Sun, 23 May 2004, Jeff Garzik wrote:
> > 
> > Sorta like I'm hoping that cheap and prevalent 64-bit CPUs make PAE36 
> > and PAE40 on ia32 largely unnecessary.  Addressing more memory than 32 
> > bits of memory on a 32-bit CPU always seemed like a hack to me, and a 
> > source of bugs and lost performance...
> 
> I agree. I held out on PAE for a longish while, in the unrealistic hope 
> that people would switch to alpha's. 
> 
> Oh, well. I don't expect _everybody_ to switch to x86-64 immediately, but 
> I hope we can hold out long enough without 4g that it does work out this 
> time.

Sounds sane. 

One of the real problems on machines with more than 4GB in 32bit mode
is where do you put all of the pci resources.  Especially when you start
getting into machines with large memory mapped resources of 128MB or more.

On Intel chipsets that is usually not a problem because you can relocate the
memory from under those resources and still get at it with PAE36.  Other
chipsets don't really have that kind of capability.

>From what I can tell all of the large PCI memory mapped resources are
64bit.  Which gives another solution of simply moving all of those
large memory mapped I/O resources above the memory entirely.  Besides
solving my immediate problem of loosing 1/2GB of memory in some
cases, that looks like the only sane long term solution.

So to encourage x86-64 usage, I am going to implement that in
LinuxBIOS and encourage any other BIOS vendor I run into to
follow suit.  That way the only painful customer questions I will
get will be why doesn't this high performance card work with a
32bit kernel :)  Which is much easier to explain :)

Can anyone think of a reason that would not be a good solution?


Eric