Re: linux+glibc memory allocator, poor performance

Re: linux+glibc memory allocator, poor performance

[J.C. Pizarro]

On Wed, 12 Mar 2008 19:14:06 +0100, Xose Vazquez Perez wrote:
> hi,
>
> More tests from FreeBSD people <http://people.freebsd.org/~kris/scaling/smp.html>
>
> Poor performance with 2.6.24 linux kernel and glibc 2.7 provided by Fedora 8:
>
> *Memory allocation performance on FreeBSD and Linux with ebizzy (Mar 2008)*
> <http://people.freebsd.org/~kris/scaling/ebizzy.html>
>
> -thanks-
>
> regards,
> --
> so much to do, so little time.

Assume a SMP system that has 8 CPUs. The main problem of requesting
pages is the BKL (Big Kernel Lock) in this SMP system used for mutual
exclusion of the shared resource (the memory).

To solve this major problem, i propose you freely to allocate 8 local caches
of (e.g.) 2 MiB each CPU (total 2MiB x 8 CPUs = 16 MiB) acting as
8 producer buffers for globally many consumer tasks (e.g. >= 20).

When the some producer buffer is empty then it does unfrequently BKL to
allocate another 2 MiB more from the shared resource (the memory).

In the reverse, it's simple, return back the unused pages to the local buffer
of the producer, and when this full then to do BKL too to unallocate its half
to the shared resource (the memory).

   Sincerely, J.C.Pizarro

Re: linux+glibc memory allocator, poor performance

[Al Viro]

On Wed, Mar 12, 2008 at 09:09:04PM +0100, J.C. Pizarro wrote:

> Assume a SMP system that has 8 CPUs. The main problem of requesting
> pages is the BKL (Big Kernel Lock) in this SMP system used for mutual
> exclusion of the shared resource (the memory).

Really?  Used _where_?

> To solve this major problem, i propose you freely

I hate to think of what you'd propose under duress...

> to allocate 8 local caches
> of (e.g.) 2 MiB each CPU (total 2MiB x 8 CPUs = 16 MiB) acting as
> 8 producer buffers for globally many consumer tasks (e.g. >= 20).
> 
> When the some producer buffer is empty then it does unfrequently BKL to
> allocate another 2 MiB more from the shared resource (the memory).

Excellent advice.  One question: why the bleeding hell would we use BKL
in allocator, frequently or not, when we do not share the crucial resource
needed to come up with such ideas?  Or are you proposing to share your
crack pipe as well?

> In the reverse, it's simple, return back the unused pages to the local buffer
> of the producer, and when this full then to do BKL too to unallocate its half
> to the shared resource (the memory).

I wonder...  Are you seriously implying that nobody here had ever been able
to come up with anything better than _that_ (e.g. with an amazingly advanced
idea of googling for papers on allocators for all of two minutes)?  And that
you are so sure of it that you couldn't be arsed to check what the hell is
allocator really doing, on the off-chance that somebody might have been able
to match your brilliant intellectual feat?

And then we are told that linux-kernel regulars are mean and insulting...

FWIW, the quality of proposed "solution" is not an issue; hell, even "what
if that code is really so dumb that <....> would be an improvement" is not
a problem - the estimate is too low, but far be it from me to suggest that
one should apriori assume that unreviewed code is good.  What _really_
makes the whole thing incredibly insulting is that you had not bothered
to even look at the damn thing and just went on with "of course none of
you could have ever come up with anything better - I don't even need to
look at it to tell you that".

Re: linux+glibc memory allocator, poor performance

[David Newall]

Al Viro wrote:
> And then we are told that linux-kernel regulars are mean and insulting...
>   

You really are being rude and insulting.  Unnecessarily so.  Maybe
you're just recalcitrant (in the sense that President Suharto inferred,
not the sense that Prime Minister Keating meant.)

Re: linux+glibc memory allocator, poor performance

[Mike Galbraith]

On Thu, 2008-03-13 at 16:27 +1030, David Newall wrote:
> Al Viro wrote:
> > And then we are told that linux-kernel regulars are mean and insulting...
> >   
> 
> You really are being rude and insulting.

FWIW, I thought it was a well deserved "Al" thump.  (not that my opinion
or sense of humor matters, but it made me chuckle mightily)

	-Mike

Re: linux+glibc memory allocator, poor performance

[David Newall]

Mike Galbraith wrote:
> On Thu, 2008-03-13 at 16:27 +1030, David Newall wrote:
>   
>> Al Viro wrote:
>>     
>>> And then we are told that linux-kernel regulars are mean and insulting...
>>>   
>>>       
>> You really are being rude and insulting.
>>     
>
> FWIW, I thought it was a well deserved "Al" thump.

Hostility such as that gives this list a bad reputation and drives
people away.  One can (and should) tell people that an idea is
completely wrong without being rude.

Perhaps I'm just incredibly sensitive today.

Re: linux+glibc memory allocator, poor performance

[Christoph Hellwig]

On Thu, Mar 13, 2008 at 04:52:11PM +1030, David Newall wrote:
> > FWIW, I thought it was a well deserved "Al" thump.
> 
> Hostility such as that gives this list a bad reputation and drives
> people away.  One can (and should) tell people that an idea is
> completely wrong without being rude.
> 
> Perhaps I'm just incredibly sensitive today.

Or just dumb.  If you're read past post by Mr Bizzaro you'd realize that
driving him away would be a good thing.

Re: linux+glibc memory allocator, poor performance

[David Newall]

Christoph Hellwig wrote:
> On Thu, Mar 13, 2008 at 04:52:11PM +1030, David Newall wrote:
>   
>>> FWIW, I thought it was a well deserved "Al" thump.
>>>       
>> Hostility such as that gives this list a bad reputation and drives
>> people away.  One can (and should) tell people that an idea is
>> completely wrong without being rude.
>>
>> Perhaps I'm just incredibly sensitive today.
>>     
>
> Or just dumb.  If you're read past post by Mr Bizzaro you'd realize that
> driving him away would be a good thing

This is the sort of comment that gives Linux kernel developers a bad
name.  I don't plan to read his past posts, just like I don't plan to
read yours.  I don't need to read every stupid thing you've ever said to
know that I don't want anyone driven away.  Can't you just ignore him?

Re: linux+glibc memory allocator, poor performance

[Pekka Enberg]

On Thu, Mar 13, 2008 at 8:22 AM, David Newall <davidn@davidnewall.com> wrote:
>  Hostility such as that gives this list a bad reputation and drives
>  people away.  One can (and should) tell people that an idea is
>  completely wrong without being rude.

Hey, why don't we create a linux-sensitive@vger.kernel.org and you
guys can move your nice and friendly chats over there...?

Re: linux+glibc memory allocator, poor performance

[David Newall]

Pekka Enberg wrote:
> On Thu, Mar 13, 2008 at 8:22 AM, David Newall <davidn@davidnewall.com> wrote:
>   
>>  Hostility such as that gives this list a bad reputation and drives
>>  people away.  One can (and should) tell people that an idea is
>>  completely wrong without being rude.
>>     
>
> Hey, why don't we create a linux-sensitive@vger.kernel.org and you
> guys can move your nice and friendly chats over there...?
>   


Better create a linux-arseholes@vger.kernel.org, and move the attitude
there.

Re: linux+glibc memory allocator, poor performance

[Alan Cox]

On Thu, 13 Mar 2008 11:10:55 +0200
"Pekka Enberg" <penberg@cs.helsinki.fi> wrote:

> On Thu, Mar 13, 2008 at 8:22 AM, David Newall <davidn@davidnewall.com> wrote:
> >  Hostility such as that gives this list a bad reputation and drives
> >  people away.  One can (and should) tell people that an idea is
> >  completely wrong without being rude.
> 
> Hey, why don't we create a linux-sensitive@vger.kernel.org and you
> guys can move your nice and friendly chats over there...?

I think it would be far better to create linux-flame and send the hostile
stuff away. In a lot of cultures hostile responses or unfriendly
responses are taken badly. Politeness is a good thing this list needs
more of.

Re: linux+glibc memory allocator, poor performance

[Pekka Enberg]

Hi Alan,

On Thu, Mar 13, 2008 at 2:10 PM, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>  I think it would be far better to create linux-flame and send the hostile
>  stuff away. In a lot of cultures hostile responses or unfriendly
>  responses are taken badly. Politeness is a good thing this list needs
>  more of.

Do you honestly think we should be polite to people trolling on the
LKML? The problem with kill-filing people like J.C. Pizzarro (a known
troll on the GCC mailing list at least) is that other people will
engange in discussing these "proposals" and contribute in generating
noise anyway.

Re: linux+glibc memory allocator, poor performance

[J.C. Pizarro]

2008/3/13, Pekka Enberg <penberg@cs.helsinki.fi> wrote:
> Hi Alan,
>
>
>  On Thu, Mar 13, 2008 at 2:10 PM, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>  >  I think it would be far better to create linux-flame and send the hostile
>  >  stuff away. In a lot of cultures hostile responses or unfriendly
>  >  responses are taken badly. Politeness is a good thing this list needs
>  >  more of.
>
>
> Do you honestly think we should be polite to people trolling on the
>  LKML? The problem with kill-filing people like J.C. Pizzarro (a known
>  troll on the GCC mailing list at least) is that other people will
>  engange in discussing these "proposals" and contribute in generating
>  noise anyway.

I'm not a troll on the GCC mailing list. Some GCC's people is calling me troll.

   J.C.Pizarro

Re: linux+glibc memory allocator, poor performance

[David Newall]

Pekka Enberg wrote:
> Hi Alan,
>
> On Thu, Mar 13, 2008 at 2:10 PM, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>   
>>  Politeness is a good thing this list needs more of.
>>     
>
> Do you honestly think we should be polite to people trolling on the
> LKML?

I do.  It's obvious.  Rudeness detracts (and reflects poorly on one's
parents.)  Good manners cost nothing.  I might also point out that it
was bad manners, not any alleged trolling, which caused this sad
discussion.  Don't be rude; rather say nothing.

Re: linux+glibc memory allocator, poor performance

[Pekka Enberg]

Hi David,

On Thu, Mar 13, 2008 at 9:40 PM, David Newall <davidn@davidnewall.com> wrote:
>  I do.  It's obvious.  Rudeness detracts (and reflects poorly on one's
>  parents.)  Good manners cost nothing.  I might also point out that it
>  was bad manners, not any alleged trolling, which caused this sad
>  discussion.  Don't be rude; rather say nothing.

Thanks for the free lesson in manners! It's much appreciated.

                        Pekka

Re: linux+glibc memory allocator, poor performance

[Al Viro]

On Fri, Mar 14, 2008 at 06:10:01AM +1030, David Newall wrote:
> Pekka Enberg wrote:
> > Hi Alan,
> >
> > On Thu, Mar 13, 2008 at 2:10 PM, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> >   
> >>  Politeness is a good thing this list needs more of.
> >>     
> >
> > Do you honestly think we should be polite to people trolling on the
> > LKML?
> 
> I do.  It's obvious.  Rudeness detracts (and reflects poorly on one's
> parents.)  Good manners cost nothing.  I might also point out that it
> was bad manners, not any alleged trolling, which caused this sad
> discussion.

"Deliberate insult" would describe the original posting more accurately.
Whether you prefer to look at it as a show of bad manners or as trolling
is a matter of perspective; arguably it had been both.

> Don't be rude; rather say nothing.

Request duly noted and denied.  For the record: condescending twits of that
variety ("I can't be arsed to look at code I'm commenting on; here's my
valuable insight that is obviously beyond the mental capacity of anybody
here") can expect a harsh reply, whoever they are.  I'm yet to see l-k
regulars treating newbies that way, BTW, despite the much decried lack of
politeness.

Since you are postulating social norms, here's one you seem to have overlooked:
one who comments on code or design, be it review, suggestions of changes/
improvements, etc., ought to read the thing he is commenting on.  Whoever
it is and wherever the code in question might be.  And yes, it applies to
kernel hackers commenting on newbies' patches to the same extent as to
completely unknown folks commenting on the kernel, etc.

Breaking that rule and being caught at that is one hell of a way to earn
yourself a lousy reputation.  Combine that with arrogance and... well,
there's a term for what you become: target practice.

Re: linux+glibc memory allocator, poor performance

[Stephen Clark]

Al Viro wrote:
> On Fri, Mar 14, 2008 at 06:10:01AM +1030, David Newall wrote:
>   
>> Pekka Enberg wrote:
>>     
>>> Hi Alan,
>>>
>>> On Thu, Mar 13, 2008 at 2:10 PM, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>>>   
>>>       
>>>>  Politeness is a good thing this list needs more of.
>>>>     
>>>>         
>>> Do you honestly think we should be polite to people trolling on the
>>> LKML?
>>>       
>> I do.  It's obvious.  Rudeness detracts (and reflects poorly on one's
>> parents.)  Good manners cost nothing.  I might also point out that it
>> was bad manners, not any alleged trolling, which caused this sad
>> discussion.
>>     
>
> "Deliberate insult" would describe the original posting more accurately.
> Whether you prefer to look at it as a show of bad manners or as trolling
> is a matter of perspective; arguably it had been both.
>
>   
>> Don't be rude; rather say nothing.
>>     
>
> Request duly noted and denied.  For the record: condescending twits of that
> variety ("I can't be arsed to look at code I'm commenting on; here's my
> valuable insight that is obviously beyond the mental capacity of anybody
> here") can expect a harsh reply, whoever they are.  I'm yet to see l-k
> regulars treating newbies that way, BTW, despite the much decried lack of
> politeness.
>
> Since you are postulating social norms, here's one you seem to have overlooked:
> one who comments on code or design, be it review, suggestions of changes/
> improvements, etc., ought to read the thing he is commenting on.  Whoever
> it is and wherever the code in question might be.  And yes, it applies to
> kernel hackers commenting on newbies' patches to the same extent as to
> completely unknown folks commenting on the kernel, etc.
>
> Breaking that rule and being caught at that is one hell of a way to earn
> yourself a lousy reputation.  Combine that with arrogance and... well,
> there's a term for what you become: target practice.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
>   
Someone once said:
"If you don't have anything good to say, don't day anything at all".

I have been reading LKML since the mid 90's and continually see people 
jump on
what are maybe dumb/stupid questions or trolls instead of simply 
ignoring them.

My $.02
Steve

-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: linux+glibc memory allocator, poor performance

[Willy Tarreau]

On Thu, Mar 13, 2008 at 02:48:31PM +0200, Pekka Enberg wrote:
> Hi Alan,
> 
> On Thu, Mar 13, 2008 at 2:10 PM, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> >  I think it would be far better to create linux-flame and send the hostile
> >  stuff away. In a lot of cultures hostile responses or unfriendly
> >  responses are taken badly. Politeness is a good thing this list needs
> >  more of.
> 
> Do you honestly think we should be polite to people trolling on the
> LKML? The problem with kill-filing people like J.C. Pizzarro (a known
> troll on the GCC mailing list at least) is that other people will
> engange in discussing these "proposals" and contribute in generating
> noise anyway.

Being polite, understanding and grateful generally encourages people
to contribute and progress. Being sometimes rude helps them understand
they went too far, and gives more credit to your politeness in normal
time because you're seen as balanced. Being always rude or always
excessively polite just assigns you a style by which people don't know
what you really think. So using all the capabilities of the language
is finally the best solution to precisely express what you think.

Willy

Re: linux+glibc memory allocator, poor performance

[David Newall]

David Newall wrote:
> Al Viro wrote:
>   
>> And then we are told that linux-kernel regulars are mean and insulting...
>>   
>>     
>
> You really are being rude and insulting.  Unnecessarily so.  Maybe
> you're just recalcitrant (in the sense that President Suharto inferred,
> not the sense that Prime Minister Keating meant.)

*blink* D'oh! Mahatir, of course.

Re: linux+glibc memory allocator, poor performance

[J.C. Pizarro]

On 2008/3/12, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Wed, Mar 12, 2008 at 09:09:04PM +0100, J.C. Pizarro wrote:
>
>  > Assume a SMP system that has 8 CPUs. The main problem of requesting
>  > pages is the BKL (Big Kernel Lock) in this SMP system used for mutual
>  > exclusion of the shared resource (the memory).
>
>
> Really?  Used _where_?

To see "Lock ordering:" in mm/filemap.c

I tell you an example of this used big kernel lock
write_lock_irq/write_unlock_irq:

     void __remove_from_page_cache(struct page *page):
      * Remove a page from the page cache and free it. Caller has to make
      * sure the page is locked and that nobody else uses it - or that usage
      * is safe.  The caller must hold a write_lock on the mapping's tree_lock.

     void remove_from_page_cache(struct page *page) {
	     struct address_space *mapping = page->mapping;

	     BUG_ON(!PageLocked(page));

	     write_lock_irq(&mapping->tree_lock);
	     __remove_from_page_cache(page);
	     write_unlock_irq(&mapping->tree_lock);
     }

     * Shared mappings 30.11.1994, 15.8.1995  Bruno.
     * 'unifying' the page and buffer cache and SMP-threaded the
        page-cache, 21.05.1999, Ingo Molnar.
     * SMP-threaded pagemap-LRU 1999, Andrea Arcangeli.

Here doesn't say multiple SMP buffers of pages for SMP.
For UP, no SMP buffer is needed.

>  > To solve this major problem, i propose you freely
>
>
> I hate to think of what you'd propose under duress...

Don't hate to me, you've not reasons to insult me or to somebody.
To think of the proposals posted by any person, not of me.

>  > to allocate 8 local caches
>  > of (e.g.) 2 MiB each CPU (total 2MiB x 8 CPUs = 16 MiB) acting as
>  > 8 producer buffers for globally many consumer tasks (e.g. >= 20).
>  >
>  > When the some producer buffer is empty then it does unfrequently BKL to
>  > allocate another 2 MiB more from the shared resource (the memory).
>
>
> Excellent advice.  One question: why the bleeding hell would we use BKL
>  in allocator, frequently or not, when we do not share the crucial resource
>  needed to come up with such ideas?  Or are you proposing to share your
>  crack pipe as well?

You're contradictory. It uses BKL or slower spinlocks/rwlocks in allocator.
It shares the crucial resource.

The physical RAM of the SMP system is a shared resource for 2 or more CPUs
(even in NUMA systems), so to avoid the race conditions, it needs to be
temporaly locked to comply the mutual exclusion of this shared resource.

I'm saying that it will try to reduce the frequency of BKLs to maximize the
OS performance in the mind of Huffman (freq.) & Amdhal (hot code).

I don't understand you saying "to share your crack pipe", what means it?
Nothing is crack. It's not an unix pipe. It's 8 local SMP buffers for 8 CPUs.

>  > In the reverse, it's simple, return back the unused pages to the local buffer
>  > of the producer, and when this full then to do BKL too to unallocate its half
>  > to the shared resource (the memory).
>
>
> I wonder...  Are you seriously implying that nobody here had ever been able
>  to come up with anything better than _that_ (e.g. with an amazingly advanced
>  idea of googling for papers on allocators for all of two minutes)?  And that
>  you are so sure of it that you couldn't be arsed to check what the hell is
>  allocator really doing, on the off-chance that somebody might have been able
>  to match your brilliant intellectual feat?

Don't point me! Don't menace me!

I know how to work internally a monolithic O.S., so i'm not googling papers from
others and i'm thinking from my brain how to work a little better some things
that i proposed.

>  And then we are told that linux-kernel regulars are mean and insulting...
>
>  FWIW, the quality of proposed "solution" is not an issue; hell, even "what
>  if that code is really so dumb that <....> would be an improvement" is not
>  a problem - the estimate is too low, but far be it from me to suggest that
>  one should apriori assume that unreviewed code is good.  What _really_
>  makes the whole thing incredibly insulting is that you had not bothered
>  to even look at the damn thing and just went on with "of course none of
>  you could have ever come up with anything better - I don't even need to
>  look at it to tell you that".

You're a distractor man that distracts to the people.

   J.C.Pizarro

Re: linux+glibc memory allocator, poor performance

[Rik van Riel]

On Wed, 12 Mar 2008 21:09:04 +0100
"J.C. Pizarro" <jcpiza@gmail.com> wrote:

> Assume a SMP system that has 8 CPUs. The main problem of requesting
> pages is the BKL (Big Kernel Lock) in this SMP system used for mutual
> exclusion of the shared resource (the memory).
> 
> To solve this major problem, i propose you freely to allocate 8 local caches
> of (e.g.) 2 MiB each CPU (total 2MiB x 8 CPUs = 16 MiB) acting as
> 8 producer buffers for globally many consumer tasks (e.g. >= 20).
> 
> When the some producer buffer is empty then it does unfrequently BKL to
> allocate another 2 MiB more from the shared resource (the memory).

You really should read the source code before proposing ideas.

The kernel has done roughly what you describe since a little before
2.6.0.

-- 
All rights reversed.

Re: linux+glibc memory allocator, poor performance

[Al Viro]

On Wed, Mar 12, 2008 at 05:22:21PM -0400, Rik van Riel wrote:
> On Wed, 12 Mar 2008 21:09:04 +0100
> "J.C. Pizarro" <jcpiza@gmail.com> wrote:
> 
> > Assume a SMP system that has 8 CPUs. The main problem of requesting
> > pages is the BKL (Big Kernel Lock) in this SMP system used for mutual
> > exclusion of the shared resource (the memory).
> > 
> > To solve this major problem, i propose you freely to allocate 8 local caches
> > of (e.g.) 2 MiB each CPU (total 2MiB x 8 CPUs = 16 MiB) acting as
> > 8 producer buffers for globally many consumer tasks (e.g. >= 20).
> > 
> > When the some producer buffer is empty then it does unfrequently BKL to
> > allocate another 2 MiB more from the shared resource (the memory).
> 
> You really should read the source code before proposing ideas.
> 
> The kernel has done roughly what you describe since a little before
> 2.6.0.

ITYM a little before 2.4.0, and even prior to that it hadn't been under BKL.

linux+glibc memory allocator, poor performance

[Xose Vazquez Perez]

hi,

More tests from FreeBSD people <http://people.freebsd.org/~kris/scaling/smp.html>

Poor performance with 2.6.24 linux kernel and glibc 2.7 provided by Fedora 8:

*Memory allocation performance on FreeBSD and Linux with ebizzy (Mar 2008)*
<http://people.freebsd.org/~kris/scaling/ebizzy.html>


-thanks-

regards,
-- 
so much to do, so little time.

Re: linux+glibc memory allocator, poor performance

[Xose Vazquez Perez]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

Xose Vazquez Perez wrote:

> Poor performance with 2.6.24 linux kernel and glibc 2.7 provided by 
> Fedora 8:
> 
> *Memory allocation performance on FreeBSD and Linux with ebizzy (Mar 2008)*
> <http://people.freebsd.org/~kris/scaling/ebizzy.html>

Definitely it looks like a glibc bug: http://www.kernel.org/pub/linux/kernel/people/npiggin/ebizzy/
This time google libc does not help, but jemalloc did it.

yellow color(attached png) freeBSD vs. gnu malloc both in LiNUX.


-thanks for test it Nick-

regards,
-- 
so much to do, so little time.

[-- Attachment #2: ebizzy2.png --]
[-- Type: image/png, Size: 4696 bytes --]