Re: [discuss] memrlimit - potential applications that can use

[Balbir Singh <balbir@linux.vnet.ibm.com>]

KAMEZAWA Hiroyuki wrote:
> On Thu, 21 Aug 2008 15:56:41 +0530
> Balbir Singh <balbir@linux.vnet.ibm.com> wrote:
> 
>> KAMEZAWA Hiroyuki wrote:
>>> On Thu, 21 Aug 2008 08:55:52 +0530
>>> Balbir Singh <balbir@linux.vnet.ibm.com> wrote:
>>>
>>>>>>> So, before we expand the use of those features to control groups by
>>>>>>> adding a bunch of new code, let's make sure that there will be users
>>>>>> for
>>>>>>> it and that those users have no better way of doing it.
>>>>>> I am all ears to better ways of doing it. Are you suggesting that overcommit was
>>>>>> added even though we don't actually need it?
>>>>> It serves a purpose, certainly.  We have have better ways of doing it
>>>>> now, though.  "So, before we expand the use of those features to
>>>>> control groups by adding a bunch of new code, let's make sure that there
>>>>> will be users for it and that those users have no better way of doing
>>>>> it."
>>>>>
>>>>> The one concrete user that's been offered so far is postgres.  I've
>>>> No, you've been offered several, including php and apache that use memory limits.
>>>>
>>>>> suggested something that I hope will be more effective than enforcing
>>>>> overcommit.  
>>> I'm sorry I miss the point. My concern on memrlimit (for overcommiting) is that
>>> it's not fair because an application which get -ENOMEM at mmap() is just someone
>>> unlucky.
>> It can happen today with overcommit turned on. Why is it unlucky?
>>
> Today's overcommit is also unlucky ;) 
> 
> For example) process A and B is under a memrlimit.
>  process A no memory leak, it often calls malloc() and free().
>  process B does memory leak, 100MB per night.
> 
> process A cannot do anything when it notices malloc() returns NULL.
> It controls his memory usage perfectly. He is unlucky and will die.
> process B can use up VSZ which is freed by process A.
> 

Yes, true that will happen. Why will A die because it sees NULL? Yes, many
applications do die, but that is not how malloc == NULL is expected to be
handled. If that is a concern, do not use any memrlimits for A and B, if you do
you will find the bug early.

Now consider the other scenario, if there really is a memory leak and process B
is using all that memory, two things to consider

1. Without swap controller, B will start swapping out A's memory and cause
excessive swapping and performance loss
2. With swap controller enabled, at some point we will hit the swap limit, what
happens then?

> (OOM-killer, is disliked by everyone, have some kind of fairness.
>  It checks usage.)
> 
>>  I think it's better to trigger some notifier to application or daemon
>>> rather than return -ENOMEM at mmap(). Notification like "Oh, it seems the VSZ
>>> of total application exceeds the limit you set. Although you can continue your
>>> operation, it's recommended that you should fix up the  situation".
>>> will be good.
>>>
>> So you are suggesting that when we are running out of memory (as defined by our
>> current resource constraints), we don't return -ENOMEM, but instead we now
>> handle a new event that states that we are running out of memory?
>>
> Not "running out of memory" Just "VSZ is over the limit you set/expected".
> 
> My point is an application witch can handle NULL returned by malloc() is
> not very popular, I think.
> 

Yes and that's why we have the flexibility, if the application can't deal with
it don't set memrlimits for those applications :)

-- 
	Balbir