RSS Limit implementation issue

RSS Limit implementation issue

[Ram Gupta]

I am working to implement enforcing RSS limits of a process. I am
planning to make a check for rss limit when setting up pte. If the
limit is crossed I see couple of  different ways of handling .

1. Kill the process . In this case there is no swapping problem.

2. Dont kill the process but dont allocate the memory & do yield as we
do for init process. Modify the scheduler not to chose the process
which has already allocated rss upto its limit. When rss usage
fallsbelow its limit then the scheduler may chose it again to run.
Here there is a scenario when no page of the process has been freed or
swapped out because there were enough free pages? Then we need a way
to reschedule the process by forcefully freeing some pages or need to
kill the process.

I am looking forward for your comments & pros/cons of both approach &
any other alternatives you might come up with.

Thanks
Ram Gupta

Re: RSS Limit implementation issue

[Alan Cox]

On Iau, 2006-02-09 at 15:10 -0600, Ram Gupta wrote:
> I am working to implement enforcing RSS limits of a process. I am
> planning to make a check for rss limit when setting up pte. If the
> limit is crossed I see couple of  different ways of handling .
> 
> 1. Kill the process . In this case there is no swapping problem.

Not good as the process isn't responsible for the RSS size so it would
be rather random.

> 2. Dont kill the process but dont allocate the memory & do yield as we
> do for init process. Modify the scheduler not to chose the process
> which has already allocated rss upto its limit. When rss usage
> fallsbelow its limit then the scheduler may chose it again to run.
> Here there is a scenario when no page of the process has been freed or
> swapped out because there were enough free pages? Then we need a way
> to reschedule the process by forcefully freeing some pages or need to
> kill the process.

That is what I would expect. Or perhaps even allowing the process to
exceed the RSS but using the RSS limit as a swapper target so that the
process is victimised early. No point forcing swapping and the RSS limit
when there is free memory, only where the resource is contended ..

Re: RSS Limit implementation issue

[Ram Gupta]

On 2/9/06, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

>
> That is what I would expect. Or perhaps even allowing the process to
> exceed the RSS but using the RSS limit as a swapper target so that the
> process is victimised early. No point forcing swapping and the RSS limit
> when there is free memory, only where the resource is contended ..
>
>

So we will need some kind of free memory threshold . If free memory is
more than it than we can let RSS exceed & scheduler can also schedule
it in this situation but not if free memory is less than the
threshhold. Also we need to figure out a way for swapper to target
pages based on RSS limit. One possible  disadvantage I can think is
that  as the swapper swaps out a page based on RSS limit , the
process's rss will become within the rss limit & then scheduler will
schedule this process again & hence possibly same page might have to
be brought in. This may cause increase in swapping. What do you think
how much realistic is this scenario?

Re: RSS Limit implementation issue

[Kyle Moffett]

On Feb 10, 2006, at 09:50, Ram Gupta wrote:
> On 2/9/06, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>> That is what I would expect. Or perhaps even allowing the process  
>> to exceed the RSS but using the RSS limit as a swapper target so  
>> that the process is victimised early. No point forcing swapping  
>> and the RSS limit when there is free memory, only where the  
>> resource is contended ..
>
> So we will need some kind of free memory threshold . If free memory  
> is more than it than we can let RSS exceed & scheduler can also  
> schedule it in this situation but not if free memory is less than  
> the threshhold. Also we need to figure out a way for swapper to  
> target pages based on RSS limit. One possible  disadvantage I can  
> think is that  as the swapper swaps out a page based on RSS limit ,  
> the process's rss will become within the rss limit & then scheduler  
> will schedule this process again & hence possibly same page might  
> have to be brought in. This may cause increase in swapping. What do  
> you think how much realistic is this scenario?

Just use a basic hysteresis device:

When allocating resources:
if (resource > limit + delta)
	disable_process();

When freeing resources:
if (resource < limit - delta)
	enable_process();

If the delta is set to something reasonable (say 1 or 2 pages), then  
the process will only be rescheduled when it gets enough free RSS  
(one page to satisfy its latest request and a few spare).  Even  
better, you could use a running average of "time between RSS- 
triggered-pauses" to figure out how much memory you should keep free.  
Pseudocode below:

[Tuneables]
unsigned int time_quantum_factor;
unsigned int limit;
unsigned int max_delta;

[Per-process state]
unsigned int pages;
unsigned int delta;
unsigned long long last_limit_time;

[When allocating resources]
if (pages > limit + delta) {
	int time_factor = log2(now - last_limit_time)
		- time_quantum_factor;
	last_limit_time = now;
	
	if (time_factor < 0 && delta < max_delta)
		delta <<= 1;
	else if (time_factor > 0 && delta > 1)
		delta >>= 1;
	
	put_process_to_sleep();
}

[When freeing resources]
if (resource < limit - delta)
	enable_process();

The effect of this code would be that the RSS code would avoid  
rescheduling a process more often than every 1<<time_quantum_factor  
microseconds.  It would attempt to provide a safe hysteresis delta  
such that the process would have enough pages free that it could  
probably run for at least the minimum amount of time.  Note that the  
code would _only_ have an effect if the process is already about to  
sleep on an RSS limit, otherwise that code path would never get hit.   
Obviously it's possible to adjust this algorithm to react more slowly  
or quickly by adjusting the shift values, but it should work well  
enough for a beta as-is.

Cheers,
Kyle Moffett

--
I didn't say it would work as a defense, just that they can spin that  
out for years in court if it came to it.
   -- Rob Landley

Re: RSS Limit implementation issue

[Rik van Riel]

On Fri, 10 Feb 2006, Ram Gupta wrote:

> Also we need to figure out a way for swapper to target pages based on 
> RSS limit.

Indeed.  You do not want an RSS limited process to get stuck
on an idle system, because nothing wants to free its memory.

> One possible disadvantage I can think is that as the swapper 
> swaps out a page based on RSS limit, the process's rss will become 
> within the rss limit & then scheduler will schedule this process again & 
> hence possibly same page might have to be brought in. This may cause 
> increase in swapping. What do you think how much realistic is this 
> scenario?

Thanks to the swap cache, this should not be an issue.

You don't need to actually write the page to disk when removing
the page from the process RSS - you simply add it to the swap
cache, unmap it and move it to the far end of the inactive list,
where kswapd will run into it quickly if the system needs memory
again in the future.

-- 
All Rights Reversed

Re: RSS Limit implementation issue

[Ram Gupta]

On 2/9/06, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> On Iau, 2006-02-09 at 15:10 -0600, Ram Gupta wrote:
> > I am working to implement enforcing RSS limits of a process. I am
> > planning to make a check for rss limit when setting up pte. If the
> > limit is crossed I see couple of  different ways of handling .
> >
> > 1. Kill the process . In this case there is no swapping problem.
>
> Not good as the process isn't responsible for the RSS size so it would
> be rather random.
>

I doubt I am missing some point here. I dont understand why the
process isn't responsible for RSS size. This limit is process specific
& the count of rss increases when the process maps some page in its
page table.

Thanks
Ram Gupta

Re: RSS Limit implementation issue

[Bill Davidsen]

Ram Gupta wrote:
> On 2/9/06, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>> On Iau, 2006-02-09 at 15:10 -0600, Ram Gupta wrote:
>>> I am working to implement enforcing RSS limits of a process. I am
>>> planning to make a check for rss limit when setting up pte. If the
>>> limit is crossed I see couple of  different ways of handling .
>>>
>>> 1. Kill the process . In this case there is no swapping problem.
>> Not good as the process isn't responsible for the RSS size so it would
>> be rather random.
>>
> 
> I doubt I am missing some point here. I dont understand why the
> process isn't responsible for RSS size. This limit is process specific
> & the count of rss increases when the process maps some page in its
> page table.

A process has no control over its RSS size, only its virtual size. I'm 
not sure you're clear on that, or just not saying it clearly. Therefore 
the same process, say a largish perl run, may be 175mB in vsize, and 
during the day have rss of perhaps half that. At night, with next to no 
load on the machine, the rss is 175mB because there is a bunch of free 
memory available.

If you want to make rss a hard limit the result should be swapping, not 
failure to run. I'm not sure the limit in that form is a good idea, and 
before someone reminds me, I do remember liking it better a few years ago.

If you can come up with a better way to adjust rss to get better overall 
greater throughput while being fair to all processes, go to it. But in 
general these things are a tradeoff, like swappiness, you tune until the 
volume of complaints reaches a minimum.

You could do tuning to get minimum page faults overall, or assure a 
minumum size, or... I think there's room for improvement, particularly 
for servers, but a hard limit doesn't seem to be it.

Didn't Rik do something in this area back in 2.4 and decide there 
weren't many fish in that pond?

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

RE: RSS Limit implementation issue

[Roger Heflin]

 

> 
> A process has no control over its RSS size, only its virtual 
> size. I'm not sure you're clear on that, or just not saying 
> it clearly. Therefore the same process, say a largish perl 
> run, may be 175mB in vsize, and during the day have rss of 
> perhaps half that. At night, with next to no load on the 
> machine, the rss is 175mB because there is a bunch of free 
> memory available.
> 
> If you want to make rss a hard limit the result should be 
> swapping, not failure to run. I'm not sure the limit in that 
> form is a good idea, and before someone reminds me, I do 
> remember liking it better a few years ago.

working_set_size limits sucked on VMS.  The OS would limit a process to
its working set size and casue the entire machine to swap
even though there was adequate free memory.   I believe they
had a normalworkingset size variable, and a maxworkingsetsize
one indicated how much ram you could get on a memory limited 
system, the other indicated the most it would ever let you get even if
there was plenty of free ram.   The maxworkingsetsize caused
a lot of issues, as the default appeared to be defined for
much smaller systems that we were using at the time, and so
were much too low, and cause unnecessary swapping.  Part of the
issue would be that the admin would need to know what he was
doing to use the feature, and most don't.

The argument from the admins at the time was that this limited
the damage to other processes by preventing certain processes
from getting too much memory, they ignored the fact that
anything swapping (even only the one process) unnecessarly 
*KILLED* performance for the entire machine, since swapping
is rather expensive on the os.

                       Roger

Re: RSS Limit implementation issue

[Bill Davidsen]

Roger Heflin wrote:
>  
> 
>> A process has no control over its RSS size, only its virtual 
>> size. I'm not sure you're clear on that, or just not saying 
>> it clearly. Therefore the same process, say a largish perl 
>> run, may be 175mB in vsize, and during the day have rss of 
>> perhaps half that. At night, with next to no load on the 
>> machine, the rss is 175mB because there is a bunch of free 
>> memory available.
>>
>> If you want to make rss a hard limit the result should be 
>> swapping, not failure to run. I'm not sure the limit in that 
>> form is a good idea, and before someone reminds me, I do 
>> remember liking it better a few years ago.
> 
> working_set_size limits sucked on VMS.  The OS would limit a process to
> its working set size and casue the entire machine to swap
> even though there was adequate free memory.   I believe they
> had a normalworkingset size variable, and a maxworkingsetsize
> one indicated how much ram you could get on a memory limited 
> system, the other indicated the most it would ever let you get even if
> there was plenty of free ram.   The maxworkingsetsize caused
> a lot of issues, as the default appeared to be defined for
> much smaller systems that we were using at the time, and so
> were much too low, and cause unnecessary swapping.  Part of the
> issue would be that the admin would need to know what he was
> doing to use the feature, and most don't.
> 
> The argument from the admins at the time was that this limited
> the damage to other processes by preventing certain processes
> from getting too much memory, they ignored the fact that
> anything swapping (even only the one process) unnecessarly 
> *KILLED* performance for the entire machine, since swapping
> is rather expensive on the os.

After thinking about this, I have the opinion that if an RSS limit is 
working it would be a hard limit. The alternative is a process which 
gets large and then when memory pressure increases the oversize process 
either causes a lot of swapping or worse yet ties up a lot of memory if 
swap rate is limited.

There are many ways to tune Linux badly, adding one more will not add 
much to the pain if the default is off. The values available to a normal 
users might be limited to prevent the most obvious bad choices. Or a 
corresponding option could be provided to take corrective action for a 
process with RSS set (to any value) and swap rate high.

RE: RSS Limit implementation issue

[Roger Heflin]

 

> After thinking about this, I have the opinion that if an RSS 
> limit is working it would be a hard limit. The alternative is 
> a process which gets large and then when memory pressure 
> increases the oversize process either causes a lot of 
> swapping or worse yet ties up a lot of memory if swap rate is limited.
> 
> There are many ways to tune Linux badly, adding one more will 
> not add much to the pain if the default is off. The values 
> available to a normal users might be limited to prevent the 
> most obvious bad choices. Or a corresponding option could be 
> provided to take corrective action for a process with RSS set 
> (to any value) and swap rate high.
>

I think the mistake on VMS was that the defaults were horribly
low and where not changed when machines got more memory, there
may be some situations where an RSS limit is wanted, but
the kernel would need to be implementing it, the process has no
control over its RSS.

I can see even having a min RSS that a process won't get swapped
below being probably more useful, as certain interactive process
on either a server or a desktop might want to not be swapped out below
a certain level, even if other processes suffer.   I would suspect
that I would set a min limit on certain applications that needed
decent response, and had got previously bad response to because 
of swapping, though something in the kernel would need to deal
with a potential overcommitting of memory via this method, and 
deal with the associated deadlock.

                                 Roger

Re: RSS Limit implementation issue

[kingsley]

On Tue, Apr 04, 2006 at 03:28:38PM -0400, Bill Davidsen wrote:
> Roger Heflin wrote:
> > 
> >
> >>A process has no control over its RSS size, only its virtual 
> >>size. I'm not sure you're clear on that, or just not saying 
> >>it clearly. Therefore the same process, say a largish perl 
> >>run, may be 175mB in vsize, and during the day have rss of 
> >>perhaps half that. At night, with next to no load on the 
> >>machine, the rss is 175mB because there is a bunch of free 
> >>memory available.
> >>
> >>If you want to make rss a hard limit the result should be 
> >>swapping, not failure to run. I'm not sure the limit in that 
> >>form is a good idea, and before someone reminds me, I do 
> >>remember liking it better a few years ago.
> >
> >working_set_size limits sucked on VMS.  The OS would limit a process to
> >its working set size and casue the entire machine to swap
> >even though there was adequate free memory.   I believe they
> >had a normalworkingset size variable, and a maxworkingsetsize
> >one indicated how much ram you could get on a memory limited 
> >system, the other indicated the most it would ever let you get even if
> >there was plenty of free ram.   The maxworkingsetsize caused
> >a lot of issues, as the default appeared to be defined for
> >much smaller systems that we were using at the time, and so
> >were much too low, and cause unnecessary swapping.  Part of the
> >issue would be that the admin would need to know what he was
> >doing to use the feature, and most don't.
> >
> >The argument from the admins at the time was that this limited
> >the damage to other processes by preventing certain processes
> >from getting too much memory, they ignored the fact that
> >anything swapping (even only the one process) unnecessarly 
> >*KILLED* performance for the entire machine, since swapping
> >is rather expensive on the os.

Apologies for just jumping in, but this is an area of interest to me.

I agree with the above point.  Though perhaps hard limits would be
acceptable if there was mechanism for capping the swapping caused by
hard limits to minimise the effect of swapping on the rest of the
system.

> After thinking about this, I have the opinion that if an RSS limit is 
> working it would be a hard limit. The alternative is a process which 
> gets large and then when memory pressure increases the oversize process 
> either causes a lot of swapping or worse yet ties up a lot of memory if 
> swap rate is limited.
> 
> There are many ways to tune Linux badly, adding one more will not add 
> much to the pain if the default is off. The values available to a normal 
> users might be limited to prevent the most obvious bad choices. Or a 
> corresponding option could be provided to take corrective action for a 
> process with RSS set (to any value) and swap rate high.

A while back, I had a patch for limits for the old 2.4 VM subsystem.
It implemented both hard and soft limits, with the hard limit
corresponding to rlim_max and the soft limit corresponding to
rlim_cur.  

The hard limit was straight forward, serving as an upper cap on the
total RSS of the task.  In contrast the scaled value of the task (soft
limit / RSS) served as an indicator as to the proportional usage of
each address space.  It would be used by VM subsystem to select (via a
bit-search and array look up as for the O(1) scheduler) a victim for
page replacement when memory usage was low.

Anyway, while I'm not for advocating hard limits, my point is that I
think solutions with both soft and hard limits are possible.  We have
two limits after all.  If any experimental work were done, perhaps it
could support both types of limits, permitting people to play with
both and to decide the merits of each.

-- 
		Kingsley

Re: RSS Limit implementation issue

[Peter Chubb]

>>>>> "Bill" == Bill Davidsen <davidsen@tmr.com> writes:

Bill> Ram Gupta wrote:

Bill> If you want to make rss a hard limit the result should be
Bill> swapping, not failure to run. I'm not sure the limit in that
Bill> form is a good idea, and before someone reminds me, I do
Bill> remember liking it better a few years ago.

Bill> If you can come up with a better way to adjust rss to get better
Bill> overall greater throughput while being fair to all processes, go
Bill> to it. But in general these things are a tradeoff, like
Bill> swappiness, you tune until the volume of complaints reaches a
Bill> minimum.

What I did in one experiment was to:
     1.  delay swapin requests if the process was over its rsslimit,
         until it fell below, and
     2.  Poke the swapper to try to swap out the current process's
         pages in that case.

The problem with the approach is that it behaved poorly under memory
pressure.  If a process's optimum working set was larger than its RSS
limit, then either it was delayed to the point of glaciality, or it
could saturate the swap device (and so disturb other processes's
operation). 

-- 
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
http://www.ertos.nicta.com.au           ERTOS within National ICT Australia

Re: RSS Limit implementation issue

[Bill Davidsen]

Peter Chubb wrote:

>>>>>>"Bill" == Bill Davidsen <davidsen@tmr.com> writes:
>>>>>>            
>>>>>>
>
>Bill> Ram Gupta wrote:
>
>Bill> If you want to make rss a hard limit the result should be
>Bill> swapping, not failure to run. I'm not sure the limit in that
>Bill> form is a good idea, and before someone reminds me, I do
>Bill> remember liking it better a few years ago.
>
>Bill> If you can come up with a better way to adjust rss to get better
>Bill> overall greater throughput while being fair to all processes, go
>Bill> to it. But in general these things are a tradeoff, like
>Bill> swappiness, you tune until the volume of complaints reaches a
>Bill> minimum.
>
>What I did in one experiment was to:
>     1.  delay swapin requests if the process was over its rsslimit,
>         until it fell below, and
>     2.  Poke the swapper to try to swap out the current process's
>         pages in that case.
>
>The problem with the approach is that it behaved poorly under memory
>pressure.  If a process's optimum working set was larger than its RSS
>limit, then either it was delayed to the point of glaciality, or it
>could saturate the swap device (and so disturb other processes's
>operation). 
>
>  
>
I'm paying close attention, but that's kind of the problem people have, 
memory pressure gets high and the processes don't run. I thought of 
"swap out two to swap in one" as a way to dribble the rss down, but I 
doubt it's a magic solution. Swap kills, so does not swap and no memory. 
Obvious solution is to make the rss limit hard and keep it small, I 
don't think that's the answer.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: RSS Limit implementation issue

[Bernd Eckenfels]

Ram Gupta <ram.gupta5@gmail.com> wrote:
> planning to make a check for rss limit when setting up pte. If the
> limit is crossed I see couple of  different ways of handling .
> 
> 1. Kill the process . In this case there is no swapping problem.

This signal would happen on random page access, right?

> 2. Dont kill the process but dont allocate the memory & do yield as we
> do for init process. Modify the scheduler not to chose the process
> which has already allocated rss upto its limit.

Yes, that behaviour looks good. That would keep the system responsive.
Basically the same state as waiting for a page to be paged in.

(However a user based rss limit would be more interesting than process
based)

Gruss
Bernd

Re: RSS Limit implementation issue

[Bill Davidsen]

Ram Gupta wrote:
> I am working to implement enforcing RSS limits of a process. I am
> planning to make a check for rss limit when setting up pte. If the
> limit is crossed I see couple of  different ways of handling .
> 
> 1. Kill the process . In this case there is no swapping problem.

Since the process has little or no control over that it seems 
impractical. And works the wrong way, when there is a ton of free memory 
the process would get a large rss and be killed, while on a loaded 
system it would run.
> 
> 2. Dont kill the process but dont allocate the memory & do yield as we
> do for init process. Modify the scheduler not to chose the process
> which has already allocated rss upto its limit. When rss usage
> fallsbelow its limit then the scheduler may chose it again to run.
> Here there is a scenario when no page of the process has been freed or
> swapped out because there were enough free pages? Then we need a way
> to reschedule the process by forcefully freeing some pages or need to
> kill the process.
> 
> I am looking forward for your comments & pros/cons of both approach &
> any other alternatives you might come up with.

First, someone did some work on this a few years ago, you might be able 
to find info looking a rmap posts for the mid 2.4 days.

Second, I think this limitation needs to be enforced only when free 
memory is below some trigger point, when candidates for reclaim would be 
drawn from processes over their rss target.

Finally, it would be good to be aggressive about cleaning dirty pages of 
a process of the target, so pages clould be reclaimed quickly. There are 
a lot of factors in that, useless disk activity being one possible side 
effect.


-- 
bill davidsen <davidsen@tmr.com>
   CTO TMR Associates, Inc
   Doing interesting things with small computers since 1979

Re: RSS Limit implementation issue

[Ram Gupta]

On 2/11/06, Chris Siebenmann <cks@utcc.utoronto.ca> wrote:

>  I suggest a third method: steal another page from the process itself.
> This automatically keeps the process within its own RSS, slows down its
> activities, *and* lets it keep running.
>
>  Under the name 'paging against itself', I believe this has probably
> already been done by various people in the early 1990s.

This method may cause increased amount of swapping affecting the
overall system performance.

Thanks
Ram Gupta

Re: RSS Limit implementation issue

[Ram Gupta]

On 2/13/06, Chris Siebenmann <cks@utcc.utoronto.ca> wrote:

>
>  I believe that this is the inevitable result of anything that doesn't
> kill the process on the spot. When you put the process to sleep, you
> effectively reduce its RSS by reducing its page-touching activity; if
> the system is under memory pressure, other things will then steal pages
> from it anyways.
>
True  but with your approach swapping occurs right away. Objective
here is to reduce the chances of swapping as much as possible.

Regards
Ram Gupta

Re: RSS Limit implementation issue

[Bodo Eggert]

Ram Gupta <ram.gupta5@gmail.com> wrote:
> On 2/9/06, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>> On Iau, 2006-02-09 at 15:10 -0600, Ram Gupta wrote:

>> > I am working to implement enforcing RSS limits of a process. I am
>> > planning to make a check for rss limit when setting up pte. If the
>> > limit is crossed I see couple of  different ways of handling .
>> >
>> > 1. Kill the process . In this case there is no swapping problem.
>>
>> Not good as the process isn't responsible for the RSS size so it would
>> be rather random.
>>
> 
> I doubt I am missing some point here. I dont understand why the
> process isn't responsible for RSS size. This limit is process specific
> & the count of rss increases when the process maps some page in its
> page table.

It can't be responsible because the kernel is controlling the RSS size e.g.
by prefetching or swapping out. The process has very little means of
influencing these mechanisms, nor is there a way to actively shrink the RSS.

(It's perfectly legal to e.g. mmap a large file (1 GB) and memcopy that area
into another mmaped file, and the kernel is expected to keep sane even on a
16 MB machine.)

You may introduce a mechanism that allows a process to keep it's RSS < n,
but you'll have to deal with legacy processes at least for the next 20 years.

-- 
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF
verbreiteten Lügen zu sabotieren.