Re: OOM at low page cache?

Re: OOM at low page cache?

[Vlastimil Babka]

CC linux-mm in case somebody has a good answer but missed this in lkml traffic

On 01/23/2015 11:18 PM, John Moser wrote:
> Why is there no tunable to OOM at low page cache?
> 
> I have no swap configured.  I have 16GB RAM.  If Chrome or Gimp or some
> other stupid program goes off the deep end and eats up my RAM, I hit
> some 15.5GB or 15.75GB usage and stay there for about 40 minutes.  Every
> time the program tries to do something to eat more RAM, it cranks disk
> hard; the disk starts thrashing, the mouse pointer stops moving, and
> nothing goes on.  It's like swapping like crazy, except you're reading
> library files instead of paged anonymous RAM.
> 
> If only I could tell the system to OOM kill at 512MB or 1GB or 95%
> non-evictable RAM, it would recover on its own.  As-is, I need to wait
> or trigger the OOM killer by sysrq.
> 
> Am I just the only person in the world who's ever had that problem?  Or
> is it a matter of questions fast popping up when you try to do this
> *and* enable paging to disk?  (In my experience, that's a matter of too
> much swap space:  if you have 16GB RAM and your computer dies at 15.25GB
> usage, your swap space should be no larger than 750MB plus inactive
> working RAM; obviously, your computer can't handle paging 750MB back and
> forth.  If you make it 8GB wide and you start swap thrashing at 2GB
> usage, you have too much swap available).
> 
> I guess you could try to detect excessive swap and page cache thrashing,
> but that's complex; if anyone really wanted to do that, it would be done
> by now.  A low-barrier OOM is much simpler.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: OOM at low page cache?

[Minchan Kim]

Hello,

On Tue, Jan 27, 2015 at 12:03:34PM +0100, Vlastimil Babka wrote:
> CC linux-mm in case somebody has a good answer but missed this in lkml traffic
> 
> On 01/23/2015 11:18 PM, John Moser wrote:
> > Why is there no tunable to OOM at low page cache?

AFAIR, there were several trial although there wasn't acceptable
at that time. One thing I can remember is min_filelist_kbytes.
FYI, http://lwn.net/Articles/412313/

> > 
> > I have no swap configured.  I have 16GB RAM.  If Chrome or Gimp or some
> > other stupid program goes off the deep end and eats up my RAM, I hit
> > some 15.5GB or 15.75GB usage and stay there for about 40 minutes.  Every
> > time the program tries to do something to eat more RAM, it cranks disk
> > hard; the disk starts thrashing, the mouse pointer stops moving, and
> > nothing goes on.  It's like swapping like crazy, except you're reading
> > library files instead of paged anonymous RAM.
> > 
> > If only I could tell the system to OOM kill at 512MB or 1GB or 95%
> > non-evictable RAM, it would recover on its own.  As-is, I need to wait
> > or trigger the OOM killer by sysrq.
> > 
> > Am I just the only person in the world who's ever had that problem?  Or
> > is it a matter of questions fast popping up when you try to do this
> > *and* enable paging to disk?  (In my experience, that's a matter of too
> > much swap space:  if you have 16GB RAM and your computer dies at 15.25GB
> > usage, your swap space should be no larger than 750MB plus inactive
> > working RAM; obviously, your computer can't handle paging 750MB back and
> > forth.  If you make it 8GB wide and you start swap thrashing at 2GB
> > usage, you have too much swap available).
> > 
> > I guess you could try to detect excessive swap and page cache thrashing,
> > but that's complex; if anyone really wanted to do that, it would be done
> > by now.  A low-barrier OOM is much simpler.

I'm far away from reclaim code for a long time but when I read again,
I found something strange.

With having swap in get_scan_count, we keep a mount of file LRU + free
as above than high wmark to prevent file LRU thrashing but we don't
with no swap. Why?

Anyway, I believe we should fix it and we now have workingset.c so
there might be more ways to be smart than old(although I am concern
about that shadow shrinker blows out lots of information to be useful
to detect in heavy memory pressure like page thrashing)

Below could be band-aid until we find a elegant solution?

Re: OOM at low page cache?

[Rik van Riel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/28/2015 01:26 AM, Minchan Kim wrote:
> Hello,
> 
> On Tue, Jan 27, 2015 at 12:03:34PM +0100, Vlastimil Babka wrote:
>> CC linux-mm in case somebody has a good answer but missed this in
>> lkml traffic
>> 
>> On 01/23/2015 11:18 PM, John Moser wrote:
>>> Why is there no tunable to OOM at low page cache?
> 
> AFAIR, there were several trial although there wasn't acceptable at
> that time. One thing I can remember is min_filelist_kbytes. FYI,
> http://lwn.net/Articles/412313/

The Android low memory killer does exactly what you want, and
for very much the same reasons.

See drivers/staging/android/lowmemorykiller.c

However, in the mainline kernel I think it does make sense to
apply something like the patch that Minchan cooked up with, to OOM
if freeing all the page cache could not bring us back up to the high
watermark, across all the memory zones.

- -- 
All rights reversed
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJUyNewAAoJEM553pKExN6DlKQH/3PprrXF7IOyjiXnO+2Qqbau
wgWXO7mQWGFi+zNqSUzmWtfTCFVx6BxLi23MCQG1RqKGnQI4DehdOKMDidFwoC8D
2grKe9ELp04mEbyG0aipdxSw6FouIDFhC2FzmU7oQDZX5RKmLuxY7QPU4NTCitcR
xHp6jWrvyY2CDiSpA2QSAaAAIG21BtPJvXQg3WvY/LI03N1edqZnExt5Po8CY7oe
EeiO7ZtYISl/wRIoribEafZF4rMAfJ5A36kdbulqCqVtgCWEDPV0RCXimc5EtDIt
bFDiv924+YMiuEFULJlEqLGqTJOtfJ+NlBIn8nVRk5P1pOGEbO05zE+XV1Vea6k=
=8351
-----END PGP SIGNATURE-----

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: OOM at low page cache?

[John Moser]

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

On 01/28/2015 07:36 AM, Rik van Riel wrote:
> On 01/28/2015 01:26 AM, Minchan Kim wrote:
> > Hello,
>
> > On Tue, Jan 27, 2015 at 12:03:34PM +0100, Vlastimil Babka wrote:
> >> CC linux-mm in case somebody has a good answer but missed this in
> >> lkml traffic
> >>
> >> On 01/23/2015 11:18 PM, John Moser wrote:
> >>> Why is there no tunable to OOM at low page cache?
>
> > AFAIR, there were several trial although there wasn't acceptable at
> > that time. One thing I can remember is min_filelist_kbytes. FYI,
> > http://lwn.net/Articles/412313/
>
> The Android low memory killer does exactly what you want, and
> for very much the same reasons.
>
> See drivers/staging/android/lowmemorykiller.c
>

Haven't seen that; it's been a long time since I bothered myself with
kernel code, so I'm out-of-touch.

Wow lots of good responses coming quick.
> However, in the mainline kernel I think it does make sense to
> apply something like the patch that Minchan cooked up with, to OOM
> if freeing all the page cache could not bring us back up to the high
> watermark, across all the memory zones.
>
>



[-- Attachment #2: Type: text/html, Size: 1804 bytes --]

Re: OOM at low page cache?

[John Moser]

On 01/28/2015 01:26 AM, Minchan Kim wrote:
> Hello,
>
> On Tue, Jan 27, 2015 at 12:03:34PM +0100, Vlastimil Babka wrote:
>> CC linux-mm in case somebody has a good answer but missed this in lkml traffic
>>
>> On 01/23/2015 11:18 PM, John Moser wrote:
>>> Why is there no tunable to OOM at low page cache?
> AFAIR, there were several trial although there wasn't acceptable
> at that time. One thing I can remember is min_filelist_kbytes.
> FYI, http://lwn.net/Articles/412313/
>

That looks more straight-forward than http://lwn.net/Articles/422291/


> I'm far away from reclaim code for a long time but when I read again,
> I found something strange.
>
> With having swap in get_scan_count, we keep a mount of file LRU + free
> as above than high wmark to prevent file LRU thrashing but we don't
> with no swap. Why?
>

That's ... strange.  That means having a token 1MB swap file changes the
system's practical memory reclaim behavior dramatically?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: OOM at low page cache?

[Minchan Kim]

Hello,

On Wed, Jan 28, 2015 at 09:15:50AM -0500, John Moser wrote:
> On 01/28/2015 01:26 AM, Minchan Kim wrote:
> > Hello,
> >
> > On Tue, Jan 27, 2015 at 12:03:34PM +0100, Vlastimil Babka wrote:
> >> CC linux-mm in case somebody has a good answer but missed this in lkml traffic
> >>
> >> On 01/23/2015 11:18 PM, John Moser wrote:
> >>> Why is there no tunable to OOM at low page cache?
> > AFAIR, there were several trial although there wasn't acceptable
> > at that time. One thing I can remember is min_filelist_kbytes.
> > FYI, http://lwn.net/Articles/412313/
> >
> 
> That looks more straight-forward than http://lwn.net/Articles/422291/
> 
> 
> > I'm far away from reclaim code for a long time but when I read again,
> > I found something strange.
> >
> > With having swap in get_scan_count, we keep a mount of file LRU + free
> > as above than high wmark to prevent file LRU thrashing but we don't
> > with no swap. Why?
> >
> 
> That's ... strange.  That means having a token 1MB swap file changes the
> system's practical memory reclaim behavior dramatically?

Basically, yes but 1M is too small. If all of swap consumed, the behavior
will be same so I think we need more explicit logic to prevent cache
thrashing. Could you test below patch?

Thanks.

Re: OOM at low page cache?

[John Moser]

On 01/28/2015 01:26 AM, Minchan Kim wrote:
>
> Below could be band-aid until we find a elegant solution?
>
>

I don't know about elegant; but I'd be impressed if anyone figured out
how to just go Windows 95 with it and build a Task Master interface.  It
would be useful to have a kernel interface that allows a service to
attach, delegate an interface program, etc., and then pull it up under
certain conditions (low memory, heavy scheduling due to lots of
fork()ing, etc.) and assign temporary high priority.  Basically,
nearly-pause the system and allow the user to select and kill/term
processes, or bring a process forward (for like 10 seconds, then kick it
back again) so the user can save their work and exit gracefully.  At
hard OOM, you could either OOM or pause everything (you'd need a
zero-allocation path to kill things in a user-end OOM handler).

Yeah, imaginative fantasies.  Totally doable, but probably too complex
to bother.  There's all kinds of semaphore inversion or some such to
worry about; how do you ensure an X11 program is 100% snappy when the
system is being thrashed by fork() bombs and memory pressure?

Actually, I have no idea what I'm talking about.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>