Kswapd flaw

Kswapd flaw

[Al Boldi]

In 2.4.31 kswapd starts paging during OOMs even w/o swap enabled.

Is there a way to fix/disable this behaviour?

Thanks!

Re: Kswapd flaw

[Marcelo Tosatti]

On Mon, Jun 27, 2005 at 11:04:08PM +0300, Al Boldi wrote:
> 
> In 2.4.31 kswapd starts paging during OOMs even w/o swap enabled.
> 
> Is there a way to fix/disable this behaviour?

Al,

What do you mean by paging? Can you be more detailed on the problem 
description please?

RE: Kswapd flaw

[Al Boldi]

On Mon, Jun 27, 2005 at 11:04:08PM +0300, Al Boldi wrote:
> 
> In 2.4.31 kswapd starts paging during OOMs even w/o swap enabled.
> 
> Is there a way to fix/disable this behaviour?

Marcelo,

Kswapd starts evicting processes to fullfil a malloc, when it should just
deny it because there is no swap.

Re: Kswapd flaw

[Michal Schmidt]

Al Boldi wrote:
> Kswapd starts evicting processes to fullfil a malloc, when it should just
> deny it because there is no swap.

The kernel can always discard pages of executables and read-only mapped 
files, because it can load them back if necessary. If you don't like 
this, call mlockall() in your program.

Michal

RE: Kswapd flaw

[Al Boldi]

Michal Schmidt wrote:
> The kernel can always discard pages of executables and read-only mapped
files,
> because it can load them back if necessary.
> If you don't like this, call mlockall() in your program.

Thanks for the pointer, but what if I don't own the source.
Is there a way to tell kswapd to disable this feature?

Re: Kswapd flaw

[Michal Schmidt]

Al Boldi wrote:
> Michal Schmidt wrote:
>>If you don't like this, call mlockall() in your program.
> 
> 
> Thanks for the pointer, but what if I don't own the source.

An LD_PRELOAD trick might work.

Michal

Re: Kswapd flaw

[Nix]

On 28 Jun 2005, Al Boldi murmured woefully:
> Kswapd starts evicting processes to fullfil a malloc, when it should just
> deny it because there is no swap.

This is how the kernel has always worked. If you have no swap, memory
pressure is higher than it otherwise might be, because dirty pages must
be kept in physical RAM even if they are rarely used. So when memory gets
low the kernel *has* to evict pages which aren't dirty, and write out and
flush dirty pages corresponding to files on disk.

What else could it possibly do? Stop evicting everything?

I can't even tell what you're expecting. Surely not that no pages are
ever evicted or flushed; your memory would fill up with page cache in
no time.

-- 
`I lost interest in "blade servers" when I found they didn't throw knives
 at people who weren't supposed to be in your machine room.'
    --- Anthony de Boer

RE: Kswapd flaw

[Al Boldi]

On 28 Jun 2005, Al Boldi murmured woefully:
> Kswapd starts evicting processes to fullfil a malloc, when it should 
> just deny it because there is no swap.
Nix wrote:
> I can't even tell what you're expecting. Surely not that no pages are ever
evicted or flushed;
> your memory would fill up with page cache in no time.

Nix,
Please do flush anytime, and do it in sync during OOMs; but don't evict
procs especially not RUNNING procs, that is overkill.

Re: Kswapd flaw

[Nix]

On 28 Jun 2005, Al Boldi yowled:
> Nix wrote:
>> On 28 Jun 2005, Al Boldi murmured woefully:
>>> Kswapd starts evicting processes to fullfil a malloc, when it should 
>>> just deny it because there is no swap.
>> I can't even tell what you're expecting. Surely not that no pages are ever
>> evicted or flushed; your memory would fill up with page cache in no time.
> 
> Please do flush anytime, and do it in sync during OOMs; but don't evict
> procs especially not RUNNING procs, that is overkill.

But processes (really, mapped text pages; really, read-only mapped pages
of all kinds) are loaded piecemeal in any case. Would you really like a
system where once something was faulted in, it could never leave? You'd
run out of memory *awfully* fast.

A system in which pages can be faulted in *and* out is consistent: one
in which they can only be faulted in is both inconsistent and very
deadlock-prone.

-- 
`I lost interest in "blade servers" when I found they didn't throw knives
 at people who weren't supposed to be in your machine room.'
    --- Anthony de Boer

RE: Kswapd flaw

[Al Boldi]

Hi Nix, how are you?
You wrote: {
On 28 Jun 2005, Al Boldi yowled:
> Nix wrote:
>> On 28 Jun 2005, Al Boldi murmured woefully:
>>> Kswapd starts evicting processes to fullfil a malloc, when it should 
>>> just deny it because there is no swap.
>> I can't even tell what you're expecting. Surely not that no pages are 
>> ever evicted or flushed; your memory would fill up with page cache in no
time.
> 
> Please do flush anytime, and do it in sync during OOMs; but don't 
> evict procs especially not RUNNING procs, that is overkill.

Would you really like a system where once something was faulted in, it could
never leave? You'd run out of memory *awfully* fast.
}

Nix,
You should only fault if you have a place to fault to, as into a swap.
Without swap faulting is overkill.

Is it possible to change kswapd's default behaviour to not fault if there is
no swap?

Thanks!

Re: Kswapd flaw

[Nix]

On Tue, 28 Jun 2005, Al Boldi wrote:
> Hi Nix, how are you?
> You wrote: {
> On 28 Jun 2005, Al Boldi yowled:
>> Please do flush anytime, and do it in sync during OOMs; but don't 
>> evict procs especially not RUNNING procs, that is overkill.
> 
> Would you really like a system where once something was faulted in, it could
> never leave? You'd run out of memory *awfully* fast.
> }
> 
> You should only fault if you have a place to fault to, as into a swap.
> Without swap faulting is overkill.

That's `swapping', i.e. writing out dirty data pages (and text pages
dirtied by e.g. relocation) which otherwise have nowhere to go.

Pages mapped from files with read-only mappings and non-dirty pages of
read-write mappings get discarded when memory is tight; pages mapped
with read-write non-private maps and which have been modified get
written back to their file and discarded in the same situation.

(Private modified read-write mappings of data files have to stay in
memory: the only place they could go is swap, and there is none.)

> Is it possible to change kswapd's default behaviour to not fault if
> there is no swap?

I don't think so, except on a process-by-process basis via mlockall().
(/proc/sys/vm/swappiness lets you say that swapping is more or less
desirable, but under enough memory pressure paging *will* happen
regardless of the value of that variable.)

You can kludge it to some extent by putting your binaries in a tmpfs and
running from there: then you'll be paging from memory to memory ;)

But I'm mystified as to why you might want to suppress paging. The only
effect of suppressing it is to reduce the amount of memory you can
allocate before you run completely out and start killing things. Surely
slow execution is generally preferable to *no* execution?

(Or is it that you're trying to suppress *disk activity*, perhaps to
keep power consumption down? If so, running from tmpfs seems the best
option.)

-- 
`I lost interest in "blade servers" when I found they didn't throw knives
 at people who weren't supposed to be in your machine room.'
    --- Anthony de Boer

RE: Kswapd flaw

[Al Boldi]

Nix wrote:{
On Tue, 28 Jun 2005, Al Boldi wrote:
> Hi Nix, how are you?
> You wrote: {
> On 28 Jun 2005, Al Boldi yowled:
>> Please do flush anytime, and do it in sync during OOMs; but don't 
>> evict procs especially not RUNNING procs, that is overkill.
> 
> Would you really like a system where once something was faulted in, it 
> could never leave? You'd run out of memory *awfully* fast.
> }
> 
> You should only fault if you have a place to fault to, as into a swap.
> Without swap faulting is overkill.
>
> Is it possible to change kswapd's default behaviour to not fault if 
> there is no swap?

I don't think so, except on a process-by-process basis via mlockall().
(/proc/sys/vm/swappiness lets you say that swapping is more or less
desirable, but under enough memory pressure paging *will* happen regardless
of the value of that variable.)

But I'm mystified as to why you might want to suppress paging. The only
effect of suppressing it is to reduce the amount of memory you can allocate
before you run completely out and start killing things.
}

Nix,
You start killing things because you overcommitted, when you were not
supposed to fault in the first place because you have no swap.

( I couldn't find /proc/sys/vm/swappiness in 2.4, although I heard about it
in 2.6.)

Re: Kswapd flaw

[Nix]

On Tue, 28 Jun 2005, Al Boldi murmured woefully:
> Nix wrote:{
> I don't think so, except on a process-by-process basis via mlockall().
> (/proc/sys/vm/swappiness lets you say that swapping is more or less
> desirable, but under enough memory pressure paging *will* happen regardless
> of the value of that variable.)
> 
> But I'm mystified as to why you might want to suppress paging. The only
> effect of suppressing it is to reduce the amount of memory you can allocate
> before you run completely out and start killing things.
> }
> 
> Nix,
> You start killing things because you overcommitted, when you were not
> supposed to fault in the first place because you have no swap.

The only thing that is blocked by having no swap is swapping. The only
things that go to swap are things that can't go anywhere else (because
swap is comparatievly slow). As far as I know, the only things that land
in swap are modified private writable file-backed mappings
(e.g. relocations in text sections of programs), and allocated anonymous
mappings (what most people think of as `data'). (Data allocated via
brk() is of the latter type.)

So if swap is turned off, the kernel can still acquire more memory when
short by discarding file-backed pages from the page cache --- this
includes non-modified text pages of programs and modified non-private
file mappings, as well as data read/written via read() and write().

That is, with no swapfile, the kernel can't swap. It can still *page*,
pushing stuff out to files if modified, throwing them out of memory if
not, and reloading them from their original file when needed.

> ( I couldn't find /proc/sys/vm/swappiness in 2.4, although I heard about it
> in 2.6.)

Yeah, it's a 2.6 thing.

-- 
`I lost interest in "blade servers" when I found they didn't throw knives
 at people who weren't supposed to be in your machine room.'
    --- Anthony de Boer

Re: Kswapd flaw

[Helge Hafting]

On Tue, Jun 28, 2005 at 02:47:15PM +0300, Al Boldi wrote:
> On 28 Jun 2005, Al Boldi murmured woefully:
> > Kswapd starts evicting processes to fullfil a malloc, when it should 
> > just deny it because there is no swap.
> Nix wrote:
> > I can't even tell what you're expecting. Surely not that no pages are ever
> evicted or flushed;
> > your memory would fill up with page cache in no time.
> 
> Nix,
> Please do flush anytime, and do it in sync during OOMs; but don't evict
> procs especially not RUNNING procs, that is overkill.

Something running rarely gets thrown out - It is usually only pages
that see little use that gets evicted.  Such as startup code.

Of course you can improve the situation a lot by adding swap - the kernel
will then be free to swap out dirty but little-used pages instead
of not-dirty but more used executable mappings. That will improve
your performance.

Getting more memory or running fewer programs also helps, obviously.

Helge Hafting

Re: Kswapd flaw

[Paulo Marques]

Al Boldi wrote:
> On Mon, Jun 27, 2005 at 11:04:08PM +0300, Al Boldi wrote:
> 
>>In 2.4.31 kswapd starts paging during OOMs even w/o swap enabled.
>>
>>Is there a way to fix/disable this behaviour?
> 
> 
> Marcelo,
> 
> Kswapd starts evicting processes to fullfil a malloc, when it should just
> deny it because there is no swap.

I think what you really want is to adjust your "overcommit" settings.

See: "Documentation/vm/overcommit-accounting"

-- 
Paulo Marques - www.grupopie.com

It is a mistake to think you can solve any major problems
just with potatoes.
Douglas Adams

Re: Kswapd flaw

[Marcelo Tosatti]

On Tue, Jun 28, 2005 at 03:55:29PM +0100, Paulo Marques wrote:
> Al Boldi wrote:
> >On Mon, Jun 27, 2005 at 11:04:08PM +0300, Al Boldi wrote:
> >
> >>In 2.4.31 kswapd starts paging during OOMs even w/o swap enabled.
> >>
> >>Is there a way to fix/disable this behaviour?
> >
> >
> >Marcelo,
> >
> >Kswapd starts evicting processes to fullfil a malloc, when it should just
> >deny it because there is no swap.
> 
> I think what you really want is to adjust your "overcommit" settings.
> 
> See: "Documentation/vm/overcommit-accounting"

Al, 

You should set overcommit accordingly as Paulo mentions.

You might also want to disable the OOM killer (CONFIG_OOM_KILLER).

RE: Kswapd flaw

[Al Boldi]

Marcelo wrote: {
On Tue, Jun 28, 2005 at 03:55:29PM +0100, Paulo Marques wrote:
> Al Boldi wrote:
> >On Mon, Jun 27, 2005 at 11:04:08PM +0300, Al Boldi wrote:
> >
> >>In 2.4.31 kswapd starts paging during OOMs even w/o swap enabled.
> >>
> >>Is there a way to fix/disable this behaviour?
> >
> >
> >Marcelo,
> >
> >Kswapd starts evicting processes to fullfil a malloc, when it should 
> >just deny it because there is no swap.
> 
> I think what you really want is to adjust your "overcommit" settings.
> 
> See: "Documentation/vm/overcommit-accounting"

You might also want to disable the OOM killer (CONFIG_OOM_KILLER).
}

Paulo,
Thanks for the pointer!  Overcommit is the problem, and kswapd is not
honoring it.
Can this be fixed/adjusted?

Marcelo,
Disable OOM killer? Is that an option in 2.4 or 2.6?

Thanks!

Re: Kswapd flaw

[Robert Hancock]

Al Boldi wrote:
> Nix,
> You should only fault if you have a place to fault to, as into a swap.
> Without swap faulting is overkill.
> 
> Is it possible to change kswapd's default behaviour to not fault if there is
> no swap?

See Michal Schmidt's posting. It CAN fault if there is no swap.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

RE: Kswapd flaw

[Al Boldi]

Marcelo wrote: {
On Tue, Jun 28, 2005 at 11:00:14PM +0300, Al Boldi wrote:
> Hi,
> I read Documentation/vm/overcommit-accounting, thank you!
> It's for 2.6. Values are 0,1,2. What is needed is 2.
> In 2.4 values are 0,1. What is needed is 0, which is default, which is 
> the problem.
> Kswapd overcommits anyway.
> Can this be fixed/adjusted?

Can you please try this

--- linux-2.4.30/mm/mmap.c.orig 2005-06-28 17:15:27.000000000 -0300
+++ linux-2.4.30/mm/mmap.c      2005-06-28 17:15:29.000000000 -0300
@@ -70,7 +70,7 @@
            return 1;

        /* The page cache contains buffer pages these days.. */
-       free = page_cache_size;
+//     free = page_cache_size;
        free += nr_free_pages();
        free += nr_swap_pages;

> CONFIG_OOM_KILLER does not disable/enable OOM-Killer.
> It merely selects a different style of killing.

What you mean? What happens if you disable CONFIG_OOM_KILLER?
}

1. Thanks for the patch pointer. Can someone work on this?

2 .You only need OOM-Killer if you overcommit.