Re: [Bug 107771] New: Single process tries to use more than 1/2 physical RAM, OS starts thrashing

Re: [Bug 107771] New: Single process tries to use more than 1/2 physical RAM, OS starts thrashing

[Andrew Morton]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

This is ... interesting.

On Thu, 12 Nov 2015 18:46:35 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=107771
> 
>             Bug ID: 107771
>            Summary: Single process tries to use more than 1/2 physical
>                     RAM, OS starts thrashing
>            Product: Memory Management
>            Version: 2.5
>     Kernel Version: 4.3.0-040300-generic (Ubuntu)
>           Hardware: All
>                 OS: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Page Allocator
>           Assignee: akpm@linux-foundation.org
>           Reporter: theosib@gmail.com
>         Regression: No
> 
> I have a 24-core (48 thread) system with 64GB of RAM.  
> 
> When I run multiple processes, I can use all of physical RAM before swapping
> starts.  However, if I'm running only a *single* process, the system will start
> swapping after I've exceeded only 1/2 of available physical RAM.  Only after
> swap fills does it start using more of the physical RAM.  
> 
> I can't find any ulimit settings or anything else that would cause this to
> happen intentionally. 
> 
> I had originally filed this against Ubuntu, but I'm now running a more recent
> kernel, and the problem persists, so I think it's more appropriate to file
> here.  There are some logs that they had me collect, so if you want to see
> them, the are here:
> 
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1513673
> 
> I don't recall this problem happening with older kernels (whatever came with
> Ubuntu 15.04), although I may just not have noticed.  By swapping early, I'm
> limited by the speed of my SSD, which is moving only about 20MB/sec in each
> direction, and that makes what I'm running take 10 times as long to complete.
> 
> -- 
> You are receiving this mail because:
> You are the assignee for the bug.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 107771] New: Single process tries to use more than 1/2 physical RAM, OS starts thrashing

[Rik van Riel]

[-- Attachment #1: Type: text/plain, Size: 2608 bytes --]

On Fri, 2016-04-15 at 12:15 -0700, Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via
> the
> bugzilla web interface).
> 
> This is ... interesting.

First things first. What is the value of
/proc/sys/vm/zone_reclaim?

I am assuming this is a two socket system,
with two 12-core CPUs. Am I right?

> On Thu, 12 Nov 2015 18:46:35 +0000 bugzilla-
> daemon@bugzilla.kernel.org wrote:
> 
> > 
> > https://bugzilla.kernel.org/show_bug.cgi?id=107771
> > 
> >             Bug ID: 107771
> >            Summary: Single process tries to use more than 1/2
> > physical
> >                     RAM, OS starts thrashing
> >            Product: Memory Management
> >            Version: 2.5
> >     Kernel Version: 4.3.0-040300-generic (Ubuntu)
> >           Hardware: All
> >                 OS: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: Page Allocator
> >           Assignee: akpm@linux-foundation.org
> >           Reporter: theosib@gmail.com
> >         Regression: No
> > 
> > I have a 24-core (48 thread) system with 64GB of RAM.  
> > 
> > When I run multiple processes, I can use all of physical RAM before
> > swapping
> > starts.  However, if I'm running only a *single* process, the
> > system will start
> > swapping after I've exceeded only 1/2 of available physical
> > RAM.  Only after
> > swap fills does it start using more of the physical RAM.  
> > 
> > I can't find any ulimit settings or anything else that would cause
> > this to
> > happen intentionally. 
> > 
> > I had originally filed this against Ubuntu, but I'm now running a
> > more recent
> > kernel, and the problem persists, so I think it's more appropriate
> > to file
> > here.  There are some logs that they had me collect, so if you want
> > to see
> > them, the are here:
> > 
> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1513673
> > 
> > I don't recall this problem happening with older kernels (whatever
> > came with
> > Ubuntu 15.04), although I may just not have noticed.  By swapping
> > early, I'm
> > limited by the speed of my SSD, which is moving only about 20MB/sec
> > in each
> > direction, and that makes what I'm running take 10 times as long to
> > complete.
> > 
-- 
All Rights Reversed.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Bug 107771] New: Single process tries to use more than 1/2 physical RAM, OS starts thrashing

[Michal Hocko]

On Fri 15-04-16 15:31:22, Rik van Riel wrote:
> On Fri, 2016-04-15 at 12:15 -0700, Andrew Morton wrote:
> > (switched to email.  Please respond via emailed reply-to-all, not via
> > the
> > bugzilla web interface).
> > 
> > This is ... interesting.
> 
> First things first. What is the value of
> /proc/sys/vm/zone_reclaim?

Also snapshots of /proc/vmstat taken every 1s or so while you see the
trashing would be helpful.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 107771] New: Single process tries to use more than 1/2 physical RAM, OS starts thrashing

[Timothy Normand Miller]

On Fri, Apr 15, 2016 at 3:31 PM, Rik van Riel <riel@redhat.com> wrote:
> On Fri, 2016-04-15 at 12:15 -0700, Andrew Morton wrote:
>> (switched to email.  Please respond via emailed reply-to-all, not via
>> the
>> bugzilla web interface).
>>
>> This is ... interesting.
>
> First things first. What is the value of
> /proc/sys/vm/zone_reclaim?

There is no such thing on this system.  However:

$ cat /proc/sys/vm/zone_reclaim_mode
0

>
> I am assuming this is a two socket system,
> with two 12-core CPUs. Am I right?

Yes.

>
>> On Thu, 12 Nov 2015 18:46:35 +0000 bugzilla-
>> daemon@bugzilla.kernel.org wrote:
>>
>> >
>> > https://bugzilla.kernel.org/show_bug.cgi?id=107771
>> >
>> >             Bug ID: 107771
>> >            Summary: Single process tries to use more than 1/2
>> > physical
>> >                     RAM, OS starts thrashing
>> >            Product: Memory Management
>> >            Version: 2.5
>> >     Kernel Version: 4.3.0-040300-generic (Ubuntu)
>> >           Hardware: All
>> >                 OS: Linux
>> >               Tree: Mainline
>> >             Status: NEW
>> >           Severity: normal
>> >           Priority: P1
>> >          Component: Page Allocator
>> >           Assignee: akpm@linux-foundation.org
>> >           Reporter: theosib@gmail.com
>> >         Regression: No
>> >
>> > I have a 24-core (48 thread) system with 64GB of RAM.
>> >
>> > When I run multiple processes, I can use all of physical RAM before
>> > swapping
>> > starts.  However, if I'm running only a *single* process, the
>> > system will start
>> > swapping after I've exceeded only 1/2 of available physical
>> > RAM.  Only after
>> > swap fills does it start using more of the physical RAM.
>> >
>> > I can't find any ulimit settings or anything else that would cause
>> > this to
>> > happen intentionally.
>> >
>> > I had originally filed this against Ubuntu, but I'm now running a
>> > more recent
>> > kernel, and the problem persists, so I think it's more appropriate
>> > to file
>> > here.  There are some logs that they had me collect, so if you want
>> > to see
>> > them, the are here:
>> >
>> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1513673
>> >
>> > I don't recall this problem happening with older kernels (whatever
>> > came with
>> > Ubuntu 15.04), although I may just not have noticed.  By swapping
>> > early, I'm
>> > limited by the speed of my SSD, which is moving only about 20MB/sec
>> > in each
>> > direction, and that makes what I'm running take 10 times as long to
>> > complete.
>> >
> --
> All Rights Reversed.
>



-- 
Timothy Normand Miller, PhD
Assistant Professor of Computer Science, Binghamton University
http://www.cs.binghamton.edu/~millerti/
Open Graphics Project

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 107771] New: Single process tries to use more than 1/2 physical RAM, OS starts thrashing

[Timothy Normand Miller]

On Sun, Apr 17, 2016 at 2:53 PM, Michal Hocko <mhocko@kernel.org> wrote:
> On Fri 15-04-16 15:31:22, Rik van Riel wrote:
>> On Fri, 2016-04-15 at 12:15 -0700, Andrew Morton wrote:
>> > (switched to email.  Please respond via emailed reply-to-all, not via
>> > the
>> > bugzilla web interface).
>> >
>> > This is ... interesting.
>>
>> First things first. What is the value of
>> /proc/sys/vm/zone_reclaim?
>
> Also snapshots of /proc/vmstat taken every 1s or so while you see the
> trashing would be helpful.

It's been so long since I reported this bug that I don't recall
exactly what I was doing.  I think I was running Synopsys.  I tried
artificially reproducing this by just allocating a huge amount of
memory and touching all the pages, but the problem didn't manifest.  I
wonder if Synopsys was allocating pages it didn't touch or mmaping
files or some other weird thing.


> --
> Michal Hocko
> SUSE Labs



-- 
Timothy Normand Miller, PhD
Assistant Professor of Computer Science, Binghamton University
http://www.cs.binghamton.edu/~millerti/
Open Graphics Project

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>