Re: [Bug 99471] System locks with kswapd0 and kworker taking full IO and mem

Re: [Bug 99471] System locks with kswapd0 and kworker taking full IO and mem

[Andrew Morton]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=99471

Guys, could you take a look please?

The machine went oom when there's heaps of unused swap and most memory
is being used on active_anon and inactive_anon.  We should have just
swapped that stuff out and kept going.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 99471] System locks with kswapd0 and kworker taking full IO and mem

[Raymond Jennings]

On 09/10/15 14:04, Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
>
> On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
>
>> https://bugzilla.kernel.org/show_bug.cgi?id=99471
> Guys, could you take a look please?
>
> The machine went oom when there's heaps of unused swap and most memory
> is being used on active_anon and inactive_anon.  We should have just
> swapped that stuff out and kept going.

Isn't there already logic in the kernel that disables OOM if there's 
swap space available?

I saw it once, what happened to it?
>
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 99471] System locks with kswapd0 and kworker taking full IO and mem

[Johannes Weiner]

On Thu, Sep 10, 2015 at 02:04:18PM -0700, Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > https://bugzilla.kernel.org/show_bug.cgi?id=99471
> 
> Guys, could you take a look please?
> 
> The machine went oom when there's heaps of unused swap and most memory
> is being used on active_anon and inactive_anon.  We should have just
> swapped that stuff out and kept going.

I think we need to re-evaluate the way we balance file and anon scan
pressure. It's not just the "not swapping" aspect that bugs me, it's
also the fact that the machine has been thrashing page cache at full
load for *minutes* before signalling the OOM.

SSDs can flush and reload pages quick enough that on memory pressure
there are always reclaimable cache pages and the scanner never goes
after anonymous memory. If anonymous memory does not leave enough room
for page cache to hold the libraries and executables, userspace goes
into a state where it's mostly waiting for cache to become uptodate.

It's a very frustrating problem because it's hard to even detect.

One idea I had to address the LRU balance problem in the past was to
always reclaim the pages in the following order: inactive file, active
file, anon*. As one set becomes empty, go after the next one. If the
workingset code detects cache thrashing, it depends on the refault
distances what to do: if they are smaller than the active file size,
deactivate; if they are bigger than that, but smaller than active file
+ anon, we need to start swapping to alleviate the cache thrashing.

Now, if the refault distances are bigger than active file + anon, no
amount of deactivating and swapping are going to stop the thrashing
and we have to think about triggering OOM. But OOM is drastic and the
refaults might happen at a very slow pace (or, with sparse files, not
require any IO at all) and the system might be completely fine. So in
addition this would require a measure of overall time spent on
thrashing IO, comparable to what Tejun proposed in "[RFD] memory
pressure and sizing problem", where we say if thrashing IO takes up X
percent of all execution time spent, we trigger the OOM killer--not to
free memory, but to reduce the tasks that contribute to the thrashing
and let the remaining tasks make progress, similar to the swap token
or a BSD style memory scheduler.

* we can ignore the difference between inactive and active anon here
  as anon is not aged the same way as the file LRU is aged

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 99471] System locks with kswapd0 and kworker taking full IO and mem

[Rik van Riel]

On 09/15/2015 04:39 AM, Johannes Weiner wrote:
> On Thu, Sep 10, 2015 at 02:04:18PM -0700, Andrew Morton wrote:
>> (switched to email.  Please respond via emailed reply-to-all, not via the
>> bugzilla web interface).
>>
>> On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
>>
>>> https://bugzilla.kernel.org/show_bug.cgi?id=99471
>>
>> Guys, could you take a look please?
>>
>> The machine went oom when there's heaps of unused swap and most memory
>> is being used on active_anon and inactive_anon.  We should have just
>> swapped that stuff out and kept going.
> 
> I think we need to re-evaluate the way we balance file and anon scan
> pressure. It's not just the "not swapping" aspect that bugs me, it's
> also the fact that the machine has been thrashing page cache at full
> load for *minutes* before signalling the OOM.
> 
> SSDs can flush and reload pages quick enough that on memory pressure
> there are always reclaimable cache pages and the scanner never goes
> after anonymous memory. If anonymous memory does not leave enough room
> for page cache to hold the libraries and executables, userspace goes
> into a state where it's mostly waiting for cache to become uptodate.
> 
> It's a very frustrating problem because it's hard to even detect.
> 
> One idea I had to address the LRU balance problem in the past was to
> always reclaim the pages in the following order: inactive file, active
> file, anon*. As one set becomes empty, go after the next one. If the
> workingset code detects cache thrashing, it depends on the refault
> distances what to do: if they are smaller than the active file size,
> deactivate; if they are bigger than that, but smaller than active file
> + anon, we need to start swapping to alleviate the cache thrashing.
> 
> Now, if the refault distances are bigger than active file + anon, no
> amount of deactivating and swapping are going to stop the thrashing
> and we have to think about triggering OOM. But OOM is drastic and the
> refaults might happen at a very slow pace (or, with sparse files, not
> require any IO at all) and the system might be completely fine.

We already measure how much the system is slowed down by waiting
on IO - iowait time.  It's not perfect, but it can give us some
indication whether or not we are thrashing on page cache access.

> So in
> addition this would require a measure of overall time spent on
> thrashing IO, comparable to what Tejun proposed in "[RFD] memory
> pressure and sizing problem", where we say if thrashing IO takes up X
> percent of all execution time spent, we trigger the OOM killer--not to
> free memory, but to reduce the tasks that contribute to the thrashing
> and let the remaining tasks make progress, similar to the swap token
> or a BSD style memory scheduler.

The BSD style process swapping only takes mapped memory into
account. Page cache thrashing is pretty much ignored.

Maybe we should only count page fault stalls (do we track
those already?) in addition to refault distances?

-- 
All rights reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 99471] System locks with kswapd0 and kworker taking full IO and mem

[Michal Hocko]

[Sorry for replying here but I couldn't find the original Andrew's email
in my mailbox]

On Tue 15-09-15 10:39:19, Johannes Weiner wrote:
> On Thu, Sep 10, 2015 at 02:04:18PM -0700, Andrew Morton wrote:
> > (switched to email.  Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
> > 
> > On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
> > 
> > > https://bugzilla.kernel.org/show_bug.cgi?id=99471
> > 
> > Guys, could you take a look please?
> > 
> > The machine went oom when there's heaps of unused swap and most memory
> > is being used on active_anon and inactive_anon.  We should have just
> > swapped that stuff out and kept going.

I would strongly suspect the memory is pinned by somebody which
completely ruins all the get_scan_count assumptions. The first
referenced OOM report might contain a hint:
[ 2162.123944] Purging GPU memory, 368640 bytes freed, 615292928 bytes still pinned.
[ 2175.996060] Purging GPU memory, 499712 bytes freed, 615251968 bytes still pinned.
[ 2175.998841] bash invoked oom-killer: gfp_mask=0x20858, order=0, oom_score_adj=0
[ 2175.998844] bash cpuset=/ mems_allowed=0
[...]
[ 2175.999016] active_anon:305425 inactive_anon:141206 isolated_anon:0
                active_file:5109 inactive_file:4666 isolated_file:0
                unevictable:4 dirty:2 writeback:0 unstable:0
                free:13218 slab_reclaimable:6552 slab_unreclaimable:11310
                mapped:21203 shmem:155079 pagetables:10921 bounce:0
                free_cma:0
[...]
[ 2175.999074] 169619 total pagecache pages
[ 2175.999076] 4752 pages in swap cache
[ 2175.999078] Swap cache stats: add 468915, delete 464163, find 76521/98873
[ 2175.999080] Free swap  = 1615656kB
[ 2175.999082] Total swap = 2097148kB
[ 2175.999083] 521838 pages RAM
[ 2175.999084] 0 pages HighMem/MovableOnly
[ 2175.999086] 11811 pages reserved
[ 2175.999087] 0 pages hwpoisoned

So there is more than 600MB used by the GPU. Later OOM invocations do
not mention GPU OOM shrinker at all. Anon+File+Unevict+Free+Slab+Pagetbl
gives us 1.9G so considerable amount of pinned memory has to be sitting
on LRU lists. I would bet it is shmem here but there is still more than
1G on the anon LRU lists. Is it possible they are pinned indirectly?
I am CCing Daniel for the GPU memory consumption. Maybe there is some
additional diagnostic to look at.

Another interesting thing to note is that
[ 2175.999473] Out of memory: Kill process 3566 (java) score 170 or sacrifice child
[ 2175.999477] Killed process 3566 (java) total-vm:3417044kB, anon-rss:703656kB, file-rss:0kB
[...]
[ 2176.000641] bash invoked oom-killer: gfp_mask=0x20858, order=0, oom_score_adj=0
[ 2176.000644] bash cpuset=/ mems_allowed=0
[...]
[ 2176.000798] active_anon:305425 inactive_anon:141206 isolated_anon:0
                active_file:5109 inactive_file:4666 isolated_file:0
                unevictable:4 dirty:2 writeback:0 unstable:0
                free:13187 slab_reclaimable:6552 slab_unreclaimable:11310
                mapped:21203 shmem:155079 pagetables:10921 bounce:0
                free_cma:0

So the anon LRU lists are intact even after java has exited so something
is clearly wrong the anon LRU list and it looks like a leak via elevated
page ref. counting.

It sounds like this is reproducible for you Gonzalo, could you invoke a
crash dump and save the vmcore so that the LRU can be investigated? We
would see the state after something went wrong but maybe there will be
some pattern to help us.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 99471] System locks with kswapd0 and kworker taking full IO and mem

[Andrew Morton]

On Mon, 5 Oct 2015 22:03:46 +0200 Michal Hocko <mhocko@kernel.org> wrote:

> On Tue 15-09-15 10:39:19, Johannes Weiner wrote:
> > On Thu, Sep 10, 2015 at 02:04:18PM -0700, Andrew Morton wrote:
> > > (switched to email.  Please respond via emailed reply-to-all, not via the
> > > bugzilla web interface).
> > > 
> > > On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
> > > 
> > > > https://bugzilla.kernel.org/show_bug.cgi?id=99471
> > > 
> > > Guys, could you take a look please?

So this isn't fixed and a number of new reporters (cc'ed) are chiming
in (let's please keep this going via email, not via the bugzilla UI!).

We have various theories but I don't think we've nailed it down yet.

Are any of the reporters able to come up with a set of instructions
which will permit the developers to reproduce this bug locally?

Can we think up a way of adding some form of debug/instrumentation to
the kernel which will permit us to diagnose and fix this?  It could be
something which a tester manually adds or it could be something
permanent, perhaps controlled via a procfs knob.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 99471] System locks with kswapd0 and kworker taking full IO and mem

[Mel Gorman]

On Tue, Feb 16, 2016 at 02:41:59PM -0800, Andrew Morton wrote:
> On Mon, 5 Oct 2015 22:03:46 +0200 Michal Hocko <mhocko@kernel.org> wrote:
> 
> > On Tue 15-09-15 10:39:19, Johannes Weiner wrote:
> > > On Thu, Sep 10, 2015 at 02:04:18PM -0700, Andrew Morton wrote:
> > > > (switched to email.  Please respond via emailed reply-to-all, not via the
> > > > bugzilla web interface).
> > > > 
> > > > On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
> > > > 
> > > > > https://bugzilla.kernel.org/show_bug.cgi?id=99471
> > > > 
> > > > Guys, could you take a look please?
> 
> So this isn't fixed and a number of new reporters (cc'ed) are chiming
> in (let's please keep this going via email, not via the bugzilla UI!).
> 
> We have various theories but I don't think we've nailed it down yet.
> 

So, I'm nowhere close to this at the moment. I was aware of at least one
swapping-related problem that was introduced between 4.0 and 4.1. The
commit that introduced it only affects NUMA so there is no chance they
are related. However, I'll still need to chase that down early next week
before considering this problem. Someone else may figure it out faster.

As the problem I'm aware of is NUMA only, I took a momentary look at
this. The first log shows MCE errors but they may be overheating related
so I'm willing to ignore that.

The log clearly states that a lot of memory is pinned by the GPU just
before the OOM triggers.

[ 2175.996060] Purging GPU memory, 499712 bytes freed, 615251968 bytes still pinned.

So that in itself is a major problem. Next the memory usage at the time
of failure was

[ 2175.999016] active_anon:305425 inactive_anon:141206 isolated_anon:0
                active_file:5109 inactive_file:4666 isolated_file:0
                unevictable:4 dirty:2 writeback:0 unstable:0
                free:13218 slab_reclaimable:6552 slab_unreclaimable:11310
                mapped:21203 shmem:155079 pagetables:10921 bounce:0
                free_cma:0

1.8G of anony memory usage with almost 600M of that being GPU-related.
The file usage is negligible so this is looking closer to being a true
OOM situation

[ 2175.999080] Free swap  = 1615656kB
[ 2175.999082] Total swap = 2097148kB

Load of swap available. The IO is likely high because files are probably
being continually reclaimed and paged back in so it's thrashing.
Johannes is likely correct when he says there is a problem with
balancing when the storage is fast. That's one aspect of the problem
but it does not explain why the problem is recent. The one major
candidate I can spot is this

1da58ee2: mm: vmscan: count only dirty pages as congested

That alters how and when processes are put to sleep waiting on
congestion to clear. While I can see the logic behind the patch, the
impact was no quantified and it can mean that kswapd is no longer
throttling when it used to. Try something like this untested

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 2aec4241b42a..50b24a022db0 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -953,8 +953,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 		 * end of the LRU a second time.
 		 */
 		mapping = page_mapping(page);
-		if (((dirty || writeback) && mapping &&
-		     inode_write_congested(mapping->host)) ||
+		if ((mapping && inode_write_congested(mapping->host)) ||
 		    (writeback && PageReclaim(page)))
 			nr_congested++;
 
This is not necessary the right fix, it just may narrow down where the
problem is.

The problem is compounded probably by scasnning one third of the LRU before
any reclaim candidates are found. Is it known if all the people reporting
problems are using an i915 GPU? If so, Daniel, are you aware of any commits
between 3.18 and 4.1 that would potentially pin GPU memory permanently or
alternative would have busted the shrinker?

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>