[REGRESSION] [BISECTED] kswapd high CPU usage

[REGRESSION] [BISECTED] kswapd high CPU usage

[Nalorokk]

[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]

It appears that kernels newer than 4.1 have kswapd-related bug resulting in
high CPU usage. CPU 100% usage could last for several minutes or several
days, with CPU being busy entirely with serving kswapd. It happens usually
after server being mostly idle, sometimes after days, sometimes after weeks
of uptime. But the issue appears much sooner if the machine is loaded with
something like building a kernel.

Here are the graphs of CPU load: first
<http://i.piccy.info/i9/9ee6c0620c9481a974908484b2a52a0f/1453384595/44012/994698/cpu_month.png>,
second
<http://i.piccy.info/i9/7c97c2f39620bb9d7ea93096312dbbb6/1453384649/41222/994698/cpu_year.png>.
Perf top output is here <http://pastebin.com/aRzTjb2x>as well.

To find the cause of this problem I've started with the fact that the issue
appeared after 4.1 kernel update. Then I performed longterm test of 3.18,
and discovered that 3.18 is unaffected by this bug. Then I did some tests
of 4.0 to confirm that this version behaves well too.

Then I performed git bisect from tag v4.0 to v4.1-rc1 and found exact
commits that seem to be reason of high CPU usage.

The first really "bad" commit is 79553da293d38d63097278de13e28a3b371f43c1.
2 previous commits cause weird behavior as well resulting in kswapd
consuming more CPU than unaffected kernels, but not that much as the commit
pointed above. I believe those commits are related to the same mm tree
merge.

I tried to add transparent_hugepage=never to kernel boot parameters, but it
did not change anything. Changing allocator to SLAB from SLUB alters
behavior and makes CPU load lower, but don't solve a problem at all.

Here <https://bugzilla.kernel.org/show_bug.cgi?id=110501>is kernel bugzilla
bugreport as well.

Ideas? ​

[-- Attachment #2: Type: text/html, Size: 1956 bytes --]

Fwd: [REGRESSION] [BISECTED] kswapd high CPU usage

[Nalorokk]

It appears that kernels newer than 4.1 have kswapd-related bug
resulting in high CPU usage. CPU 100% usage could last for several
minutes or several days, with CPU being busy entirely with serving
kswapd. It happens usually after server being mostly idle, sometimes
after days, sometimes after weeks of uptime. But the issue appears
much sooner if the machine is loaded with something like building a
kernel.

Here are the graphs of CPU load: first [1], second [2]. Perf top
output is here [3] as well.

To find the cause of this problem I've started with the fact that the
issue appeared after 4.1 kernel update. Then I performed longterm test
of 3.18, and discovered that 3.18 is unaffected by this bug. Then I
did some tests of 4.0 to confirm that this version behaves well too.

Then I performed git bisect from tag v4.0 to v4.1-rc1 and found exact
commits that seem to be reason of high CPU usage.

The first really "bad" commit is
79553da293d38d63097278de13e28a3b371f43c1. 2 previous commits cause
weird behavior as well resulting in kswapd consuming more CPU than
unaffected kernels, but not that much as the commit pointed above. I
believe those commits are related to the same mm tree merge.

I tried to add transparent_hugepage=never to kernel boot parameters,
but it did not change anything. Changing allocator to SLAB from SLUB
alters behavior and makes CPU load lower, but don't solve a problem at
all.

Here [4] is kernel bugzilla bugreport as well.

Ideas?

[1] http://i.piccy.info/i9/9ee6c0620c9481a974908484b2a52a0f/1453384595/44012/994698/cpu_month.png
[2] http://i.piccy.info/i9/7c97c2f39620bb9d7ea93096312dbbb6/1453384649/41222/994698/cpu_year.png
[3] http://pastebin.com/aRzTjb2x
[4] https://bugzilla.kernel.org/show_bug.cgi?id=110501

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Kirill A. Shutemov]

On Fri, Jan 22, 2016 at 12:28:10AM +1000, Nalorokk wrote:
> It appears that kernels newer than 4.1 have kswapd-related bug resulting in
> high CPU usage. CPU 100% usage could last for several minutes or several
> days, with CPU being busy entirely with serving kswapd. It happens usually
> after server being mostly idle, sometimes after days, sometimes after weeks
> of uptime. But the issue appears much sooner if the machine is loaded with
> something like building a kernel.
> 
> Here are the graphs of CPU load: first
> <http://i.piccy.info/i9/9ee6c0620c9481a974908484b2a52a0f/1453384595/44012/994698/cpu_month.png>,
> second
> <http://i.piccy.info/i9/7c97c2f39620bb9d7ea93096312dbbb6/1453384649/41222/994698/cpu_year.png>.
> Perf top output is here <http://pastebin.com/aRzTjb2x>as well.
> 
> To find the cause of this problem I've started with the fact that the issue
> appeared after 4.1 kernel update. Then I performed longterm test of 3.18,
> and discovered that 3.18 is unaffected by this bug. Then I did some tests
> of 4.0 to confirm that this version behaves well too.
> 
> Then I performed git bisect from tag v4.0 to v4.1-rc1 and found exact
> commits that seem to be reason of high CPU usage.
> 
> The first really "bad" commit is 79553da293d38d63097278de13e28a3b371f43c1.
> 2 previous commits cause weird behavior as well resulting in kswapd
> consuming more CPU than unaffected kernels, but not that much as the commit
> pointed above. I believe those commits are related to the same mm tree
> merge.
> 
> I tried to add transparent_hugepage=never to kernel boot parameters, but it
> did not change anything. Changing allocator to SLAB from SLUB alters
> behavior and makes CPU load lower, but don't solve a problem at all.
> 
> Here <https://bugzilla.kernel.org/show_bug.cgi?id=110501>is kernel bugzilla
> bugreport as well.
> 
> Ideas? a??

Could you try to insert "late_initcall(set_recommended_min_free_kbytes);"
back and check if makes any difference.

-- 
 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Hugh Greenberg]

Kirill A. Shutemov <kirill <at> shutemov.name> writes:
> 
> Could you try to insert "late_initcall(set_recommended_min_free_kbytes);"
> back and check if makes any difference.
> 

We tested adding late_initcall(set_recommended_min_free_kbytes); 
back in 4.1.14 and it made a huge difference. We aren't sure if the
issue is 100% fixed, but it could be. We will keep testing it.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Kirill A. Shutemov]

On Sat, Jan 23, 2016 at 03:57:21PM +0000, Hugh Greenberg wrote:
> Kirill A. Shutemov <kirill <at> shutemov.name> writes:
> > 
> > Could you try to insert "late_initcall(set_recommended_min_free_kbytes);"
> > back and check if makes any difference.
> > 
> 
> We tested adding late_initcall(set_recommended_min_free_kbytes); 
> back in 4.1.14 and it made a huge difference. We aren't sure if the
> issue is 100% fixed, but it could be. We will keep testing it.

It would be nice to have values of min_free_kbytes before and after
set_recommended_min_free_kbytes() in your configuration.

-- 
 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Hugh Greenberg]

Kirill A. Shutemov <kirill <at> shutemov.name> writes:

> 
> On Sat, Jan 23, 2016 at 03:57:21PM +0000, Hugh Greenberg wrote:
> > Kirill A. Shutemov <kirill <at> shutemov.name> writes:
> > > 
> > > Could you try to insert 
"late_initcall(set_recommended_min_free_kbytes);"
> > > back and check if makes any difference.
> > > 
> > 
> > We tested adding late_initcall(set_recommended_min_free_kbytes); 
> > back in 4.1.14 and it made a huge difference. We aren't sure if the
> > issue is 100% fixed, but it could be. We will keep testing it.
> 
> It would be nice to have values of min_free_kbytes before and after
> set_recommended_min_free_kbytes() in your configuration.
> 

We'll try to get this for you. 

The reports are coming from 2GB Haswell chromebooks. I have a 4GB haswell 
chromebook and I cannot reproduce the issue, even if I boot the kernel with 
2GB or less.

After more testing, we were still able to reproduce the issue. It seems to 
have taken longer to show up this time.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Hugh Greenberg]

Kirill A. Shutemov <kirill <at> shutemov.name> writes:

> 
> On Sat, Jan 23, 2016 at 03:57:21PM +0000, Hugh Greenberg wrote:
> > Kirill A. Shutemov <kirill <at> shutemov.name> writes:
> > > 
> > > Could you try to insert 
"late_initcall(set_recommended_min_free_kbytes);"
> > > back and check if makes any difference.
> > > 
> > 
> > We tested adding late_initcall(set_recommended_min_free_kbytes); 
> > back in 4.1.14 and it made a huge difference. We aren't sure if the
> > issue is 100% fixed, but it could be. We will keep testing it.
> 
> It would be nice to have values of min_free_kbytes before and after
> set_recommended_min_free_kbytes() in your configuration.
> 

Before adding set_recommended_min_free_kbytes: 5391
After: 67584



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Kirill A. Shutemov]

On Mon, Jan 25, 2016 at 04:46:58PM +0000, Hugh Greenberg wrote:
> Kirill A. Shutemov <kirill <at> shutemov.name> writes:
> 
> > 
> > On Sat, Jan 23, 2016 at 03:57:21PM +0000, Hugh Greenberg wrote:
> > > Kirill A. Shutemov <kirill <at> shutemov.name> writes:
> > > > 
> > > > Could you try to insert 
> "late_initcall(set_recommended_min_free_kbytes);"
> > > > back and check if makes any difference.
> > > > 
> > > 
> > > We tested adding late_initcall(set_recommended_min_free_kbytes); 
> > > back in 4.1.14 and it made a huge difference. We aren't sure if the
> > > issue is 100% fixed, but it could be. We will keep testing it.
> > 
> > It would be nice to have values of min_free_kbytes before and after
> > set_recommended_min_free_kbytes() in your configuration.
> > 
> 
> Before adding set_recommended_min_free_kbytes: 5391
> After: 67584

[ add more people to the thread ]

The 'before' value look low to me for machine with 2G of RAM.

In the bugzilla[1], you've mentioned zram. I wounder if we need to
increase min_free_kbytes when zram is in use as we do for THP.

[1] https://bugzilla.kernel.org/show_bug.cgi?id=110501

-- 
 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Sergey Senozhatsky]

On (02/02/16 15:59), Kirill A. Shutemov wrote:
> > > On Sat, Jan 23, 2016 at 03:57:21PM +0000, Hugh Greenberg wrote:
> > > > Kirill A. Shutemov <kirill <at> shutemov.name> writes:
> > > > > 
> > > > > Could you try to insert 
> > "late_initcall(set_recommended_min_free_kbytes);"
> > > > > back and check if makes any difference.
> > > > > 
> > > > 
> > > > We tested adding late_initcall(set_recommended_min_free_kbytes); 
> > > > back in 4.1.14 and it made a huge difference. We aren't sure if the
> > > > issue is 100% fixed, but it could be. We will keep testing it.
> > > 
> > > It would be nice to have values of min_free_kbytes before and after
> > > set_recommended_min_free_kbytes() in your configuration.
> > > 
> > 
> > Before adding set_recommended_min_free_kbytes: 5391
> > After: 67584
> 
> [ add more people to the thread ]
> 
> The 'before' value look low to me for machine with 2G of RAM.
> 
> In the bugzilla[1], you've mentioned zram. I wounder if we need to
> increase min_free_kbytes when zram is in use as we do for THP.
> 
> [1] https://bugzilla.kernel.org/show_bug.cgi?id=110501

[add Seth Jennings]


additional info: http://marc.info/?l=linux-mm&m=145373987224270
> The reports are coming from 2GB Haswell chromebooks. I have a 4GB haswell
> chromebook and I cannot reproduce the issue, even if I boot the kernel with
> 2GB or less.
>
> After more testing, we were still able to reproduce the issue. It seems to
> have taken longer to show up this time.


a small note,
I assume, IF min_free_kbytes must be set then probably in zsmalloc (zbud?)
init, not in zram. zswap can use both -- zsmalloc and zbud.

	-ss

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Minchan Kim]

On Tue, Feb 02, 2016 at 03:59:50PM +0200, Kirill A. Shutemov wrote:
> On Mon, Jan 25, 2016 at 04:46:58PM +0000, Hugh Greenberg wrote:
> > Kirill A. Shutemov <kirill <at> shutemov.name> writes:
> > 
> > > 
> > > On Sat, Jan 23, 2016 at 03:57:21PM +0000, Hugh Greenberg wrote:
> > > > Kirill A. Shutemov <kirill <at> shutemov.name> writes:
> > > > > 
> > > > > Could you try to insert 
> > "late_initcall(set_recommended_min_free_kbytes);"
> > > > > back and check if makes any difference.
> > > > > 
> > > > 
> > > > We tested adding late_initcall(set_recommended_min_free_kbytes); 
> > > > back in 4.1.14 and it made a huge difference. We aren't sure if the
> > > > issue is 100% fixed, but it could be. We will keep testing it.
> > > 
> > > It would be nice to have values of min_free_kbytes before and after
> > > set_recommended_min_free_kbytes() in your configuration.
> > > 
> > 
> > Before adding set_recommended_min_free_kbytes: 5391
> > After: 67584
> 
> [ add more people to the thread ]
> 
> The 'before' value look low to me for machine with 2G of RAM.
> 
> In the bugzilla[1], you've mentioned zram. I wounder if we need to
> increase min_free_kbytes when zram is in use as we do for THP.
> 
> [1] https://bugzilla.kernel.org/show_bug.cgi?id=110501

Normally, it's recommended to increate min_free_kbytes when zram is
used for swap because zram should allocate a page in reclaim path
dynamically to keep compressed page.

However, when I read bugzilla's perf profile, I can't find
any zram related things and if there is lack of free memory for
zram page allocation due to changing min_free_kbytes, user will see
below error message.

pr_err("Error allocating memory for compressed page: %u, size=%zu\n"

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Hugh Greenberg]

Kirill A. Shutemov <kirill <at> shutemov.name> writes:


> In the bugzilla[1], you've mentioned zram. I wounder if we need to
> increase min_free_kbytes when zram is in use as we do for THP.
> 
> [1] https://bugzilla.kernel.org/show_bug.cgi?id=110501
> 

We've tried increasing the min_free_kbytes. It may help for a time, 
but then the issue returns.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Hugh Greenberg]

A lot of users are still reporting the issue. 
The bugzilla report has some new information.
https://bugzilla.kernel.org/show_bug.cgi?id=110501
 




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [REGRESSION] [BISECTED] kswapd high CPU usage

[Hugh Greenberg]

There is also another bug report for the same issue 
that has some good information: 
https://bugzilla.kernel.org/show_bug.cgi?id=65201

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>