kswapd stuck using 100% CPU

kswapd stuck using 100% CPU

[Anton Blanchard]

Hi,

I booted the latest git today on a ppc64 box. When I pushed it into
swap I noticed both kswapd's were using 100% CPU and the soft lockup
detector suggested it was stuck in balance_pgdat:

BUG: soft lockup - CPU#7 stuck for 23s! [kswapd1:359]
Call Trace:
[c00000000015e190] .balance_pgdat+0x150/0x940 
[c00000000015eb2c] .kswapd+0x1ac/0x490
[c00000000009edbc] .kthread+0xbc/0xd0
[c00000000002142c] .kernel_thread+0x54/0x70

I haven't had time to bisect but I did notice we were looping here:

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 7658fd6..c92bad2 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2945,9 +2959,11 @@ out:
 			if (zone->all_unreclaimable && priority != DEF_PRIORITY)
 				continue;
 
+#if 0
 			/* Would compaction fail due to lack of free memory? */
 			if (compaction_suitable(zone, order) == COMPACT_SKIPPED)
 				goto loop_again;
+#endif
 
 			/* Confirm the zone is balanced for order-0 */
 			if (!zone_watermark_ok(zone, 0,


After commenting it out the box is happy again.

Anton

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH] Re: kswapd stuck using 100% CPU

[Rik van Riel]

On Sat, 24 Mar 2012 13:03:53 +1100
Anton Blanchard <anton@samba.org> wrote:

> I booted the latest git today on a ppc64 box. When I pushed it into
> swap I noticed both kswapd's were using 100% CPU and the soft lockup
> detector suggested it was stuck in balance_pgdat:
> 
> BUG: soft lockup - CPU#7 stuck for 23s! [kswapd1:359]
> Call Trace:
> [c00000000015e190] .balance_pgdat+0x150/0x940 
> [c00000000015eb2c] .kswapd+0x1ac/0x490
> [c00000000009edbc] .kthread+0xbc/0xd0
> [c00000000002142c] .kernel_thread+0x54/0x70

Are you running without CONFIG_COMPACTION enabled by any chance?

Because if you do, the stub function compaction_suitable will always
return COMPACT_SKIPPED:
 
> I haven't had time to bisect but I did notice we were looping here:
> 
> +++ b/mm/vmscan.c
> @@ -2945,9 +2959,11 @@ out:
>  			if (zone->all_unreclaimable && priority != DEF_PRIORITY)
>  				continue;
>  
> +#if 0
>  			/* Would compaction fail due to lack of free memory? */
>  			if (compaction_suitable(zone, order) == COMPACT_SKIPPED)
>  				goto loop_again;
> +#endif

The patch below should fix it.

-----

Only test compaction_suitable if the kernel is built with CONFIG_COMPACTION,
otherwise the stub compaction_suitable function will always return
COMPACT_SKIPPED and send kswapd into an infinite loop.

Signed-off-by: Rik van Riel <riel@redhat.com>
Reported-by: Anton Blanchard <anton@samba.org>

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 7658fd6..33c332b 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2946,7 +2946,8 @@ out:
 				continue;
 
 			/* Would compaction fail due to lack of free memory? */
-			if (compaction_suitable(zone, order) == COMPACT_SKIPPED)
+			if (COMPACTION_BUILD &&
+			    compaction_suitable(zone, order) == COMPACT_SKIPPED)
 				goto loop_again;
 
 			/* Confirm the zone is balanced for order-0 */

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Re: kswapd stuck using 100% CPU

[Hugh Dickins]

On Sat, 24 Mar 2012, Rik van Riel wrote:
> 
> Only test compaction_suitable if the kernel is built with CONFIG_COMPACTION,
> otherwise the stub compaction_suitable function will always return
> COMPACT_SKIPPED and send kswapd into an infinite loop.
> 
> Signed-off-by: Rik van Riel <riel@redhat.com>
> Reported-by: Anton Blanchard <anton@samba.org>

Thank you, Anton and Rik.  I never quite got around to investigating
why swapping had been nearly twice as slow with linux-next on my Aspire
One (with a relatively minimal config, omitting COMPACTION).  That was
the reason (one half of the HT cpu busy in kswapd), and this fixes it.

Tested-by: Hugh Dickins <hughd@google.com>

> 
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 7658fd6..33c332b 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -2946,7 +2946,8 @@ out:
>  				continue;
>  
>  			/* Would compaction fail due to lack of free memory? */
> -			if (compaction_suitable(zone, order) == COMPACT_SKIPPED)
> +			if (COMPACTION_BUILD &&
> +			    compaction_suitable(zone, order) == COMPACT_SKIPPED)
>  				goto loop_again;
>  
>  			/* Confirm the zone is balanced for order-0 */

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Re: kswapd stuck using 100% CPU

[Mel Gorman]

On Sat, Mar 24, 2012 at 10:26:21AM -0400, Rik van Riel wrote:
> 
> Only test compaction_suitable if the kernel is built with CONFIG_COMPACTION,
> otherwise the stub compaction_suitable function will always return
> COMPACT_SKIPPED and send kswapd into an infinite loop.
> 
> Signed-off-by: Rik van Riel <riel@redhat.com>
> Reported-by: Anton Blanchard <anton@samba.org>
> 

Acked-by: Mel Gorman <mel@csn.ul.ie>

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Re: kswapd stuck using 100% CPU

[Pekka Enberg]

On Mon, Mar 26, 2012 at 12:32 PM, Mel Gorman <mel@csn.ul.ie> wrote:
> On Sat, Mar 24, 2012 at 10:26:21AM -0400, Rik van Riel wrote:
>>
>> Only test compaction_suitable if the kernel is built with CONFIG_COMPACTION,
>> otherwise the stub compaction_suitable function will always return
>> COMPACT_SKIPPED and send kswapd into an infinite loop.
>>
>> Signed-off-by: Rik van Riel <riel@redhat.com>
>> Reported-by: Anton Blanchard <anton@samba.org>
>
> Acked-by: Mel Gorman <mel@csn.ul.ie>

The API looks fragile and this patch isn't exactly making it any
better. Why don't we make compaction_suitable() return something other
than COMPACT_SKIPPED for !CONFIG_COMPACTION case?

                        Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Re: kswapd stuck using 100% CPU

[Mel Gorman]

On Mon, Mar 26, 2012 at 01:40:41PM +0300, Pekka Enberg wrote:
> On Mon, Mar 26, 2012 at 12:32 PM, Mel Gorman <mel@csn.ul.ie> wrote:
> > On Sat, Mar 24, 2012 at 10:26:21AM -0400, Rik van Riel wrote:
> >>
> >> Only test compaction_suitable if the kernel is built with CONFIG_COMPACTION,
> >> otherwise the stub compaction_suitable function will always return
> >> COMPACT_SKIPPED and send kswapd into an infinite loop.
> >>
> >> Signed-off-by: Rik van Riel <riel@redhat.com>
> >> Reported-by: Anton Blanchard <anton@samba.org>
> >
> > Acked-by: Mel Gorman <mel@csn.ul.ie>
> 
> The API looks fragile and this patch isn't exactly making it any
> better. Why don't we make compaction_suitable() return something other
> than COMPACT_SKIPPED for !CONFIG_COMPACTION case?
> 

Returning COMPACT_PARTIAL or COMPACT_CONTINUE would confuse the check in
should_continue_reclaim. A fourth return type could be added but an
obvious name does not spring to mind that would end up being similar to
just adding a CONFIG_COMPACTION check.

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] Re: kswapd stuck using 100% CPU

[Pekka Enberg]

Hi Mel,

On Mon, Mar 26, 2012 at 4:56 PM, Mel Gorman <mel@csn.ul.ie> wrote:
>> The API looks fragile and this patch isn't exactly making it any
>> better. Why don't we make compaction_suitable() return something other
>> than COMPACT_SKIPPED for !CONFIG_COMPACTION case?
>
> Returning COMPACT_PARTIAL or COMPACT_CONTINUE would confuse the check in
> should_continue_reclaim. A fourth return type could be added but an
> obvious name does not spring to mind that would end up being similar to
> just adding a CONFIG_COMPACTION check.

How about COMPACT_DISABLED?

The current API just doesn't make sense from practical point of view.
Anyone calling compaction_suitable() needs to do the COMPAT_BUILD
check first which is a non-obvious and error-prone API.

                        Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>