kswapd craziness round 2

kswapd craziness round 2

[Jiri Slaby]

Hi,

You still feel the sour taste of the "kswapd craziness in v3.7" thread,
right? Welcome to the hell, part two :{.

I believe this started happening after update from
3.8.0-rc4-next-20130125 to 3.8.0-rc7-next-20130211. The same as before,
many hours of uptime are needed and perhaps some suspend/resume cycles
too. Memory pressure is not high, plenty of I/O cache:
# free
             total       used       free     shared    buffers     cached
Mem:       6026692    5571184     455508          0     351252    2016648
-/+ buffers/cache:    3203284    2823408
Swap:            0          0          0

kswap is working very toughly though:
root       580  0.6  0.0      0     0 ?        S    uno12  46:21 [kswapd0]

This happens on I/O activity right now. For example by updatedb or find
/. This is what the stack trace of kswapd0 looks like:
[<ffffffff8113c431>] shrink_slab+0xa1/0x2d0
[<ffffffff8113ecd1>] kswapd+0x541/0x930
[<ffffffff810a3000>] kthread+0xc0/0xd0
[<ffffffff816beb5c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff

Any ideas?

thanks,
-- 
js
suse labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd craziness round 2

[Hillf Danton]

On Fri, Mar 8, 2013 at 3:37 AM, Jiri Slaby <jslaby@suse.cz> wrote:
> On 03/01/2013 03:02 PM, Hillf Danton wrote:
>> On Fri, Mar 1, 2013 at 1:02 AM, Jiri Slaby <jslaby@suse.cz> wrote:
>>>
>>> Ok, no difference, kswap is still crazy. I'm attaching the output of
>>> "grep -vw '0' /proc/vmstat" if you see something there.
>>>
>> Thanks to you for test and data.
>>
>> Lets try to restore the deleted nap, then.
>
> Oh, it seems to be nice now:
> root       579  0.0  0.0      0     0 ?        S    Mar04   0:13 [kswapd0]
>
Double thanks.

But Mel does not like it, probably.
Lets try nap in another way.

Hillf

--- a/mm/vmscan.c	Thu Feb 21 20:01:02 2013
+++ b/mm/vmscan.c	Fri Mar  8 14:36:10 2013
@@ -2793,6 +2793,10 @@ loop_again:
 				 * speculatively avoid congestion waits
 				 */
 				zone_clear_flag(zone, ZONE_CONGESTED);
+
+			else if (sc.priority > 2 &&
+				 sc.priority < DEF_PRIORITY - 2)
+				wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
 		}

 		/*
--

>>
>> --- a/mm/vmscan.c     Thu Feb 21 20:01:02 2013
>> +++ b/mm/vmscan.c     Fri Mar  1 21:55:40 2013
>> @@ -2817,6 +2817,10 @@ loop_again:
>>                */
>>               if (sc.nr_reclaimed >= SWAP_CLUSTER_MAX)
>>                       break;
>> +
>> +             if (sc.priority < DEF_PRIORITY - 2)
>> +                     congestion_wait(BLK_RW_ASYNC, HZ/10);
>> +
>>       } while (--sc.priority >= 0);
>>
>>  out:
>> --
>>
>
>
> --
> js
> suse labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd craziness round 2

[Zlatko Calusic]

On 08.03.2013 07:42, Hillf Danton wrote:
> On Fri, Mar 8, 2013 at 3:37 AM, Jiri Slaby <jslaby@suse.cz> wrote:
>> On 03/01/2013 03:02 PM, Hillf Danton wrote:
>>> On Fri, Mar 1, 2013 at 1:02 AM, Jiri Slaby <jslaby@suse.cz> wrote:
>>>>
>>>> Ok, no difference, kswap is still crazy. I'm attaching the output of
>>>> "grep -vw '0' /proc/vmstat" if you see something there.
>>>>
>>> Thanks to you for test and data.
>>>
>>> Lets try to restore the deleted nap, then.
>>
>> Oh, it seems to be nice now:
>> root       579  0.0  0.0      0     0 ?        S    Mar04   0:13 [kswapd0]
>>
> Double thanks.
>
> But Mel does not like it, probably.
> Lets try nap in another way.
>
> Hillf
>
> --- a/mm/vmscan.c	Thu Feb 21 20:01:02 2013
> +++ b/mm/vmscan.c	Fri Mar  8 14:36:10 2013
> @@ -2793,6 +2793,10 @@ loop_again:
>   				 * speculatively avoid congestion waits
>   				 */
>   				zone_clear_flag(zone, ZONE_CONGESTED);
> +
> +			else if (sc.priority > 2 &&
> +				 sc.priority < DEF_PRIORITY - 2)
> +				wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
>   		}
>
>   		/*
> --
>

There's another bug in there, which I'm still chasing. Artificial sleeps 
like this just mask the real bug and introduce new problems (on my 4GB 
server kswapd spends all the time in those congestion wait calls). The 
problem is that the bug needs about 5 days of uptime to reveal it's ugly 
head. So far I can only tell that it was introduced somewhere between 
3.1 & 3.4.

Also, check shrink_inactive_list(), it already sleeps if really needed:

if (nr_writeback && nr_writeback >=
		(nr_taken >> (DEF_PRIORITY - sc->priority)))
	wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);

Regards,
-- 
Zlatko

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd craziness round 2

[Hillf Danton]

On Fri, Mar 8, 2013 at 3:29 PM, Zlatko Calusic <zcalusic@bitsync.net> wrote:
> There's another bug in there, which I'm still chasing.
>
I am busy in discovering an employer(a really hard work?) so
I dunno the hours I have for that bug.

Hmm, take a look at Mels thoughts?
http://marc.info/?l=linux-mm&m=136189593423501&w=2

BTW, he will be online next week.

Hillf

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd craziness round 2

[Jiri Slaby]

On 03/08/2013 07:42 AM, Hillf Danton wrote:
> On Fri, Mar 8, 2013 at 3:37 AM, Jiri Slaby <jslaby@suse.cz> wrote:
>> On 03/01/2013 03:02 PM, Hillf Danton wrote:
>>> On Fri, Mar 1, 2013 at 1:02 AM, Jiri Slaby <jslaby@suse.cz> wrote:
>>>>
>>>> Ok, no difference, kswap is still crazy. I'm attaching the output of
>>>> "grep -vw '0' /proc/vmstat" if you see something there.
>>>>
>>> Thanks to you for test and data.
>>>
>>> Lets try to restore the deleted nap, then.
>>
>> Oh, it seems to be nice now:
>> root       579  0.0  0.0      0     0 ?        S    Mar04   0:13 [kswapd0]
>>
> Double thanks.

There is one downside. I'm not sure whether that patch was the culprit.
My Thunderbird is jerky when scrolling and lags while writing this
message. The letters sometimes appear later than typed and in groups. Like
I (kbd): My Thunder
TB: My Thunder
I (kbd): b-i-r-d
TB: is silent
I (kbd): still typing...
TB: bird is

Perhaps it's not only TB.

> But Mel does not like it, probably.
> Lets try nap in another way.

Will try next week.

> --- a/mm/vmscan.c	Thu Feb 21 20:01:02 2013
> +++ b/mm/vmscan.c	Fri Mar  8 14:36:10 2013
> @@ -2793,6 +2793,10 @@ loop_again:
>  				 * speculatively avoid congestion waits
>  				 */
>  				zone_clear_flag(zone, ZONE_CONGESTED);
> +
> +			else if (sc.priority > 2 &&
> +				 sc.priority < DEF_PRIORITY - 2)
> +				wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
>  		}
> 
>  		/*

-- 
js
suse labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd craziness round 2

[Pádraig Brady]

On 03/08/2013 11:21 PM, Jiri Slaby wrote:
> On 03/08/2013 07:42 AM, Hillf Danton wrote:
>> On Fri, Mar 8, 2013 at 3:37 AM, Jiri Slaby <jslaby@suse.cz> wrote:
>>> On 03/01/2013 03:02 PM, Hillf Danton wrote:
>>>> On Fri, Mar 1, 2013 at 1:02 AM, Jiri Slaby <jslaby@suse.cz> wrote:
>>>>>
>>>>> Ok, no difference, kswap is still crazy. I'm attaching the output of
>>>>> "grep -vw '0' /proc/vmstat" if you see something there.
>>>>>
>>>> Thanks to you for test and data.
>>>>
>>>> Lets try to restore the deleted nap, then.
>>>
>>> Oh, it seems to be nice now:
>>> root       579  0.0  0.0      0     0 ?        S    Mar04   0:13 [kswapd0]
>>>
>> Double thanks.
> 
> There is one downside. I'm not sure whether that patch was the culprit.
> My Thunderbird is jerky when scrolling and lags while writing this
> message. The letters sometimes appear later than typed and in groups. Like
> I (kbd): My Thunder
> TB: My Thunder
> I (kbd): b-i-r-d
> TB: is silent
> I (kbd): still typing...
> TB: bird is
> 
> Perhaps it's not only TB.

I notice the same thunderbird issue on the much older 2.6.40.4-5.fc15.x86_64
which I'd hoped would be fixed on upgrade :(

My Thunderbird is using 1957m virt, 722m RSS on my 3G system.
What are your corresponding mem values?

For reference:
http://marc.info/?t=130865025500001&r=1&w=2
https://bugzilla.redhat.com/show_bug.cgi?id=712019

thanks,
PA!draig.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd craziness round 2

[Hillf Danton]

On Wed, Mar 20, 2013 at 12:59 AM, Pádraig Brady <P@draigbrady.com> wrote:
>
> I notice the same thunderbird issue on the much older 2.6.40.4-5.fc15.x86_64
> which I'd hoped would be fixed on upgrade :(
>
> My Thunderbird is using 1957m virt, 722m RSS on my 3G system.
> What are your corresponding mem values?
>
> For reference:
> http://marc.info/?t=130865025500001&r=1&w=2
> https://bugzilla.redhat.com/show_bug.cgi?id=712019
>
Hey, would you all please try Mels new work?
http://marc.info/?l=linux-mm&m=136352546814642&w=4

thanks
Hillf

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd craziness round 2

[Jiri Slaby]

On 03/20/2013 05:12 AM, Hillf Danton wrote:
> Hey, would you all please try Mels new work?
> http://marc.info/?l=linux-mm&m=136352546814642&w=4

Yeah, I was in CC and also asked Mel if I should apply those. I will as
soon as I'm back home (next week).

thanks,
-- 
js
suse labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>