From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx184.postini.com [74.125.245.184]) by kanga.kvack.org (Postfix) with SMTP id 7042F6B004D for ; Tue, 27 Nov 2012 15:49:41 -0500 (EST) From: Johannes Weiner Subject: kswapd craziness in 3.7 Date: Tue, 27 Nov 2012 15:48:34 -0500 Message-Id: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: Mel Gorman , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org Hi everyone, I hope I included everybody that participated in the various threads on kswapd getting stuck / exhibiting high CPU usage. We were looking at at least three root causes as far as I can see, so it's not really clear who observed which problem. Please correct me if the reported-by, tested-by, bisected-by tags are incomplete. One problem was, as it seems, overly aggressive reclaim due to scaling up reclaim goals based on compaction failures. This one was reverted in 9671009 mm: revert "mm: vmscan: scale number of pages reclaimed by reclaim/compaction based on failures". Another one was an accounting problem where a freed higher order page was underreported, and so kswapd had trouble restoring watermarks. This one was fixed in ef6c5be fix incorrect NR_FREE_PAGES accounting (appears like memory leak). The third one is a problem with small zones, like the DMA zone, where the high watermark is lower than the low watermark plus compaction gap (2 * allocation size). The zonelist reclaim in kswapd would do nothing because all high watermarks are met, but the compaction logic would find its own requirements unmet and loop over the zones again. Indefinitely, until some third party would free enough memory to help meet the higher compaction watermark. The problematic code has been there since the 3.4 merge window for non-THP higher order allocations but has been more prominent since the 3.7 merge window, where kswapd is also woken up for the much more common THP allocations. The following patch should fix the third issue by making both reclaim and compaction code in kswapd use the same predicate to determine whether a zone is balanced or not. Hopefully, the sum of all three fixes should tame kswapd enough for 3.7. Johannes -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx135.postini.com [74.125.245.135]) by kanga.kvack.org (Postfix) with SMTP id E3E0C6B005A for ; Tue, 27 Nov 2012 15:49:45 -0500 (EST) From: Johannes Weiner Subject: [patch] mm: vmscan: fix kswapd endless loop on higher order allocation Date: Tue, 27 Nov 2012 15:48:35 -0500 Message-Id: <1354049315-12874-2-git-send-email-hannes@cmpxchg.org> In-Reply-To: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: Mel Gorman , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org Kswapd does not in all places have the same criteria for a balanced zone. Zones are only being reclaimed when their high watermark is breached, but compaction checks loop over the zonelist again when the zone does not meet the low watermark plus two times the size of the allocation. This gets kswapd stuck in an endless loop over a small zone, like the DMA zone, where the high watermark is smaller than the compaction requirement. Add a function, zone_balanced(), that checks the watermark, and, for higher order allocations, if compaction has enough free memory. Then use it uniformly to check for balanced zones. This makes sure that when the compaction watermark is not met, at least reclaim happens and progress is made - or the zone is declared unreclaimable at some point and skipped entirely. Reported-by: George Spelvin Reported-by: Johannes Hirte Reported-by: Tomas Racek Signed-off-by: Johannes Weiner Tested-by: Johannes Hirte Reviewed-by: Rik van Riel --- mm/vmscan.c | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 48550c6..3b0aef4 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2397,6 +2397,19 @@ static void age_active_anon(struct zone *zone, struct scan_control *sc) } while (memcg); } +static bool zone_balanced(struct zone *zone, int order, + unsigned long balance_gap, int classzone_idx) +{ + if (!zone_watermark_ok_safe(zone, order, high_wmark_pages(zone) + + balance_gap, classzone_idx, 0)) + return false; + + if (COMPACTION_BUILD && order && !compaction_suitable(zone, order)) + return false; + + return true; +} + /* * pgdat_balanced is used when checking if a node is balanced for high-order * allocations. Only zones that meet watermarks and are in a zone allowed @@ -2475,8 +2488,7 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining, continue; } - if (!zone_watermark_ok_safe(zone, order, high_wmark_pages(zone), - i, 0)) + if (!zone_balanced(zone, order, 0, i)) all_zones_ok = false; else balanced += zone->present_pages; @@ -2585,8 +2597,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, break; } - if (!zone_watermark_ok_safe(zone, order, - high_wmark_pages(zone), 0, 0)) { + if (!zone_balanced(zone, order, 0, 0)) { end_zone = i; break; } else { @@ -2662,9 +2673,8 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, testorder = 0; if ((buffer_heads_over_limit && is_highmem_idx(i)) || - !zone_watermark_ok_safe(zone, testorder, - high_wmark_pages(zone) + balance_gap, - end_zone, 0)) { + !zone_balanced(zone, testorder, + balance_gap, end_zone)) { shrink_zone(zone, &sc); reclaim_state->reclaimed_slab = 0; @@ -2691,8 +2701,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, continue; } - if (!zone_watermark_ok_safe(zone, testorder, - high_wmark_pages(zone), end_zone, 0)) { + if (!zone_balanced(zone, testorder, 0, end_zone)) { all_zones_ok = 0; /* * We are still under min water mark. This -- 1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx172.postini.com [74.125.245.172]) by kanga.kvack.org (Postfix) with SMTP id 50C536B004D for ; Tue, 27 Nov 2012 15:58:40 -0500 (EST) Received: by mail-we0-f169.google.com with SMTP id t49so4048143wey.14 for ; Tue, 27 Nov 2012 12:58:38 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> From: Linus Torvalds Date: Tue, 27 Nov 2012 12:58:18 -0800 Message-ID: Subject: Re: kswapd craziness in 3.7 Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Andrew Morton , Mel Gorman , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List Note that in the meantime, I've also applied (through Andrew) the patch that reverts commit c654345924f7 (see commit 82b212f40059 'Revert "mm: remove __GFP_NO_KSWAPD"'). I wonder if that revert may be bogus, and a result of this same issue. Maybe that revert should be reverted, and replaced with your patch? Mel? Zdenek? What's the status here? Linus On Tue, Nov 27, 2012 at 12:48 PM, Johannes Weiner wrote: > Hi everyone, > > I hope I included everybody that participated in the various threads > on kswapd getting stuck / exhibiting high CPU usage. We were looking > at at least three root causes as far as I can see, so it's not really > clear who observed which problem. Please correct me if the > reported-by, tested-by, bisected-by tags are incomplete. > > One problem was, as it seems, overly aggressive reclaim due to scaling > up reclaim goals based on compaction failures. This one was reverted > in 9671009 mm: revert "mm: vmscan: scale number of pages reclaimed by > reclaim/compaction based on failures". > > Another one was an accounting problem where a freed higher order page > was underreported, and so kswapd had trouble restoring watermarks. > This one was fixed in ef6c5be fix incorrect NR_FREE_PAGES accounting > (appears like memory leak). > > The third one is a problem with small zones, like the DMA zone, where > the high watermark is lower than the low watermark plus compaction gap > (2 * allocation size). The zonelist reclaim in kswapd would do > nothing because all high watermarks are met, but the compaction logic > would find its own requirements unmet and loop over the zones again. > Indefinitely, until some third party would free enough memory to help > meet the higher compaction watermark. The problematic code has been > there since the 3.4 merge window for non-THP higher order allocations > but has been more prominent since the 3.7 merge window, where kswapd > is also woken up for the much more common THP allocations. > > The following patch should fix the third issue by making both reclaim > and compaction code in kswapd use the same predicate to determine > whether a zone is balanced or not. > > Hopefully, the sum of all three fixes should tame kswapd enough for > 3.7. > > Johannes > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx118.postini.com [74.125.245.118]) by kanga.kvack.org (Postfix) with SMTP id 792536B004D for ; Tue, 27 Nov 2012 16:17:10 -0500 (EST) Message-ID: <50B52DC4.5000109@redhat.com> Date: Tue, 27 Nov 2012 16:16:52 -0500 From: Rik van Riel MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Johannes Weiner , Andrew Morton , Mel Gorman , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On 11/27/2012 03:58 PM, Linus Torvalds wrote: > Note that in the meantime, I've also applied (through Andrew) the > patch that reverts commit c654345924f7 (see commit 82b212f40059 > 'Revert "mm: remove __GFP_NO_KSWAPD"'). > > I wonder if that revert may be bogus, and a result of this same issue. > Maybe that revert should be reverted, and replaced with your patch? > > Mel? Zdenek? What's the status here? Mel posted several patches to fix the kswapd issue. This one is slightly more risky than the outright revert, but probably preferred from a performance point of view: https://lkml.org/lkml/2012/11/12/151 It works by skipping the kswapd wakeup for THP allocations, only if compaction is deferred or contended. -- All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx193.postini.com [74.125.245.193]) by kanga.kvack.org (Postfix) with SMTP id 8A19C6B004D for ; Tue, 27 Nov 2012 16:30:10 -0500 (EST) Date: Tue, 27 Nov 2012 16:29:11 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121127212911.GR24381@cmpxchg.org> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Andrew Morton , Mel Gorman , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On Tue, Nov 27, 2012 at 12:58:18PM -0800, Linus Torvalds wrote: > Note that in the meantime, I've also applied (through Andrew) the > patch that reverts commit c654345924f7 (see commit 82b212f40059 > 'Revert "mm: remove __GFP_NO_KSWAPD"'). > > I wonder if that revert may be bogus, and a result of this same issue. > Maybe that revert should be reverted, and replaced with your patch? The __GFP_NO_KSWAPD removal woke kswapd for THP reclaim and so it exposed all these bugs that accumulated in there when higher order kswapd reclaim was excercised less often. The revert will hide the problem again, but doesn't make it go away entirely, so I think we need my fix either way. Whether you want to put the full THP weight back on the freshly fixed higher order kswapd code for 3.7 is a different matter :-) At least we would see quickly if it's still not working correctly... -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx150.postini.com [74.125.245.150]) by kanga.kvack.org (Postfix) with SMTP id DE1786B0062 for ; Tue, 27 Nov 2012 16:50:33 -0500 (EST) Date: Tue, 27 Nov 2012 16:49:28 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121127214928.GA20253@cmpxchg.org> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50B52DC4.5000109@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50B52DC4.5000109@redhat.com> Sender: owner-linux-mm@kvack.org List-ID: To: Rik van Riel Cc: Linus Torvalds , Andrew Morton , Mel Gorman , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On Tue, Nov 27, 2012 at 04:16:52PM -0500, Rik van Riel wrote: > On 11/27/2012 03:58 PM, Linus Torvalds wrote: > >Note that in the meantime, I've also applied (through Andrew) the > >patch that reverts commit c654345924f7 (see commit 82b212f40059 > >'Revert "mm: remove __GFP_NO_KSWAPD"'). > > > >I wonder if that revert may be bogus, and a result of this same issue. > >Maybe that revert should be reverted, and replaced with your patch? > > > >Mel? Zdenek? What's the status here? > > Mel posted several patches to fix the kswapd issue. This one is > slightly more risky than the outright revert, but probably preferred > from a performance point of view: > > https://lkml.org/lkml/2012/11/12/151 > > It works by skipping the kswapd wakeup for THP allocations, only > if compaction is deferred or contended. Just to clarify, this would be a replacement strictly for the __GFP_NO_KSWAPD removal revert, to control how often kswapd is woken up for higher order allocations like THP. My patch is to fix how kswapd actually does higher order reclaim, and it is required either way. [ But isn't the _reason_ why the "wake up kswapd more carefully for THP" patch was written kind of moot now since it was developed against a crazy kswapd? It would certainly need to be re-evaluated. My (limited) testing didn't show any issues anymore with waking kswapd unconditionally once it's fixed. ] -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx159.postini.com [74.125.245.159]) by kanga.kvack.org (Postfix) with SMTP id D689F6B004D for ; Tue, 27 Nov 2012 17:02:49 -0500 (EST) Message-ID: <50B5387C.1030005@redhat.com> Date: Tue, 27 Nov 2012 17:02:36 -0500 From: Rik van Riel MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50B52DC4.5000109@redhat.com> <20121127214928.GA20253@cmpxchg.org> In-Reply-To: <20121127214928.GA20253@cmpxchg.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Linus Torvalds , Andrew Morton , Mel Gorman , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On 11/27/2012 04:49 PM, Johannes Weiner wrote: > On Tue, Nov 27, 2012 at 04:16:52PM -0500, Rik van Riel wrote: >> On 11/27/2012 03:58 PM, Linus Torvalds wrote: >>> Note that in the meantime, I've also applied (through Andrew) the >>> patch that reverts commit c654345924f7 (see commit 82b212f40059 >>> 'Revert "mm: remove __GFP_NO_KSWAPD"'). >>> >>> I wonder if that revert may be bogus, and a result of this same issue. >>> Maybe that revert should be reverted, and replaced with your patch? >>> >>> Mel? Zdenek? What's the status here? >> >> Mel posted several patches to fix the kswapd issue. This one is >> slightly more risky than the outright revert, but probably preferred >> from a performance point of view: >> >> https://lkml.org/lkml/2012/11/12/151 >> >> It works by skipping the kswapd wakeup for THP allocations, only >> if compaction is deferred or contended. > > Just to clarify, this would be a replacement strictly for the > __GFP_NO_KSWAPD removal revert, to control how often kswapd is woken > up for higher order allocations like THP. > > My patch is to fix how kswapd actually does higher order reclaim, and > it is required either way. > > [ But isn't the _reason_ why the "wake up kswapd more carefully for > THP" patch was written kind of moot now since it was developed > against a crazy kswapd? It would certainly need to be re-evaluated. > My (limited) testing didn't show any issues anymore with waking > kswapd unconditionally once it's fixed. ] Kswapd going crazy is certainly a large part of the problem. However, that leaves the issue of page_alloc.c waking up kswapd when the system is not actually low on memory. Instead, kswapd is woken up because memory compaction failed, potentially even due to lock contention during compaction! Ideally the allocation code would only wake up kswapd if memory needs to be freed, or in order for kswapd to do memory compaction (so the allocator does not have to). -- All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx107.postini.com [74.125.245.107]) by kanga.kvack.org (Postfix) with SMTP id D2C2A6B0070 for ; Tue, 27 Nov 2012 17:27:38 -0500 (EST) Date: Tue, 27 Nov 2012 17:26:37 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121127222637.GG2301@cmpxchg.org> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50B52DC4.5000109@redhat.com> <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50B5387C.1030005@redhat.com> Sender: owner-linux-mm@kvack.org List-ID: To: Rik van Riel Cc: Linus Torvalds , Andrew Morton , Mel Gorman , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On Tue, Nov 27, 2012 at 05:02:36PM -0500, Rik van Riel wrote: > On 11/27/2012 04:49 PM, Johannes Weiner wrote: > >On Tue, Nov 27, 2012 at 04:16:52PM -0500, Rik van Riel wrote: > >>On 11/27/2012 03:58 PM, Linus Torvalds wrote: > >>>Note that in the meantime, I've also applied (through Andrew) the > >>>patch that reverts commit c654345924f7 (see commit 82b212f40059 > >>>'Revert "mm: remove __GFP_NO_KSWAPD"'). > >>> > >>>I wonder if that revert may be bogus, and a result of this same issue. > >>>Maybe that revert should be reverted, and replaced with your patch? > >>> > >>>Mel? Zdenek? What's the status here? > >> > >>Mel posted several patches to fix the kswapd issue. This one is > >>slightly more risky than the outright revert, but probably preferred > >>from a performance point of view: > >> > >>https://lkml.org/lkml/2012/11/12/151 > >> > >>It works by skipping the kswapd wakeup for THP allocations, only > >>if compaction is deferred or contended. > > > >Just to clarify, this would be a replacement strictly for the > >__GFP_NO_KSWAPD removal revert, to control how often kswapd is woken > >up for higher order allocations like THP. > > > >My patch is to fix how kswapd actually does higher order reclaim, and > >it is required either way. > > > >[ But isn't the _reason_ why the "wake up kswapd more carefully for > > THP" patch was written kind of moot now since it was developed > > against a crazy kswapd? It would certainly need to be re-evaluated. > > My (limited) testing didn't show any issues anymore with waking > > kswapd unconditionally once it's fixed. ] > > Kswapd going crazy is certainly a large part of the problem. > > However, that leaves the issue of page_alloc.c waking up > kswapd when the system is not actually low on memory. > > Instead, kswapd is woken up because memory compaction failed, > potentially even due to lock contention during compaction! > > Ideally the allocation code would only wake up kswapd if > memory needs to be freed, or in order for kswapd to do > memory compaction (so the allocator does not have to). Maybe I missed something, but shouldn't this be solved with my patch? The first scan over the zones finds the higher order watermark breached, but the reclaim scan over the zones tests against order-0 (testorder) watermarks when compaction is suitable, i.e. no reclaim if there are enough order-0 pages for compaction to work. It should just fall through to that zones_need_compaction condition at the end and run compaction. As such, it should always be approriate to wake kswapd if allocations fail. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx131.postini.com [74.125.245.131]) by kanga.kvack.org (Postfix) with SMTP id C25EA6B009C for ; Tue, 27 Nov 2012 18:20:01 -0500 (EST) Received: by mail-we0-f169.google.com with SMTP id t49so4110078wey.14 for ; Tue, 27 Nov 2012 15:20:00 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <20121127222637.GG2301@cmpxchg.org> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50B52DC4.5000109@redhat.com> <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> <20121127222637.GG2301@cmpxchg.org> From: Linus Torvalds Date: Tue, 27 Nov 2012 15:19:38 -0800 Message-ID: Subject: Re: kswapd craziness in 3.7 Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Rik van Riel , Andrew Morton , Mel Gorman , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On Tue, Nov 27, 2012 at 2:26 PM, Johannes Weiner wrote: > On Tue, Nov 27, 2012 at 05:02:36PM -0500, Rik van Riel wrote: >> >> Kswapd going crazy is certainly a large part of the problem. >> >> However, that leaves the issue of page_alloc.c waking up >> kswapd when the system is not actually low on memory. >> >> Instead, kswapd is woken up because memory compaction failed, >> potentially even due to lock contention during compaction! >> >> Ideally the allocation code would only wake up kswapd if >> memory needs to be freed, or in order for kswapd to do >> memory compaction (so the allocator does not have to). > > Maybe I missed something, but shouldn't this be solved with my patch? Ok, guys. Cage fight! The rules are simple: two men enter, one man leaves. And the one who comes out gets to explain to me which patch(es) I should apply, and which I should revert, if any. My current guess is that I should apply the one Johannes just sent ("mm: vmscan: fix kswapd endless loop on higher order allocation") after having added the cc to stable to it, and then revert the recent revert (commit 82b212f40059). But I await the Thunderdome. Linus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx142.postini.com [74.125.245.142]) by kanga.kvack.org (Postfix) with SMTP id 81DBF6B004D for ; Wed, 28 Nov 2012 04:45:19 -0500 (EST) Date: Wed, 28 Nov 2012 09:45:11 +0000 From: Mel Gorman Subject: Re: kswapd craziness in 3.7 Message-ID: <20121128094511.GS8218@suse.de> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Thorsten Leemhuis , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Jiri Slaby , Zdenek Kabelac , Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org (Adding Thorsten to cc) On Tue, Nov 27, 2012 at 03:48:34PM -0500, Johannes Weiner wrote: > Hi everyone, > > I hope I included everybody that participated in the various threads > on kswapd getting stuck / exhibiting high CPU usage. We were looking > at at least three root causes as far as I can see, so it's not really > clear who observed which problem. Please correct me if the > reported-by, tested-by, bisected-by tags are incomplete. > > One problem was, as it seems, overly aggressive reclaim due to scaling > up reclaim goals based on compaction failures. This one was reverted > in 9671009 mm: revert "mm: vmscan: scale number of pages reclaimed by > reclaim/compaction based on failures". > This particular one would have been made worse by the accounting bug and if kswapd was staying awake longer than necessary. As scaling the amount of reclaim only for direct reclaim helped this problem a lot, I strongly suspect the accounting bug was a factor. However the benefit for this is marginal -- it primarily affects how many THP pages we can allocate under stress. There is already a graceful fallback path and a system under heavy reclaim pressure is not going to notice the performance benefit of THP. > Another one was an accounting problem where a freed higher order page > was underreported, and so kswapd had trouble restoring watermarks. > This one was fixed in ef6c5be fix incorrect NR_FREE_PAGES accounting > (appears like memory leak). > This almost certainly also requires the follow-on fix at https://lkml.org/lkml/2012/11/26/225 for reasons I explained in https://lkml.org/lkml/2012/11/27/190 . > The third one is a problem with small zones, like the DMA zone, where > the high watermark is lower than the low watermark plus compaction gap > (2 * allocation size). The zonelist reclaim in kswapd would do > nothing because all high watermarks are met, but the compaction logic > would find its own requirements unmet and loop over the zones again. > Indefinitely, until some third party would free enough memory to help > meet the higher compaction watermark. The problematic code has been > there since the 3.4 merge window for non-THP higher order allocations > but has been more prominent since the 3.7 merge window, where kswapd > is also woken up for the much more common THP allocations. > Yes. > The following patch should fix the third issue by making both reclaim > and compaction code in kswapd use the same predicate to determine > whether a zone is balanced or not. > > Hopefully, the sum of all three fixes should tame kswapd enough for > 3.7. > Not exactly sure of that. With just those patches it is possible for allocations for THP entering the slow path to keep kswapd continually awake doing busy work. This was an alternative to the revert that covered that https://lkml.org/lkml/2012/11/12/151 but it was not enough because kswapd would stay awake due to the bug you identified and fixed. I went with the __GFP_NO_KSWAPD patch in this cycle because 3.6 was/is very poor in how it handles THP after the removal of lumpy reclaim. 3.7 was shaping up to be even worse with multiple root causes too close to the release date. Taking kswapd out of the equation covered some of the problems (yes, by hiding them) so it could be revisited but Johannes may have finally squashed it. However, if we revert the revert then I strongly recommend that it be replaced with "Avoid waking kswapd for THP allocations when compaction is deferred or contended". -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx177.postini.com [74.125.245.177]) by kanga.kvack.org (Postfix) with SMTP id 3C3AE6B004D for ; Wed, 28 Nov 2012 05:14:04 -0500 (EST) Date: Wed, 28 Nov 2012 10:13:59 +0000 From: Mel Gorman Subject: Re: kswapd craziness in 3.7 Message-ID: <20121128101359.GT8218@suse.de> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50B52DC4.5000109@redhat.com> <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> <20121127222637.GG2301@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Johannes Weiner , Rik van Riel , Andrew Morton , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On Tue, Nov 27, 2012 at 03:19:38PM -0800, Linus Torvalds wrote: > On Tue, Nov 27, 2012 at 2:26 PM, Johannes Weiner wrote: > > On Tue, Nov 27, 2012 at 05:02:36PM -0500, Rik van Riel wrote: > >> > >> Kswapd going crazy is certainly a large part of the problem. > >> > >> However, that leaves the issue of page_alloc.c waking up > >> kswapd when the system is not actually low on memory. > >> > >> Instead, kswapd is woken up because memory compaction failed, > >> potentially even due to lock contention during compaction! > >> > >> Ideally the allocation code would only wake up kswapd if > >> memory needs to be freed, or in order for kswapd to do > >> memory compaction (so the allocator does not have to). > > > > Maybe I missed something, but shouldn't this be solved with my patch? > > Ok, guys. Cage fight! > > The rules are simple: two men enter, one man leaves. > I'm fairly scorch damaged from this whole cycle already. I won't need a prop master to look the part for a thunderdome match. > And the one who comes out gets to explain to me which patch(es) I > should apply, and which I should revert, if any. > Based on the reports I've seen I expect the following to work for 3.7 Keep 96710098 mm: revert "mm: vmscan: scale number of pages reclaimed by reclaim/compaction based on failures" ef6c5be6 fix incorrect NR_FREE_PAGES accounting (appears like memory leak) Revert 82b212f4 Revert "mm: remove __GFP_NO_KSWAPD" Merge mm: vmscan: fix kswapd endless loop on higher order allocation mm: Avoid waking kswapd for THP allocations when compaction is deferred or contended Johannes' patch should remove the necessity for __GFP_NO_KSWAPD revert but I think we should also avoid waking kswapd for THP allocations if compaction is deferred. Johannes' patch might mean that kswapd goes quickly go back to sleep but it's still busy work. 3.6 is still known to be screwed in terms of THP because of the amount of time it can spend in compaction after lumpy reclaim was removed. This is my old list of patches I felt needed to be backported after 3.7 came out. They are not tagged -stable, I'll be sending it to Greg manually. e64c523 mm: compaction: abort compaction loop if lock is contended or run too long 3cc668f mm: compaction: move fatal signal check out of compact_checklock_irqsave 661c4cb mm: compaction: Update try_to_compact_pages()kerneldoc comment 2a1402a mm: compaction: acquire the zone->lru_lock as late as possible f40d1e4 mm: compaction: acquire the zone->lock as late as possible 753341a revert "mm: have order > 0 compaction start off where it left" bb13ffe mm: compaction: cache if a pageblock was scanned and no pages were isolated c89511a mm: compaction: Restart compaction from near where it left off 6299702 mm: compaction: clear PG_migrate_skip based on compaction and reclaim activity 0db63d7 mm: compaction: correct the nr_strict va isolated check for CMA Only Johannes' patch needs to be added to this list. kswapd is not woken for THP in 3.6 but as it calls compaction for other high-order allocations it still makes sense. -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx206.postini.com [74.125.245.206]) by kanga.kvack.org (Postfix) with SMTP id 085DF6B004D for ; Wed, 28 Nov 2012 05:51:20 -0500 (EST) Message-ID: <50B5ECA3.9040407@leemhuis.info> Date: Wed, 28 Nov 2012 11:51:15 +0100 From: Thorsten Leemhuis MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50B52DC4.5000109@redhat.com> <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> In-Reply-To: <20121128101359.GT8218@suse.de> Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: Linus Torvalds , Johannes Weiner , Rik van Riel , Andrew Morton , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List Mel Gorman wrote on 28.11.2012 11:13: > On Tue, Nov 27, 2012 at 03:19:38PM -0800, Linus Torvalds wrote: >> On Tue, Nov 27, 2012 at 2:26 PM, Johannes Weiner wrote: >> > On Tue, Nov 27, 2012 at 05:02:36PM -0500, Rik van Riel wrote: > >> And the one who comes out gets to explain to me which patch(es) I >> should apply, and which I should revert, if any. > > Based on the reports I've seen I expect the following to work for 3.7 > > Keep > 96710098 mm: revert "mm: vmscan: scale number of pages reclaimed by reclaim/compaction based on failures" > ef6c5be6 fix incorrect NR_FREE_PAGES accounting (appears like memory leak) > > Revert > 82b212f4 Revert "mm: remove __GFP_NO_KSWAPD" > > Merge > mm: vmscan: fix kswapd endless loop on higher order allocation > mm: Avoid waking kswapd for THP allocations when compaction is deferred or contended I'll build a kernel with this combination and will give it a try. Maybe one of those people that reported problems in https://bugzilla.redhat.com/show_bug.cgi?id=866988 can try them, too. There two people recently reported their problems were gone with kernels that contained 82b212f4. > Johannes' patch should remove the necessity for __GFP_NO_KSWAPD revert but I > think we should also avoid waking kswapd for THP allocations if compaction > is deferred. Johannes' patch might mean that kswapd goes quickly go back > to sleep but it's still busy work. Is there a way to trigger (some benchmark?) and detect (something in /proc/vmstat ?) the problem Hannes patch tries to fix? Background: The two main problems that got me into this discussion vanished thx to 9671009 (mm: revert "mm: vmscan: scale number of pages reclaimed by reclaim/compaction based on failures") and ef6c5be (fix incorrect NR_FREE_PAGES accounting (appears like memory leak)). I thought all my problems had gone, but after a few days of uptime (suspended and resumed the particular machine a few times in between, as I was using it just in the evenings) kswap now and then started consuming nearly 100% of one cpu core for 10 to 15 seconds intervals (it seems watching a YouTube video triggered it; and the machine was using a little bit swap space). I just had started debugging this, but due to some stupid mistake (https://plus.google.com/107616711159256259828/posts/GXuhf1LTien ) then rebooted the machine :-/ So maybe I hit the problem Hannes patch tries to solve, but I'm not sure; and I have no easy way to verify quickly if the proposed patch combination helps. Thorsten -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx127.postini.com [74.125.245.127]) by kanga.kvack.org (Postfix) with SMTP id 24CE26B0068 for ; Wed, 28 Nov 2012 08:35:48 -0500 (EST) Message-ID: <50B6131E.2020805@redhat.com> Date: Wed, 28 Nov 2012 14:35:26 +0100 From: Zdenek Kabelac MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Johannes Weiner , Andrew Morton , Mel Gorman , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Bruno Wolff III , linux-mm , Linux Kernel Mailing List Dne 27.11.2012 21:58, Linus Torvalds napsal(a): > Note that in the meantime, I've also applied (through Andrew) the > patch that reverts commit c654345924f7 (see commit 82b212f40059 > 'Revert "mm: remove __GFP_NO_KSWAPD"'). > > I wonder if that revert may be bogus, and a result of this same issue. > Maybe that revert should be reverted, and replaced with your patch? > > Mel? Zdenek? What's the status here? > I've tried for longer term: https://lkml.org/lkml/2012/11/5/308 https://lkml.org/lkml/2012/11/12/113 these 2 seems to be now merge in -rc7 (since they disappeared after my git rebase) and added slightly modified patch from Jiri (https://lkml.org/lkml/2012/11/15/950 (Unsure where it still applies for -rc7??) Also I've Jan Kara fs: Fix imbalance in freeze protection in mark_files_ro() (which is still not applied to upstream) And I think I'm NOT seeing huge load from kswapd0. (At least related to my not really long uptimes) But also I'm now frequent victim of my other report: https://lkml.org/lkml/2012/11/15/369 Which turns into a problem, that if my T61 docking station has enabled support for 'old hw' for docking in BIOS - i.e. serial output' it becomes unstable and either 1st. or 2nd. resume deadlocks machine - and serial port gives just garbage) Zdenek > Linus > > On Tue, Nov 27, 2012 at 12:48 PM, Johannes Weiner wrote: >> Hi everyone, >> >> I hope I included everybody that participated in the various threads >> on kswapd getting stuck / exhibiting high CPU usage. We were looking >> at at least three root causes as far as I can see, so it's not really >> clear who observed which problem. Please correct me if the >> reported-by, tested-by, bisected-by tags are incomplete. >> >> One problem was, as it seems, overly aggressive reclaim due to scaling >> up reclaim goals based on compaction failures. This one was reverted >> in 9671009 mm: revert "mm: vmscan: scale number of pages reclaimed by >> reclaim/compaction based on failures". >> >> Another one was an accounting problem where a freed higher order page >> was underreported, and so kswapd had trouble restoring watermarks. >> This one was fixed in ef6c5be fix incorrect NR_FREE_PAGES accounting >> (appears like memory leak). >> >> The third one is a problem with small zones, like the DMA zone, where >> the high watermark is lower than the low watermark plus compaction gap >> (2 * allocation size). The zonelist reclaim in kswapd would do >> nothing because all high watermarks are met, but the compaction logic >> would find its own requirements unmet and loop over the zones again. >> Indefinitely, until some third party would free enough memory to help >> meet the higher compaction watermark. The problematic code has been >> there since the 3.4 merge window for non-THP higher order allocations >> but has been more prominent since the 3.7 merge window, where kswapd >> is also woken up for the much more common THP allocations. >> >> The following patch should fix the third issue by making both reclaim >> and compaction code in kswapd use the same predicate to determine >> whether a zone is balanced or not. >> >> Hopefully, the sum of all three fixes should tame kswapd enough for >> 3.7. >> >> Johannes >> -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx196.postini.com [74.125.245.196]) by kanga.kvack.org (Postfix) with SMTP id 7335E6B0081 for ; Wed, 28 Nov 2012 09:05:05 -0500 (EST) Received: by mail-ee0-f41.google.com with SMTP id d41so9315224eek.14 for ; Wed, 28 Nov 2012 06:05:03 -0800 (PST) Message-ID: <50B61A0B.5080006@suse.cz> Date: Wed, 28 Nov 2012 15:04:59 +0100 From: Jiri Slaby MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50B6131E.2020805@redhat.com> In-Reply-To: <50B6131E.2020805@redhat.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Zdenek Kabelac Cc: Linus Torvalds , Johannes Weiner , Andrew Morton , Mel Gorman , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Thorsten Leemhuis , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On 11/28/2012 02:35 PM, Zdenek Kabelac wrote: > and added slightly modified patch from Jiri > (https://lkml.org/lkml/2012/11/15/950 > (Unsure where it still applies for -rc7??) It is needed for -next only. And if you have recent -next, it's already there... thanks, -- js suse labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx177.postini.com [74.125.245.177]) by kanga.kvack.org (Postfix) with SMTP id BF2EC6B006E for ; Wed, 28 Nov 2012 11:42:24 -0500 (EST) Date: Wed, 28 Nov 2012 16:42:18 +0000 From: Mel Gorman Subject: Re: kswapd craziness in 3.7 Message-ID: <20121128164217.GV8218@suse.de> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50B52DC4.5000109@redhat.com> <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <20121128101359.GT8218@suse.de> Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Johannes Weiner , Rik van Riel , Andrew Morton , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On Wed, Nov 28, 2012 at 10:13:59AM +0000, Mel Gorman wrote: > On Tue, Nov 27, 2012 at 03:19:38PM -0800, Linus Torvalds wrote: > > On Tue, Nov 27, 2012 at 2:26 PM, Johannes Weiner wrote: > > > On Tue, Nov 27, 2012 at 05:02:36PM -0500, Rik van Riel wrote: > > >> > > >> Kswapd going crazy is certainly a large part of the problem. > > >> > > >> However, that leaves the issue of page_alloc.c waking up > > >> kswapd when the system is not actually low on memory. > > >> > > >> Instead, kswapd is woken up because memory compaction failed, > > >> potentially even due to lock contention during compaction! > > >> > > >> Ideally the allocation code would only wake up kswapd if > > >> memory needs to be freed, or in order for kswapd to do > > >> memory compaction (so the allocator does not have to). > > > > > > Maybe I missed something, but shouldn't this be solved with my patch? > > > > Ok, guys. Cage fight! > > > > The rules are simple: two men enter, one man leaves. > > > > I'm fairly scorch damaged from this whole cycle already. I won't need a > prop master to look the part for a thunderdome match. > > > And the one who comes out gets to explain to me which patch(es) I > > should apply, and which I should revert, if any. > > > > Based on the reports I've seen I expect the following to work for 3.7 > > Keep > 96710098 mm: revert "mm: vmscan: scale number of pages reclaimed by reclaim/compaction based on failures" > ef6c5be6 fix incorrect NR_FREE_PAGES accounting (appears like memory leak) > > Revert > 82b212f4 Revert "mm: remove __GFP_NO_KSWAPD" > > Merge > mm: vmscan: fix kswapd endless loop on higher order allocation > mm: Avoid waking kswapd for THP allocations when compaction is deferred or contended > and mm: compaction: Fix return value of capture_free_page but this one may already be in flight from Andrew's tree as he picked it up already. -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx124.postini.com [74.125.245.124]) by kanga.kvack.org (Postfix) with SMTP id A814A6B0062 for ; Wed, 28 Nov 2012 17:52:17 -0500 (EST) Date: Wed, 28 Nov 2012 14:52:15 -0800 From: Andrew Morton Subject: Re: kswapd craziness in 3.7 Message-Id: <20121128145215.d23aeb1b.akpm@linux-foundation.org> In-Reply-To: <20121128101359.GT8218@suse.de> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50B52DC4.5000109@redhat.com> <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: Linus Torvalds , Johannes Weiner , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On Wed, 28 Nov 2012 10:13:59 +0000 Mel Gorman wrote: > Based on the reports I've seen I expect the following to work for 3.7 > > Keep > 96710098 mm: revert "mm: vmscan: scale number of pages reclaimed by reclaim/compaction based on failures" > ef6c5be6 fix incorrect NR_FREE_PAGES accounting (appears like memory leak) > > Revert > 82b212f4 Revert "mm: remove __GFP_NO_KSWAPD" > > Merge > mm: vmscan: fix kswapd endless loop on higher order allocation > mm: Avoid waking kswapd for THP allocations when compaction is deferred or contended "mm: Avoid waking kswapd for THP ..." is marked "I have not tested it myself" and when Zdenek tested it he hit an unexplained oom. > Johannes' patch should remove the necessity for __GFP_NO_KSWAPD revert but I > think we should also avoid waking kswapd for THP allocations if compaction > is deferred. Johannes' patch might mean that kswapd goes quickly go back > to sleep but it's still busy work. > > 3.6 is still known to be screwed in terms of THP because of the amount of > time it can spend in compaction after lumpy reclaim was removed. This is > my old list of patches I felt needed to be backported after 3.7 came out. > They are not tagged -stable, I'll be sending it to Greg manually. > > e64c523 mm: compaction: abort compaction loop if lock is contended or run too long > 3cc668f mm: compaction: move fatal signal check out of compact_checklock_irqsave > 661c4cb mm: compaction: Update try_to_compact_pages()kerneldoc comment > 2a1402a mm: compaction: acquire the zone->lru_lock as late as possible > f40d1e4 mm: compaction: acquire the zone->lock as late as possible > 753341a revert "mm: have order > 0 compaction start off where it left" > bb13ffe mm: compaction: cache if a pageblock was scanned and no pages were isolated > c89511a mm: compaction: Restart compaction from near where it left off > 6299702 mm: compaction: clear PG_migrate_skip based on compaction and reclaim activity > 0db63d7 mm: compaction: correct the nr_strict va isolated check for CMA > > Only Johannes' patch needs to be added to this list. kswapd is not woken > for THP in 3.6 but as it calls compaction for other high-order allocations > it still makes sense. Please identify "Johannes' patch"? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx170.postini.com [74.125.245.170]) by kanga.kvack.org (Postfix) with SMTP id 0541F6B004D for ; Wed, 28 Nov 2012 18:54:18 -0500 (EST) Date: Wed, 28 Nov 2012 23:54:12 +0000 From: Mel Gorman Subject: Re: kswapd craziness in 3.7 Message-ID: <20121128235412.GW8218@suse.de> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50B52DC4.5000109@redhat.com> <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> <20121128145215.d23aeb1b.akpm@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <20121128145215.d23aeb1b.akpm@linux-foundation.org> Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: Linus Torvalds , Johannes Weiner , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On Wed, Nov 28, 2012 at 02:52:15PM -0800, Andrew Morton wrote: > On Wed, 28 Nov 2012 10:13:59 +0000 > Mel Gorman wrote: > > > Based on the reports I've seen I expect the following to work for 3.7 > > > > Keep > > 96710098 mm: revert "mm: vmscan: scale number of pages reclaimed by reclaim/compaction based on failures" > > ef6c5be6 fix incorrect NR_FREE_PAGES accounting (appears like memory leak) > > > > Revert > > 82b212f4 Revert "mm: remove __GFP_NO_KSWAPD" > > > > Merge > > mm: vmscan: fix kswapd endless loop on higher order allocation > > mm: Avoid waking kswapd for THP allocations when compaction is deferred or contended > > "mm: Avoid waking kswapd for THP ..." is marked "I have not tested it > myself" and when Zdenek tested it he hit an unexplained oom. > I thought Zdenek was testing with __GFP_NO_KSWAPD when he hit that OOM. Further, when he hit that OOM, it looked like a genuine OOM. He had no swap configured and inactive/active file pages were very low. Finally, the free pages for Normal looked off and could also have been affected by the accounting bug. I'm looking at https://lkml.org/lkml/2012/11/18/132 here. Are you thinking of something else? I have not tested with the patch admittedly but Thorsten has and seemed to be ok with it https://lkml.org/lkml/2012/11/23/276. > > Johannes' patch should remove the necessity for __GFP_NO_KSWAPD revert but I > > think we should also avoid waking kswapd for THP allocations if compaction > > is deferred. Johannes' patch might mean that kswapd goes quickly go back > > to sleep but it's still busy work. > > > > 3.6 is still known to be screwed in terms of THP because of the amount of > > time it can spend in compaction after lumpy reclaim was removed. This is > > my old list of patches I felt needed to be backported after 3.7 came out. > > They are not tagged -stable, I'll be sending it to Greg manually. > > > > e64c523 mm: compaction: abort compaction loop if lock is contended or run too long > > 3cc668f mm: compaction: move fatal signal check out of compact_checklock_irqsave > > 661c4cb mm: compaction: Update try_to_compact_pages()kerneldoc comment > > 2a1402a mm: compaction: acquire the zone->lru_lock as late as possible > > f40d1e4 mm: compaction: acquire the zone->lock as late as possible > > 753341a revert "mm: have order > 0 compaction start off where it left" > > bb13ffe mm: compaction: cache if a pageblock was scanned and no pages were isolated > > c89511a mm: compaction: Restart compaction from near where it left off > > 6299702 mm: compaction: clear PG_migrate_skip based on compaction and reclaim activity > > 0db63d7 mm: compaction: correct the nr_strict va isolated check for CMA > > > > Only Johannes' patch needs to be added to this list. kswapd is not woken > > for THP in 3.6 but as it calls compaction for other high-order allocations > > it still makes sense. > > Please identify "Johannes' patch"? mm: vmscan: fix kswapd endless loop on higher order allocation -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx183.postini.com [74.125.245.183]) by kanga.kvack.org (Postfix) with SMTP id 678626B005A for ; Wed, 28 Nov 2012 19:14:54 -0500 (EST) Date: Wed, 28 Nov 2012 16:14:52 -0800 From: Andrew Morton Subject: Re: kswapd craziness in 3.7 Message-Id: <20121128161452.c1ddc9a3.akpm@linux-foundation.org> In-Reply-To: <20121128235412.GW8218@suse.de> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50B52DC4.5000109@redhat.com> <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: Linus Torvalds , Johannes Weiner , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On Wed, 28 Nov 2012 23:54:12 +0000 Mel Gorman wrote: > On Wed, Nov 28, 2012 at 02:52:15PM -0800, Andrew Morton wrote: > > On Wed, 28 Nov 2012 10:13:59 +0000 > > Mel Gorman wrote: > > > > > Based on the reports I've seen I expect the following to work for 3.7 > > > > > > Keep > > > 96710098 mm: revert "mm: vmscan: scale number of pages reclaimed by reclaim/compaction based on failures" > > > ef6c5be6 fix incorrect NR_FREE_PAGES accounting (appears like memory leak) > > > > > > Revert > > > 82b212f4 Revert "mm: remove __GFP_NO_KSWAPD" > > > > > > Merge > > > mm: vmscan: fix kswapd endless loop on higher order allocation > > > mm: Avoid waking kswapd for THP allocations when compaction is deferred or contended > > > > "mm: Avoid waking kswapd for THP ..." is marked "I have not tested it > > myself" and when Zdenek tested it he hit an unexplained oom. > > > > I thought Zdenek was testing with __GFP_NO_KSWAPD when he hit that OOM. > Further, when he hit that OOM, it looked like a genuine OOM. He had no > swap configured and inactive/active file pages were very low. Finally, > the free pages for Normal looked off and could also have been affected by > the accounting bug. I'm looking at https://lkml.org/lkml/2012/11/18/132 > here. Are you thinking of something else? who, me, think? I was trying to work out why I hadn't merged or queued a patch which you felt was important. Turned out it was because it didn't look very tested and final. > I have not tested with the patch admittedly but Thorsten has and seemed > to be ok with it https://lkml.org/lkml/2012/11/23/276. OK, I'll queue revert-revert-mm-remove-__gfp_no_kswapd.patch and the patch from https://patchwork.kernel.org/patch/1728081/. So what I'm currently sitting on for 3.7 is mm-compaction-fix-return-value-of-capture_free_page.patch mm-vmemmap-fix-wrong-use-of-virt_to_page.patch mm-vmscan-fix-endless-loop-in-kswapd-balancing.patch revert-revert-mm-remove-__gfp_no_kswapd.patch mm-avoid-waking-kswapd-for-thp-allocations-when-compaction-is-deferred-or-contended.patch mm-soft-offline-split-thp-at-the-beginning-of-soft_offline_page.patch > > Please identify "Johannes' patch"? > > mm: vmscan: fix kswapd endless loop on higher order allocation OK, we have that. I'll start a round of testing, do another -next drop and send the above Linuswards tomorrow. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx195.postini.com [74.125.245.195]) by kanga.kvack.org (Postfix) with SMTP id 1EFF76B0068 for ; Thu, 29 Nov 2012 10:26:47 -0500 (EST) Message-ID: <50B77F84.1030907@leemhuis.info> Date: Thu, 29 Nov 2012 16:30:12 +0100 From: Thorsten Leemhuis MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50B52DC4.5000109@redhat.com> <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> In-Reply-To: <20121128235412.GW8218@suse.de> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: Andrew Morton , Linus Torvalds , Johannes Weiner , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List Mel Gorman wrote on 29.11.2012 00:54: > On Wed, Nov 28, 2012 at 02:52:15PM -0800, Andrew Morton wrote: >> On Wed, 28 Nov 2012 10:13:59 +0000 >> Mel Gorman wrote: >> >> > Based on the reports I've seen I expect the following to work for 3.7 >> > Keep >> > 96710098 mm: revert "mm: vmscan: scale number of pages reclaimed by reclaim/compaction based on failures" >> > ef6c5be6 fix incorrect NR_FREE_PAGES accounting (appears like memory leak) >> > Revert >> > 82b212f4 Revert "mm: remove __GFP_NO_KSWAPD" >> > Merge >> > mm: vmscan: fix kswapd endless loop on higher order allocation >> > mm: Avoid waking kswapd for THP allocations when compaction is deferred or contended >> "mm: Avoid waking kswapd for THP ..." is marked "I have not tested it >> myself" and when Zdenek tested it he hit an unexplained oom. > I thought Zdenek was testing with __GFP_NO_KSWAPD when he hit that OOM. > Further, when he hit that OOM, it looked like a genuine OOM. He had no > swap configured and inactive/active file pages were very low. Finally, > the free pages for Normal looked off and could also have been affected by > the accounting bug. I'm looking at https://lkml.org/lkml/2012/11/18/132 > here. Are you thinking of something else? > > I have not tested with the patch admittedly but Thorsten has and seemed > to be ok with it https://lkml.org/lkml/2012/11/23/276. Yeah, on my two main work horses a few different kernels based on rc6 or rc7 worked fine with this patch. But sorry, it seems the patch doesn't fix the problems Fedora user John Ellson sees, who tried kernels I built in the Fedora buildsystem. Details: In https://bugzilla.redhat.com/show_bug.cgi?id=866988#c35 he mentioned his machine worked fine with a rc6 based kernel I built that contained 82b212f4 (Revert "mm: remove __GFP_NO_KSWAPD"). Before that he had tried a kernel with the same baseline that contained "Avoid waking kswapd for THP allocations when [a?|]" instead and reported it didn't help on his i686 machine (seems it helped the x86-64 one): https://bugzilla.redhat.com/show_bug.cgi?id=866988#c33 He now tried a recent mainline kernel I built 20 hours ago that is based on a git checkout from round about two days ago, reverts 82b212f4, and had * fix-kswapd-endless-loop-on-higher-order-allocation.patch * Avoid-waking-kswapd-for-THP-allocations-when.patch * mm-compaction-Fix-return-value-of-capture_free_page.patch applied. In https://bugzilla.redhat.com/show_bug.cgi?id=866988#c39 and comment 41 he reported that this kernel on his i686 host showed 100%cpu usage by kswapd0 :-/ Build log for said kernel rpms (I quite sure I applied the patches properly, but you know: mistakes happen, so be careful, maybe I did something stupid somewhere...): http://kojipkgs.fedoraproject.org//work/tasks/8253/4738253/build.log I know, this makes things more complicated again; but I wanted to let you guys know that some problem might still be lurking somewhere. Side note: right now it seems John with kernels that contain "Avoid-waking-kswapd-for-THP-allocations-when" can trigger the problem quicker (or only?) on i686 than on x86-64. CU Thorsten -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx165.postini.com [74.125.245.165]) by kanga.kvack.org (Postfix) with SMTP id 774856B0078 for ; Thu, 29 Nov 2012 12:06:11 -0500 (EST) Date: Thu, 29 Nov 2012 12:05:12 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121129170512.GI2301@cmpxchg.org> References: <50B52DC4.5000109@redhat.com> <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <50B77F84.1030907@leemhuis.info> Sender: owner-linux-mm@kvack.org List-ID: To: Thorsten Leemhuis Cc: Mel Gorman , Andrew Morton , Linus Torvalds , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List On Thu, Nov 29, 2012 at 04:30:12PM +0100, Thorsten Leemhuis wrote: > Mel Gorman wrote on 29.11.2012 00:54: > > On Wed, Nov 28, 2012 at 02:52:15PM -0800, Andrew Morton wrote: > >> On Wed, 28 Nov 2012 10:13:59 +0000 > >> Mel Gorman wrote: > >> > >> > Based on the reports I've seen I expect the following to work for 3.7 > >> > Keep > >> > 96710098 mm: revert "mm: vmscan: scale number of pages reclaimed by reclaim/compaction based on failures" > >> > ef6c5be6 fix incorrect NR_FREE_PAGES accounting (appears like memory leak) > >> > Revert > >> > 82b212f4 Revert "mm: remove __GFP_NO_KSWAPD" > >> > Merge > >> > mm: vmscan: fix kswapd endless loop on higher order allocation > >> > mm: Avoid waking kswapd for THP allocations when compaction is deferred or contended > >> "mm: Avoid waking kswapd for THP ..." is marked "I have not tested it > >> myself" and when Zdenek tested it he hit an unexplained oom. > > I thought Zdenek was testing with __GFP_NO_KSWAPD when he hit that OOM. > > Further, when he hit that OOM, it looked like a genuine OOM. He had no > > swap configured and inactive/active file pages were very low. Finally, > > the free pages for Normal looked off and could also have been affected by > > the accounting bug. I'm looking at https://lkml.org/lkml/2012/11/18/132 > > here. Are you thinking of something else? > > > > I have not tested with the patch admittedly but Thorsten has and seemed > > to be ok with it https://lkml.org/lkml/2012/11/23/276. > > Yeah, on my two main work horses a few different kernels based on rc6 or > rc7 worked fine with this patch. But sorry, it seems the patch doesn't > fix the problems Fedora user John Ellson sees, who tried kernels I built > in the Fedora buildsystem. Details: > > In https://bugzilla.redhat.com/show_bug.cgi?id=866988#c35 he mentioned > his machine worked fine with a rc6 based kernel I built that contained > 82b212f4 (Revert "mm: remove __GFP_NO_KSWAPD"). Before that he had tried > a kernel with the same baseline that contained "Avoid waking kswapd for > THP allocations when [a?|]" instead and reported it didn't help on his > i686 machine (seems it helped the x86-64 one): > https://bugzilla.redhat.com/show_bug.cgi?id=866988#c33 > > He now tried a recent mainline kernel I built 20 hours ago that is based > on a git checkout from round about two days ago, reverts 82b212f4, and had > * fix-kswapd-endless-loop-on-higher-order-allocation.patch > * Avoid-waking-kswapd-for-THP-allocations-when.patch > * mm-compaction-Fix-return-value-of-capture_free_page.patch > applied. In https://bugzilla.redhat.com/show_bug.cgi?id=866988#c39 and > comment 41 he reported that this kernel on his i686 host showed 100%cpu > usage by kswapd0 :-/ > > Build log for said kernel rpms (I quite sure I applied the patches > properly, but you know: mistakes happen, so be careful, maybe I did > something stupid somewhere...): > http://kojipkgs.fedoraproject.org//work/tasks/8253/4738253/build.log > > I know, this makes things more complicated again; but I wanted to let > you guys know that some problem might still be lurking somewhere. Side > note: right now it seems John with kernels that contain > "Avoid-waking-kswapd-for-THP-allocations-when" can trigger the problem > quicker (or only?) on i686 than on x86-64. Humm, highmem... Could this be the lowmem protection forcing kswapd to reclaim highmem at DEF_PRIORITY (not useful but burns CPU) every time it's woken up? This requires somebody to wake up kswapd regularly, though and from his report it's not quite clear to me if kswapd gets stuck or just has really high CPU usage while the system is still under load. The initial post says he would expect "<5% cpu when idling" but his top snippet in there shows there are other tasks running as well. So does it happen while the system is busy or when it's otherwise idle? [ On the other hand, not waking kswapd from THP allocations seems to not show this problem on his i686 machine. But it could also just be a tiny window of conditions aligning perfectly that drops kswapd in an endless loop, and the increased wakeups increase the probability of hitting it. So, yeah, this would be good to know. ] As the system is still responsive when this happens, any chance he could capture /proc/zoneinfo and /proc/vmstat when kswapd goes haywire? Or even run perf record -a -g sleep 5; perf report > kswapd.txt? Preferrably with this patch applied, to rule out faulty lowmem protection: buffer_heads_over_limit can put kswapd into reclaim, but it's ignored when figuring out whether the zone is balanced and so priority levels are not descended and no progress is ever made. diff --git a/mm/vmscan.c b/mm/vmscan.c index 3b0aef4..73c4f5f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2400,6 +2400,14 @@ static void age_active_anon(struct zone *zone, struct scan_control *sc) static bool zone_balanced(struct zone *zone, int order, unsigned long balance_gap, int classzone_idx) { + /* + * If the number of buffer_heads in the machine exceeds the + * maximum allowed level and this node has a highmem zone, + * force kswapd to reclaim from it to relieve lowmem pressure. + */ + if (is_highmem(zone) && buffer_heads_over_limit) + return false; + if (!zone_watermark_ok_safe(zone, order, high_wmark_pages(zone) + balance_gap, classzone_idx, 0)) return false; @@ -2586,17 +2594,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, */ age_active_anon(zone, &sc); - /* - * If the number of buffer_heads in the machine - * exceeds the maximum allowed level and this node - * has a highmem zone, force kswapd to reclaim from - * it to relieve lowmem pressure. - */ - if (buffer_heads_over_limit && is_highmem_idx(i)) { - end_zone = i; - break; - } - if (!zone_balanced(zone, order, 0, 0)) { end_zone = i; break; @@ -2672,8 +2669,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, COMPACT_SKIPPED) testorder = 0; - if ((buffer_heads_over_limit && is_highmem_idx(i)) || - !zone_balanced(zone, testorder, + if (!zone_balanced(zone, testorder, balance_gap, end_zone)) { shrink_zone(zone, &sc); -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx113.postini.com [74.125.245.113]) by kanga.kvack.org (Postfix) with SMTP id A2E016B009F for ; Fri, 30 Nov 2012 07:35:37 -0500 (EST) Message-ID: <50B8A8E7.4030108@leemhuis.info> Date: Fri, 30 Nov 2012 13:39:03 +0100 From: Thorsten Leemhuis MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <50B52DC4.5000109@redhat.com> <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> In-Reply-To: <20121129170512.GI2301@cmpxchg.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Mel Gorman , Andrew Morton , Linus Torvalds , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List Johannes Weiner wrote on 29.11.2012 18:05: > On Thu, Nov 29, 2012 at 04:30:12PM +0100, Thorsten Leemhuis wrote: >> Mel Gorman wrote on 29.11.2012 00:54: >> > On Wed, Nov 28, 2012 at 02:52:15PM -0800, Andrew Morton wrote: >> >> On Wed, 28 Nov 2012 10:13:59 +0000 >> >> Mel Gorman wrote: >> >> > Based on the reports I've seen I expect the following to work for 3.7 >> >> > Keep >> >> > 96710098 mm: revert "mm: vmscan: scale number of pages reclaimed by reclaim/compaction based on failures" >> >> > ef6c5be6 fix incorrect NR_FREE_PAGES accounting (appears like memory leak) >> >> > Revert >> >> > 82b212f4 Revert "mm: remove __GFP_NO_KSWAPD" >> >> > Merge >> >> > mm: vmscan: fix kswapd endless loop on higher order allocation >> >> > mm: Avoid waking kswapd for THP allocations when compaction is deferred or contended >> >> "mm: Avoid waking kswapd for THP ..." is marked "I have not tested it >> >> myself" and when Zdenek tested it he hit an unexplained oom. >> > I thought Zdenek was testing with __GFP_NO_KSWAPD when he hit that OOM. >> > Further, when he hit that OOM, it looked like a genuine OOM. He had no >> > swap configured and inactive/active file pages were very low. Finally, >> > the free pages for Normal looked off and could also have been affected by >> > the accounting bug. I'm looking at https://lkml.org/lkml/2012/11/18/132 >> > here. Are you thinking of something else? >> > I have not tested with the patch admittedly but Thorsten has and seemed >> > to be ok with it https://lkml.org/lkml/2012/11/23/276. >> Yeah, on my two main work horses a few different kernels based on rc6 or >> rc7 worked fine with this patch. But sorry, it seems the patch doesn't >> fix the problems Fedora user John Ellson sees, who tried kernels I built >> in the Fedora buildsystem. Details: > [...] >> I know, this makes things more complicated again; but I wanted to let >> you guys know that some problem might still be lurking somewhere. Side >> note: right now it seems John with kernels that contain >> "Avoid-waking-kswapd-for-THP-allocations-when" can trigger the problem >> quicker (or only?) on i686 than on x86-64. > > Humm, highmem... Could this be the lowmem protection forcing kswapd > to reclaim highmem at DEF_PRIORITY (not useful but burns CPU) every > time it's woken up? > > This requires somebody to wake up kswapd regularly, though and from > his report it's not quite clear to me if kswapd gets stuck or just has > really high CPU usage while the system is still under load. The > initial post says he would expect "<5% cpu when idling" but his top > snippet in there shows there are other tasks running as well. So does > it happen while the system is busy or when it's otherwise idle? > > [ On the other hand, not waking kswapd from THP allocations seems to > not show this problem on his i686 machine. But it could also just > be a tiny window of conditions aligning perfectly that drops kswapd > in an endless loop, and the increased wakeups increase the > probability of hitting it. So, yeah, this would be good to know. ] > > As the system is still responsive when this happens, any chance he > could capture /proc/zoneinfo and /proc/vmstat when kswapd goes > haywire? > > Or even run perf record -a -g sleep 5; perf report > kswapd.txt? > > Preferrably with this patch applied, to rule out faulty lowmem > protection: > > buffer_heads_over_limit can put kswapd into reclaim, but it's ignored > when figuring out whether the zone is balanced and so priority levels > are not descended and no progress is ever made. /me wonders how to elegantly get out of his man-in-the-middle position John was able to reproduce the problem quickly with a kernel that contained the patch from your mail. For details see https://bugzilla.redhat.com/show_bug.cgi?id=866988#c42 and later He provided the informations there. Parts of it: /proc/vmstat while kswad0 at 100%cpu nr_free_pages 196858 nr_inactive_anon 15804 nr_active_anon 65 nr_inactive_file 20792 nr_active_file 11307 nr_unevictable 0 nr_mlock 0 nr_anon_pages 14385 nr_mapped 2393 nr_file_pages 32563 nr_dirty 5 nr_writeback 0 nr_slab_reclaimable 3113 nr_slab_unreclaimable 4725 nr_page_table_pages 271 nr_kernel_stack 96 nr_unstable 0 nr_bounce 0 nr_vmscan_write 1487 nr_vmscan_immediate_reclaim 3 nr_writeback_temp 0 nr_isolated_anon 0 nr_isolated_file 0 nr_shmem 381 nr_dirtied 388323 nr_written 361128 nr_anon_transparent_hugepages 1 nr_free_cma 0 nr_dirty_threshold 38188 nr_dirty_background_threshold 19094 pgpgin 1057223 pgpgout 1552306 pswpin 8 pswpout 1487 pgalloc_dma 5548 pgalloc_normal 10651864 pgalloc_high 2191246 pgalloc_movable 0 pgfree 13055503 pgactivate 440358 pgdeactivate 259724 pgfault 31423675 pgmajfault 3760 pgrefill_dma 2174 pgrefill_normal 212914 pgrefill_high 51755 pgrefill_movable 0 pgsteal_kswapd_dma 1 pgsteal_kswapd_normal 202106 pgsteal_kswapd_high 36515 pgsteal_kswapd_movable 0 pgsteal_direct_dma 18 pgsteal_direct_normal 0 pgsteal_direct_high 3818 pgsteal_direct_movable 0 pgscan_kswapd_dma 1 pgscan_kswapd_normal 203044 pgscan_kswapd_high 40407 pgscan_kswapd_movable 0 pgscan_direct_dma 18 pgscan_direct_normal 0 pgscan_direct_high 4409 pgscan_direct_movable 0 pgscan_direct_throttle 0 pginodesteal 0 slabs_scanned 264192 kswapd_inodesteal 171676 kswapd_low_wmark_hit_quickly 0 kswapd_high_wmark_hit_quickly 26 kswapd_skip_congestion_wait 0 pageoutrun 117729182 allocstall 5 pgrotated 1628 compact_blocks_moved 313 compact_pages_moved 7192 compact_pagemigrate_failed 265 compact_stall 13 compact_fail 9 compact_success 4 htlb_buddy_alloc_success 0 htlb_buddy_alloc_fail 0 unevictable_pgs_culled 2985 unevictable_pgs_scanned 0 unevictable_pgs_rescued 1877 unevictable_pgs_mlocked 3965 unevictable_pgs_munlocked 3965 unevictable_pgs_cleared 0 unevictable_pgs_stranded 0 thp_fault_alloc 636 thp_fault_fallback 10 thp_collapse_alloc 342 thp_collapse_alloc_failed 2 thp_split 6 /proc/zoneinfo with kswapd0 at 100% cpu Node 0, zone DMA pages free 1655 min 196 low 245 high 294 scanned 0 spanned 4080 present 3951 nr_free_pages 1655 nr_inactive_anon 0 nr_active_anon 0 nr_inactive_file 0 nr_active_file 0 nr_unevictable 0 nr_mlock 0 nr_anon_pages 0 nr_mapped 0 nr_file_pages 0 nr_dirty 0 nr_writeback 0 nr_slab_reclaimable 3 nr_slab_unreclaimable 1 nr_page_table_pages 0 nr_kernel_stack 0 nr_unstable 0 nr_bounce 0 nr_vmscan_write 0 nr_vmscan_immediate_reclaim 0 nr_writeback_temp 0 nr_isolated_anon 0 nr_isolated_file 0 nr_shmem 0 nr_dirtied 315 nr_written 315 nr_anon_transparent_hugepages 0 nr_free_cma 0 protection: (0, 861, 1000, 1000) pagesets cpu: 0 count: 0 high: 0 batch: 1 vm stats threshold: 2 all_unreclaimable: 1 start_pfn: 16 inactive_ratio: 1 Node 0, zone Normal pages free 186234 min 10953 low 13691 high 16429 scanned 0 spanned 222206 present 220470 nr_free_pages 186234 nr_inactive_anon 3147 nr_active_anon 2 nr_inactive_file 14064 nr_active_file 4672 nr_unevictable 0 nr_mlock 0 nr_anon_pages 3028 nr_mapped 216 nr_file_pages 18857 nr_dirty 8 nr_writeback 0 nr_slab_reclaimable 3110 nr_slab_unreclaimable 4729 nr_page_table_pages 62 nr_kernel_stack 96 nr_unstable 0 nr_bounce 0 nr_vmscan_write 311 nr_vmscan_immediate_reclaim 2 nr_writeback_temp 0 nr_isolated_anon 0 nr_isolated_file 0 nr_shmem 114 nr_dirtied 339809 nr_written 315061 nr_anon_transparent_hugepages 0 nr_free_cma 0 protection: (0, 0, 1111, 1111) pagesets cpu: 0 count: 81 high: 186 batch: 31 vm stats threshold: 8 all_unreclaimable: 0 start_pfn: 4096 inactive_ratio: 1 Node 0, zone HighMem pages free 8983 min 34 low 475 high 917 scanned 0 spanned 35840 present 35560 nr_free_pages 8983 nr_inactive_anon 12661 nr_active_anon 64 nr_inactive_file 6849 nr_active_file 6500 nr_unevictable 0 nr_mlock 0 nr_anon_pages 11357 nr_mapped 2177 nr_file_pages 13692 nr_dirty 0 nr_writeback 0 nr_slab_reclaimable 0 nr_slab_unreclaimable 0 nr_page_table_pages 209 nr_kernel_stack 0 nr_unstable 0 nr_bounce 0 nr_vmscan_write 1176 nr_vmscan_immediate_reclaim 1 nr_writeback_temp 0 nr_isolated_anon 0 nr_isolated_file 0 nr_shmem 267 nr_dirtied 48189 nr_written 45739 nr_anon_transparent_hugepages 1 nr_free_cma 0 protection: (0, 0, 0, 0) pagesets cpu: 0 count: 20 high: 42 batch: 7 vm stats threshold: 4 all_unreclaimable: 0 start_pfn: 226302 inactive_ratio: 1 First few lines of /proc/vmstat while kswad0 at 100%cpu # ======== # captured on: Fri Nov 30 07:22:00 2012 # hostname : rawhide # os release : 3.7.0-0.rc7.git1.2.van.main.knurd.kswap.3.fc18.i686 # perf version : 3.7.0-0.rc7.git1.2.van.main.knurd.kswap.3.fc18.i686 # arch : i686 # nrcpus online : 1 # nrcpus avail : 1 # cpudesc : QEMU Virtual CPU version 1.0.1 # cpuid : AuthenticAMD,6,2,3 # total memory : 1027716 kB # cmdline : /usr/bin/perf record -g -a sleep 5 # event : name = cpu-clock, type = 1, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1, precise_ip = 0, id = { 7 } # HEADER_CPU_TOPOLOGY info available, use -I to display # pmu mappings: software = 1, tracepoint = 2, breakpoint = 5 # ======== # # Samples: 20K of event 'cpu-clock' # Event count (approx.): 20016 # # Overhead Command Shared Object Symbol # ........ ........... ......................... ................................... # 16.52% kswapd0 [kernel.kallsyms] [k] idr_get_next | --- idr_get_next | |--99.76%-- css_get_next | mem_cgroup_iter | | | |--50.49%-- shrink_zone | | kswapd | | kthread | | ret_from_kernel_thread | | | --49.51%-- kswapd | kthread | ret_from_kernel_thread --0.24%-- [...] 11.23% kswapd0 [kernel.kallsyms] [k] prune_super | --- prune_super | |--86.74%-- shrink_slab | kswapd | kthread | ret_from_kernel_thread | --13.26%-- kswapd kthread ret_from_kernel_thread 10.73% kswapd0 [kernel.kallsyms] [k] shrink_slab | --- shrink_slab | |--99.58%-- kswapd | kthread | ret_from_kernel_thread --0.42%-- [...] 7.36% kswapd0 [kernel.kallsyms] [k] grab_super_passive | --- grab_super_passive | |--92.46%-- prune_super | shrink_slab | kswapd | kthread | ret_from_kernel_thread | --7.54%-- shrink_slab kswapd kthread ret_from_kernel_thread 5.82% kswapd0 [kernel.kallsyms] [k] _raw_spin_lock | --- _raw_spin_lock | |--34.28%-- put_super | drop_super | prune_super | shrink_slab | kswapd | kthread | ret_from_kernel_thread | |--30.50%-- grab_super_passive | prune_super | shrink_slab | kswapd | kthread | ret_from_kernel_thread | |--17.27%-- prune_super | shrink_slab | kswapd | kthread | ret_from_kernel_thread | |--16.15%-- drop_super | prune_super | shrink_slab | kswapd | kthread | ret_from_kernel_thread | |--1.20%-- mb_cache_shrink_fn | shrink_slab | kswapd | kthread | ret_from_kernel_thread | --0.60%-- shrink_slab kswapd kthread ret_from_kernel_thread 4.43% kswapd0 [kernel.kallsyms] [k] fill_contig_page_info | --- fill_contig_page_info | |--99.10%-- fragmentation_index | compaction_suitable | kswapd | kthread | ret_from_kernel_thread | --0.90%-- compaction_suitable kswapd kthread ret_from_kernel_thread 3.81% kswapd0 [kernel.kallsyms] [k] shrink_lruvec | --- shrink_lruvec | |--99.34%-- shrink_zone | kswapd | kthread | ret_from_kernel_thread | --0.66%-- kswapd kthread ret_from_kernel_thread The rest at https://bugzilla.redhat.com/attachment.cgi?id=654977 CU Thorsten -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx117.postini.com [74.125.245.117]) by kanga.kvack.org (Postfix) with SMTP id 33CD66B004D for ; Fri, 30 Nov 2012 19:46:33 -0500 (EST) Date: Fri, 30 Nov 2012 19:45:20 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121201004520.GK2301@cmpxchg.org> References: <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50B8A8E7.4030108@leemhuis.info> Sender: owner-linux-mm@kvack.org List-ID: To: Thorsten Leemhuis Cc: Mel Gorman , Andrew Morton , Linus Torvalds , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List , John Ellson Hi Thorsten, On Fri, Nov 30, 2012 at 01:39:03PM +0100, Thorsten Leemhuis wrote: > /me wonders how to elegantly get out of his man-in-the-middle position You control the mighty koji :-) But seriously, this is very helpful, thank you! John now also Cc'd directly. > John was able to reproduce the problem quickly with a kernel that > contained the patch from your mail. For details see > > https://bugzilla.redhat.com/show_bug.cgi?id=866988#c42 and later > > He provided the informations there. Parts of it: > /proc/vmstat while kswad0 at 100%cpu > /proc/zoneinfo with kswapd0 at 100% cpu > perf profile Thanks. I'm quoting the interesting bits in order of the cars on my possibly derailing train of thought: > pageoutrun 117729182 > allocstall 5 Okay, so kswapd is stupidly looping but it's still managing to do it's actual job; nobody is dropping into direct reclaim. > pgsteal_kswapd_dma 1 > pgsteal_kswapd_normal 202106 > pgsteal_kswapd_high 36515 > pgsteal_kswapd_movable 0 > pgscan_kswapd_dma 1 > pgscan_kswapd_normal 203044 > pgscan_kswapd_high 40407 > pgscan_kswapd_movable 0 Does not seem excessive, so apparently it also does not overreclaim. > Node 0, zone DMA > pages free 1655 > min 196 > low 245 > high 294 > Node 0, zone Normal > pages free 186234 > min 10953 > low 13691 > high 16429 > Node 0, zone HighMem > pages free 8983 > min 34 > low 475 > high 917 These are all well above their watermarks, yet kswapd is definitely finding something wrong with one of these as it actually does drop into the reclaim loop, so zone_balanced() must be returning false: > 16.52% kswapd0 [kernel.kallsyms] [k] idr_get_next > | > --- idr_get_next > | > |--99.76%-- css_get_next > | mem_cgroup_iter > | | > | |--50.49%-- shrink_zone > | | kswapd > | | kthread > | | ret_from_kernel_thread > | | > | --49.51%-- kswapd > | kthread > | ret_from_kernel_thread > --0.24%-- [...] > > 11.23% kswapd0 [kernel.kallsyms] [k] prune_super > | > --- prune_super > | > |--86.74%-- shrink_slab > | kswapd > | kthread > | ret_from_kernel_thread > | > --13.26%-- kswapd > kthread > ret_from_kernel_thread Spending so much time in shrink_zone and shrink_slab without overreclaiming a zone, I would say that a) this always stays on the DEF_PRIORITY and b) only loops on the DMA zone. At DEF_PRIORITY, the scan goal for filepages in the other zones would be > 0 e.g. As the DMA zone watermarks are fine, it must be the fragmentation index that indicates a lack of memory. Filling in the 1655 free pages into the fragmentation index formula indicates lack of free memory when these 1655 pages are lumped together in less than 9 page blocks. Not unrealistic, I think: on my desktop machine, the DMA zone's free 3975 pages are lumped together in only 12 blocks. But on my system, the DMA zone is either never used and there is always at least one page block available that could satisfy a huge page allocation (fragmentation index == -1000). Unless the system gets really close to OOM, at which point the DMA zone is highly fragmented. And keep in mind that if the priority level goes below DEF_PRIORITY, as it does close to OOM, the unreclaimable DMA zone is ignored anyway. But the DMA zone here is just barely used: > Node 0, zone DMA [...] > nr_slab_reclaimable 3 > nr_slab_unreclaimable 1 [...] > nr_dirtied 315 > nr_written 315 which could explain a fragmentation index that asks for more free memory while the watermarks are fine. Why this all loops: there is one more inconsistency where the conditions for reclaim and the conditions for compaction contradict each other: reclaim also does not consider the DMA zone balanced, but it needs only 25% of the whole node to be balanced, while compaction requires every single zone to be balanced individually. So these strict per-zone checks for compaction at the end of balance_pgdat() are likely to be the culprits that keep kswapd looping forever on this machine, trying to balance DMA for compaction while reclaim decides it has enough balanced memory in the node overall. I think we can just remove them: whenever the compaction code is reached, the reclaim code balanced 25% of the memory available for the classzone to be suitable for compaction. Mel? Rik? --- From: Johannes Weiner Subject: [patch] mm: vmscan: do not keep kswapd looping forever due to individual uncompactable zones When a zone meets its high watermark and is compactable in case of higher order allocations, it contributes to the percentage of the node's memory that is considered balanced. This requirement, that a node be only partially balanced, came about when kswapd was desparately trying to balance tiny zones when all bigger zones in the node had plenty of free memory. Arguably, the same should apply to compaction: if a significant part of the node is balanced enough to run compaction, do not get hung up on that tiny zone that might never get in shape. When the compaction logic in kswapd is reached, we know that at least 25% of the node's memory is balanced properly for compaction (see zone_balanced and pgdat_balanced). Remove the individual zone checks that restart the kswapd cycle. Otherwise, we may observe more endless looping in kswapd where the compaction code loops back to reclaim because of a single zone and reclaim does nothing because the node is considered balanced overall. Reported-by: Thorsten Leemhuis Signed-off-by: Johannes Weiner --- mm/vmscan.c | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 3b0aef4..486100f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2806,22 +2806,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, if (!populated_zone(zone)) continue; - if (zone->all_unreclaimable && - sc.priority != DEF_PRIORITY) - continue; - - /* Would compaction fail due to lack of free memory? */ - if (COMPACTION_BUILD && - compaction_suitable(zone, order) == COMPACT_SKIPPED) - goto loop_again; - - /* Confirm the zone is balanced for order-0 */ - if (!zone_watermark_ok(zone, 0, - high_wmark_pages(zone), 0, 0)) { - order = sc.order = 0; - goto loop_again; - } - /* Check if the memory needs to be defragmented. */ if (zone_watermark_ok(zone, order, low_wmark_pages(zone), *classzone_idx, 0)) -- 1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx192.postini.com [74.125.245.192]) by kanga.kvack.org (Postfix) with SMTP id 88F0A6B002B for ; Mon, 3 Dec 2012 03:26:42 -0500 (EST) Message-ID: <50BC6314.7060106@leemhuis.info> Date: Mon, 03 Dec 2012 09:30:12 +0100 From: Thorsten Leemhuis MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <20121127214928.GA20253@cmpxchg.org> <50B5387C.1030005@redhat.com> <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> In-Reply-To: <20121201004520.GK2301@cmpxchg.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Mel Gorman , Andrew Morton , Linus Torvalds , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List , John Ellson Hi! Johannes Weiner wrote on 01.12.2012 01:45: > On Fri, Nov 30, 2012 at 01:39:03PM +0100, Thorsten Leemhuis wrote: >> /me wonders how to elegantly get out of his man-in-the-middle position > You control the mighty koji :-) Something even a journalist can ;-) > But seriously, this is very helpful, thank you! Np; BTW, in case anybody here on LKML cares: I started maintaining a side repo (PPA in ubuntu speak) a few weeks ago that offers kernel vanilla builds (mainline and stable) for the Fedora 17 and 18; see https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories for details. It's not as good and up2date yet as I would like it, but one has to start somewhere. Back to topic: > John now also Cc'd directly. > >> John was able to reproduce the problem quickly with a kernel that >> contained the patch from your mail. For details see > > [stripped: all the glory details of what likely went wrong and lead > to the problem john sees or saw] > > --- > From: Johannes Weiner > Subject: [patch] mm: vmscan: do not keep kswapd looping forever due > to individual uncompactable zones > > When a zone meets its high watermark and is compactable in case of > higher order allocations, it contributes to the percentage of the > node's memory that is considered balanced. > [...] FYI: I built a kernel with that patch. I've been running on my x86_64 machine at home over the weekend and everything was working fine (just as without the patch). John gave it a quick try and in https://bugzilla.redhat.com/show_bug.cgi?id=866988#c57 reported: """ I just installed kernel-3.7.0-0.rc7.git1.2.van.main.knurd.kswap.4.fc18.i686 and ran my usual load that triggers the problem. OK so far. I'll check again in 24hours, but looking good so far. """ BTW, I built that kernel without the patch you mentioned in http://thread.gmane.org/gmane.linux.kernel.mm/90911/focus=91153 ("buffer_heads_over_limit can put kswapd into reclaim, but it's ignored [...]) It looked to me like that patch was only meant for debugging. Let me know if that was wrong. Ohh, and I didn't update to a fresher mainline checkout yet to make sure the base for John's testing didn't change. CU Thorsten -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx104.postini.com [74.125.245.104]) by kanga.kvack.org (Postfix) with SMTP id B93B56B004D for ; Mon, 3 Dec 2012 08:08:37 -0500 (EST) Date: Mon, 3 Dec 2012 14:08:34 +0100 From: Borislav Petkov Subject: Fedora repo (was: Re: kswapd craziness in 3.7) Message-ID: <20121203130834.GB32243@liondog.tnic> References: <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <50BC6314.7060106@leemhuis.info> Sender: owner-linux-mm@kvack.org List-ID: To: Thorsten Leemhuis Cc: Johannes Weiner , Mel Gorman , Andrew Morton , Linus Torvalds , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List , John Ellson On Mon, Dec 03, 2012 at 09:30:12AM +0100, Thorsten Leemhuis wrote: > Np; BTW, in case anybody here on LKML cares: I started maintaining a > side repo (PPA in ubuntu speak) a few weeks ago that offers kernel > vanilla builds (mainline and stable) for the Fedora 17 and 18; see > https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories > for details. It's not as good and up2date yet as I would like it, but > one has to start somewhere. Once you have this ready, you should send a more official mail with "[ANNOUNCE]" in its subject and containing explanations how to use the repo to lkml and relevant lists so that more people know about it. Thanks. -- Regards/Gruss, Boris. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx193.postini.com [74.125.245.193]) by kanga.kvack.org (Postfix) with SMTP id ECB496B004D for ; Mon, 3 Dec 2012 08:14:10 -0500 (EST) Received: by mail-ee0-f41.google.com with SMTP id d41so1928575eek.14 for ; Mon, 03 Dec 2012 05:14:09 -0800 (PST) Message-ID: <50BCA59D.6040704@suse.cz> Date: Mon, 03 Dec 2012 14:14:05 +0100 From: Jiri Slaby MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> In-Reply-To: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Andrew Morton , Mel Gorman , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org On 11/27/2012 09:48 PM, Johannes Weiner wrote: > I hope I included everybody that participated in the various threads > on kswapd getting stuck / exhibiting high CPU usage. We were looking > at at least three root causes as far as I can see, so it's not really > clear who observed which problem. Please correct me if the > reported-by, tested-by, bisected-by tags are incomplete. Hi, I reported the problem for the first time but I got lost in the patches flying around very early. Whatever is in the current -next, works for me since -next was resurrected after the 2 weeks gap last week... thanks, -- js suse labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx133.postini.com [74.125.245.133]) by kanga.kvack.org (Postfix) with SMTP id 16E956B002B for ; Mon, 3 Dec 2012 10:23:33 -0500 (EST) Message-ID: <50BCC3E3.40804@redhat.com> Date: Mon, 03 Dec 2012 16:23:15 +0100 From: Zdenek Kabelac MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <20121128094511.GS8218@suse.de> In-Reply-To: <20121128094511.GS8218@suse.de> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: Johannes Weiner , Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Thorsten Leemhuis , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Jiri Slaby , Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org Dne 28.11.2012 10:45, Mel Gorman napsal(a): > (Adding Thorsten to cc) > > On Tue, Nov 27, 2012 at 03:48:34PM -0500, Johannes Weiner wrote: >> Hi everyone, >> >> I hope I included everybody that participated in the various threads >> on kswapd getting stuck / exhibiting high CPU usage. We were looking >> at at least three root causes as far as I can see, so it's not really >> clear who observed which problem. Please correct me if the >> reported-by, tested-by, bisected-by tags are incomplete. >> >> One problem was, as it seems, overly aggressive reclaim due to scaling >> up reclaim goals based on compaction failures. This one was reverted >> in 9671009 mm: revert "mm: vmscan: scale number of pages reclaimed by >> reclaim/compaction based on failures". >> > > This particular one would have been made worse by the accounting bug and > if kswapd was staying awake longer than necessary. As scaling the amount > of reclaim only for direct reclaim helped this problem a lot, I strongly > suspect the accounting bug was a factor. > > However the benefit for this is marginal -- it primarily affects how > many THP pages we can allocate under stress. There is already a graceful > fallback path and a system under heavy reclaim pressure is not going to > notice the performance benefit of THP. > >> Another one was an accounting problem where a freed higher order page >> was underreported, and so kswapd had trouble restoring watermarks. >> This one was fixed in ef6c5be fix incorrect NR_FREE_PAGES accounting >> (appears like memory leak). >> > > This almost certainly also requires the follow-on fix at > https://lkml.org/lkml/2012/11/26/225 for reasons I explained in > https://lkml.org/lkml/2012/11/27/190 . > >> The third one is a problem with small zones, like the DMA zone, where >> the high watermark is lower than the low watermark plus compaction gap >> (2 * allocation size). The zonelist reclaim in kswapd would do >> nothing because all high watermarks are met, but the compaction logic >> would find its own requirements unmet and loop over the zones again. >> Indefinitely, until some third party would free enough memory to help >> meet the higher compaction watermark. The problematic code has been >> there since the 3.4 merge window for non-THP higher order allocations >> but has been more prominent since the 3.7 merge window, where kswapd >> is also woken up for the much more common THP allocations. >> > > Yes. > >> The following patch should fix the third issue by making both reclaim >> and compaction code in kswapd use the same predicate to determine >> whether a zone is balanced or not. >> >> Hopefully, the sum of all three fixes should tame kswapd enough for >> 3.7. >> > > Not exactly sure of that. With just those patches it is possible for > allocations for THP entering the slow path to keep kswapd continually awake > doing busy work. This was an alternative to the revert that covered that > https://lkml.org/lkml/2012/11/12/151 but it was not enough because kswapd > would stay awake due to the bug you identified and fixed. > > I went with the __GFP_NO_KSWAPD patch in this cycle because 3.6 was/is > very poor in how it handles THP after the removal of lumpy reclaim. 3.7 > was shaping up to be even worse with multiple root causes too close to the > release date. Taking kswapd out of the equation covered some of the > problems (yes, by hiding them) so it could be revisited but Johannes may > have finally squashed it. > > However, if we revert the revert then I strongly recommend that it be > replaced with "Avoid waking kswapd for THP allocations when compaction is > deferred or contended". > Ok, bad news - I've been hit by kswapd0 loop again - my kernel git commit cc19528bd3084c3c2d870b31a3578da8c69952f3 again shown kswapd0 for couple minutes on CPU. It seemed to go instantly away when I've drop caches (echo 3 >/proc/sys/vm/drop_cache) (After that I've had over 1G free memory) Here are some stats before drop while kswapd0 was running: kswapd0 R running task 0 30 2 0x00000000 ffff880133207b08 0000000000000082 ffff880133207b18 0000000000000246 ffff880135b92340 ffff880133207fd8 ffff880133207fd8 ffff880133207fd8 ffff880103098000 ffff880135b92340 0000000000000000 ffff880133206000 Call Trace: [] preempt_schedule+0x42/0x60 [] _raw_spin_unlock+0x55/0x60 [] grab_super_passive+0x3c/0x90 [] prune_super+0x46/0x1b0 [] shrink_slab+0xba/0x510 [] ? mem_cgroup_iter+0x17a/0x2e0 [] ? mem_cgroup_iter+0xca/0x2e0 [] balance_pgdat+0x621/0x7e0 [] kswapd+0x174/0x640 [] ? __init_waitqueue_head+0x60/0x60 [] ? balance_pgdat+0x7e0/0x7e0 [] kthread+0xdb/0xe0 [] ? kthread_create_on_node+0x140/0x140 [] ret_from_fork+0x7c/0xb0 [] ? kthread_create_on_node+0x140/0x140 runnable tasks: task PID tree-key switches prio exec-runtime sum-exec sum-sleep ---------------------------------------------------------------------------------------------------------- kswapd0 30 8087056.792356 30543 120 8087056.792356 158938.479290 137131605.711862 / kworker/0:3 29833 8087050.792356 526664 120 8087050.792356 24710.527691 24775203.529553 / R bash 24767 43813.836355 121 120 43813.836355 40.855087 10579.107486 /autogroup-392 ---- Showing all locks held in the system: 1 lock held by bash/10668: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x610/0x990 1 lock held by bash/10756: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x610/0x990 1 lock held by bash/26989: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x610/0x990 1 lock held by less/10268: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x610/0x990 1 lock held by less/19112: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x610/0x990 1 lock held by bash/13774: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x610/0x990 1 lock held by less/32444: #0: (&tty->atomic_read_lock){+.+.+.}, at: [] n_tty_read+0x610/0x990 2 locks held by bash/24767: #0: (sysrq_key_table_lock){......}, at: [] __handle_sysrq+0x33/0x190 #1: (tasklist_lock){.+.+..}, at: [] debug_show_all_locks+0x43/0x2a0 ============================================= SysRq : HELP : loglevel(0-9) reBoot Crash show-all-locks(D) terminate-all-tasks(E) memory-full-oom-kill(F) kill-all-tasks(I) thaw-filesystems(J) saK show-backtrace-all-active-cpus(L) show-memory-usage(M) nice-all-RT-tasks(N) powerOff show-registers(P) show-all-timers(Q) unRaw Sync show-task-states(T) Unmount force-fb(V) show-blocked-tasks(W) dump-ftrace-buffer(Z) SysRq : Show Memory Mem-Info: DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 147 CPU 1: hi: 186, btch: 31 usd: 157 Normal per-cpu: CPU 0: hi: 186, btch: 31 usd: 154 CPU 1: hi: 186, btch: 31 usd: 182 active_anon:610014 inactive_anon:16551 isolated_anon:0 active_file:83258 inactive_file:151927 isolated_file:0 unevictable:16 dirty:12 writeback:0 unstable:0 free:72021 slab_reclaimable:18685 slab_unreclaimable:13682 mapped:23445 shmem:29913 pagetables:7689 bounce:0 free_cma:0 DMA free:15892kB min:260kB low:324kB high:388kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15644kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 2982 3927 3927 DMA32 free:251976kB min:51124kB low:63904kB high:76684kB active_anon:1738128kB inactive_anon:58108kB active_file:316652kB inactive_file:591328kB unevictable:16kB isolated(anon):0kB isolated(file):0kB present:3054528kB mlocked:16kB dirty:40kB writeback:0kB mapped:58684kB shmem:108216kB slab_reclaimable:38888kB slab_unreclaimable:15988kB kernel_stack:1416kB pagetables:8684kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 0 945 945 Normal free:20216kB min:16196kB low:20244kB high:24292kB active_anon:701928kB inactive_anon:8096kB active_file:16380kB inactive_file:16380kB unevictable:48kB isolated(anon):0kB isolated(file):0kB present:967680kB mlocked:48kB dirty:8kB writeback:0kB mapped:35096kB shmem:11436kB slab_reclaimable:35852kB slab_unreclaimable:38732kB kernel_stack:3200kB pagetables:22072kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:42 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 DMA: 1*4kB 0*8kB 1*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15892kB DMA32: 56*4kB 577*8kB 754*16kB 1192*32kB 713*64kB 484*128kB 223*256kB 57*512kB 1*1024kB 1*2048kB 0*4096kB = 251976kB Normal: 526*4kB 350*8kB 181*16kB 152*32kB 66*64kB 18*128kB 2*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 20216kB 265099 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 0kB Total swap = 0kB 1032176 pages RAM 42790 pages reserved 672981 pages shared 820401 pages non-shared vmstat: nr_free_pages 72360 nr_inactive_anon 16501 nr_active_anon 609811 nr_inactive_file 151932 nr_active_file 83212 nr_unevictable 16 nr_mlock 16 nr_anon_pages 503314 nr_mapped 23443 nr_file_pages 264982 nr_dirty 234 nr_writeback 0 nr_slab_reclaimable 18685 nr_slab_unreclaimable 13682 nr_page_table_pages 7690 nr_kernel_stack 577 nr_unstable 0 nr_bounce 0 nr_vmscan_write 0 nr_vmscan_immediate_reclaim 29 nr_writeback_temp 0 nr_isolated_anon 0 nr_isolated_file 0 nr_shmem 29838 nr_dirtied 2206202 nr_written 2066654 nr_anon_transparent_hugepages 182 nr_free_cma 0 nr_dirty_threshold 13870 nr_dirty_background_threshold 6935 pgpgin 3224666 pgpgout 9329522 pswpin 0 pswpout 0 pgalloc_dma 2 pgalloc_dma32 100605413 pgalloc_normal 25009399 pgalloc_movable 0 pgfree 126647271 pgactivate 1185101 pgdeactivate 214747 pgfault 106494704 pgmajfault 9834 pgrefill_dma 0 pgrefill_dma32 99747 pgrefill_normal 232841 pgrefill_movable 0 pgsteal_kswapd_dma 0 pgsteal_kswapd_dma32 208294 pgsteal_kswapd_normal 162100 pgsteal_kswapd_movable 0 pgsteal_direct_dma 0 pgsteal_direct_dma32 11942 pgsteal_direct_normal 91155 pgsteal_direct_movable 0 pgscan_kswapd_dma 0 pgscan_kswapd_dma32 211693 pgscan_kswapd_normal 182157 pgscan_kswapd_movable 0 pgscan_direct_dma 0 pgscan_direct_dma32 12129 pgscan_direct_normal 96028 pgscan_direct_movable 0 pgscan_direct_throttle 0 pginodesteal 77546 slabs_scanned 784384 kswapd_inodesteal 47090 kswapd_low_wmark_hit_quickly 57 kswapd_high_wmark_hit_quickly 275 kswapd_skip_congestion_wait 0 pageoutrun 1636173 allocstall 175 pgrotated 73 compact_blocks_moved 80209 compact_pages_moved 345293 compact_pagemigrate_failed 64875 compact_stall 736 compact_fail 314 compact_success 422 htlb_buddy_alloc_success 0 htlb_buddy_alloc_fail 0 unevictable_pgs_culled 2848 unevictable_pgs_scanned 0 unevictable_pgs_rescued 3330 unevictable_pgs_mlocked 3346 unevictable_pgs_munlocked 3330 unevictable_pgs_cleared 0 unevictable_pgs_stranded 0 thp_fault_alloc 53631 thp_fault_fallback 1682 thp_collapse_alloc 13390 thp_collapse_alloc_failed 643 thp_split 2387 Zdenek -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx166.postini.com [74.125.245.166]) by kanga.kvack.org (Postfix) with SMTP id 57F2C6B0070 for ; Mon, 3 Dec 2012 14:20:04 -0500 (EST) Date: Mon, 3 Dec 2012 14:18:58 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121203191858.GY24381@cmpxchg.org> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <20121128094511.GS8218@suse.de> <50BCC3E3.40804@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50BCC3E3.40804@redhat.com> Sender: owner-linux-mm@kvack.org List-ID: To: Zdenek Kabelac Cc: Mel Gorman , Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Thorsten Leemhuis , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Jiri Slaby , Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org Szia Zdenek, On Mon, Dec 03, 2012 at 04:23:15PM +0100, Zdenek Kabelac wrote: > Ok, bad news - I've been hit by kswapd0 loop again - > my kernel git commit cc19528bd3084c3c2d870b31a3578da8c69952f3 again > shown kswapd0 for couple minutes on CPU. > > It seemed to go instantly away when I've drop caches > (echo 3 >/proc/sys/vm/drop_cache) > (After that I've had over 1G free memory) Any chance you could retry with this patch on top? Thanks! --- From: Johannes Weiner Subject: [patch] mm: vmscan: do not keep kswapd looping forever due to individual uncompactable zones When a zone meets its high watermark and is compactable in case of higher order allocations, it contributes to the percentage of the node's memory that is considered balanced. This requirement, that a node be only partially balanced, came about when kswapd was desparately trying to balance tiny zones when all bigger zones in the node had plenty of free memory. Arguably, the same should apply to compaction: if a significant part of the node is balanced enough to run compaction, do not get hung up on that tiny zone that might never get in shape. When the compaction logic in kswapd is reached, we know that at least 25% of the node's memory is balanced properly for compaction (see zone_balanced and pgdat_balanced). Remove the individual zone checks that restart the kswapd cycle. Otherwise, we may observe more endless looping in kswapd where the compaction code loops back to reclaim because of a single zone and reclaim does nothing because the node is considered balanced overall. Reported-by: Thorsten Leemhuis Signed-off-by: Johannes Weiner --- mm/vmscan.c | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 3b0aef4..486100f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2806,22 +2806,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, if (!populated_zone(zone)) continue; - if (zone->all_unreclaimable && - sc.priority != DEF_PRIORITY) - continue; - - /* Would compaction fail due to lack of free memory? */ - if (COMPACTION_BUILD && - compaction_suitable(zone, order) == COMPACT_SKIPPED) - goto loop_again; - - /* Confirm the zone is balanced for order-0 */ - if (!zone_watermark_ok(zone, 0, - high_wmark_pages(zone), 0, 0)) { - order = sc.order = 0; - goto loop_again; - } - /* Check if the memory needs to be defragmented. */ if (zone_watermark_ok(zone, order, low_wmark_pages(zone), *classzone_idx, 0)) -- 1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx193.postini.com [74.125.245.193]) by kanga.kvack.org (Postfix) with SMTP id EF1C56B004D for ; Mon, 3 Dec 2012 14:43:05 -0500 (EST) Date: Mon, 3 Dec 2012 14:42:08 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121203194208.GZ24381@cmpxchg.org> References: <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50BC6314.7060106@leemhuis.info> Sender: owner-linux-mm@kvack.org List-ID: To: Thorsten Leemhuis Cc: Mel Gorman , Andrew Morton , Linus Torvalds , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List , John Ellson On Mon, Dec 03, 2012 at 09:30:12AM +0100, Thorsten Leemhuis wrote: > >> John was able to reproduce the problem quickly with a kernel that > >> contained the patch from your mail. For details see > > > > [stripped: all the glory details of what likely went wrong and lead > > to the problem john sees or saw] > > > > --- > > From: Johannes Weiner > > Subject: [patch] mm: vmscan: do not keep kswapd looping forever due > > to individual uncompactable zones > > > > When a zone meets its high watermark and is compactable in case of > > higher order allocations, it contributes to the percentage of the > > node's memory that is considered balanced. > > [...] > > FYI: I built a kernel with that patch. I've been running on my x86_64 > machine at home over the weekend and everything was working fine (just > as without the patch). John gave it a quick try and in > https://bugzilla.redhat.com/show_bug.cgi?id=866988#c57 reported: > > """ > I just installed > kernel-3.7.0-0.rc7.git1.2.van.main.knurd.kswap.4.fc18.i686 and ran my > usual load that triggers the problem. OK so far. I'll check again in > 24hours, but looking good so far. > """ w00t! > BTW, I built that kernel without the patch you mentioned in > http://thread.gmane.org/gmane.linux.kernel.mm/90911/focus=91153 > ("buffer_heads_over_limit can put kswapd into reclaim, but it's ignored > [...]) It looked to me like that patch was only meant for debugging. Let > me know if that was wrong. Ohh, and I didn't update to a fresher > mainline checkout yet to make sure the base for John's testing didn't > change. Ah, yes, the ApplyPatch is commented out. I think we want that upstream as well, but it's not critical. It'll reduce kswapd CPU usage marginally on highmem systems in certain situations, but I don't think any of the 100% CPU usage problems are fixed by it. Not rebasing sounds reasonable to me to verify the patch. It might be worth testing that the final version that will be 3.8 still works for John, however, once that is done. Just to be sure. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx130.postini.com [74.125.245.130]) by kanga.kvack.org (Postfix) with SMTP id 9771E6B0062 for ; Tue, 4 Dec 2012 03:55:07 -0500 (EST) Received: by mail-ee0-f41.google.com with SMTP id d41so2560666eek.14 for ; Tue, 04 Dec 2012 00:55:06 -0800 (PST) Message-ID: <50BDBA64.8080404@suse.cz> Date: Tue, 04 Dec 2012 09:55:00 +0100 From: Jiri Slaby MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <50BCA59D.6040704@suse.cz> In-Reply-To: <50BCA59D.6040704@suse.cz> Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Andrew Morton , Mel Gorman , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Thorsten Leemhuis , Zdenek Kabelac , Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org On 12/03/2012 02:14 PM, Jiri Slaby wrote: > On 11/27/2012 09:48 PM, Johannes Weiner wrote: >> I hope I included everybody that participated in the various threads >> on kswapd getting stuck / exhibiting high CPU usage. We were looking >> at at least three root causes as far as I can see, so it's not really >> clear who observed which problem. Please correct me if the >> reported-by, tested-by, bisected-by tags are incomplete. > > Hi, I reported the problem for the first time but I got lost in the > patches flying around very early. > > Whatever is in the current -next, works for me since -next was > resurrected after the 2 weeks gap last week... Bah, I always need to write an email to reproduce that. It's back: 3.7.0-rc7-next-20121130 [] __cond_resched+0x2a/0x40 [] shrink_slab+0x1c0/0x2d0 [] kswapd+0x65d/0xb50 [] kthread+0xc0/0xd0 [] ret_from_fork+0x7c/0xb0 [] 0xffffffffffffffff Going to apply this: https://lkml.org/lkml/2012/12/3/407 and wait another 5 days to see the results... thanks, -- js suse labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx203.postini.com [74.125.245.203]) by kanga.kvack.org (Postfix) with SMTP id C60D46B005D for ; Tue, 4 Dec 2012 04:05:46 -0500 (EST) Message-ID: <50BDBCD9.9060509@redhat.com> Date: Tue, 04 Dec 2012 10:05:29 +0100 From: Zdenek Kabelac MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <20121128094511.GS8218@suse.de> <50BCC3E3.40804@redhat.com> <20121203191858.GY24381@cmpxchg.org> In-Reply-To: <20121203191858.GY24381@cmpxchg.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Mel Gorman , Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Thorsten Leemhuis , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Jiri Slaby , Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org Dne 3.12.2012 20:18, Johannes Weiner napsal(a): > Szia Zdenek, > > On Mon, Dec 03, 2012 at 04:23:15PM +0100, Zdenek Kabelac wrote: >> Ok, bad news - I've been hit by kswapd0 loop again - >> my kernel git commit cc19528bd3084c3c2d870b31a3578da8c69952f3 again >> shown kswapd0 for couple minutes on CPU. >> >> It seemed to go instantly away when I've drop caches >> (echo 3 >/proc/sys/vm/drop_cache) >> (After that I've had over 1G free memory) > > Any chance you could retry with this patch on top? > > --- > From: Johannes Weiner > Subject: [patch] mm: vmscan: do not keep kswapd looping forever due > to individual uncompactable zones > > --- > mm/vmscan.c | 16 ---------------- > 1 file changed, 16 deletions(-) > > diff --git a/mm/vmscan.c b/mm/vmscan.c Ok, I'm running now b69f0859dc8e633c5d8c06845811588fe17e68b3 (-rc8) with your patch. I'll be able to give some feedback after couple days (if I keep my machine running without reboot - since before I had occasional problems with ACPI now resolved. (https://bugzilla.kernel.org/show_bug.cgi?id=51071) (patch not yet in -rc8) I'm also using this extra patch: https://patchwork.kernel.org/patch/1792531/ What seems to be triggering condition on my machine - running laptop for some days - and having Thunderbird reaching 0.8G (I guess they must keep all my news messages in memory to consume that size) and Firefox 1.3GB of consumed memory (assuming massive leaking with combination of flash) Zdenek -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx128.postini.com [74.125.245.128]) by kanga.kvack.org (Postfix) with SMTP id 36DA26B006E for ; Tue, 4 Dec 2012 04:15:15 -0500 (EST) Received: by mail-ee0-f41.google.com with SMTP id d41so2574546eek.14 for ; Tue, 04 Dec 2012 01:15:13 -0800 (PST) Message-ID: <50BDBF1D.60105@suse.cz> Date: Tue, 04 Dec 2012 10:15:09 +0100 From: Jiri Slaby MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <20121128094511.GS8218@suse.de> <50BCC3E3.40804@redhat.com> <20121203191858.GY24381@cmpxchg.org> <50BDBCD9.9060509@redhat.com> In-Reply-To: <50BDBCD9.9060509@redhat.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Zdenek Kabelac , Mel Gorman , Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Thorsten Leemhuis , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org On 12/04/2012 10:05 AM, Zdenek Kabelac wrote: > Dne 3.12.2012 20:18, Johannes Weiner napsal(a): >> Szia Zdenek, >> >> On Mon, Dec 03, 2012 at 04:23:15PM +0100, Zdenek Kabelac wrote: >>> Ok, bad news - I've been hit by kswapd0 loop again - >>> my kernel git commit cc19528bd3084c3c2d870b31a3578da8c69952f3 again >>> shown kswapd0 for couple minutes on CPU. >>> >>> It seemed to go instantly away when I've drop caches >>> (echo 3 >/proc/sys/vm/drop_cache) >>> (After that I've had over 1G free memory) >> >> Any chance you could retry with this patch on top? It does not apply to -next :/. Should I try anything else? >> From: Johannes Weiner >> Subject: [patch] mm: vmscan: do not keep kswapd looping forever due >> to individual uncompactable zones ... > What seems to be triggering condition on my machine - running laptop for > some days - and having Thunderbird reaching 0.8G (I guess they must > keep all my news messages in memory to consume that size) and Firefox > 1.3GB of consumed > memory (assuming massive leaking with combination of flash) Similar here, 5 days of uptime (suspend/resumes in between). FF 900M, TB 250M, java 1.1G, kvm 550M, X 400M, cache 1.5G out of 6G total mem. And boom. thanks, -- js suse labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx143.postini.com [74.125.245.143]) by kanga.kvack.org (Postfix) with SMTP id 689966B004D for ; Tue, 4 Dec 2012 11:12:38 -0500 (EST) Date: Tue, 4 Dec 2012 11:11:31 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121204161131.GB24381@cmpxchg.org> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <20121128094511.GS8218@suse.de> <50BCC3E3.40804@redhat.com> <20121203191858.GY24381@cmpxchg.org> <50BDBCD9.9060509@redhat.com> <50BDBF1D.60105@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50BDBF1D.60105@suse.cz> Sender: owner-linux-mm@kvack.org List-ID: To: Jiri Slaby Cc: Zdenek Kabelac , Mel Gorman , Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Thorsten Leemhuis , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org On Tue, Dec 04, 2012 at 10:15:09AM +0100, Jiri Slaby wrote: > On 12/04/2012 10:05 AM, Zdenek Kabelac wrote: > > Dne 3.12.2012 20:18, Johannes Weiner napsal(a): > >> Szia Zdenek, > >> > >> On Mon, Dec 03, 2012 at 04:23:15PM +0100, Zdenek Kabelac wrote: > >>> Ok, bad news - I've been hit by kswapd0 loop again - > >>> my kernel git commit cc19528bd3084c3c2d870b31a3578da8c69952f3 again > >>> shown kswapd0 for couple minutes on CPU. > >>> > >>> It seemed to go instantly away when I've drop caches > >>> (echo 3 >/proc/sys/vm/drop_cache) > >>> (After that I've had over 1G free memory) > >> > >> Any chance you could retry with this patch on top? > > It does not apply to -next :/. Should I try anything else? The COMPACTION_BUILD changed to IS_ENABLED(CONFIG_COMPACTION), below is a -next patch. I hope you don't run into other problems that come out of -next craziness, because Linus is kinda waiting for this to be resolved to release 3.8. If you've always tested against -next so far and it worked otherwise, don't change the environment now, please. If you just started, it would make more sense to test based on 3.7-rc8. Thanks! --- From: Johannes Weiner Subject: [patch] mm: vmscan: do not keep kswapd looping forever due to individual uncompactable zones When a zone meets its high watermark and is compactable in case of higher order allocations, it contributes to the percentage of the node's memory that is considered balanced. This requirement, that a node be only partially balanced, came about when kswapd was desparately trying to balance tiny zones when all bigger zones in the node had plenty of free memory. Arguably, the same should apply to compaction: if a significant part of the node is balanced enough to run compaction, do not get hung up on that tiny zone that might never get in shape. When the compaction logic in kswapd is reached, we know that at least 25% of the node's memory is balanced properly for compaction (see zone_balanced and pgdat_balanced). Remove the individual zone checks that restart the kswapd cycle. Otherwise, we may observe more endless looping in kswapd where the compaction code loops back to reclaim because of a single zone and reclaim does nothing because the node is considered balanced overall. Reported-by: Thorsten Leemhuis Signed-off-by: Johannes Weiner --- mm/vmscan.c | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 3b0aef4..486100f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2806,22 +2806,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, if (!populated_zone(zone)) continue; - if (zone->all_unreclaimable && - sc.priority != DEF_PRIORITY) - continue; - - /* Would compaction fail due to lack of free memory? */ - if (IS_ENABLED(CONFIG_COMPACTION) && - compaction_suitable(zone, order) == COMPACT_SKIPPED) - goto loop_again; - - /* Confirm the zone is balanced for order-0 */ - if (!zone_watermark_ok(zone, 0, - high_wmark_pages(zone), 0, 0)) { - order = sc.order = 0; - goto loop_again; - } - /* Check if the memory needs to be defragmented. */ if (zone_watermark_ok(zone, order, low_wmark_pages(zone), *classzone_idx, 0)) -- 1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx199.postini.com [74.125.245.199]) by kanga.kvack.org (Postfix) with SMTP id B88116B0062 for ; Tue, 4 Dec 2012 11:16:36 -0500 (EST) Date: Tue, 4 Dec 2012 11:15:33 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121204161533.GC24381@cmpxchg.org> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <20121128094511.GS8218@suse.de> <50BCC3E3.40804@redhat.com> <20121203191858.GY24381@cmpxchg.org> <50BDBCD9.9060509@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50BDBCD9.9060509@redhat.com> Sender: owner-linux-mm@kvack.org List-ID: To: Zdenek Kabelac Cc: Mel Gorman , Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Thorsten Leemhuis , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Jiri Slaby , Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org On Tue, Dec 04, 2012 at 10:05:29AM +0100, Zdenek Kabelac wrote: > Dne 3.12.2012 20:18, Johannes Weiner napsal(a): > >Szia Zdenek, > > > >On Mon, Dec 03, 2012 at 04:23:15PM +0100, Zdenek Kabelac wrote: > >>Ok, bad news - I've been hit by kswapd0 loop again - > >>my kernel git commit cc19528bd3084c3c2d870b31a3578da8c69952f3 again > >>shown kswapd0 for couple minutes on CPU. > >> > >>It seemed to go instantly away when I've drop caches > >>(echo 3 >/proc/sys/vm/drop_cache) > >>(After that I've had over 1G free memory) > > > >Any chance you could retry with this patch on top? > > > >--- > >From: Johannes Weiner > >Subject: [patch] mm: vmscan: do not keep kswapd looping forever due > > to individual uncompactable zones > > > >--- > > mm/vmscan.c | 16 ---------------- > > 1 file changed, 16 deletions(-) > > > >diff --git a/mm/vmscan.c b/mm/vmscan.c > > > Ok, I'm running now b69f0859dc8e633c5d8c06845811588fe17e68b3 (-rc8) > with your patch. I'll be able to give some feedback after couple > days (if I keep my machine running without reboot - since before > I had occasional problems with ACPI now resolved. > (https://bugzilla.kernel.org/show_bug.cgi?id=51071) > (patch not yet in -rc8) > I'm also using this extra patch: https://patchwork.kernel.org/patch/1792531/ Okay, fingers crossed! Thanks for persisting. > What seems to be triggering condition on my machine - running laptop > for some days - and having Thunderbird reaching 0.8G (I guess they > must keep all my news messages in memory to consume that size) and > Firefox 1.3GB of consumed > memory (assuming massive leaking with combination of flash) Were you able speed this process up in the past? I.e. by doing a search over all mail? Watching 8 nyan cat videos in parallel? If not, it's probably better not to change anything now... Thanks! -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx151.postini.com [74.125.245.151]) by kanga.kvack.org (Postfix) with SMTP id D2B456B006C for ; Tue, 4 Dec 2012 11:22:43 -0500 (EST) Received: by mail-ee0-f41.google.com with SMTP id d41so2882699eek.14 for ; Tue, 04 Dec 2012 08:22:42 -0800 (PST) Message-ID: <50BE234E.7000603@suse.cz> Date: Tue, 04 Dec 2012 17:22:38 +0100 From: Jiri Slaby MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <20121128094511.GS8218@suse.de> <50BCC3E3.40804@redhat.com> <20121203191858.GY24381@cmpxchg.org> <50BDBCD9.9060509@redhat.com> <50BDBF1D.60105@suse.cz> <20121204161131.GB24381@cmpxchg.org> In-Reply-To: <20121204161131.GB24381@cmpxchg.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Zdenek Kabelac , Mel Gorman , Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Thorsten Leemhuis , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org On 12/04/2012 05:11 PM, Johannes Weiner wrote: >>>> Any chance you could retry with this patch on top? >> >> It does not apply to -next :/. Should I try anything else? > > The COMPACTION_BUILD changed to IS_ENABLED(CONFIG_COMPACTION), below > is a -next patch. I hope you don't run into other problems that come > out of -next craziness, because Linus is kinda waiting for this to be > resolved to release 3.8. If you've always tested against -next so far > and it worked otherwise, don't change the environment now, please. If > you just started, it would make more sense to test based on 3.7-rc8. I reported the issue as soon as it appeared in -next for the first time on Oct 12. Since then I'm constantly hitting the issue (well, there were more than one I suppose, but not all of them were fixed by now) until now. I run only -next... Going to apply the patch now. -- js suse labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx105.postini.com [74.125.245.105]) by kanga.kvack.org (Postfix) with SMTP id 4830D6B0062 for ; Tue, 4 Dec 2012 14:51:12 -0500 (EST) Date: Tue, 4 Dec 2012 14:50:08 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121204195008.GD24381@cmpxchg.org> References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <20121128094511.GS8218@suse.de> <50BCC3E3.40804@redhat.com> <20121203191858.GY24381@cmpxchg.org> <50BDBCD9.9060509@redhat.com> <50BDBF1D.60105@suse.cz> <20121204161131.GB24381@cmpxchg.org> <50BE234E.7000603@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50BE234E.7000603@suse.cz> Sender: owner-linux-mm@kvack.org List-ID: To: Jiri Slaby Cc: Zdenek Kabelac , Mel Gorman , Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Thorsten Leemhuis , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org On Tue, Dec 04, 2012 at 05:22:38PM +0100, Jiri Slaby wrote: > On 12/04/2012 05:11 PM, Johannes Weiner wrote: > >>>> Any chance you could retry with this patch on top? > >> > >> It does not apply to -next :/. Should I try anything else? > > > > The COMPACTION_BUILD changed to IS_ENABLED(CONFIG_COMPACTION), below > > is a -next patch. I hope you don't run into other problems that come > > out of -next craziness, because Linus is kinda waiting for this to be > > resolved to release 3.8. If you've always tested against -next so far > > and it worked otherwise, don't change the environment now, please. If > > you just started, it would make more sense to test based on 3.7-rc8. > > I reported the issue as soon as it appeared in -next for the first time > on Oct 12. Since then I'm constantly hitting the issue (well, there were > more than one I suppose, but not all of them were fixed by now) until > now. I run only -next... Okay. Yes, it was a couple of problems, but not everybody hit the same subset. > Going to apply the patch now. Thanks! -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx204.postini.com [74.125.245.204]) by kanga.kvack.org (Postfix) with SMTP id D772B6B0044 for ; Tue, 4 Dec 2012 16:43:13 -0500 (EST) Date: Tue, 4 Dec 2012 16:42:10 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121204214210.GB20253@cmpxchg.org> References: <20121128101359.GT8218@suse.de> <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20121203194208.GZ24381@cmpxchg.org> Sender: owner-linux-mm@kvack.org List-ID: To: Thorsten Leemhuis Cc: Mel Gorman , Andrew Morton , Linus Torvalds , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List , John Ellson On Mon, Dec 03, 2012 at 02:42:08PM -0500, Johannes Weiner wrote: > On Mon, Dec 03, 2012 at 09:30:12AM +0100, Thorsten Leemhuis wrote: > > >> John was able to reproduce the problem quickly with a kernel that > > >> contained the patch from your mail. For details see > > > > > > [stripped: all the glory details of what likely went wrong and lead > > > to the problem john sees or saw] > > > > > > --- > > > From: Johannes Weiner > > > Subject: [patch] mm: vmscan: do not keep kswapd looping forever due > > > to individual uncompactable zones > > > > > > When a zone meets its high watermark and is compactable in case of > > > higher order allocations, it contributes to the percentage of the > > > node's memory that is considered balanced. > > > [...] > > > > FYI: I built a kernel with that patch. I've been running on my x86_64 > > machine at home over the weekend and everything was working fine (just > > as without the patch). John gave it a quick try and in > > https://bugzilla.redhat.com/show_bug.cgi?id=866988#c57 reported: > > > > """ > > I just installed > > kernel-3.7.0-0.rc7.git1.2.van.main.knurd.kswap.4.fc18.i686 and ran my > > usual load that triggers the problem. OK so far. I'll check again in > > 24hours, but looking good so far. > > """ > > w00t! Update from John in the BZ (https://bugzilla.redhat.com/show_bug.cgi?id=866988#c62): "Good news. I've now been running both kernel-3.7.0-0.rc7.git1.2.van.main.knurd.kswap.4.fc18.i686 and kernel-3.7.0-0.rc7.git1.2.van.main.knurd.kswap.4.fc18.x86_64 for over 24hours with no evidence of problems with kswapd" Now waiting for results from Jiri, Zdenek and Bruno... -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx134.postini.com [74.125.245.134]) by kanga.kvack.org (Postfix) with SMTP id CB9786B006C for ; Tue, 4 Dec 2012 22:02:48 -0500 (EST) Date: Tue, 4 Dec 2012 21:01:33 -0600 From: Bruno Wolff III Subject: Re: kswapd craziness in 3.7 Message-ID: <20121205030133.GA17438@wolff.to> References: <20121128101359.GT8218@suse.de> <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20121204214210.GB20253@cmpxchg.org> Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Thorsten Leemhuis , Mel Gorman , Andrew Morton , Linus Torvalds , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , linux-mm , Linux Kernel Mailing List , John Ellson On Tue, Dec 04, 2012 at 16:42:10 -0500, Johannes Weiner wrote: > kernel-3.7.0-0.rc7.git1.2.van.main.knurd.kswap.4.fc18.i686 >and > kernel-3.7.0-0.rc7.git1.2.van.main.knurd.kswap.4.fc18.x86_64 >for over 24hours with no evidence of problems with kswapd" > >Now waiting for results from Jiri, Zdenek and Bruno... I have been running 3.7.0-0.rc7.git1.2.van.main.knurd.kswap.4.fc18.i686.PAE a bit over 23 hours and kswapd has accumalated one minute 8 seconds of CPU time. I did several yum operations during that time and didn't see kswapd spike to 90+% CPU usage as I had seen in the past. With some kernels I wasn't reliably triggering the kswapd issue, so it may not be long enough to know for sure that the problem is fixed. I also should note that when I tried 3.7.0-0.rc7.git3.2.fc19.i686.PAE I did see problems with kswapd hitting 90+% usage of a CPU. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx111.postini.com [74.125.245.111]) by kanga.kvack.org (Postfix) with SMTP id 4F6456B005D for ; Thu, 6 Dec 2012 03:09:09 -0500 (EST) Message-ID: <50C052A2.90303@leemhuis.info> Date: Thu, 06 Dec 2012 09:09:06 +0100 From: Thorsten Leemhuis MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <20121127222637.GG2301@cmpxchg.org> <20121128101359.GT8218@suse.de> <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> In-Reply-To: <20121203194208.GZ24381@cmpxchg.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Mel Gorman , Andrew Morton , Linus Torvalds , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , Bruno Wolff III , linux-mm , Linux Kernel Mailing List , John Ellson Hi! Just a quick update Johannes Weiner wrote on 03.12.2012 20:42: > On Mon, Dec 03, 2012 at 09:30:12AM +0100, Thorsten Leemhuis wrote: > >> BTW, I built that kernel without the patch you mentioned in >> http://thread.gmane.org/gmane.linux.kernel.mm/90911/focus=91153 >> ("buffer_heads_over_limit can put kswapd into reclaim, but it's ignored >> [...]) It looked to me like that patch was only meant for debugging. Let >> me know if that was wrong. Ohh, and I didn't update to a fresher >> mainline checkout yet to make sure the base for John's testing didn't >> change. > > Ah, yes, the ApplyPatch is commented out. > > I think we want that upstream as well, but it's not critical. > [...] Sorry, it had no "Singed-off-by", so I assumed it was just for debugging. > Not rebasing sounds reasonable to me to verify the patch. It might be > worth testing that the final version that will be 3.8 still works for > John, however, once that is done. Just to be sure. Just to be sure, I yesterday built a rc8 kernel with the patch referenced above and the one that is not yet merged (these two, to be precise: http://thread.gmane.org/gmane.linux.kernel.mm/90911/focus=91153 http://thread.gmane.org/gmane.linux.kernel.mm/90911/focus=91300 ; all the others patches my kswap test kernels contained earlier were afaics merged a few days ago) and mentioned it in the Fedora bug report. John gave them a try and in https://bugzilla.redhat.com/show_bug.cgi?id=866988#c65 reported "No problems so far. I'll check back again in ~24hours." CU, Thorsten -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx204.postini.com [74.125.245.204]) by kanga.kvack.org (Postfix) with SMTP id D01E96B007D for ; Thu, 6 Dec 2012 08:52:02 -0500 (EST) Message-ID: <50C0A2EF.6010404@redhat.com> Date: Thu, 06 Dec 2012 14:51:43 +0100 From: Zdenek Kabelac MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <20121128094511.GS8218@suse.de> <50BCC3E3.40804@redhat.com> <20121203191858.GY24381@cmpxchg.org> <50BDBCD9.9060509@redhat.com> In-Reply-To: <50BDBCD9.9060509@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Mel Gorman , Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Thorsten Leemhuis , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Jiri Slaby , Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org Dne 4.12.2012 10:05, Zdenek Kabelac napsal(a): > Dne 3.12.2012 20:18, Johannes Weiner napsal(a): >> Szia Zdenek, >> >> On Mon, Dec 03, 2012 at 04:23:15PM +0100, Zdenek Kabelac wrote: >>> Ok, bad news - I've been hit by kswapd0 loop again - >>> my kernel git commit cc19528bd3084c3c2d870b31a3578da8c69952f3 again >>> shown kswapd0 for couple minutes on CPU. >>> >>> It seemed to go instantly away when I've drop caches >>> (echo 3 >/proc/sys/vm/drop_cache) >>> (After that I've had over 1G free memory) >> >> Any chance you could retry with this patch on top? >> >> --- >> From: Johannes Weiner >> Subject: [patch] mm: vmscan: do not keep kswapd looping forever due >> to individual uncompactable zones >> >> --- >> mm/vmscan.c | 16 ---------------- >> 1 file changed, 16 deletions(-) >> >> diff --git a/mm/vmscan.c b/mm/vmscan.c > > > Ok, I'm running now b69f0859dc8e633c5d8c06845811588fe17e68b3 (-rc8) > with your patch. I'll be able to give some feedback after couple > days (if I keep my machine running without reboot - since before So to just give some positive info - with 2 1/2 day uptime, several suspend/resumes, ff at 1.4GB I still have just 29 seconds for kswapd0 process. So the patch above might have helped - but I'll look for a few more days. Zdenek -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx130.postini.com [74.125.245.130]) by kanga.kvack.org (Postfix) with SMTP id A2C646B002B for ; Thu, 6 Dec 2012 12:39:09 -0500 (EST) Date: Thu, 6 Dec 2012 11:37:42 -0600 From: Bruno Wolff III Subject: Re: kswapd craziness in 3.7 Message-ID: <20121206173742.GA27297@wolff.to> References: <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20121205030133.GA17438@wolff.to> Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Thorsten Leemhuis , Mel Gorman , Andrew Morton , Linus Torvalds , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , linux-mm , Linux Kernel Mailing List , John Ellson On Tue, Dec 04, 2012 at 21:01:33 -0600, Bruno Wolff III wrote: >On Tue, Dec 04, 2012 at 16:42:10 -0500, > Johannes Weiner wrote: >> kernel-3.7.0-0.rc7.git1.2.van.main.knurd.kswap.4.fc18.i686 >>and >> kernel-3.7.0-0.rc7.git1.2.van.main.knurd.kswap.4.fc18.x86_64 >>for over 24hours with no evidence of problems with kswapd" >> >>Now waiting for results from Jiri, Zdenek and Bruno... > >I have been running >3.7.0-0.rc7.git1.2.van.main.knurd.kswap.4.fc18.i686.PAE a bit over 23 >hours and kswapd has accumalated one minute 8 seconds of CPU time. I >did several yum operations during that time and didn't see kswapd >spike to 90+% CPU usage as I had seen in the past. With some kernels >I wasn't reliably triggering the kswapd issue, so it may not be long >enough to know for sure that the problem is fixed. I am now at a bit over 2 and 1/2 days with kswapd having used 1 minute 53 seconds of CPU time. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx184.postini.com [74.125.245.184]) by kanga.kvack.org (Postfix) with SMTP id CB04F8D0006 for ; Thu, 6 Dec 2012 14:31:43 -0500 (EST) Received: by mail-wg0-f47.google.com with SMTP id dq11so3347155wgb.26 for ; Thu, 06 Dec 2012 11:31:42 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <20121206173742.GA27297@wolff.to> References: <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> From: Linus Torvalds Date: Thu, 6 Dec 2012 11:31:21 -0800 Message-ID: Subject: Re: kswapd craziness in 3.7 Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Bruno Wolff III Cc: Johannes Weiner , Thorsten Leemhuis , Mel Gorman , Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , linux-mm , Linux Kernel Mailing List , John Ellson Ok, people seem to be reporting success. I've applied Johannes' last patch with the new tested-by tags. Johannes (or anybody else, for that matter), please holler LOUDLY if you disagreed.. (or if I used the wrong version of the patch, there's been several, afaik). Linus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx149.postini.com [74.125.245.149]) by kanga.kvack.org (Postfix) with SMTP id B693C6B00D2 for ; Thu, 6 Dec 2012 14:43:55 -0500 (EST) Message-ID: <50C0F55F.6030405@redhat.com> Date: Thu, 06 Dec 2012 14:43:27 -0500 From: Rik van Riel MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Bruno Wolff III , Johannes Weiner , Thorsten Leemhuis , Mel Gorman , Andrew Morton , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , linux-mm , Linux Kernel Mailing List , John Ellson On 12/06/2012 02:31 PM, Linus Torvalds wrote: > Ok, people seem to be reporting success. > > I've applied Johannes' last patch with the new tested-by tags. > > Johannes (or anybody else, for that matter), please holler LOUDLY if > you disagreed.. (or if I used the wrong version of the patch, there's > been several, afaik). Johannes's patch is a fairly big hammer, with kswapd not looping back to the start when zones are still unbalanced. However, the next allocation will wake up kswapd again, and having kswapd stop early beats having it in an infinite loop. I believe Johannes's patch will be fine for 3.7. -- All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx109.postini.com [74.125.245.109]) by kanga.kvack.org (Postfix) with SMTP id 7BCD98D0011 for ; Thu, 6 Dec 2012 15:24:28 -0500 (EST) Date: Thu, 6 Dec 2012 15:23:25 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121206202325.GA1498@cmpxchg.org> References: <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Bruno Wolff III , Thorsten Leemhuis , Mel Gorman , Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , linux-mm , Linux Kernel Mailing List , John Ellson On Thu, Dec 06, 2012 at 11:31:21AM -0800, Linus Torvalds wrote: > Ok, people seem to be reporting success. > > I've applied Johannes' last patch with the new tested-by tags. > > Johannes (or anybody else, for that matter), please holler LOUDLY if > you disagreed.. (or if I used the wrong version of the patch, there's > been several, afaik). I just went back one more time and of course that's when I spot that I forgot to remove the zone congestion clearing that depended on the now removed checks to ensure the zone is balanced. It's not too big of a deal, just the /risk/ of increased CPU use from reclaim because we go back to scanning zones that we previously deemed congested and slept a little bit before continuing reclaim. Sorry, I should have seen that earlier. Removing it is a low risk fix, the clearing was kinda redundant anyway (the preliminary zone check clears it for OK zones, so does the reclaim loop under the same criteria), letting it stay is probably more problematic for 3.8 than just dropping it... --- From: Johannes Weiner Subject: [patch] mm: vmscan: fix inappropriate zone congestion clearing c702418 ("mm: vmscan: do not keep kswapd looping forever due to individual uncompactable zones") removed zone watermark checks from the compaction code in kswapd but left in the zone congestion clearing, which now happens unconditionally on higher order reclaim. This messes up the reclaim throttling logic for zones with dirty/writeback pages, where zones should only lose their congestion status when their watermarks have been restored. Remove the clearing from the zone compaction section entirely. The preliminary zone check and the reclaim loop in kswapd will clear it if the zone is considered balanced. Signed-off-by: Johannes Weiner --- mm/vmscan.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 124bbfe..b7ed376 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2827,9 +2827,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, if (zone_watermark_ok(zone, order, low_wmark_pages(zone), *classzone_idx, 0)) zones_need_compaction = 0; - - /* If balanced, clear the congested flag */ - zone_clear_flag(zone, ZONE_CONGESTED); } if (zones_need_compaction) -- 1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx158.postini.com [74.125.245.158]) by kanga.kvack.org (Postfix) with SMTP id 389C18D0011 for ; Thu, 6 Dec 2012 15:32:36 -0500 (EST) Message-ID: <50C100D2.4010103@redhat.com> Date: Thu, 06 Dec 2012 15:32:18 -0500 From: Rik van Riel MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <20121206202325.GA1498@cmpxchg.org> In-Reply-To: <20121206202325.GA1498@cmpxchg.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Linus Torvalds , Bruno Wolff III , Thorsten Leemhuis , Mel Gorman , Andrew Morton , George Spelvin , Johannes Hirte , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis Kletnieks , Jiri Slaby , Zdenek Kabelac , linux-mm , Linux Kernel Mailing List , John Ellson On 12/06/2012 03:23 PM, Johannes Weiner wrote: > From: Johannes Weiner > Subject: [patch] mm: vmscan: fix inappropriate zone congestion clearing > > c702418 ("mm: vmscan: do not keep kswapd looping forever due to > individual uncompactable zones") removed zone watermark checks from > the compaction code in kswapd but left in the zone congestion > clearing, which now happens unconditionally on higher order reclaim. > > This messes up the reclaim throttling logic for zones with > dirty/writeback pages, where zones should only lose their congestion > status when their watermarks have been restored. > > Remove the clearing from the zone compaction section entirely. The > preliminary zone check and the reclaim loop in kswapd will clear it if > the zone is considered balanced. > > Signed-off-by: Johannes Weiner Reviewed-by: Rik van Riel -- All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx188.postini.com [74.125.245.188]) by kanga.kvack.org (Postfix) with SMTP id 3F7DC6B005D for ; Sat, 8 Dec 2012 05:35:52 -0500 (EST) Received: by mail-ee0-f41.google.com with SMTP id d41so860065eek.14 for ; Sat, 08 Dec 2012 02:35:50 -0800 (PST) Message-ID: <50C31802.7030506@suse.cz> Date: Sat, 08 Dec 2012 11:35:46 +0100 From: Jiri Slaby MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <1354049315-12874-1-git-send-email-hannes@cmpxchg.org> <20121128094511.GS8218@suse.de> <50BCC3E3.40804@redhat.com> <20121203191858.GY24381@cmpxchg.org> <50BDBCD9.9060509@redhat.com> <50BDBF1D.60105@suse.cz> <20121204161131.GB24381@cmpxchg.org> In-Reply-To: <20121204161131.GB24381@cmpxchg.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Zdenek Kabelac , Mel Gorman , Andrew Morton , Rik van Riel , George Spelvin , Johannes Hirte , Thorsten Leemhuis , Tomas Racek , Jan Kara , Dave Hansen , Josh Boyer , Valdis.Kletnieks@vt.edu, Bruno Wolff III , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org On 12/04/2012 05:11 PM, Johannes Weiner wrote: > On Tue, Dec 04, 2012 at 10:15:09AM +0100, Jiri Slaby wrote: >> It does not apply to -next :/. Should I try anything else? > > The COMPACTION_BUILD changed to IS_ENABLED(CONFIG_COMPACTION), below > is a -next patch. I hope you don't run into other problems that come > out of -next craziness, because Linus is kinda waiting for this to be > resolved to release 3.8. If you've always tested against -next so far > and it worked otherwise, don't change the environment now, please. If > you just started, it would make more sense to test based on 3.7-rc8. > > Thanks! > > --- > From: Johannes Weiner > Subject: [patch] mm: vmscan: do not keep kswapd looping forever due > to individual uncompactable zones > > When a zone meets its high watermark and is compactable in case of > higher order allocations, it contributes to the percentage of the > node's memory that is considered balanced. > > This requirement, that a node be only partially balanced, came about > when kswapd was desparately trying to balance tiny zones when all > bigger zones in the node had plenty of free memory. Arguably, the > same should apply to compaction: if a significant part of the node is > balanced enough to run compaction, do not get hung up on that tiny > zone that might never get in shape. > > When the compaction logic in kswapd is reached, we know that at least > 25% of the node's memory is balanced properly for compaction (see > zone_balanced and pgdat_balanced). Remove the individual zone checks > that restart the kswapd cycle. > > Otherwise, we may observe more endless looping in kswapd where the > compaction code loops back to reclaim because of a single zone and > reclaim does nothing because the node is considered balanced overall. > > Reported-by: Thorsten Leemhuis > Signed-off-by: Johannes Weiner Looks like it's gone with this patch now. Hopefully the send button won't trigger the issue the same as the last time :). > --- > mm/vmscan.c | 16 ---------------- > 1 file changed, 16 deletions(-) > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 3b0aef4..486100f 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -2806,22 +2806,6 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, > if (!populated_zone(zone)) > continue; > > - if (zone->all_unreclaimable && > - sc.priority != DEF_PRIORITY) > - continue; > - > - /* Would compaction fail due to lack of free memory? */ > - if (IS_ENABLED(CONFIG_COMPACTION) && > - compaction_suitable(zone, order) == COMPACT_SKIPPED) > - goto loop_again; > - > - /* Confirm the zone is balanced for order-0 */ > - if (!zone_watermark_ok(zone, 0, > - high_wmark_pages(zone), 0, 0)) { > - order = sc.order = 0; > - goto loop_again; > - } > - > /* Check if the memory needs to be defragmented. */ > if (zone_watermark_ok(zone, order, > low_wmark_pages(zone), *classzone_idx, 0)) > -- js suse labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx148.postini.com [74.125.245.148]) by kanga.kvack.org (Postfix) with SMTP id 10E046B005D for ; Sat, 8 Dec 2012 07:06:14 -0500 (EST) Date: Sat, 08 Dec 2012 13:06:10 +0100 From: Zlatko Calusic MIME-Version: 1.0 References: <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> In-Reply-To: Message-ID: <50C32D32.6040800@iskon.hr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: kswapd craziness in 3.7 Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Johannes Weiner , linux-mm , Linux Kernel Mailing List On 06.12.2012 20:31, Linus Torvalds wrote: > Ok, people seem to be reporting success. > > I've applied Johannes' last patch with the new tested-by tags. > I've been testing this patch since it was applied, and it certainly fixes the kswapd craziness issue, good work Johannes! But, it's still not perfect yet, because I see that the system keeps lots of memory unused (free), where it previously used it all for the page cache (there's enough fs activity to warrant it). I'm now testing the last piece of Johannes' changes (still not in git tree), and can report results in 24-48 hours. Regards, -- Zlatko -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx184.postini.com [74.125.245.184]) by kanga.kvack.org (Postfix) with SMTP id F17BE6B0072 for ; Sat, 8 Dec 2012 16:22:13 -0500 (EST) Date: Sat, 08 Dec 2012 22:22:08 +0100 From: Zlatko Calusic MIME-Version: 1.0 References: <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> In-Reply-To: <50C32D32.6040800@iskon.hr> Message-ID: <50C3AF80.8040700@iskon.hr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: kswapd craziness in 3.7 Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Johannes Weiner , linux-mm , Linux Kernel Mailing List On 08.12.2012 13:06, Zlatko Calusic wrote: > On 06.12.2012 20:31, Linus Torvalds wrote: >> Ok, people seem to be reporting success. >> >> I've applied Johannes' last patch with the new tested-by tags. >> > > I've been testing this patch since it was applied, and it certainly > fixes the kswapd craziness issue, good work Johannes! > > But, it's still not perfect yet, because I see that the system keeps > lots of memory unused (free), where it previously used it all for the > page cache (there's enough fs activity to warrant it). > > I'm now testing the last piece of Johannes' changes (still not in git > tree), and can report results in 24-48 hours. > > Regards, Or sooner... in short: nothing's changed! On a 4GB RAM system, where applications use close to 2GB, kswapd likes to keep around 1GB free (unused), leaving only 1GB for page/buffer cache. If I force bigger page cache by reading a big file and thus use the unused 1GB of RAM, kswapd will soon (in a matter of minutes) evict those (or other) pages out and once again keep unused memory close to 1GB. I guess it's not a showstopper, but it still counts as a very bad memory management, wasting lots of RAM. As an additional data point, if memory pressure is slightly higher (say backup kicks in, keeping page cache mostly full) kswapd gets in D (uninterruptible sleep) state (function: congestion_wait) and load average goes up by 1. It recovers only when it successfully throws out half of page cache again. Hope it helps. -- Zlatko -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx138.postini.com [74.125.245.138]) by kanga.kvack.org (Postfix) with SMTP id 7AD456B0074 for ; Sat, 8 Dec 2012 20:01:58 -0500 (EST) Received: by mail-pb0-f41.google.com with SMTP id xa7so1172104pbc.14 for ; Sat, 08 Dec 2012 17:01:57 -0800 (PST) Date: Sat, 8 Dec 2012 17:01:42 -0800 (PST) From: Linus Torvalds Subject: Re: kswapd craziness in 3.7 In-Reply-To: <50C3AF80.8040700@iskon.hr> Message-ID: References: <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Zlatko Calusic Cc: Linus Torvalds , Johannes Weiner , Mel Gorman , linux-mm , Linux Kernel Mailing List On Sat, 8 Dec 2012, Zlatko Calusic wrote: > > Or sooner... in short: nothing's changed! > > On a 4GB RAM system, where applications use close to 2GB, kswapd likes to keep > around 1GB free (unused), leaving only 1GB for page/buffer cache. If I force > bigger page cache by reading a big file and thus use the unused 1GB of RAM, > kswapd will soon (in a matter of minutes) evict those (or other) pages out and > once again keep unused memory close to 1GB. Ok, guys, what was the reclaim or kswapd patch during the merge window that actually caused all of these insane problems? It seems it was more fundamentally buggered than the fifteen-million fixes for kswapd we have already picked up. (Ok, I may be exaggerating the number of patches, but it's starting to feel that way - I thought that 3.7 was going to be a calm and easy release, but the kswapd issues seem to just keep happening. We've been fighting the kswapd changes for a while now.) Trying to keep a gigabyte free (presumably because that way we have lots of high-order alloction pages) is ridiculous. Is it one of the compaction changes? Mel? Ideas? Linus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx139.postini.com [74.125.245.139]) by kanga.kvack.org (Postfix) with SMTP id 37E3D6B005D for ; Sun, 9 Dec 2012 16:59:57 -0500 (EST) Received: by mail-we0-f169.google.com with SMTP id t49so1174736wey.14 for ; Sun, 09 Dec 2012 13:59:55 -0800 (PST) Message-ID: <50C509D3.3070108@gmail.com> Date: Sun, 09 Dec 2012 22:59:47 +0100 From: Zdenek Kabelac MIME-Version: 1.0 Subject: Re: kswapd craziness in 3.7 References: <20121128145215.d23aeb1b.akpm@linux-foundation.org> <20121128235412.GW8218@suse.de> <50B77F84.1030907@leemhuis.info> <20121129170512.GI2301@cmpxchg.org> <50B8A8E7.4030108@leemhuis.info> <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Zlatko Calusic , Johannes Weiner , Mel Gorman , linux-mm , Linux Kernel Mailing List Dne 9.12.2012 02:01, Linus Torvalds napsal(a): > > > On Sat, 8 Dec 2012, Zlatko Calusic wrote: >> >> Or sooner... in short: nothing's changed! >> >> On a 4GB RAM system, where applications use close to 2GB, kswapd likes to keep >> around 1GB free (unused), leaving only 1GB for page/buffer cache. If I force >> bigger page cache by reading a big file and thus use the unused 1GB of RAM, >> kswapd will soon (in a matter of minutes) evict those (or other) pages out and >> once again keep unused memory close to 1GB. > > Ok, guys, what was the reclaim or kswapd patch during the merge window > that actually caused all of these insane problems? It seems it was more > fundamentally buggered than the fifteen-million fixes for kswapd we have > already picked up. > > (Ok, I may be exaggerating the number of patches, but it's starting to > feel that way - I thought that 3.7 was going to be a calm and easy > release, but the kswapd issues seem to just keep happening. We've been > fighting the kswapd changes for a while now.) > > Trying to keep a gigabyte free (presumably because that way we have lots > of high-order alloction pages) is ridiculous. Is it one of the compaction > changes? > > Mel? Ideas? > Very true It's just as simple a making dd if=/dev/zero of=/tmp/zero bs=1M count=0 seek=1000000 and now dd if=/tmp/zero of=/dev/null bs=1M and kswapd fights with dd for CPU time.... Zdenek -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx148.postini.com [74.125.245.148]) by kanga.kvack.org (Postfix) with SMTP id 683606B005A for ; Mon, 10 Dec 2012 06:03:42 -0500 (EST) Date: Mon, 10 Dec 2012 11:03:37 +0000 From: Mel Gorman Subject: Re: kswapd craziness in 3.7 Message-ID: <20121210110337.GH1009@suse.de> References: <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Zlatko Calusic , Rik van Riel , Johannes Weiner , linux-mm , Linux Kernel Mailing List On Sat, Dec 08, 2012 at 05:01:42PM -0800, Linus Torvalds wrote: > > > On Sat, 8 Dec 2012, Zlatko Calusic wrote: > > > > Or sooner... in short: nothing's changed! > > > > On a 4GB RAM system, where applications use close to 2GB, kswapd likes to keep > > around 1GB free (unused), leaving only 1GB for page/buffer cache. If I force > > bigger page cache by reading a big file and thus use the unused 1GB of RAM, > > kswapd will soon (in a matter of minutes) evict those (or other) pages out and > > once again keep unused memory close to 1GB. > > Ok, guys, what was the reclaim or kswapd patch during the merge window > that actually caused all of these insane problems? I believe commit c6543459 (mm: remove __GFP_NO_KSWAPD) is the primary candidate. __GFP_NO_KSWAPD was originally introduced by THP because kswapd was excessively reclaiming. kswapd would stay awake aggressively reclaiming even if compaction was deferred. The flag was removed in this cycle when it was expected that it was no longer necessary. I'm not foisting the blame on Rik here, I was on the review list for that patch and did not identify that it would cause this many problems either. > It seems it was more > fundamentally buggered than the fifteen-million fixes for kswapd we have > already picked up. > It was already fundamentally buggered up. The difference was it stayed asleep for THP requests in earlier kernels. There is a big difference between a direct reclaim/compaction for THP and kswapd doing the same work. Direct reclaim/compaction will try once, give up quickly and defer requests in the near future to avoid impacting the system heavily for THP. The same applies for khugepaged. kswapd is different. It can keep going until it meets its watermarks for a THP allocation are met. Two reasons why it might keep going for a long time are that compaction is being inefficient which we know it may be due to crap like this end_pfn = ALIGN(low_pfn + pageblock_nr_pages, pageblock_nr_pages); and the second reason is if the highest zone is relatively because compaction_suitable will keep saying that allocations are failing due to insufficient amounts of memory in the highest zone. It'll reclaim a little from this highest zone and then shrink_slab() potentially dumping a large amount of memory. This may be the case for Zlatko as with a 4G machine his ZONE_NORMAL could be small depending on how the 32-bit address space is used by his hardware. > (Ok, I may be exaggerating the number of patches, but it's starting to > feel that way - I thought that 3.7 was going to be a calm and easy > release, but the kswapd issues seem to just keep happening. We've been > fighting the kswapd changes for a while now.) > Yes. > Trying to keep a gigabyte free (presumably because that way we have lots > of high-order alloction pages) is ridiculous. Is it one of the compaction > changes? > Not directly. Compaction has been a bigger factor after 3.5 due to the removal of lumpy reclaim but it's not directly responsible for excessive amounts of memory being kept free. The closest patch I'm aware of that would cause problems of that nature would be commit 83fde0f2 (mm: vmscan: scale number of pages reclaimed by reclaim/compaction based on failures) and it has already been reverted by 96710098. > Mel? Ideas? > Consider reverting the revert of __GFP_NO_KSWAPD again until this can be ironed out at a more reasonable pace. Rik? Johannes? Verify if the shrinking slab is the issue with this brutually ugly hack. Zlatko? diff --git a/mm/vmscan.c b/mm/vmscan.c index b7ed376..2189d20 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2550,6 +2550,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, unsigned long balanced; int i; int end_zone = 0; /* Inclusive. 0 = ZONE_DMA */ + bool should_shrink_slab = true; unsigned long total_scanned; struct reclaim_state *reclaim_state = current->reclaim_state; unsigned long nr_soft_reclaimed; @@ -2695,7 +2696,8 @@ loop_again: shrink_zone(zone, &sc); reclaim_state->reclaimed_slab = 0; - nr_slab = shrink_slab(&shrink, sc.nr_scanned, lru_pages); + if (should_shrink_slab) + nr_slab = shrink_slab(&shrink, sc.nr_scanned, lru_pages); sc.nr_reclaimed += reclaim_state->reclaimed_slab; total_scanned += sc.nr_scanned; @@ -2817,6 +2819,16 @@ out: if (order) { int zones_need_compaction = 1; + /* + * Shrinking slab for high-order allocs can cause an excessive + * amount of memory to be dumped. Only shrink slab once per + * round for high-order allocs. + * + * This is a very stupid hack. balance_pgdat() is in serious + * need of a rework + */ + should_shrink_slab = false; + for (i = 0; i <= end_zone; i++) { struct zone *zone = pgdat->node_zones + i; -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx135.postini.com [74.125.245.135]) by kanga.kvack.org (Postfix) with SMTP id B62F56B002B for ; Mon, 10 Dec 2012 11:39:22 -0500 (EST) Date: Mon, 10 Dec 2012 11:39:04 -0500 From: Johannes Weiner Subject: Re: kswapd craziness in 3.7 Message-ID: <20121210163904.GA22101@cmpxchg.org> References: <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20121210110337.GH1009@suse.de> Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: Linus Torvalds , Zlatko Calusic , Rik van Riel , linux-mm , Linux Kernel Mailing List On Mon, Dec 10, 2012 at 11:03:37AM +0000, Mel Gorman wrote: > On Sat, Dec 08, 2012 at 05:01:42PM -0800, Linus Torvalds wrote: > > On Sat, 8 Dec 2012, Zlatko Calusic wrote: > > > Or sooner... in short: nothing's changed! > > > > > > On a 4GB RAM system, where applications use close to 2GB, kswapd likes to keep > > > around 1GB free (unused), leaving only 1GB for page/buffer cache. If I force > > > bigger page cache by reading a big file and thus use the unused 1GB of RAM, > > > kswapd will soon (in a matter of minutes) evict those (or other) pages out and > > > once again keep unused memory close to 1GB. > > > > Ok, guys, what was the reclaim or kswapd patch during the merge window > > that actually caused all of these insane problems? > > I believe commit c6543459 (mm: remove __GFP_NO_KSWAPD) is the primary > candidate. __GFP_NO_KSWAPD was originally introduced by THP because kswapd > was excessively reclaiming. kswapd would stay awake aggressively reclaiming > even if compaction was deferred. The flag was removed in this cycle when it > was expected that it was no longer necessary. I'm not foisting the blame > on Rik here, I was on the review list for that patch and did not identify > that it would cause this many problems either. > > > It seems it was more > > fundamentally buggered than the fifteen-million fixes for kswapd we have > > already picked up. > > It was already fundamentally buggered up. The difference was it stayed > asleep for THP requests in earlier kernels. > > There is a big difference between a direct reclaim/compaction for THP > and kswapd doing the same work. Direct reclaim/compaction will try once, > give up quickly and defer requests in the near future to avoid impacting > the system heavily for THP. The same applies for khugepaged. > > kswapd is different. It can keep going until it meets its watermarks for > a THP allocation are met. Two reasons why it might keep going for a long > time are that compaction is being inefficient which we know it may be due > to crap like this > > end_pfn = ALIGN(low_pfn + pageblock_nr_pages, pageblock_nr_pages); > > and the second reason is if the highest zone is relatively because > compaction_suitable will keep saying that allocations are failing due to > insufficient amounts of memory in the highest zone. It'll reclaim a little > from this highest zone and then shrink_slab() potentially dumping a large > amount of memory. This may be the case for Zlatko as with a 4G machine > his ZONE_NORMAL could be small depending on how the 32-bit address space > is used by his hardware. Unlike direct reclaim, kswapd also never does sync migration. Since the fragmentation index is a ratio of free pages over free page blocks, doing lightweight compaction that reduces the page blocks but never really follows through to compact a THP block increases the free memory requirement. I thought about the small Normal zone too. Direct reclaim/compaction is fine with one zone being able to provide a THP, but kswapd requires 25% of the node. A small ZONE_NORMAL would not be able to meet this and so the bigger DMA32 zone would also be required to be balanced for the THP allocation. > > Mel? Ideas? > > Consider reverting the revert of __GFP_NO_KSWAPD again until this can be > ironed out at a more reasonable pace. Rik? Johannes? Yes, I also think we need more time for this. > Verify if the shrinking slab is the issue with this brutually ugly > hack. Zlatko? > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index b7ed376..2189d20 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -2550,6 +2550,7 @@ static unsigned long balance_pgdat(pg_data_t *pgdat, int order, > unsigned long balanced; > int i; > int end_zone = 0; /* Inclusive. 0 = ZONE_DMA */ > + bool should_shrink_slab = true; > unsigned long total_scanned; > struct reclaim_state *reclaim_state = current->reclaim_state; > unsigned long nr_soft_reclaimed; > @@ -2695,7 +2696,8 @@ loop_again: > shrink_zone(zone, &sc); > > reclaim_state->reclaimed_slab = 0; > - nr_slab = shrink_slab(&shrink, sc.nr_scanned, lru_pages); > + if (should_shrink_slab) > + nr_slab = shrink_slab(&shrink, sc.nr_scanned, lru_pages); > sc.nr_reclaimed += reclaim_state->reclaimed_slab; > total_scanned += sc.nr_scanned; > > @@ -2817,6 +2819,16 @@ out: > if (order) { > int zones_need_compaction = 1; > > + /* > + * Shrinking slab for high-order allocs can cause an excessive > + * amount of memory to be dumped. Only shrink slab once per > + * round for high-order allocs. > + * > + * This is a very stupid hack. balance_pgdat() is in serious > + * need of a rework > + */ > + should_shrink_slab = false; > + > for (i = 0; i <= end_zone; i++) { > struct zone *zone = pgdat->node_zones + i; I don't see a shrink_slab() invocation after this point since the loop_again jumps in this loop where removed, so this shouldn't change anything? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx170.postini.com [74.125.245.170]) by kanga.kvack.org (Postfix) with SMTP id EDA8C6B002B for ; Mon, 10 Dec 2012 13:01:46 -0500 (EST) Date: Mon, 10 Dec 2012 18:01:41 +0000 From: Mel Gorman Subject: Re: kswapd craziness in 3.7 Message-ID: <20121210180141.GK1009@suse.de> References: <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <20121210163904.GA22101@cmpxchg.org> Sender: owner-linux-mm@kvack.org List-ID: To: Johannes Weiner Cc: Linus Torvalds , Zlatko Calusic , Rik van Riel , linux-mm , Linux Kernel Mailing List On Mon, Dec 10, 2012 at 11:39:04AM -0500, Johannes Weiner wrote: > On Mon, Dec 10, 2012 at 11:03:37AM +0000, Mel Gorman wrote: > > On Sat, Dec 08, 2012 at 05:01:42PM -0800, Linus Torvalds wrote: > > > On Sat, 8 Dec 2012, Zlatko Calusic wrote: > > > > Or sooner... in short: nothing's changed! > > > > > > > > On a 4GB RAM system, where applications use close to 2GB, kswapd likes to keep > > > > around 1GB free (unused), leaving only 1GB for page/buffer cache. If I force > > > > bigger page cache by reading a big file and thus use the unused 1GB of RAM, > > > > kswapd will soon (in a matter of minutes) evict those (or other) pages out and > > > > once again keep unused memory close to 1GB. > > > > > > Ok, guys, what was the reclaim or kswapd patch during the merge window > > > that actually caused all of these insane problems? > > > > I believe commit c6543459 (mm: remove __GFP_NO_KSWAPD) is the primary > > candidate. __GFP_NO_KSWAPD was originally introduced by THP because kswapd > > was excessively reclaiming. kswapd would stay awake aggressively reclaiming > > even if compaction was deferred. The flag was removed in this cycle when it > > was expected that it was no longer necessary. I'm not foisting the blame > > on Rik here, I was on the review list for that patch and did not identify > > that it would cause this many problems either. > > > > > It seems it was more > > > fundamentally buggered than the fifteen-million fixes for kswapd we have > > > already picked up. > > > > It was already fundamentally buggered up. The difference was it stayed > > asleep for THP requests in earlier kernels. > > > > There is a big difference between a direct reclaim/compaction for THP > > and kswapd doing the same work. Direct reclaim/compaction will try once, > > give up quickly and defer requests in the near future to avoid impacting > > the system heavily for THP. The same applies for khugepaged. > > > > kswapd is different. It can keep going until it meets its watermarks for > > a THP allocation are met. Two reasons why it might keep going for a long > > time are that compaction is being inefficient which we know it may be due > > to crap like this > > > > end_pfn = ALIGN(low_pfn + pageblock_nr_pages, pageblock_nr_pages); > > > > and the second reason is if the highest zone is relatively because > > compaction_suitable will keep saying that allocations are failing due to > > insufficient amounts of memory in the highest zone. It'll reclaim a little > > from this highest zone and then shrink_slab() potentially dumping a large > > amount of memory. This may be the case for Zlatko as with a 4G machine > > his ZONE_NORMAL could be small depending on how the 32-bit address space > > is used by his hardware. > > Unlike direct reclaim, kswapd also never does sync migration. Since > the fragmentation index is a ratio of free pages over free page > blocks, doing lightweight compaction that reduces the page blocks but > never really follows through to compact a THP block increases the free > memory requirement. > True. > I thought about the small Normal zone too. Direct reclaim/compaction > is fine with one zone being able to provide a THP, but kswapd requires > 25% of the node. A small ZONE_NORMAL would not be able to meet this > and so the bigger DMA32 zone would also be required to be balanced for > the THP allocation. > Also true. > > > Mel? Ideas? > > > > Consider reverting the revert of __GFP_NO_KSWAPD again until this can be > > ironed out at a more reasonable pace. Rik? Johannes? > > Yes, I also think we need more time for this. > Yes, the last minute band-aids are just getting worse and the result is more mess. > > > I don't see a shrink_slab() invocation after this point since the > loop_again jumps in this loop where removed, so this shouldn't change > anything? /me slaps self In this last-minute disaster, I'm not thinking properly at all any more. The shrink slab disabling should have happened before the loop_again but even then it's wrong because it's just covering over the problem. The way order and testorder interact with how balanced is calculated means that we potentially call shrink_slab() multiple times and that thing is global in nature and basically uncontrolled. You could argue that we should only call shrink_slab() if order-0 watermarks are not met but that will not necessarily prevent kswapd reclaiming too much. It keeps going back to balance_pgdat needing its list of requirements drawn up and receive some major surgery and we're not going to do that as a quick hack. -- Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx104.postini.com [74.125.245.104]) by kanga.kvack.org (Postfix) with SMTP id 6C3436B006C for ; Mon, 10 Dec 2012 13:29:06 -0500 (EST) Date: Mon, 10 Dec 2012 19:29:00 +0100 From: Zlatko Calusic MIME-Version: 1.0 References: <20121201004520.GK2301@cmpxchg.org> <50BC6314.7060106@leemhuis.info> <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> In-Reply-To: <20121210110337.GH1009@suse.de> Message-ID: <50C629EC.6080800@iskon.hr> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: kswapd craziness in 3.7 Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: Linus Torvalds , Rik van Riel , Johannes Weiner , linux-mm , Linux Kernel Mailing List On 10.12.2012 12:03, Mel Gorman wrote: > There is a big difference between a direct reclaim/compaction for THP > and kswapd doing the same work. Direct reclaim/compaction will try once, > give up quickly and defer requests in the near future to avoid impacting > the system heavily for THP. The same applies for khugepaged. > > kswapd is different. It can keep going until it meets its watermarks for > a THP allocation are met. Two reasons why it might keep going for a long > time are that compaction is being inefficient which we know it may be due > to crap like this > > end_pfn = ALIGN(low_pfn + pageblock_nr_pages, pageblock_nr_pages); > > and the second reason is if the highest zone is relatively because > compaction_suitable will keep saying that allocations are failing due to > insufficient amounts of memory in the highest zone. It'll reclaim a little > from this highest zone and then shrink_slab() potentially dumping a large > amount of memory. This may be the case for Zlatko as with a 4G machine > his ZONE_NORMAL could be small depending on how the 32-bit address space > is used by his hardware. > The kernel is 64-bit, if it makes any difference (userspace, though is still 32-bit). There's no swap (swap support not even compiled in). The zones are as follows: On node 0 totalpages: 1048019 DMA zone: 64 pages used for memmap DMA zone: 6 pages reserved DMA zone: 3913 pages, LIFO batch:0 DMA32 zone: 16320 pages used for memmap DMA32 zone: 831109 pages, LIFO batch:31 Normal zone: 3072 pages used for memmap Normal zone: 193535 pages, LIFO batch:31 If I understand correctly, you think that because 193535 pages in ZONE_NORMAL is relatively small compared to 831109 pages of ZONE_DMA32 the system has hard time balancing itself? Is there any way I could force and test different memory layout? I'm slightly lost at all the memory models (if I have a choice at all), so if you have any suggestions, I'm all ears. Maybe I could limit available memory and thus have only DMA32 zone, just to prove your theory? I remember doing tuning like that many years ago when I had more time to play with Linux MM, unfortunately didn't have much time lately, so I'm a bit rusty, but I'm willing to help testing and resolving this issue. -- Zlatko -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx182.postini.com [74.125.245.182]) by kanga.kvack.org (Postfix) with SMTP id D3D706B0070 for ; Mon, 10 Dec 2012 13:33:15 -0500 (EST) Date: Mon, 10 Dec 2012 19:33:10 +0100 From: Zlatko Calusic MIME-Version: 1.0 References: <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> <20121210180141.GK1009@suse.de> In-Reply-To: <20121210180141.GK1009@suse.de> Message-ID: <50C62AE6.3030000@iskon.hr> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: kswapd craziness in 3.7 Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: Johannes Weiner , Linus Torvalds , Rik van Riel , linux-mm , Linux Kernel Mailing List On 10.12.2012 19:01, Mel Gorman wrote: > In this last-minute disaster, I'm not thinking properly at all any more. The > shrink slab disabling should have happened before the loop_again but even > then it's wrong because it's just covering over the problem. > > The way order and testorder interact with how balanced is calculated means > that we potentially call shrink_slab() multiple times and that thing is > global in nature and basically uncontrolled. You could argue that we should > only call shrink_slab() if order-0 watermarks are not met but that will > not necessarily prevent kswapd reclaiming too much. It keeps going back > to balance_pgdat needing its list of requirements drawn up and receive > some major surgery and we're not going to do that as a quick hack. > I was about to apply the patch that you sent, and reboot the server, but it seems there's no point because the patch is flawed? Anyway, if and when you have a proper one, I'll be glad to test it for you and report results. -- Zlatko -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx138.postini.com [74.125.245.138]) by kanga.kvack.org (Postfix) with SMTP id 0EA6C6B002B for ; Mon, 10 Dec 2012 14:13:46 -0500 (EST) Received: by mail-wi0-f169.google.com with SMTP id hq12so1361869wib.2 for ; Mon, 10 Dec 2012 11:13:45 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <50C62AE6.3030000@iskon.hr> References: <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> <20121210180141.GK1009@suse.de> <50C62AE6.3030000@iskon.hr> From: Linus Torvalds Date: Mon, 10 Dec 2012 11:13:25 -0800 Message-ID: Subject: Re: kswapd craziness in 3.7 Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Zlatko Calusic , Andrew Morton Cc: Mel Gorman , Johannes Weiner , Rik van Riel , linux-mm , Linux Kernel Mailing List On Mon, Dec 10, 2012 at 10:33 AM, Zlatko Calusic wrote: > > I was about to apply the patch that you sent, and reboot the server, but it > seems there's no point because the patch is flawed? > > Anyway, if and when you have a proper one, I'll be glad to test it for you > and report results. I have reverted (again) the __GFP_NO_KSWAPD removal, and considering that it really looks like there are overwhelming reasons to have that flag, I will *not* take some new patch to revert it. I'm getting convinced that the original removal really was bogus, and had no actual valid reason for it. Part of that is that I noticed that non-THP allocations wanted to use it too. The i915 driver had wanted to use __GFP_NO_KSWAPD because it too didn't want to start some cleaning thread. The whole mindset kswapd is somehow better than direct reclaim or needed when it fails is broken. Some allocations simply *will* fail, without necessarily wanting kswapd to be started. THP - where the high order of the allocation means that failure is inevitable under some fragmentation circumstances - is just one such case. I also reverted one of the "fix up the mess from removing __GFP_NO_KSWAPD" patch, because that one was an obvious workaround that tried to re-introduce the "let's not wake up kswapd after all for that case". It clashed with a clean revert, and it was pointless in the presense of __GFP_NO_KSWAPD anyway. I did *not* revert some of the other fixup patches that tried to help kswapd balancing decisions and avoid excessive CPU use other ways. So some remains of this whole saga do still remain, but they look fairly minimal. It's worth giving this as much testing as is at all possible, but at the same time I really don't think I can delay 3.7 any more without messing up the holiday season too much. So unless something obvious pops up, I will do the release tonight. So testing will be minimal - but it's not like we haven't gone back-and-forth on this several times already, and we revert to *mostly* the same old state as 3.6 anyway, so it should be fairly safe. Linus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx116.postini.com [74.125.245.116]) by kanga.kvack.org (Postfix) with SMTP id CB5676B005A for ; Mon, 10 Dec 2012 15:35:12 -0500 (EST) Date: Mon, 10 Dec 2012 21:35:06 +0100 From: Zlatko Calusic MIME-Version: 1.0 References: <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> <20121210180141.GK1009@suse.de> <50C62AE6.3030000@iskon.hr> In-Reply-To: Message-ID: <50C6477A.4090005@iskon.hr> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: kswapd craziness in 3.7 Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Andrew Morton , Mel Gorman , Johannes Weiner , Rik van Riel , linux-mm , Linux Kernel Mailing List On 10.12.2012 20:13, Linus Torvalds wrote: > > It's worth giving this as much testing as is at all possible, but at > the same time I really don't think I can delay 3.7 any more without > messing up the holiday season too much. So unless something obvious > pops up, I will do the release tonight. So testing will be minimal - > but it's not like we haven't gone back-and-forth on this several times > already, and we revert to *mostly* the same old state as 3.6 anyway, > so it should be fairly safe. > It compiles and boots without a hitch, so it must be perfect. :) Seriously, a few more hours need to pass, until I can provide more convincing data. That's how long it takes on this particular machine for memory pressure to build up and memory fragmentation to ensue. Only then I'll be able to tell how it really behaves. I promise to get back as soon as I can. And funny thing that you mention i915, because yesterday my daughter managed to lock up our laptop hard (that was a first), and this is what I found in kern.log after restart: Dec 9 21:29:42 titan vmunix: general protection fault: 0000 [#1] PREEMPT SMP Dec 9 21:29:42 titan vmunix: Modules linked in: vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) [last unloaded: microcode] Dec 9 21:29:42 titan vmunix: CPU 2 Dec 9 21:29:42 titan vmunix: Pid: 2523, comm: Xorg Tainted: G O 3.7.0-rc8 #1 Hewlett-Packard HP Pavilion dv7 Notebook PC/144B Dec 9 21:29:42 titan vmunix: RIP: 0010:[] [] find_get_page+0x3c/0x90 Dec 9 21:29:42 titan vmunix: RSP: 0018:ffff88014d9f7928 EFLAGS: 00010246 Dec 9 21:29:42 titan vmunix: RAX: ffff880052594bc8 RBX: 0200000000000000 RCX: 00000000fffffffa Dec 9 21:29:42 titan vmunix: RDX: 0000000000000001 RSI: ffff880052594bc8 RDI: 0000000000000000 Dec 9 21:29:42 titan vmunix: RBP: ffff88014d9f7948 R08: 0200000000000000 R09: ffff880052594b18 Dec 9 21:29:42 titan vmunix: R10: 57ffe4cbb74d1280 R11: 0000000000000000 R12: ffff88011c959a90 Dec 9 21:29:42 titan vmunix: R13: 0000000000000053 R14: 0000000000000000 R15: 0000000000000053 Dec 9 21:29:42 titan vmunix: FS: 00007fcd8d413880(0000) GS:ffff880157c80000(0000) knlGS:0000000000000000 Dec 9 21:29:42 titan vmunix: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 9 21:29:42 titan vmunix: CR2: ffffffffff600400 CR3: 000000014d937000 CR4: 00000000000007e0 Dec 9 21:29:42 titan vmunix: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Dec 9 21:29:42 titan vmunix: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Dec 9 21:29:42 titan vmunix: Process Xorg (pid: 2523, threadinfo ffff88014d9f6000, task ffff88014d9c1260) Dec 9 21:29:42 titan vmunix: Stack: Dec 9 21:29:42 titan vmunix: ffff88014d9f7958 ffff88011c959a88 0000000000000053 ffff88011c959a88 Dec 9 21:29:42 titan vmunix: ffff88014d9f7978 ffffffff81090e21 0000000000000001 ffffea00014d1280 Dec 9 21:29:42 titan vmunix: ffff88011c959960 0000000000000001 ffff88014d9f7a28 ffffffff810a1b60 Dec 9 21:29:42 titan vmunix: Call Trace: Dec 9 21:29:42 titan vmunix: [] find_lock_page+0x21/0x80 Dec 9 21:29:42 titan vmunix: [] shmem_getpage_gfp+0xa0/0x620 Dec 9 21:29:42 titan vmunix: [] shmem_read_mapping_page_gfp+0x2c/0x50 Dec 9 21:29:42 titan vmunix: [] i915_gem_object_get_pages_gtt+0xe1/0x270 Dec 9 21:29:42 titan vmunix: [] i915_gem_object_get_pages+0x4f/0x90 Dec 9 21:29:42 titan vmunix: [] i915_gem_object_bind_to_gtt+0xc3/0x4c0 Dec 9 21:29:42 titan vmunix: [] i915_gem_object_pin+0x123/0x190 Dec 9 21:29:42 titan vmunix: [] i915_gem_execbuffer_reserve_object.isra.13+0x77/0x190 Dec 9 21:29:42 titan vmunix: [] i915_gem_execbuffer_reserve.isra.14+0x2c1/0x320 Dec 9 21:29:42 titan vmunix: [] i915_gem_do_execbuffer.isra.17+0x5e2/0x11b0 Dec 9 21:29:42 titan vmunix: [] i915_gem_execbuffer2+0x94/0x280 Dec 9 21:29:42 titan vmunix: [] drm_ioctl+0x493/0x530 Dec 9 21:29:42 titan vmunix: [] ? i915_gem_execbuffer+0x480/0x480 Dec 9 21:29:42 titan vmunix: [] do_vfs_ioctl+0x8f/0x530 Dec 9 21:29:42 titan vmunix: [] sys_ioctl+0x4b/0x90 Dec 9 21:29:42 titan vmunix: [] ? sys_read+0x4d/0xa0 Dec 9 21:29:42 titan vmunix: [] system_call_fastpath+0x16/0x1b Dec 9 21:29:42 titan vmunix: Code: 63 08 48 83 ec 08 e8 84 9c fb ff 4c 89 ee 4c 89 e7 e8 89 b7 15 00 48 85 c0 48 89 c6 74 41 48 8b 18 48 85 db 74 1f f6 c3 03 75 3c <8b> 53 1c 85 d2 74 d9 8d 7a 01 89 d0 f0 0f b1 7b 1c 39 c2 75 23 Dec 9 21:29:42 titan vmunix: RIP [] find_get_page+0x3c/0x90 Dec 9 21:29:42 titan vmunix: RSP It seems that whenever (if ever?) GFP_NO_KSWAPD removal is attempted again, the i915 driver will need to be taken better care of. -- Zlatko -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx183.postini.com [74.125.245.183]) by kanga.kvack.org (Postfix) with SMTP id 2ECC26B006E for ; Mon, 10 Dec 2012 16:29:16 -0500 (EST) Received: by mail-wi0-f181.google.com with SMTP id hm9so1389324wib.8 for ; Mon, 10 Dec 2012 13:29:14 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <50C6477A.4090005@iskon.hr> References: <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> <20121210180141.GK1009@suse.de> <50C62AE6.3030000@iskon.hr> <50C6477A.4090005@iskon.hr> From: Linus Torvalds Date: Mon, 10 Dec 2012 13:28:54 -0800 Message-ID: Subject: Re: kswapd craziness in 3.7 Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Zlatko Calusic Cc: Andrew Morton , Mel Gorman , Johannes Weiner , Rik van Riel , linux-mm , Linux Kernel Mailing List , Hugh Dickins [ Adding High Dickins because of the shmem oops. ] On Mon, Dec 10, 2012 at 12:35 PM, Zlatko Calusic wrote: > > And funny thing that you mention i915, because yesterday my daughter managed to lock up our laptop hard (that was a first), and this is what I found in kern.log after restart: > > Dec 9 21:29:42 titan vmunix: general protection fault: 0000 [#1] PREEMPT SMP > Dec 9 21:29:42 titan vmunix: Modules linked in: vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) [last unloaded: microcode] > Dec 9 21:29:42 titan vmunix: CPU 2 > Dec 9 21:29:42 titan vmunix: Pid: 2523, comm: Xorg Tainted: G O 3.7.0-rc8 #1 Hewlett-Packard HP Pavilion dv7 Notebook PC/144B > Dec 9 21:29:42 titan vmunix: RIP: 0010:[] [] find_get_page+0x3c/0x90 Ho humm.. I'm not convinced this is related. > Dec 9 21:29:42 titan vmunix: Call Trace: > Dec 9 21:29:42 titan vmunix: [] find_lock_page+0x21/0x80 > Dec 9 21:29:42 titan vmunix: [] shmem_getpage_gfp+0xa0/0x620 > Dec 9 21:29:42 titan vmunix: [] shmem_read_mapping_page_gfp+0x2c/0x50 > Dec 9 21:29:42 titan vmunix: [] i915_gem_object_get_pages_gtt+0xe1/0x270 > Dec 9 21:29:42 titan vmunix: [] i915_gem_object_get_pages+0x4f/0x90 > Dec 9 21:29:42 titan vmunix: [] i915_gem_object_bind_to_gtt+0xc3/0x4c0 > Dec 9 21:29:42 titan vmunix: [] i915_gem_object_pin+0x123/0x190 > Dec 9 21:29:42 titan vmunix: [] i915_gem_execbuffer_reserve_object.isra.13+0x77/0x190 > Dec 9 21:29:42 titan vmunix: [] i915_gem_execbuffer_reserve.isra.14+0x2c1/0x320 > Dec 9 21:29:42 titan vmunix: [] i915_gem_do_execbuffer.isra.17+0x5e2/0x11b0 > Dec 9 21:29:42 titan vmunix: [] i915_gem_execbuffer2+0x94/0x280 > Dec 9 21:29:42 titan vmunix: [] drm_ioctl+0x493/0x530 > Dec 9 21:29:42 titan vmunix: [] do_vfs_ioctl+0x8f/0x530 > Dec 9 21:29:42 titan vmunix: [] sys_ioctl+0x4b/0x90 > Dec 9 21:29:42 titan vmunix: [] system_call_fastpath+0x16/0x1b > > It seems that whenever (if ever?) GFP_NO_KSWAPD removal is attempted again, the i915 driver will need to be taken better care of. That decodes to 11: e8 89 b7 15 00 callq 0x15b79f # radix_tree_lookup_slot 16: 48 85 c0 test %rax,%rax 19: 48 89 c6 mov %rax,%rsi 1c: 74 41 je 0x5f 1e: 48 8b 18 mov (%rax),%rbx # 21: 48 85 db test %rbx,%rbx 24: 74 1f je 0x45 26: f6 c3 03 test $0x3,%bl 29: 75 3c jne 0x67 2b:* 8b 53 1c mov 0x1c(%rbx),%edx <-- trapping instruction 2e: 85 d2 test %edx,%edx 30: 74 d9 je 0xb where %rbx is 0x0200000000000000. That looks like it could be a single-bit error, and should have been zero. It's the "atomic_read(&page->counter)" which is part of "page_cache_get_speculative()" as far as I can tell, and it's the "page" pointer that is that odd (non-pointer) value. The fact that %ecx contains the value "-6" makes me wonder if there was a -ENXIO somewhere, though. None of it looks all that much related to whether the i915 driver uses GFP_NO_KSWAPD or not, though. Linus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx165.postini.com [74.125.245.165]) by kanga.kvack.org (Postfix) with SMTP id E2DD26B005A for ; Mon, 10 Dec 2012 16:42:59 -0500 (EST) Date: Mon, 10 Dec 2012 22:42:56 +0100 From: Borislav Petkov Subject: Re: kswapd craziness in 3.7 Message-ID: <20121210214256.GB23484@liondog.tnic> References: <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> <20121210180141.GK1009@suse.de> <50C62AE6.3030000@iskon.hr> <50C6477A.4090005@iskon.hr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Zlatko Calusic , Andrew Morton , Mel Gorman , Johannes Weiner , Rik van Riel , linux-mm , Linux Kernel Mailing List , Hugh Dickins On Mon, Dec 10, 2012 at 01:28:54PM -0800, Linus Torvalds wrote: > [ Adding High Dickins because of the shmem oops. ] > > On Mon, Dec 10, 2012 at 12:35 PM, Zlatko Calusic > wrote: > > > > And funny thing that you mention i915, because yesterday my daughter managed to lock up our laptop hard (that was a first), and this is what I found in kern.log after restart: > > > > Dec 9 21:29:42 titan vmunix: general protection fault: 0000 [#1] PREEMPT SMP > > Dec 9 21:29:42 titan vmunix: Modules linked in: vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) [last unloaded: microcode] > > Dec 9 21:29:42 titan vmunix: CPU 2 > > Dec 9 21:29:42 titan vmunix: Pid: 2523, comm: Xorg Tainted: G O 3.7.0-rc8 #1 Hewlett-Packard HP Pavilion dv7 Notebook PC/144B > > Dec 9 21:29:42 titan vmunix: RIP: 0010:[] [] find_get_page+0x3c/0x90 > > Ho humm.. > > I'm not convinced this is related. > > > Dec 9 21:29:42 titan vmunix: Call Trace: > > Dec 9 21:29:42 titan vmunix: [] find_lock_page+0x21/0x80 > > Dec 9 21:29:42 titan vmunix: [] shmem_getpage_gfp+0xa0/0x620 > > Dec 9 21:29:42 titan vmunix: [] shmem_read_mapping_page_gfp+0x2c/0x50 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_object_get_pages_gtt+0xe1/0x270 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_object_get_pages+0x4f/0x90 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_object_bind_to_gtt+0xc3/0x4c0 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_object_pin+0x123/0x190 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_execbuffer_reserve_object.isra.13+0x77/0x190 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_execbuffer_reserve.isra.14+0x2c1/0x320 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_do_execbuffer.isra.17+0x5e2/0x11b0 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_execbuffer2+0x94/0x280 > > Dec 9 21:29:42 titan vmunix: [] drm_ioctl+0x493/0x530 > > Dec 9 21:29:42 titan vmunix: [] do_vfs_ioctl+0x8f/0x530 > > Dec 9 21:29:42 titan vmunix: [] sys_ioctl+0x4b/0x90 > > Dec 9 21:29:42 titan vmunix: [] system_call_fastpath+0x16/0x1b > > > > It seems that whenever (if ever?) GFP_NO_KSWAPD removal is attempted again, the i915 driver will need to be taken better care of. > > That decodes to > > 11: e8 89 b7 15 00 callq 0x15b79f # radix_tree_lookup_slot > 16: 48 85 c0 test %rax,%rax > 19: 48 89 c6 mov %rax,%rsi > 1c: 74 41 je 0x5f > 1e: 48 8b 18 mov (%rax),%rbx # > 21: 48 85 db test %rbx,%rbx > 24: 74 1f je 0x45 > 26: f6 c3 03 test $0x3,%bl > 29: 75 3c jne 0x67 > 2b:* 8b 53 1c mov 0x1c(%rbx),%edx <-- trapping instruction > 2e: 85 d2 test %edx,%edx > 30: 74 d9 je 0xb > > where %rbx is 0x0200000000000000. That looks like it could be a > single-bit error, and should have been zero. > > It's the "atomic_read(&page->counter)" which is part of > "page_cache_get_speculative()" as far as I can tell, and it's the > "page" pointer that is that odd (non-pointer) value. The fact that > %ecx contains the value "-6" makes me wonder if there was a -ENXIO > somewhere, though. > > None of it looks all that much related to whether the i915 driver uses > GFP_NO_KSWAPD or not, though. Aren't we gonna consider the out-of-tree vbox modules being loaded and causing some corruptions like maybe the single-bit error above? I'm also thinking of this here: https://lkml.org/lkml/2011/10/6/317 Hmm. -- Regards/Gruss, Boris. Sent from a fat crate under my desk. Formatting is fine. -- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx189.postini.com [74.125.245.189]) by kanga.kvack.org (Postfix) with SMTP id 9EB1D6B005A for ; Mon, 10 Dec 2012 16:47:46 -0500 (EST) Received: by mail-wg0-f47.google.com with SMTP id dq11so1717959wgb.26 for ; Mon, 10 Dec 2012 13:47:45 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <20121210214256.GB23484@liondog.tnic> References: <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> <20121210180141.GK1009@suse.de> <50C62AE6.3030000@iskon.hr> <50C6477A.4090005@iskon.hr> <20121210214256.GB23484@liondog.tnic> From: Linus Torvalds Date: Mon, 10 Dec 2012 13:47:23 -0800 Message-ID: Subject: Re: kswapd craziness in 3.7 Content-Type: text/plain; charset=ISO-8859-1 Sender: owner-linux-mm@kvack.org List-ID: To: Borislav Petkov , Linus Torvalds , Zlatko Calusic , Andrew Morton , Mel Gorman , Johannes Weiner , Rik van Riel , linux-mm , Linux Kernel Mailing List , Hugh Dickins On Mon, Dec 10, 2012 at 1:42 PM, Borislav Petkov wrote: > > Aren't we gonna consider the out-of-tree vbox modules being loaded and > causing some corruptions like maybe the single-bit error above? > > I'm also thinking of this here: https://lkml.org/lkml/2011/10/6/317 Yup, that looks more likely, I agree. Linus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx160.postini.com [74.125.245.160]) by kanga.kvack.org (Postfix) with SMTP id E01E56B005A for ; Mon, 10 Dec 2012 16:54:38 -0500 (EST) Date: Mon, 10 Dec 2012 22:54:36 +0100 From: Borislav Petkov Subject: Re: kswapd craziness in 3.7 Message-ID: <20121210215436.GA31536@liondog.tnic> References: <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> <20121210180141.GK1009@suse.de> <50C62AE6.3030000@iskon.hr> <50C6477A.4090005@iskon.hr> <20121210214256.GB23484@liondog.tnic> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Zlatko Calusic , Andrew Morton , Mel Gorman , Johannes Weiner , Rik van Riel , linux-mm , Linux Kernel Mailing List , Hugh Dickins On Mon, Dec 10, 2012 at 01:47:23PM -0800, Linus Torvalds wrote: > On Mon, Dec 10, 2012 at 1:42 PM, Borislav Petkov wrote: > > > > Aren't we gonna consider the out-of-tree vbox modules being loaded and > > causing some corruptions like maybe the single-bit error above? > > > > I'm also thinking of this here: https://lkml.org/lkml/2011/10/6/317 > > Yup, that looks more likely, I agree. @Zlatko: can your daughter try to retrigger the freeze without the vbox modules loaded? Thanks. -- Regards/Gruss, Boris. Sent from a fat crate under my desk. Formatting is fine. -- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx136.postini.com [74.125.245.136]) by kanga.kvack.org (Postfix) with SMTP id 47E0E6B0044 for ; Mon, 10 Dec 2012 17:15:57 -0500 (EST) Date: Mon, 10 Dec 2012 23:15:50 +0100 From: Zlatko Calusic MIME-Version: 1.0 References: <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> <20121210180141.GK1009@suse.de> <50C62AE6.3030000@iskon.hr> <50C6477A.4090005@iskon.hr> <20121210214256.GB23484@liondog.tnic> <20121210215436.GA31536@liondog.tnic> In-Reply-To: <20121210215436.GA31536@liondog.tnic> Message-ID: <50C65F16.1060809@iskon.hr> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: kswapd craziness in 3.7 Sender: owner-linux-mm@kvack.org List-ID: To: Borislav Petkov , Linus Torvalds , Andrew Morton , Mel Gorman , Johannes Weiner , Rik van Riel , linux-mm , Linux Kernel Mailing List , Hugh Dickins On 10.12.2012 22:54, Borislav Petkov wrote: > On Mon, Dec 10, 2012 at 01:47:23PM -0800, Linus Torvalds wrote: >> On Mon, Dec 10, 2012 at 1:42 PM, Borislav Petkov wrote: >>> >>> Aren't we gonna consider the out-of-tree vbox modules being loaded and >>> causing some corruptions like maybe the single-bit error above? >>> >>> I'm also thinking of this here: https://lkml.org/lkml/2011/10/6/317 >> >> Yup, that looks more likely, I agree. > > @Zlatko: can your daughter try to retrigger the freeze without the vbox > modules loaded? > Sure thing! :) Although, the vbox modules were only loaded, no VM was running at the time lockup happened. But, I've just read the whole thread you mention above and I understand the concern. I'll make sure the vbox modules are unloaded when not really needed (most of the time on that machine), in case lockup happens again. Next time my daughter plays online games, I'll tell her she's actually serving a greater purpose, and let her take her time. :) -- Zlatko -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx198.postini.com [74.125.245.198]) by kanga.kvack.org (Postfix) with SMTP id 2BEB86B0069 for ; Mon, 10 Dec 2012 18:27:23 -0500 (EST) Received: by mail-da0-f41.google.com with SMTP id e20so1471776dak.14 for ; Mon, 10 Dec 2012 15:27:22 -0800 (PST) Date: Mon, 10 Dec 2012 15:27:08 -0800 (PST) From: Hugh Dickins Subject: Re: kswapd craziness in 3.7 In-Reply-To: Message-ID: References: <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> <20121210180141.GK1009@suse.de> <50C62AE6.3030000@iskon.hr> <50C6477A.4090005@iskon.hr> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Zlatko Calusic , Borislav Petkov , Andrew Morton , Mel Gorman , Johannes Weiner , Rik van Riel , linux-mm , Linux Kernel Mailing List On Mon, 10 Dec 2012, Linus Torvalds wrote: > [ Adding High Dickins because of the shmem oops. ] I had already noticed, and was about to reply; but only then refreshed my mbox window, to find that you've already done it all for me: thanks. > > On Mon, Dec 10, 2012 at 12:35 PM, Zlatko Calusic > wrote: > > > > And funny thing that you mention i915, because yesterday my daughter managed to lock up our laptop hard (that was a first), and this is what I found in kern.log after restart: > > > > Dec 9 21:29:42 titan vmunix: general protection fault: 0000 [#1] PREEMPT SMP > > Dec 9 21:29:42 titan vmunix: Modules linked in: vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) [last unloaded: microcode] > > Dec 9 21:29:42 titan vmunix: CPU 2 > > Dec 9 21:29:42 titan vmunix: Pid: 2523, comm: Xorg Tainted: G O 3.7.0-rc8 #1 Hewlett-Packard HP Pavilion dv7 Notebook PC/144B > > Dec 9 21:29:42 titan vmunix: RIP: 0010:[] [] find_get_page+0x3c/0x90 > > Ho humm.. > > I'm not convinced this is related. > > > Dec 9 21:29:42 titan vmunix: Call Trace: > > Dec 9 21:29:42 titan vmunix: [] find_lock_page+0x21/0x80 > > Dec 9 21:29:42 titan vmunix: [] shmem_getpage_gfp+0xa0/0x620 > > Dec 9 21:29:42 titan vmunix: [] shmem_read_mapping_page_gfp+0x2c/0x50 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_object_get_pages_gtt+0xe1/0x270 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_object_get_pages+0x4f/0x90 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_object_bind_to_gtt+0xc3/0x4c0 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_object_pin+0x123/0x190 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_execbuffer_reserve_object.isra.13+0x77/0x190 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_execbuffer_reserve.isra.14+0x2c1/0x320 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_do_execbuffer.isra.17+0x5e2/0x11b0 > > Dec 9 21:29:42 titan vmunix: [] i915_gem_execbuffer2+0x94/0x280 > > Dec 9 21:29:42 titan vmunix: [] drm_ioctl+0x493/0x530 > > Dec 9 21:29:42 titan vmunix: [] do_vfs_ioctl+0x8f/0x530 > > Dec 9 21:29:42 titan vmunix: [] sys_ioctl+0x4b/0x90 > > Dec 9 21:29:42 titan vmunix: [] system_call_fastpath+0x16/0x1b > > > > It seems that whenever (if ever?) GFP_NO_KSWAPD removal is attempted again, the i915 driver will need to be taken better care of. > > That decodes to > > 11: e8 89 b7 15 00 callq 0x15b79f # radix_tree_lookup_slot > 16: 48 85 c0 test %rax,%rax > 19: 48 89 c6 mov %rax,%rsi > 1c: 74 41 je 0x5f > 1e: 48 8b 18 mov (%rax),%rbx # > 21: 48 85 db test %rbx,%rbx > 24: 74 1f je 0x45 > 26: f6 c3 03 test $0x3,%bl > 29: 75 3c jne 0x67 > 2b:* 8b 53 1c mov 0x1c(%rbx),%edx <-- trapping instruction > 2e: 85 d2 test %edx,%edx > 30: 74 d9 je 0xb > > where %rbx is 0x0200000000000000. That looks like it could be a > single-bit error, and should have been zero. > > It's the "atomic_read(&page->counter)" which is part of > "page_cache_get_speculative()" as far as I can tell, and it's the > "page" pointer that is that odd (non-pointer) value. The fact that > %ecx contains the value "-6" makes me wonder if there was a -ENXIO > somewhere, though. Yes, just what I was about to say; except I never considered the -6. I was going to suggest it's a new notebook with not-so-good memory, but see that Borislav has since made a better suggestion. > > None of it looks all that much related to whether the i915 driver uses > GFP_NO_KSWAPD or not, though. Yes, no evidence here of anything to delay 3.7 further. I'm running on current git, and no problems observed; but then, I never did see any of these kswapd problems anyway. And, in particular, I was unable to reproduce Zlatko's 1GB of 4GB kept free (on yesterday's tree, with no swap) - I saw about 100MB kept free. Hugh -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx160.postini.com [74.125.245.160]) by kanga.kvack.org (Postfix) with SMTP id 457E46B002B for ; Mon, 10 Dec 2012 19:19:39 -0500 (EST) Date: Tue, 11 Dec 2012 01:19:31 +0100 From: Zlatko Calusic MIME-Version: 1.0 References: <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> <20121210180141.GK1009@suse.de> <50C62AE6.3030000@iskon.hr> <50C6477A.4090005@iskon.hr> In-Reply-To: <50C6477A.4090005@iskon.hr> Message-ID: <50C67C13.6090702@iskon.hr> Content-Type: multipart/mixed; boundary="------------070809000706090606020605" Subject: Re: kswapd craziness in 3.7 Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Andrew Morton , Mel Gorman , Johannes Weiner , Rik van Riel , linux-mm , Linux Kernel Mailing List , Hugh Dickins This is a multi-part message in MIME format. --------------070809000706090606020605 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit > On 10.12.2012 20:13, Linus Torvalds wrote: >> >> It's worth giving this as much testing as is at all possible, but at >> the same time I really don't think I can delay 3.7 any more without >> messing up the holiday season too much. So unless something obvious >> pops up, I will do the release tonight. So testing will be minimal - >> but it's not like we haven't gone back-and-forth on this several times >> already, and we revert to *mostly* the same old state as 3.6 anyway, >> so it should be fairly safe. >> So, here's what I found. In short: close, but no cigar! Kswapd is certainly no more CPU pig, and memory seems to be utilized properly (the kernel still likes to keep 400MB free, somebody else can confirm if that's to be expected on a 4GB THP-enabled machine). So it looks very decent, and much better than anything I run in last 10 days, barring !THP kernel. What remains a mystery is that kswapd occassionaly still likes to get stuck in a D state, only now it recovers faster than before (sometimes in a matter of seconds, but sometimes it takes a few minutes). Now, I admit it's a small, maybe even cosmetic issue. But, it could also be a warning sign of a bigger problem that will reveal itself on a more loaded machine. I will now make one last attempt, I've just reverted 2 Johannes' commits that were also applied in attempt to fix breakage that removing gfp_no_kswapd introduced, namely ed23ec4 & c702418. For various reasons the results of this test will be available tommorow, so it's your call Linus. To better document this whole mess from my point of view, I've attached two graphs. First one nicely shows kswapd frenzy a week ago (blue mountains on a CPU graph). On Thu 06 & Mon 10 (until few hours ago) I run !THP kernels, better memory utilization is, I think, obvious (look at the bottom graph, lots of green is evil). CPU spikes at the end of every day are daily backup runs, which are CPU, NOT I/O bound. Notice L.A. close to 1 on !THP kernels (as it should be), and almost 2 (Fri & Sat 08) when the backup fought with kswapd (and also big CPU iowait times in that case). Finally, todays run is somewhere in between, that's why it deserves "close, but no cigar" qualification. ;) The last graph shows CPU usage in more detail, yesterdays run was on a !THP kernel, todays THP run is the one with red spikes. That was kswapd in D state, in congestion_wait(). -- Zlatko --------------070809000706090606020605 Content-Type: image/png; name="screenshot1.png" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="screenshot1.png" iVBORw0KGgoAAAANSUhEUgAAAkYAAAJQCAYAAABrUqvqAAAABmJLR0QA/wD/AP+gvaeTAAAg AElEQVR4nOyde3gU1f3/3zO7KnfIBgiXgEiohCxGLSFtgGiICt5AqPX2GPRbpX1qLVpFBBXL VykKoaBWhNoq4LfiBbzECyBeIsQm+kiQJhCsSvgpBCSEZCOlEXLZ+f2xnGF2dmZ2dndmd87m 83qefXZ3dubMez7nzOyZz3w+5wib3n5VEkURgiAAogQAgc+n3tln5XJJkuR39hkABClQjiiK EEVRXs7WU5al3l6SJIiiKC9jy1l5giCgo6MjRJNSp9vtht/vR0dHBzo6OuD3+zW1a+1DEAT4 /X6IogiXyyXvU+s4tZap9Sv36ff7Q46f7VP58vv9QeuwbZT2VK7LjlO5P+VxS5Ikl+lyuUKO G4BcNtU/1T/VP9W/UjfVP9V/Z61/N8KgPCBBEOTv6gNVNxKlKLUR1AZmy1gDUhuSVa6WcZW0 t7fD7/fLL/X+zBynXuUpj1upXX3CKI9H/TmcDr0KD9cg2XdlY2TbsEYlimKQHfV0q6H6p/qn +qf6Z2VS/VP9d4b6dytF6m2ubhRaCIIAAdo9bGWFajUSpQG0Gh7rSWodgJL29nZ536xijIyi 1RC1NBsdj9a66vLCNUZlw1ZWpvqk0CuDNQSlFnUZ7OTQ0smWUf1T/evpo/qn+qf6D/1Na111 eVT//NW/W/ml/4ARuoUQBEEQBEEkI42NjZDajgEARDNuJoIgCIIgiGSkoaEB+/fvBxDwSrkB hLjJevXqlRh1BEEQBEEQcWTnzp0ATj/CE7We+REEQRAEQXQmWJyTGH5VgiAIgiCI5EfuGCkj 1gmCIAiCIDobLKzIbZSCRxAEQRAEwSulpaWaywsLC0OWyUMbsGdq6gGZCIIgkpGMjAzLtsnI yJBfBEE4j8LCQhQWFmLChAnydz1Yx8gNBA+8RBAEQQRTW1truJw6RgThXE6cOIF///vfcLvd +OGHH+B2a0/6IWelxVMcQRCRU1dXhzlz5iAvLw8jR47E5MmT8fbbbwMI9lh4vV5cdtllWLFi BVpbW+XftaA/coIgOgP79+/H9u3b0aNHD4wePRpVVVVIT0/XXJd1jMLOlUYQROKoq6vDTTfd hNtuuw333HMPUlNTsXv3bjz99NOYOnUqgNNeixMnTuCrr77CY489hsbGRixYsCCR0gmCIBJO c3MzcnJy0LVrVwDARRddpLuu7DEymoOFIIjE8uSTT2LGjBn41a9+hUGDBuGss87CmDFjsHbt 2pB1u3TpgvPPPx9PPvkkNm3aZMn+tTxLymV79uzBr371K4wZMwaZmZm45ppr8N577wWt//bb b+OSSy7BqFGjcPPNN6Ourk7+ra2tDQsXLkRubi7Gjh2LtWvXBpXf0dGBZ555BgUFBbjwwgtx 3333oaWlxbT+kydP4pFHHkFubi5ycnLw/PPPh6xjpC/WGKKpU6fik08+CdIzZswYNDY2RlUe QRCRcfToUXz66acoLS0NehlBj9IIwsH885//xBVXXBHRNvFMpLjjjjswffp0fPLJJ6iqqsLD Dz+MN998M2idLVu24LnnnsMXX3yBCRMm4OGHH5Z/W7VqFb799lu888472Lx5Mz777LOgbdes WYPPPvsML774IrZu3Qq3243ly5eb1rdixQocOnQIGzduxMaNG0PKD6evtrZWN77IDDfeeCNe ffVV+XtpaSkuuOACpKamRl0mQRDmiST4miGPY0TB1wThPJqamtC/f39T6544cQJVVVX4wx/+ EHFnKloEQcDevXuxe/du/PDDD/jpT3+KZ599NmidZ555Bueccw66du2KW2+9Fdu3b5d/e/PN NzF//nykpaWhb9++mDdvXtC2r776KhYtWoT09HT07t0bc+bMwZYtW0zre+edd/DAAw+gX79+ SEtLwwMPPBCyjpG+WLnmmmtQUVEhe4jeeustTJs2zbLyCYIIz4kTJ7Bnz56wwddBWWnUMSII Z+LxeHDkyBEMGTJEdx32mKdLly4YOHAgrr76atxxxx0AoDtwq1UDuj7//PN47rnnsHDhQuzb tw/dunXD//7v/+Kqq64CANTU1GDJkiVyx0lNfX09Bg8eLH8fNGhQ0O+HDh3CxIkTo9Z+5MiR oDKV+zKjL1a6d++OyZMn44033sD111+P7du344knnrB8PwRBaLN//3589913GDhwIIYOHWoY fC0P8AhQuj5BOJXx48dj8+bN+M1vfqO7jtGjnh49eqC9vT3oDqm1tRU9evQwtf8uXbrgxx9/ lAMXv//++6DfMzIy8PjjjwMIxANt2rQJCxYskDtGv//973HHHXdgxYoV6NWrl+xVYqSlpeHA gQNy5+7gwYNB5aenp2P9+vXo3bu3Kb1q+vfvj0OHDmHYsGEAAh0tJeH0maVLly44efIkzjrr rJDfbrzxRsyePRs9e/bExIkTZVsSBGE/kQRfMyj4miAczB/+8Ae88MILWLNmDb7//nu0trai qqoKt912m6nt8/LysHTpUhw8eBDt7e04cuQI/vznPyMvL8/U9qNGjcLq1avR0tKCQ4cOYdGi RUG/33rrrdi2bRtaWlpw8uRJtLS0oE+fPvLvJ06cQGpqKrp27Yq6urqQTLnp06fjT3/6Ew4f PozGxkYsXbo06Pcbb7wR8+bNw/79+9Ha2oqdO3cadhLVTJ06FYsXL8bRo0dx5MgRLF68OOj3 cPrMMmrUKLz++uvo6OgI+e38889Hly5d8MQTT8TtESdBEAEiCb4OmUSWRr4mCOcxZMgQvPLK K9i9ezemTZuG0aNH4/777zcdp7JkyRL4/X7ceOONOO+883DttddCkiQsWbLE1PaPPvoo3n// fYwZMwa33HILpkyZEvT7bbfdhmeffRZjxoxBfn4+3n//faxcuVL+vbi4GEuXLsV5552HoqIi XHbZZUHb33HHHRg+fDimTJmCyy+/HHl5eXC5XPLvt956K3JzczFz5kxccMEFWLBgAa699lpT 2gHgzjvvxMCBA3HllVfiiiuuQG5ubtDv4fQpM9KMstMWLFiAF154ASNHjtRc54YbbsDx48cx fvx409oJgogdURRRUFAgB2EXFBToPo73+/3w+/0QNr39quwqGjjkXABAr1694qOYIAhCQX19 Pa6//nps27Yt0VIs5Z133sG7774bEphOEIS97NixA/3795djDQ8dOoSGhoagR+bsejOgb3cA gOhyuSCKomXBmARBEJFw7733Yt++fTh27BieeuqpEK8N77S2tuLll1/G5MmTEy2FIDodmZmZ OHr0KMrLy1FeXo6jR48iMzNTc11BECCKItyiKFKMEUEQCSM/Px9FRUVobW1FYWEh7rnnnkRL spRRo0bhZz/7WchjSIIg7Kd79+648MILg5aVlpZqjmfEHERuURQpvoggiIQxffp0TJ8+PdEy bCOWASIJgogfLL7RTZ4igiAIgiCSkXDTf2jhpnR9giAIgiCSEa1HZnqdJXmAR5a3T4/TCIIg CILozAiCEBjgkeXuEwRBEARBJDPhJpJ1s3nSaEqQ0xz6/HMAwCDVYHAEQRAEQfCDz+dDbW0t WlpaAADdunXDiBEjgkboV+N2uVzUMVLQfuIEPlu2DILLhalr18J15pmJlkQQBEEQRBTs2bMH I0eOhMfjAQA0NjaipqZGcxR6SZLYuI4iXC5X0CSTnZmaV17Bf48cwfHvv0fNyy8nWg5BEARB EHFETtcnjxFw7MAB7H7pJfn77pdfxvBJk9Bj4MAEqiIIgiAIIhqysrKwd+9e1NTUAAg8SsvK yjLcRgTAZbo+m9BRPWFjfX09ioqKkJ2djRkzZqChocFwuZIdq1bB39Ymf+84eRLbV6yw90AI giAIgrCFlJQUjB07FhdffDEuvvhijB07FikpKZrrsmQ0UfmFJ2prazVHlC0uLkZWVhbKy8uR mZmJ4uJiw+VKJj72GGZ8/DGOXH01Znz8MWZ8/DEmLlpkqe7GxkZLy1MjlJXZWj7pN4b0G0P6 jSH9xpB+Y0h/KD6fD5WVlSgrK0NZWRkqKyvR3NysuS7rC7mVC5KBiooKlJSUoGfPnpg5c6Y8 1YDe8njTpvBI2YHUrZut5ZN+Y0i/MaTfGNJvDOk3hvSHEknwNYCAx4jFFiVLjJHP50Nqaipm zpyJ1NRUNDU1GS4PorkZ2LABOHYM/ffvx/Dhwy1/7/fdd7aUy957HDxoa/mkn/STftJP+km/ XfqH9uoFbNgAIYFzDAql778lsRij1LRhAIBevXolTFCkZGRkBD1Sy8vLQ0lJCdLS0lBfX4/p 06ejoqJCd7kWy5YtwzPPPBOvQyAIwiJ8goCUJPF+E0RnZd++fZaV5fP5sHfv3pBxjJRxRtu2 bQMApA/ofdpjJAgCRFG0TEgiycvLw5o1a3D8+HGsXr0a48aNM1web6yscC2Ob91qa/mk3xjS b4zd+lvT020tn+xvDOk3hvQbY4f+SIKvGcLWD9+RWHxRn75DAPDhMVJnowGBgOz6+nrce++9 qK6uRnZ2NpYvXy57ibSWa0EeI4LgE0HwQZKML3oEQTgbKztgehPGKqcFCfEYdUjt8KMDHVK7 ZULiActKU74AIC0tDevWrcOuXbuwbt06ufOjtzze8NjjVkL6jSH9xtil33cqRnJauvbjcasg +xtD+o0h/cbYpb+wsDDkpUV7ezs6OjogfPTBm/IDeU/fYQD48BjZCXmM+MFpMSVO02MW1rHg UTtwWr8HTeQxIgjOsdpjFG7SWOYxGtC3OwCcjjFKlqy0WGhra8Phw4fxww8/ADhdOVa+s5dd 5R/futXW8km/9rtPEFA9ejRa09O51A8AB7xebu3P9E9Lr+BWP8/tn/STfqv0f/XVVzh8+LDt YyYZIZR+WCIBgXT9Pp6hAMhjRB4jfnCKh8anuLFwgp5IIY8RQRBOgXWW4gXzGA3s1yMwkSwL vE6WAR6djt0VTs+YjbFbP+9ZUfVDhthavt36KcbIGNJvDOk3hnf94ZDnjv3w/Tck9hiNYowC kMeIH8hjZA3Jop88RgTBP4nyGKWlBkbeFtngjuQxig+897idpN8XRVwceYyM4V0/eYyMsVO/ TxC41g/wbX+A9MeK3++HJEkQ3n/vNdlj1Lf/cADkMSKPER84KS4mWTwuAN/6yWOUOJzivSX4 J1Eeo759zgoMeN3R0YGOjg74/f64CnEilJXGl/4DXq+j9B/wernPSuNdP2WlJdb+POvn3f7J ot/KrLTKykrU19ebfiImj2P03qb1clZa/wEjAJDHiDxGfEAeI+tIFv3kMUoMTjoXo4G8Xc6C dZZi5dixY6irq0NzczMGDhyIwYMH48wzzwxZj3mMencXIYpiYBwjURRpHKM4YVWF63F869ao Ym/MEg/9dmK3ft5jdHjXTzFGxpB+Y0i/Mbzp79WrF7KyspCTkwMg4EHas2cPjh07ZrgdxRhp wLvHqLPc/TjpLjVZPC4Af/qV2sljlBh4bj/A6WtmZ7l2Oh27OmB+vx8NDQ04cOCA3FkCTnuM enULOIpE5i0SRdEWIUQwvPW41ZB+Y3j3uPCunzxGxlD7MYZ3/by3H7v1i6KItLS0oE6REuYk Ej764E2JTQlCI18HII8RH5DHyDp41k8eo8TDc/sB+NefbNjdAVPDPEZ9ergA0FxpQSRLVlo0 2VpO0m92fcpKI/1MN3unrDRqP51VP+/th+lP5Fxpcn+o9MMSiT1O69Un4EYkjxF5jHiAPEbW wbN+8hglHp7bD8C//mSDdZbiBfMYeXqdEZgrze12AwA6OjriKqSzYneF0zNmY+zWz3uMAu/6 KcbIGGo/xvCun/f2k+iRr91udyD4Wp40LUkepZWWlmLSpEkYNWoUJk2ahI8//hgAUF9fj6Ki ImRnZ2PGjBloaGhIiL7hw4fbWn6PggJbyyf9oSjvNs+sq7O8fCV22593/SV142wtn9q/Mby3 H971895+7NZvBkEQAh2jZIoxmjt3LubPn4+qqio89NBDuP/++wEAxcXFyMrKQnl5OTIzM1Fc XJwQfU7qcTtxrjHe73h4v+PkXT95jIyh9mMM7/p5bz/x8hiVlpZqLmd9IVG5IBlIS0uDcgiC AQMGAAAqKipw++23o2fPnpg5cybKy8sToo/3HjfpN4b3O07e9ZPHyBhqP8bwrp/39pNojxHr N4iSJJmeR4QHFi1ahLvuuguZmZm4++678dhjjwEAfD4fUlNTMXPmTKSmpqKpqSl04+ZmYMMG 4Ngxuedq9fu+9ettKZe9H121ytbynaaf3eElUr8HTbKOluxsy8u3Wz+ApNG/MPsFLvXzfv4m S/vhVT/v7Uepv/3oUWDDBgi1tbCC0tLSkJceQVlp7EvvlCEA+M5Ku+yyyzB//nzk5eWhvLwc ixcvxpYtW5CXl4eSkhKkpaWhvr4e06dPR0VFhWYZnSUrjffsNadkpQmCD03wyN8TrScaeM7K oay0xMNz+wH4159s2P3IrrS0FIWFhfJ3lpU2qH/PQFaaJEnw+/3w+/22CokXzBMkCAJcLpc8 FkJeXh7WrFmD48ePY/Xq1Rg3zl6Xux68P6Ml/cbwHqPAu36KMTKG2k8oyk4Rj/qV8N5+7NZv FuGjD96UWAC2p+8wAHx7jN5//30UFxejrq4O6enpmDdvHi699FLU19fj3nvvRXV1NbKzs7F8 +XKkpaVplkEeIz4gj5F18HzHTB6jxJMs7QfgT38ykqhxjAb17wkAEF0uF1wuV9IEX0+aNAkf fvgh/v3vf+PDDz/EpZdeCiAQlL1u3Trs2rUL69at0+0U2Q3vPW7Sbwzvd5y86yePkTHUfozh XT/v7ccO/T6fD5WVlSgrK0NZWRkqKyvR3Nysua4gCJAkCcK2j9+RWAA2zZUWgDxGfEAeI+vg /Y7fgyY0wUMeowTBe/tRwpv+ZMTKDlh5eTlGjhwJjydwjW5sbMTXX3+N8ePHy+swj9HAfj0A IJCun0zjGMVCZ5sr7YDXC58gOE6/2fWdMlea13tA1sPzXEu86qe50pyhPxnaD4/6k6X9MP2J nCuNOYmEraVvyx6jlNSzAZDHqDN5jAB+75Ccop93jxHvd8zkMUo85DEirIR1lqzA5/Nh7969 aGlpAQB069YNI0aMQErK6euE0mOUlFlpTsfKCteifkhgyIVoRrU2g936eX9GznuMAu/6KcbI GGo/xvCun/f2Y4f+lJQUjB07FhdffDEuvvhijB07NqhTpIQ9ORNKPyyR/H4/JElC3/6BUS3J Y8SvxyiSOzeneFyixSn6BcEHALLXKNF6IoXZkXldeNOvtD95jBIDeYwIK7GyA6Y3oKPWOEYD +naH3++HSPFF8YXueIzh/Y6Hd/vzrp88RsZQ+zGGd/28tx+79LNOUGFhYVCHSE17ezva2too +Dre0Fw/xkQzV04kjw3jYX+7HmMC1H7CQXOlGUPtxxje9fPefuzQ73a78Z///AeCIOA///kP /vvf/+LMM8/UXLejowMdHR2nO0Zut9tyQbyRDFlprenpEWWlOU1/NFlpkRyHHfrVWWnR2DWR +pler/cAl1k5SvtTVlri9POa1UVZac7Sb3VW2sCBA7Fz506ce+652LVrF3bs2IGMjAzd9V0u V/LNlWYFFGPEB06Ja1DHGAF82ZRldQFwZIxRuCxLijFKLLzH6PCuPxlhnaV4wWKM0lK7nc5K Yy/CfuyucHpGbgzZ3xje9VOMkTHUfozhXT/v7ccO/VrB13oB2W63O/CyXAVhCD0jN4b3Z+S8 29+p+s2OzUUxRsZ01vZjFt7189x+7JyJQa8jpIfIPEUUfB0f6I7HGN7veHi3P+/6yWNkDLUf Y3jXz3v7YePwWQ3LRguXlQacSkb76IM3JVEUIYoievUJNAqKMaIYo0QQ6R0DxRhZg9NjjNjI 4lq61Nopxij+8B6jw7v+ZEH5f5SoGKP0Ab0B0FxpQVBWWmL1H/B6KSstAfqdnpXG7GvG/pSV lrj2w2tWF+/6eW8/ynpoTU93xlxppR+WSKIowuVyoUevQQDIY0Qeo8QQjcfICaM1k8fIXph9 tTxB5DFKPLx7XHjXnyw4wWM0sF8PAIDIpgOhrLT4YHeF0zNyY8j+xvCmn3WK2GeKMTKG2o8x vOun9mMNIoCk6hi1tbVh+fLlKCgowIgRI+SBnOrr61FUVITs7GzMmDEDDQ0NCdFHWRXG8JxV AfBvf271Vwa8dZSVZgy1H2N410/tJzZYX0gUBAGSJKGjo0N35YyMjJCX1+vFL3/5S3z77bfx U22ClStXYsuWLVi5ciW+/vpr1NbWAgCKi4uRlZWF8vJyZGZmori4OCH6eO9x0x2PMbzbPxL9 0Ux9Eqn+SPdBHiNjnNR+ooH0G0PtxxqE0g9LJOYt8vQdBsBcjJHf78fzzz+P7du3429/+5ud GiNi4sSJePTRR5Gfnx+0PC8vDyUlJUhLS0N9fT2mT5+OigrtiyjFGCUGJ8QYRTOWRmeNMbJz 3BHlPpg2rdghZntUeoAc/fUI++A9Rod3/cmCE2KM0lK7QRCEgMcomow0URRx3XXXYfv27VZr jInDhw+jrKwMo0ePRkFBAT766CMAgM/nQ2pqKmbOnInU1FQ0NTWFbtzcDGzYABw7Jve8rX7f t369LeWy95bsbACne97h1je7Xrz0H121KqL1mYfAKv2R2kOtg9n/+Nat8AkCN+1HS78d75Hq Z/WhV89s+bTqwHoLs1+wxe5Obf9O0c/qyWntp7Po5739sPeW7Gy0Hz0KbNgA4dTTnljx+Xyo rKxEWVkZysrKUFlZiebmZs11WX9I2Fr6tsSeq6Wkng3AfFZac3MzJk6ciJ07d1pyAFbw85// HIsXL8b48ePx+eefY/bs2fjss8/IY2SwrlPukOLhMQq3Dys8RiwzKh4eFStINo+REzPrkhne PS68608W7PIYlZeXY+TIkfB4AtfnxsZGfP311xg/fry8jjIrTRTFgMeIDfAYCX6/H6+99hrG jh1r2QFYQW5ublAgOfOG5eXlYc2aNTh+/DhWr16NcePsDdLUg/dntLw/I2djltgF7/Z3mn5l 1pkZpqVXRLxNJPDe/jtb+4kU3vVT+4kN1hcSyra+K/cievYeDCDUY8Qyu5R06dIFWVlZWLp0 KYYNG2av2gg4dOgQ7rvvPlRXVyMtLQ3z58/HxIkTUV9fj3vvvRfV1dXIzs7G8uXLkZaWplkG eYwSQ6QeCKWnxokeI0HwcRHv4iSPkbpM2SME8x4jvXV5gBcvoxLePS68608W7PIY+Xw+7N27 Fy0tLQCAbt26YcSIEUhJOX2NYB6jwWm94HK54BZFEWwsIz1qLXrWFw8GDRqEl156KWR5Wloa 1q1blwBFwezbt8/WlMfW9HRbUx7t1n9861bLUk61/mQOeL2w8y+zNT0dsDHjNB7tx06qR49G 9u7dtpU/Lb3C1pR93u1P+o3hXb+V108teLR/SkqK6SdbLEvfzb4YUVdXhzlz5mD37t0YPXo0 li1bhkGDBsWuuBPC+zgQThyHw4MmmL3HG1JTE3H5kcC7/QfUVZu2ZTTYbX9exzFinXje2w/p N8bu9sPrOEbMW2SH/UtLSzWXa00m6/f7A4/TzBT8pz/9CdnZ2aioqIDX68WiRYtiU+pQaK60 xM+Vc3yr+bl+2BxZenNpaR3fAa/Xct3qudLC6XJa+1HOlRbJXGPRtJ9o7G92rjSmX71+9ejR ltnLzvZfPXo0l3N18T7XGO/6le+RXD+dpt/OudJYJ6iwsFCzQ8SQs9I+2bZRAgIjPurFGOXk 5GDLli1ITU1FQ0MDrrzySsel6VuJ2RgjJ8UDaA2Ex1OMUTRawsWfKOuHfQ5XZ9HEBSVTjBFg Pj4nmuOLNMbLihgjJ52nekQSG+g0eI/R4V0/4KxreTSo2z/rLFlBaWkpCgsL5XflMgaLMRo6 KCUwjpGZ6UDYGEAA0K9fP+0xgAhTWFnhWlBWhTH53ioA0Y3abAYe7a/sFNk9crTdWYG8j3x9 W/q7tpZP1x9jeNXPzmHKSosNFlbkZp2iaAZ5JCLHrme0bDyfAXXVQSMwW42d+gFAKtD2QERz 168Ve1RTMyQaeabhNUaBYXeMzvk1nxjGMEUSL6aFlv5Yy1Ri9/nLa4wUcOocroOtMWoUY2QM rzFGrP3bYX/lY7Rw+P1+AKcmkTXTKVLOk6b1ndBG7Zmws8ftQROmrc62rXyAf48Riz+xAw+a uLrj1/Ka6XlcYvGwKbc1Y/+I91V5+kaAZ48Rz+cvu7Hh1f4eNHF3/gZR6YEHTVx7jOJhf7O4 2QejzhFP6fpOx/bZxQGstmcPAOzX32OM9v1muLt+sx4luz1GvNzxh3Q+TsXolNSNC4nHsqpT BERm/8C2kT2258X+ISjOXzuJj377fEa26j/V/u3EVo9RpUf3+mkVtrefHOvDdPSy0rRgA12L rEPEXEhavPzyy7jrrrtCls+aNQuvvPJKhDKdS6RZaQe8XvgEwTFR/d5GL6ZVp8txNGazMZyk P9asNGV9eL0HQurH6z0Q9D3S+tPbDyo98G7Jx7TV2dxkpSmz0QDI+pVZOkAgLsuDJl37R9LO 1PY3Wl+tz6jdyPo1stKsrId4nL92lW+XfrmeeNd/qv3zpl/ZfnjOSvM2ejFtdbblWWksE035 Coc88rUkSejVJxD4pM5KGzduHF599VUMGRJ8t/fdd9/hpptu0p1zjFciyUoDjDMBIomNiSV7 RtgRfGfelMNXVhrTL+nc8SgzoJhuZdAwG7FZaUNB8IUsZ+XoZamFy7TS2k4QfEGPc5DjvKw0 LS+QMntOqZ+1HaXNgNCsMDPHp9xPuG3Uc9+Fy5aTf1fZXr2uk+pBD+X5q3cOOBWfIMBTefo7 6Y8/4a6fTkfd/llnyS70stLOHhy4lohm4ouUWWlK+vXrB5/Pp7EFwVDP22R3hbNZxu3Cbv3q Z+RmH+MwO+vNk8WWM++Bcj0rM9R4sL+yw6E+dqV+O+YcY/Y3snm4/foEQQFpyXUAACAASURB VHd7XmN0GDy0HyNIvzF6+q26BvEcYwTYb3+GnteIjWNkaoDHc889F1VVVSHLq6qqMHLkyNgU djJSDILVrfgjKslO/qyKWOykjHGJ9WKktT2zv13DAURjf6WWcLYL135iPS51jFEkHSQz++a9 /ZN+Y+Kh365zF3DG9TMWeGw/WjFGenFHzFEkhhvDCAB+97vf4cEHH8S2bdtw7NgxHDt2DNu2 bcMDDzyAO+64IzbVnQzbx3E51eO2a4Zxu+8Y6oeY67hEc3w+QdDNiorUM6W3nVPvmJW6jWwX Tr+ep8nMfoHgrLRY/4C0jsOp9jcL6TcmWfVbdb3m0WOkvA7YZf/S0tKgVzjcZjpGkydPhsvl wtNPP40vv/wSAJCZmYmHHnoIl112WcyiOxO2zxXFetyVHtiRHRLvcUTU2WjRZCopqakZEnMZ TJfWeFHJcMe8OoL1I42L08tKU8cTaeFBE5oUcUhaqO1vRV0r4fGOWQnpNybS9h8p5DEyxq72 o350ptc5Yv0h0e/3w8zo15deeilee+011NTUoKamBq+//nrSdYqiyUozs57yPd9bpR+VH2P2 DMsKYVk6dui3OytBOVeRnj511peW3ZTeITZX1gGvF17vgaCsJw+aorIDcHoUbVkHs7/3ADxo ckxWGtPJ7KC0m1In0x+SraYx9xjbTitbzahdK7PS1HbXq1etd7Vutf319p8I+0d6/tpVvl36 kyYrTdX+edGvbD88ZqWp7W9HVpqZZQDkvpBQ+mGJxAKOeqcE7ubUWWmdDSuz0tQZMUYZMrFk z6iz0oDwGQrKrK1Ew/Qrs+mU2VBMK6DvdtZbrwke2cug9c7WUWataRGSyaUoWyszimlKNCFa tdDISmO20KoHs3OrqTPalFmBDL0MNOU8abI2rWOoVHmPcpqCMhiVWXFOhPesqGTTbyaj12pi zZzkOStNnRUbj6w0NSFZaQDQ0dGB9vZ2ww3b29vx4osvyt+XL1+O7OxsLF261C6tSYmdIy8D /D/jj2WunHBBxkovQ7jto4V3+xvp1xomgS0PZzv2Oxtbio007EFTRHYP97hNrd/qWLtE2t8K SH8oQXMFJkq/unMfJTzGGAWNXG+D/dXxRUZxRn50oENqNxd8DQButxt+vx9lZWUAgDVr1uCp p57C//3f/1l2AFayatWqoOlK6uvrUVRUhOzsbMyYMQMNDQ0J0aWVlWNlFgSPz/iVx280Vw77 Iw2H0To1NUM0U/uj/QNVb8ej/ZXYrV8vxiiaDDQtlPrtyC7i3f6kXwPFHzOP+pXtnGKMQmGD Og4dOhS9evVCfn6+7qM0URQDL6MRr9Xccsst2Lx5MxoaGlBUVIS77roLRUVFVum3jD179mDN mjVBy4qLi5GVlYXy8nJkZmaiuLg4IdqszMrRgsc7NiWxeIzMdG6i9dgp68qo3pxsfzP2Yfq1 jjHcGFFmII9pKI7wWFgE6TfGbv1ceowU2GH/EydO4F//+hd8Ph/OPvts1NTU4OTJk5rryh0j AGEDr5ctWyY/Nps7dy6WLFmCOXPmYNeuXZg7d67lBxILra2tmD17Nh566KGg5RUVFbj99tvR s2dPzJw5E+Xl5QnRZ5SVo3zX+z0cPN7xKP8Y7J7dOpa50sx0AHi0v5KS7LqovGdmt9Gyv5WP u5T2t2PICh7vmJXYff7yqF+Jnn6rbmLttj95jELZvn07UlJSkJOTg379+iEzM1POrlcjZ6WZ KXjt2rXyY7M+ffrguuuuw7PPPmudcgtZtmwZfvKTn+Caa64JWs5G7545cyZSU1PR1KRx0Wxu BjZsAI4dk3ve4d6Zh0PvdzbbNPt+58/elb970CRvr37X2o9PEPT3c6qnvXBjdtD3WPWr3/et Xx/R+qbeKz2y3pbs7ODjOmW/WN49aJK/M/uH207PTprrK/TL9tcpJ9b3SO3PxoUKayeVfjbL daT1oNf+mf3M2p/t33Q9q/TrnX/xtn+k7V+tP9Htx+x7suuP9DoZqX6z1+tw+o+uWmWJzkTa v/3oUWDDBggWTV4/ZswYnH322fLgjV26dIFXZzxBSZLQ0dEB4YMtr8vuor79A71BdVbakiVL 8OKLL6KoqEj2EG3atAmXX365PButU/jJT34SMiFubW0t8vLyUFJSgrS0NNTX12P69Om6c7zZ kZWmNVeXXqaUujx1Bo+WDmVWBYOXrDRlRp1WVpphNpUNaGWHGI6zo5EVZVRWPDEzPhAAw6w6 rewwLbSOVVmHrP1bXp86WWlK3YmuByPUGaW8ZRYlm36trLRY5rE0qyFau/E8zx6g3X6sfGSn F2itNVfakMG94ff7Ax4jv98f0plQMnfu3JDHZldeeaXjOkUA8M0336C2tha1p3qb7D0vLw9r 1qzB8ePHsXr1aowbNy5umpQdG3WMhdYfhNl4Fi14f0YeS4yRGSjGxRi79ZP9jSH9xsRDv15G qxVQjJExdrUf1gligdjhkGOMIgnC5hHWucvLy8Pu3bsxZ86chOgIF+NiFODaGWJctGKMrPQu RBtjZFYD7/ZPVFaaVZD9jXGS/mjiduKi36LUeS1o5OsAenVvd/sPhyAIp7PSJElypPcnVmoV zyjT0tKwbt067Nq1C+vWrUNaWlpCNOndMRulK0fSMeD9jo08RsbwesfG2nCi7G9V8Cyv9meQ fmN4108eo9gQRTHQH3K73XC5XEnZMXIKyo6NmTvmSCfqVOLEO85IjsPJWWlmcJr9I/W2xcNj ZGe8mJ5+q/bpJI9LNJB+Y3T1W+RFIo+RMXaNY6R8V39WwmYBEeW8feoY2TZXmtZcUUFzPem8 h5uzir3Lc3ZFOFdaNHNIRTNXTjg7Kee6Us6VFsncWWbf2ctovXD2CnnXmavLKXOlmbaPjn6z 72bbv+X1Gsb+Tp8rTa3f6vKdqt8pczWGtB+T108n2v/41q1ctB/lHItq+1s9V1okI1+zjpHw 0QdvSh0dHQCAfmmBkaJprjRrs9IYLFPGTKaP3rxgmpk/GvOkAc7JSguX0REuKy3eGGVXaeLg rDTTNrQ5Kw2wMbvQRFaanj4nkGxZXWb1253pZRajeSblbOIYssbMauhMWWnKueHszkpjtLa2 4p///CcKCwtRWlqqmZU2fFjfQFZaW1sb2traws6VRlgDmysqHGaDrdXw/oycYoyMicT+0TyK paw0Y+xu/6TfGNJvDMUYaXPixAns2bMHbrcbP/zwA9xut+66giDA3dbWlrTB106ks8W4AIFO ntn7mAF11ZDA/tSt9y4km/3Vd91mvJhGUFaaMTzGWCgh/cbwrp+bGKNKD6Dxr2CH/ffv34/v vvsOAwcOxNChQ1FVVYV0nRtweeRr9hiNOkbxge6YjZm2OtvW8qOxfySeF7vtXz16tK3lk8fI GF7vmBmk3xi1fqsf/ZLHyBg72k9zczNycnIwYsQIeDweXHTRRbodvKApQQRBgMvlslwQEYoV WTlGf9S83/GQx8KYITU1tpav1m/1RMe825/39k/6jeFdPzceIx3ssH92dja6du1qal25YyTC BZfghtRhuR7uiFdWWqxZOOr9RptVEa+stLBZSzpZLU7JStPLDkxEVtoBr1dzuU8QZG8S0ytn KzokK+2A10tZaVr2oaw0R+mnrLTE2t/qrLRIcAluiHBB2PjWK5LL5YLL5ULfAecAoKw0u7PS YkE575q8D4dnpYXLuNDKSjA9x5cNqLOXwmqJY1aaVkxRiiSFeHbYvGSmMNAfSSYZZaVFB2Wl RYbV2WxGWWns2kVZadaiPN54ZaUZIWelDe0HABBZp4gepUVPJI8brIixMPpzSZZn/HZ1iniP cTmgMyu0mmgfgdmpXzl2l10w/VY/AmTEq/3bBek3hnf9FGMUG5IksWQ0kTpGcSTWGItwF3ze n5E7McYokk5assUYRdtB1WunFGNkDOk3hvQbQzFGsSNJEkTh1AVMcsBAW07CjjtOnyBw77Hg /Y4hrP1jHPrfCR6jWLxtCbd/jNjtceS9/SeDfjsfsfNuf/IYxYbsMZIkCX6/H36/P6GCOgt0 x2yMEz1GkZBsHqNo0fvz4t3+vLd/0m8M7/rJYxQbfr8/uGNEI19Hn5UWdi4wm7JyYs2qYOWw bCa7shLC6XF6VlqsWVFWZnGos9LYnEMeNIXPnosxqyvWrDSz9qesNL6zisxuF21WmndLvq36 KSstse0nkVlpQMBrJGx+Z738DG1A+k8AUFbasmXL8KeVK8NmPigft+mta3VWDitDmWETa1Ya EP1IyWaIJistUfOkodITojWslgRmpSnn3ou6fRnoj4SgNmlyTkBLMKmfstLsId5ZaVZniFFW WvxxalbakIF9ACDgMRIEgUa+VkFZUdrY3WCTJcbFLsxmpUWLlfq14vR4tz/v7Z/0G8O7/voh 9j6q5t3+4fD7/YH+UEJV2EBJSQkKCwvh9Xrxi1/8ApWVlQCA+vp6FBUVITs7GzNmzEBDQ0PM +4qm8xRrjEW4ffL+jJxijIzhJcZIj4TZP8agegbv7Z/0GxMv/XYNJ3FmHdk/FlgyWtJlpX34 4YdYtWoVvvjiC9xyyy2YNWsWAKC4uBhZWVkoLy9HZmYmiouLjQsyeSGNpHMUz3FczBJp5473 Owan2T9S8r1VuhdVK7ycZH9jeG//pN8Y3vW36kyOahW8298sbsGmnmuiWLFihfw5JycHTU1N aG9vR0VFBUpKStCzZ0/MnDkT06dPT4g+3j0W0d4xmI0nSFqPhUUw/VaP/suwUr9WR413+/N+ x5wU+nfYVz7v9h9QUqcxZ7118N5+whHiMUo2fD4fZs2ahVtvvRVutxs+nw+pqamYOXMmUlNT 0dSkcXfd3Axs2AAcOyb3XNm4EHrv09IrTP3O3u/82buayyN9D9nPKb0LN2YHfY9Vv/p93/r1 Ea3P3tmdjK4Otf4Y7aP3Htb+OnYLW75Sf6UnYrtqvfsEIWQ508/saZl9LLJ/OHtF3f5Xh9EV Tr/J88Gu9h/2PFTpt7r8ZNH/5c9/Hlf9bLnV7Ud9PYy1fN7aj/p4lfrbjx4FNmyAUFuLRCBJ EoQtG1+T2Je0wSMA8J+VVlNTgzvvvBNXXXUVZs+eDVEUkZeXh5KSEqSlpaG+vh7Tp09HRUWF 5vbLli3DyttWhs/q2iEAOU2G843ZlZVjVVYaANvnSxN2CGjKMcjcS7asNEDOjIolE0rLK6TO JLQk+8umrDRLqfTo6zKjX6NenQJlpZnHJwjwVFpro0RkpWnNd2lFVhrAR/txalZa+oDeABQe o2TxHG3YsAHz58/H8uXLMWfOHDnbLi8vD2vWrMHx48exevVqjBs3LiH6OmuMhQdNpgIOKcbF GC39VgZycmd/VafIqe3fLKTfGJ6yMrXg3f686w+HPMBjQlXYwLx581BdXY3rrrsOGRkZyMjI QEtLC+bOnYtdu3YhLy8Pu3fvxpw5c6zZYYTZLo6KsYgiUyfqZ8yV5sbYoRgjYzql/gjaKe8x IqTfGHVWptXZXWR/Y3jXbwa5Y5RM4xjV1taGvLp164a0tDSsW7cOu3btwrp165CWlpYQfY7z WETYOYrpjsHEviLSH0XHLt72j+XCrdWRdFz7iRDe9fN+x8y7/mTxGNk1Tl4k+qO5NvHefsIh CMLpKUHYi7Af3u/4eb9jiKv9LRo7R0lNzRD5omrHWCiOs3+ENqT2b0w89Ns1Rg8AnF/ziW1l A8lhfzvhXX84BEEIOIrYMzWaRDbyudLYHC/53irj9WyaKyrWuX7YepHMWRTNXDlKfeF+n1ad Ls/dZnqOLKfNlab4zuYyi8Re6v0qlyv1q+fqs3uuMSvsFlX7D1fPnXSutEjmOIynfrPXk2jm SvN6D1g+l1m018+Y7F/pSXj7iXauOrvbTyLnSmMdI3muNEEQkiYrLVYiykoDgJwm3ewjR2el ndrOKGPMCsLN5aPW35Rj0tXMvAlRZlHplWlZVtqpTCq9thEuK4dlnGlpYZlplrjknZyVptSm k20WQifJSrNrLKtI0Tp/zeiKRj/bF7ORFTaINistln2ry0tEVloi24+Ts9KSMvg6IUTg7qcY C2MoxsUYpt/WGIUoHwGaeYSSEPvrHE9njLFwkv5o2jDzLsRShhE82N+o3fKg34hExxixkCLq GMUZijEyxnExLhFitX7lRdAnCI7Wb+ZPysn6zcB7++ddf00q33MF8m5/3vWHg8VbU/C1RZi9 ++TdYxGPO4aI7wI7gceOtS9e9TPiot+GoHcG73fMjtIfTVapymNkdV3zYH+j6yMP+pWo/zcT 7TFikMcozjjxjjmSjgjvdwxOtH8kkH5j6I5ZG/YHxKt+BnmMjIlEfzSPIYcPH25r1mGiPUYM eRyjZBn5OhaizUoLl8Xg9Ky0SLJ2oslKiDQrLZpsKrN2i3dWmpFdwmWFqPUc8Ho1s9Jibk96 +mNtl4nKSlOvr1MPic7KkbMvdbIyrc4qqh49Oi5ZRWZ1RZNd5m30OiYrLVHtx7sl31C/6XKi yNbct29fVMdtqMOJWWmb3n5VEkURgiCg/6AMAJSVFnFWGhCSfcR61XYFyVqZlWZm3ViINCvN dFZUuIylaHBYVhqbx06rPTXB3GjiYTGb2WUCSUoJzGdlVbuPJitNve6pdaLJKmJ2tyN7h+0/ 2qwuraxFo30B9hxH1PqjyMRSZ6VZMYdZtFlpkdhfidZ8b8ryw14XNDLatLSHI1rbRXvcWvtm tlDimKw0ii+yAJPPunmPEbG7wSZFjItJosrKUei3o9Nthf2NdPFuf+bNsQKt/dvV/j1oggdN UevXe3wSTYxILI9iQmKMLMbuGCner5/79u0DKj3wCYItj9QcE2NEwdfxJZ4xFnY0XCc9I4+G SO0fqQ2tmqtOnY3GcFSMjob+cPaKSH8Ufzx2zRXIjks9V5f6d73vmutr7L8kuy7kDlqXSOxT 6QEqPbJ+vfbFPqvf1egt5ynGSOsYOlOMUTSkZGSYWi/a/55ExxiFTCJLI1+bx6jSwzUIJ90x RwI7Lt7veMzaP9oTO0R/jFkzzKvA7vid3n7CeWEs1a9hWzvbj08QkO+tCrpb1vscLVZ7XNTr 6s01ZqRdrxOlRaRZpZHayyqPkd5+4+FxUXZ8rb55tVs/m+WBXY9YuwnXkdZD3VYS7THy+/3w +/0QKehan1gnANW6QDjqjj8KeL/jcZr9Tf+pnuoEOE1/pITVH2NH0ir96rpg53LNC+drrqM8 11mMlVGd6nUemP5Yrj1G255f84npNheuo6TrcTn1qEUPo45TOM9btB4js/Y0035iqRv19VPL O+ipDO+x08OOcdSUL7n9n/JAMpQ3cOoYQzPXNUaiPUYM0eVyQRRFiCJl7quz0vK3aEffs+Xq 7CP2/N5ori+rstJ8ghA2q0JPv1Y2DCtPWa7ync35xTJb2HL1dlrba2VpGeq3Miut0hP07t2S r/27Yrvq0aPl41V6aXTLjzIrTf37AW+gHtj+WfnK7DNZv/K4TmWpGB6/STtGlJVm01x1putX /XuEWWlG9cPicTxoCtwlV3qQvyWQEZXvrYKnEvL7Aa838Lu3Ch40yd8PeL1yOSHnxal24xME w6worfNKbg+K81apn3m1tM5375b8gK7K0+Xkb/HKupVxSOq5+LSWe9AUcVaXd0t+0HGrj0/L Xuw7y0rTu05pvSuz/Nj5xfTr2d+ofKX9zexfndWl/r/QapfK64/W70bXT6396rWHcHZj7Z+9 K7MCvY1e+TxApSdQLrtenfruE4Sg61q4/wcnZKWJogjhg81vSCzGiLLSArCsNACaWUXKTAUl TTmnP8s95lOZSYZozfkVZjtltpJRTIJWhojco1f01tl66kwFZeaMMjuDZSaE/K5artan3I88 J1q0c3UZZSyZ8TzkaOxbsbwp59QdXSTZUBoEZZCcyjQDTpfNssvYcuVvhvsJoz/o3QijrDSj 7c2WraVHXXdm68tIs976iqw0Lfuz9goE2qMkpQTObbO6wsDaOdtvUCaawT7U26lR6mfnnSD4 ZBs3waPdjvTqQfU7a//Kd+W+5fPD4LjZ8SozreRzX9H+zSLvT3F8yv2oMeNx0ToGVqbSfspM NaVNlMcnXw9PtTPWnoKySpXXe5WdlXr0bKRut1pZXSHXZ0UmmfKardavtJvmtdkIVbvSvH5q tCs1icxKG5wW6PuILFWfHqnpcMotzFK2WQPXSvH0VCoeoakb1Knv7O5e/l1nvZDvqnfmblU3 LPUzWraeIPjkd09l6H6YC5QdIztetVuX3a0wu3gqEWybSo+8Dy19yvK0dMj6jeyidbKazQpU xijobaOwsfxd/dJB7xm5nPJf6ZHbifz9VHthbccwqysC/SHLlNp12p+u/bXK0tqn8l1jHW+j N3ybD7ffSOyvWJe1T09lcN2ydsrao7Ku1EQT48LKZfsIGv5Bz/4IPj/Yth40hbQf5fmotH9Q +1Xq17v2qDSx7eV3tm/Vcj39bB3lo0XldUXWqWjzIftUvcv2rzw9VAXTpX40qOyMhCtXrT9o +Sl7aOlXX+eCjutU/cntC02nr/+KcpXHrtxvSFmqZXrXf7lzrFofgHxtlo+DPfpS/U9oXZuB MO1f4z9K/q44b9V1rdafSFwuF1wuF4QP33tT9hj1Gxh4/pmMHqP6+nrMnj0b1dXVOP/887F8 +XL069dPc90gjxGReMLd4TodXnUTBIfojY/T6eHkOqT0GJU98ghyfvc7dNP5r7YK5jEaOijg Ves06frFxcXIyspCeXk5MjMzUVxcnBAdjhqHIwoSoj/cHW4EJFS/BVD7MYb0G9MZ9Mte8Shw gv5YMNTP4fXzu61bUTJjBqrWrkVHa6ut+wYgPzkTtmx8TZIkCX6/HwOHnAsgOT1GeXl5KCkp QVpaGurr6zF9+nRUVFRorkseI4IgCIKIP0qP0T8mTpSX9xg4EGN//3ukjxtn+T6Zx+jswYHO o5t1ipJ9HCOfz4fU1FTMnDkTf/3rX9HUdDqOo7a2FiUlJcDJk8DBg6huasJPvx+Cbz3/xbCm 7pa+X3goHTsH1VleLnvPr+2HTzIabCuf9JN+0k/6ST/pt0v/nRkzgYMHgb590V/xH+5vb4+L 1wgA3PKkaUkefJ2SkoLGxkY899xzqK+vh8dz2q2YkZGB2bNny9+XLVsW9N1KDh8+jAEDBthS NgCgshLIyQm/XpSQ/jCQfkNIfxhIvyGkPwxJpv8fEyfC3bUrLrz9dpx7zTUQ3W779q1AZFHY LpcrLjtMFHl5eVizZg2OHz+O1atXY5wN7jgzpKam2ruD//f/bC2e9IeB9BtC+sNA+g0h/WFI Mv1nFxTgmhdeQOa118alUyRJUsBRVPr+W3JWWmraMADJGWNUX1+Pe++9F9XV1cjOzsby5cuR lpamua6dHiPbaW4G+vRJtIroIf2JhfQnFtKfWEh/YkmQfmVWmiiKED7aUiKno3n6nw0gOTtG kVBbW4sMk5PlEQTBPxkZGaitrU20DIIgEoC6YyQC6BTp+pGQzJ2iqqoqXH/99Rg9ejQmTpyI t956y9Ly6+vrUVRUhOzsbMyYMQMNDQ0AgJKSEhQWFsLr9eIXv/gFKis75yAjGRkZIS8r0bM/ Y9WqVUndvsORKPu3tbVh+fLlKCgowIgRIzptHURz/YnEVnr2Ly0txaRJkzBq1ChMmjQJH3/8 cdTHwDMZGRl4/vnnAQDPPfec5e1Q77wKd11yCiwZTVR+IZKfWbNm4YYbbkBlZSX+8Y9/oLy8 3NLy9caL+vDDD7Fq1Sp88cUXuOWWWzBr1ixL98sLtbW1smdC+dkqjMbr2rNnD9asWWPp/ngj UfZfuXIltmzZgpUrV+Lrr7/utN6pRF1/5s6di/nz56OqqgoPPfQQ7r//fkv3yxPr16+HJElY v3695WXrnVNOGUcwHEEdI7aASH7cbjcOHjyIL7/8Ev3795cbqLqHz75nZGTghRdewJgxY5Cb mxsY1sCAiooK3H777ejZsydmzpwpX/hWrFiBkSNH4qyzzkJOTg6amprQ3t5uwxHyiZ79v/32 W9x8880477zzcPnll2Pnzp2G5ejZv7W1FbNnz8ZDDz1kzwEkAVrtXK9e9NCzf0lJCebPn4+s rKxOPWG30fUnIyMD5557LqZOnYodO3bIy5W/h0PP/mlpaWDTX4miaG9mlcPp06cPVq5cGZSZ XV1djSlTpsDr9WLKlCmorq4GEPn1Xw+9enEifr8fojzSY5Kn6xMB1q5dC5/PhwULFmDcuHHY vHlz2G1cLhfKy8vx5z//OWxPXzleVGpqatB4Uez3WbNm4dZbb4U7TqmXPHP//ffjhhtuwI4d O/Dggw/iwQcfNFxfz/7Lli3DT37yE1xzzTXxkM0lkbRzPfTsf/jwYZSVlWH06NEoKCjARx99 ZKV0btC7/jBPw5dffomHH34Yd911l7xc+Xs49Oy/aNEi3HXXXcjMzMTdd9+Nxx57zKYjdD5F RUV48sknUVRUJC976KGHMGPGDOzYsQM333xz0A2UneeFU3EDgCiK5DHqJAwdOhQLFiwAAHz1 1Vf4n//5H1xxxRVB63R0dAR9v+mmm+ByuXDRRRehvr7esHyj8aJqampw55134qqrruI36y8O KO3/5Zdf4p577sE999wDAGG9DXr2X716Nfx+PzZu3AiAgo21CNfO1eeFFnr27927N8aPH4/7 778fn3/+OWbPno1LLrnE8mNwOnrXn1dffRV//etfcejQIbS3t0d9o65n//vuuw9PPPEE8vLy UF5ejvvuuw9btmyx7Lh4YsqUKZgyZQoA4O677wYA7Nu3D1OnTkWXLl1wzTXXYOHChfL6kVz/ 9TD6X3AiIhvcsTO7dzsTs2fPxt69e3Hy5El8/fXX8sW+f//++OSTT3Dy5Em88sorQdtEMsaV 3nhRGzZswPz587F8+XLMmTOH2psKPft7vV48++yz2LNnD2pra/HNm3WB1wAAIABJREFUN98Y lqNn/2+++SYkvoYIRqudG50XWujZPzc3N+jms7N66PWuP48//jgeeeQRVFVV4bnnnguyVZcu XVBXV2eqfD37Mw+FIAhwuVxobGy0+Mj4Zvjw4Xj77bdx4sQJvPXWWxg+fLj8mxVjHDplHEGz iJ1l5GsiwIQJE/DrX/8aF1xwAVatWiW7Ru+//37ce++9GDdunKk7Yz3mzp2LXbt2IS8vD7t3 78acOXMAAPPmzUN1dTWuu+46OV6gpaXFkmNKBvTsX1xcjJdeegm5ubmm4iz07E9ER6TnhVH7 //vf/44LL7wQf/zjHzvtoxy968+vf/1r3H333cjLywvp/DOvkpkYIz37P/7441i4cCHOO+88 PProo1i8eLH1B8cxixYtwj/+8Q+MGTMG69atw6JFi6IqR3mNUn7m7bokfPTBmxIQCL5O7XcO ABrHiCAIgiCIzgEbx2hA3+6QJOl0Vhp5jAiCIAiC6MwIghCYRJYgCIIgCIIAeYwIgiAIgiBY P0hk0f+Urk8QBEEQRGeF9YPc1CEiCIIgCKKzQx0jgiAIgiCIU/j9/kDwNXWMCIIgCILo7Mge o46ODhrgkSAIgiCITk17eztEUbQuXd/MqKQ0DQFB8AOd0wRBJPo6EM95HSVJgiRJgY6RZd6i SoPfcqzZBUEQ8cRoFuzETARJE+ASRHxJ5FVA71y38zpgbceIIAiCIAiCQ5jHSBRFEYIgJHy2 c7W7Tvn9o48+wlVXXYVRo0aFTKR59OhR/OY3v0F2djYuv/xy7Ny5M6Scp59+Gueddx6mTZtm 70EQBCGjd942NTVhzJgx8iTCFRUVmDp1atjt9D4zjK4FGRkZuP766/HAAw9g8uTJeOihh2w9 doLo7Ph8PowdOxY//vhj0PIff/wRubm58Pl88nl87rnnYurUqdixY4e8HvtNq29gdB2IBXmA R9YpSnTHyIj7778fjzzyCHbv3o3a2tog99mjjz6Kq6++GpWVlXjwwQfx4IMPhmz/448/4tNP P8V5550XT9kE0anRO289Hg9ycnKwceNGAMArr7yCm266Kex2ep8Z4a4FDz74INavX4/Fixdj 8+bNth03QRBASkoK8vLyQs61TZs24ec//zlSUlLk8/jLL7/Eww8/jLvuukteT+scVy/XWyda WMdIfpTm5MdpWVlZ+OMf/4hx48bhggsuwCWXXIKuXbsCAMrLy7Fx40bcc889AKDZwbvzzjvR vXt3LFy4MK66CaIzY3TeXnvttfjb3/6GSy+9FOXl5Xj88cdNbWdEuGsBuzHKzs7GsWPHrDpM giB0uOGGG/Dkk0/ioosuQl5eHj7//HO8/PLL8jlaUVGBRYsWoba2Fm1tbQnvh8geI/Yl0YJc Lhc6OjoAQHaxM9auXYsHHngAKSkpeOmll3DrrbfKvwmCgF27dsk9x2+++Sak7O7du9srniCI EIzO28LCQnz77bdYvHgxJk2aFHSOGm1nRLhrgcvlkt9p/DaCsJ9x48ahoaEBa9euhd/vx5o1 a3D06FGMGzcOQMA7fPfdd2PXrl2orq5O+Hkpd4zcbjcAyJ2SRDF48GC89957+PHHH7FmzZqg 31wuF/Lz8/Hb3/4Wd999N/bt2yf/lp+fj6VLl9IdIEE4DKPz1u12Y+rUqXjttdfwy1/+0vR2 ANCzZ0/NGyC6FhCEsxAEAddddx3+9re/YerUqfjrX/+K66+/Xu6AnDhxAh6PB62trXj66acj KlvvOhALbrc7EFrEemiJ9hjNmzcPCxcuxIQJE9C7d++g31iAldfrxSOPPIKlS5fKv/3xj39E U1MTCgoKLA/EIggieozOWwCYMGEChg4dijFjxkS03a9//Wtce+21Iec6XQsIwnn88pe/xFln nYX58+ejS5cuuPbaa+Xf5s+fj9/97nfIz8/HoEGDgrYLF2Stdx2IFUEQIGwtfVsSBAGSJKF3 yhAAQK9evSIuLNGDQBEEYS12ntMnTpzA7NmzkZWVhTvvvDOqMgiCsJ/O8N++bds2AEBaajf4 /X642Q+xeox4NwxBEMHYeU6PHj0aY8eONR0/RBBEYuhM/+0sS9/tlEdpBEF0Hvbu3ZtoCQRB EEHI6fqJjgInCIIgCIJINEEdI0mSyGNEEARBEESnxw0g4WMHEARBEARBJBpBEOBmg51R54gg CIIgiM6KIAiBrDR6hEYQBEEQRGfH7/cDOPUoTdk5amxsDJmhniAIgiAIIplhT87k4GtJktDQ 0IC6uroESyMIgiAIgogvbLBrtyRJ8Pv9kCQJZwgnMHxoP/nHjo4O2bXEBj5iv/n9frS3t8Pv 98uT0LJJGtn6aozimJQT2bL1/H6/rI19VpatXA8A2tra4HK5cMYZZwRpASBr19LHPnd0dEBp D1Y++3zGGWcE5lE59WLZfOyl3J59FgQBbrcbLpdLtp3WcYuiCMEVaifl546ODnl9pU5WjvIz 06i2lfK42G+sHJfgDiqf6p/qn+qf6l9ZJtU/1X+y17/cMZJfkOQDV7+YYZQG1kK5XLlzo+3Y Ptn6yu+iKMLv9wfNiK3eBzNCe3t7UENTrqveh9ZnZQVo6VRWjPKk0Dp2Ziv1icTelS/ZrlJ4 W6mPSWkzNhGwVuNnn9kxKMui+qf6p/qn+qf6p/qn+vefDr5WG0P5XW1ItTG1lmsdtNZvRoZV NzJlWery2UueGVfREJl+9bEoK4HR0dEhv9h+lT1NZU+aNWitylc2IFEU4XK55ApmWpW909PH ebqilHWhZR/lsSu/K383qg91GVT/VP9U/1T/VP9U/1T/HaeDrwVBAKRQl1u4g1FWnvqglAei V5beOurKU2tSViDTz9yVzJBqTepKVjc69YmhNJ6yASldrep9KddV9l6VJ4ty/WA7h/bW9ezE TkqtHrAWenZVaqb6p/qn+qf6V2szguqf6j8Z61/uGLndbvROGWJoBIIgCIIgiGSjsbER//Ed Cnj4wvUICYIgCIIgkpWGhgbs378fZ555Js444wztSWR79eqVAGkEQRAEQRDxhY3d6Ha7A48q E6yHIAiCIAjCMYjKQCmCIAiCIIjOiiAIp9P1qWNEEARBEERnhfWDgtP1CYIgCIIgkoTS0lLN 5YWFhSHL2BM0UWugI4IgkoeMjIxES+CaaOynt01GRob8IgjCfgoLC1FYWIgJEybI3/WQR75m o3FS2j5BEIQ11NbWGi6njhFBxI8TJ07g3//+N9xuN3744Qe43W7D9UVAf1RKgiCshf4QCYIg 4sf+/fuxfft29OjRA6NHj0ZVVRXS09M112V9ITfzFrFh0AmCIAiCIJKB5uZm5OTkoGvXrgCA iy66KOw2onqeE4IgEkdLSwvmz5+P3Nxc5Obm4uGHH8aPP/4YtM7bb7+NSy65BKNGjcLNN9+M uro6+beTJ0/ikUceQW5uLnJycvD8889HrMGo/MmTJ+OLL76Qv3/xxReYPHmy/L2jowPPPPMM CgoKcOGFF+K+++5DS0tLUPkZGRn4+9//jgkTJmD06NEhGo3239bWhoULFyI3Nxdjx47F2rVr g7xwZvZvhBn7GemLNYZo6tSp+OSTT4L0jBkzBo2NjVGVRxCdnaNHj+LTTz9FaWlp0EuLoOBr CrwmCGfw1FNPoaGhARs3bsS7776L77//Hn/5y1+C1tmyZQuee+45fPHFF5gwYQIefvhh+bcV K1bg0KFD2LhxIzZu3IjPPvssYg1G5V9xxRXYvHmz/H3Tpk248sor5e9r1qzBZ599hhdffBFb t26F2+3G8uXLQ/ZRXl6OF198EWVlZSgrKzO9/1WrVuHbb7/FO++8g82bN4ccn9n962HGfkb6 amtrdeOLzHDjjTfi1Vdflb+XlpbiggsuQGpqatRlEkRnJpLga3n4oq2lb0vsuVpK6tkAaEoQ grCLjIwMwz/Oiy++GC+88AKGDRsGANi3bx9uu+02bN26VXP9lpYW5ObmYvfu3QCAgoICrF27 Nmj7yy67LOo/a3X5X331FWbOnIlPPvkEkiQhPz8fq1evxrnnngsAuOyyy/D8889j6NChAAIT M06bNi3IC5KRkYGtW7diyJDwk1ar9z9x4kSsXr0a55xzDgDg22+/xSWXXCIfn5n9GxGp/dT6 lMdoZHO93//73/8iPz8fH3zwAVJTU/Hb3/4WV111FaZMmWJKP0EQobDg62PHjuH8889HVVVV 0CO1bdu2AQAG9usBURRPD/BIj9IIIvEcOXIEgwYNkr8PHjwYR44ckb/X1NRgyZIl2L17N374 4QdT20dCuPJHjhyJLl264F//+hckSUK3bt3kThEAHDp0CBMnTgzaRhRDZx7SC34Mt//6+vqg Y1IeayT71yOc/cLpi5Xu3btj8uTJeOONN3D99ddj+/bteOKJJyzfD0F0Fvbv34/vvvsOAwcO xNChQw2Dr0VRDLxYxyiSiwdBEPbQr18/HDp0SP5+6NAh9OvXT/7++9//HldffTW2bt2K2tra oHgfAOjfv3/I9pEQrnwg8Djtvffew3vvvYfLL7886Lf09HR88cUX8iOl2tpafPPNNyFl6D2+ D7f/tLQ0HDhwQP5+8ODBqPavRzj7mbGPGbp06YKTJ09q/sYep23evBkTJ06Ug0YJgogcFnw9 YsQIeDweXHTRRRg+fLju+qf6Q4EOEXmMCCLxXH755Vi8eDEaGhrQ0NCAxYsXB3U+Tpw4gdTU VHTt2hV1dXVYsGBB0PZTp07F4sWLcfToURw5cgSLFy+OaP/hygdOxxlt3rw5KL4ICPypz5s3 D/v370drayt27tyJ3/zmN5btf/r06fjTn/6Ew4cPo7GxEUuXLrV0/+HsZ8Y+Zhg1ahRef/11 zWzg888/H126dMETTzyBK664IqryCYIIEEnwNXt6JrIvFIBNEPFBmbmkzmL6wx/+AI/Hgyuv vBJXXnklUlNTcffdd8vbFhcXY+nSpTjvvPNQVFSEyy67LKjsO++8EwMHDsSVV16JK664Arm5 uRFpC1c+EPhTd7vdOOOMM5CZmRn026233orc3FzMnDkTF1xwARYsWIBrr73Wsv3fcccdGD58 OKZMmYLLL78ceXl5cLlclu0/nP3C6VPWpVF22oIFC/DCCy9g5MiRmuvccMMNOH78OMaPH29a O0EQoYiiiIKCAjkIu6CgQPcJmd/vhyRJEP5ZtkkOvu7ZO/A8nYKvCYLggfr6elx//fVy8GSy 8M477+Ddd9/Fs88+m2gpBME1O3bsQP/+/eXYwUOHDqGhoQE//elP5XXY9WNwWq9AjBE9QiMI gifuvfde7Nu3D8eOHcNTTz2l6dXimdbWVrz88stB40MRBBEdmZmZOHr0KMrLy1FeXo6jR4+G eLoZLpcrkJVGHSOCIHgiPz8fRUVFaG1tRWFhIe65555ES7KUUaNG4Wc/+xml6BOEBXTv3h0X Xnhh0LLS0lLN8YxYSJHcMaIYI4IgeGD69OmYPn16omXYRiwDRBIEET1+vx8A4AaoU0QQBEEQ RPKhl4FmhJt9oM4RQRAEQRDJhNYjM73OEstWkyeRZS4kgiAIgiCIzgoNd00QBEEQRKfBaCJZ AIG50igzLZhDn38OABgU4eB4BEEQBEE4B5/Ph9raWrS0tAAAunXrhhEjRqBPnz4h68pZaXFV yAHtJ07gs2XLILhcmLp2LVxnnploSQRBEARBRMGePXswcuRIeDweAEBjYyNqamo0R5WXpwQh b1EwNa+8gv8eOYLj33+PmpdfTrQcgiAIgiDiCA3wqODYgQPY/dJL8vfdL7+M4ZMmocfAgQlU RRAEQRBENGRlZWHv3r2oqakBEHiUlpWVpbku6w+5/X4/peqfYseqVfC3tcnfO06exPYVKzBx 0aIEqiIIgiAIIhpSUlIwduxYU+uyjpHIJpAlgImPPYYZH3+MI1dfjRkff4wZH39seaeosbHR 0vLUCGVltpZP+o0h/caQfmNIvzGk3xjSH4rP50NlZSXKyspQVlaGyspKNDc3G27jBoCOjg7L xRDatCk8UnYgdetma/mk3xjSbwzpN4b0G0P6jSH9oUQSfO1HByRJohgjPYYPH25LuW1tbbaV DQA/1NSgt43lk35jSL8xVutfcngJ5h2ch8WDF2PugLnc6VdD+o0h/cYki/59+/bZtg8jRFGE JEkQPtjyusRijFL7nQMA6NWrV0JEOYVly5bhmWeeSbQMgiDCIOw4HR8pjaGbPIJIBqzsGPl8 PuzduzdkHKOUlBR5nW3btgEAzjk7NZCuD4BijOKI3T3h41u32lo+6TeG9BtD+o0h/caQfmNI fygs+Priiy/GxRdfjLFjxwZ1ipSwvpDwwZbXJSAwsBF5jAKQx4gg+IA8RgSRfFjZAdObMFY5 LQjzGJ09JAV+v//0XGnkNYoPPPa4lZB+Y0i/MaTfGNJvDOk3hvRrU1hYGPLSwu/3w+/3BzxG fr8fANB/wAgA5DEijxFB8AF5jAgi+bDaYxRu0ljmMRoyuPdpj5EkSWCdI8JeeO1xM0i/MaTf GNJvDOk3hvQbQ/pjQxAEiKIIYcvmDZLf74coiuQxOgV5jAiCD8hjRBDJR7zT9ZnHaNhQD9rb 2yG63W64XC6Iohhm0+Snra0Nhw8fxg8//ADgdOVY+c5edpV/fOtWW8sn/aTfafq9jV6u9fNu f9JP+q3U/9VXX+Hw4cO2j7KthSAIgdfHH70ldXR0QBAEePoOA0AeI/IYEQQfkMeIIJIP1lmK F8xjNHxY34DHiEVh07Qg8cHuCqdnzMaQfmNIvzGk3xjSbwzpNybRMUZAYPRr4b1N62WP0cDB IwGQx4g8RgTBB+QxIojkI5EeI0mSILa1taGjo4Oy0uIE7z1u0m8M6TeG9BtD+o0h/caQ/mAq KytRX19vepxGeeTrkjf+TxJFEW63mzxGpyCPEUHwAXmMCCL5sKoDduzYMdTV1aG5uRkDBw7E 4MGDceaZZ4asp8xKEwQhMI6RIAhwuVyWCCGM4a3HrYb0G0P6jSH9xpB+Y0i/MaQ/mF69eiEr Kws5OTkAAh6kPXv24NixY5rryx6jt99YJ7ndboiiiLTBNI4R0Dk8RksOL8G8g/OwePBizB0w N9FyCCIqyGNEEMmHXR0wv9+PhoYGHDhwQO4sAYoYo6H9Ah4jNoYReYw61zhG8w7Og7fRi3kH 53Gpn3f7k37ryqVxjEg/6U8e/XaOYySKItLS0oI6RWokSYLw3rsbJJfLBZfLhZR+QwGQx6gz eIzoTpvgHeb1ZFA7JojkgHWW4gXzGJ0zpC8AQGTeIvIYxQe7K5yeMRtD+o3hSb+yU8TgSb8W pN8Y0m8M6Y8dSZIgfLD5DYk9Tuudmg6APEbkMSII56NswwC1Y4JIFhLlMRqWnhoYx0iSJLDR rwn74b3H7RT9Sw4vgbBDwJLDSyIq3yn6o4X0G0P6jSH9xpB+Y3jXHw6/3x/wGG3Z+JokCIE7 r34DhwMgjxF5jJwP7/qJ2CGPEUEkJ4nyGA0dlBLwGLF50miutPjAe4+b9BtD+o0h/caQfmNI vzGk3xylpaWay+VxjDa9/aokiiIEQUD/QRkAyGNEHiPnw7t+InbIY0QQyYndHbDS0lIUFhbK 35nHaHBaL7hcrsDI10T84L3HTfqNIf3GkH5jSL8xpN8Y0h9KaWlpyEsPFlYksg9mJ1lLZuIx wOPw4cNtHSCrR0GB6fWVA+OR/vjr5739JFo/q3/2vnTHUq70825/0k/67dBv9QCPhYWFIa9w CO+9u0HuEdGUIAHsfJTGGpddHN+6FT0KCsKuF+2jKNJvjFn90WK1fvXUMDzpZ21g7oC5cnbi f/7zMTf6teDJ/lqQfmNIvzFMP+ss2YXeo7QhA/tAEIRAVhrzFlHHKADFGDkf3vU7BZ7tyLRL YyT5M839RxD8Y3fHSI2yYwQgEGPEHqcR9mN3hdMzZmOs1s/GU7rimysg7BDwylv3WVq+GrK/ Nqwz9NnmFbaUzyD7G0P6jSH9xtih3+fzobKyEmVlZSgrK0NlZSWam5s115UkKTCO0fubXpc9 RpSVFoA8Rs7HKfrVmVEAX/ZU6ufN26L0GGl9JwiCT6zsgJWXl2PkyJHweDwAgMbGRnz99dcY P368vA7zGA3q3xOCIPCblVZSUoLCwkJ4vV784he/QGVlpeH69fX1KCoqQnZ2NmbMmIGGhoY4 KQ2Gxx63Eifoj3S0ayV2659WnW5r+Xbqn3dwHvfth2f7A53j/I0F0m8M6Y8d2WMEBB6n9R1w DgA+PEa///3vMWvWLAwbNgybN2/GkiVL8Omnn+quP3v2bKSmpmLWrFn4y1/+gubmZixdulRz XfIYORsnjV+TTB4jgC+vEXmMCCI5sbID5vP5sHfvXrS0tAAAunXrhhEjRiAlJUVeRzmOkd/v D8yVxl48sWLFCowcORJnnXUWcnJy0NTUhPb2dt31KyoqcPvtt6Nnz57/n71zj4+qOvf+by5S RLkkAQIlgBIvyGDEcvENcu+pQhWFHm89Qjkqtaf1UKuIQcWqtdaADbYer0cLWmqr8KqxahUv iHCS46ugDRCsR8JpCdAESAJIkUtm5v1jsjZ79uy99m2tmb0mz/fzyWcyM3uv/dvPXnvPWs9a z7Mwd+5c1NTUZFHtCVRvcZN+Pqp6LPJljo6q9meoXv9JPx/Sz0eG/oKCAowePRoTJ07ExIkT MXr06LRGkZ5QKJQaSmOLpqm6iGxbWxvmzZuHOXPmIBqNcrcrKirC3LlzUVRUhNbW1syN9u8H Vq0CDh7ULpDo1747dkgp11ih7LZjPyDslfR7ezXqcKo/KPWH6V70xVSl9butP0HR77X+k37S n6/62/ftA1atQqihASIwS/BoleSRNYxCb762UhtKUy1cv76+HjfddBMuueQSzJ8/H+Gw9ZSp 8vJyVFdXo7i4GM3NzZg5cyZqa2tNt6U8RtYEQb+foTTR+o0h4zM2leCV6xqFlW+E9J/AbChN Jf1mOL1/vUL6+ZB+PirmMWI5i/S5i6zyGJX065laRFbY0bPMqlWrsGjRIixduhQLFizgNoqA VMNo+fLlOHToEJYtW4axY8dmSWk6MisVAKk3BUD67agu2ym1fNLPR3X9qtV/lq4itDGExU2L ldNvhPTzUVF/NBrFl19+iVAohC+//BL/+Mc/0KVLF9Nt2ZQiZRtGCxcuxKZNm3DllVeitLQU paWl2uSq0tLSjO0rKiqwefNmlJeXY8uWLViwYEG2JQOQO0a7uGkxZi4f6Ctqyw4Vx5j10BwX PqSfD9X/dBbuWpj2v2r6jZB+Pirq79+/Pz799FOcddZZ2Lx5MzZu3GjaRgB0eYz+9McXtaG0 fiVnAlBnKE0WqkalOR0eY8tAONk2iAQxKk2ffVkle6qsn6LSck+Q7kUvGJfEIYJBrjJff71v dwAKe4xUJQg9Zn2jyC0q9hj0BMH+fiD9fFTXr3r9V00/exYu3LWQMtc7QEX9ZhOtrSZfM0J/ +uOLSTYTW7XJ17LId4+R6r28IOlX2eMCqK2fPEa5J0j3ohdUz0OWr4iefG2G2eTrAcWpto+2 Vhqtl5YdqMfMR/Uej+r2J/18qP7zUdXjwhpDVH/4qKp/ypQpaX9WsHZQGIByyR1VhqJy+HiJ SnAz2Zzsz4f08wli/XeDLP1sfs53S6qklM+g+sNH1frDkKHfrCHEaxwBUDfztQyOHz+OpqYm HDhwAMCJ1rHIV/Yno/xYSwwzNpUg1hKz3U7/GhT927dvx6G1ax2dJwD8NPFTAMCKTStyql9v zxmbSqTah/Rn6jbWf1X0e63/QdMfa4mhckClsvZXtf7nS/0x6v/888/R1NSElpYWZBuW8Dr0 xqsvJJn7qP/AswDQHCOaYxRsgjQvJkhavKCyfppjlHtUrj+A+vrzFdZYyhZsjlG/3qcAAMI0 tyi7yL7gNEbOh+zPh/TzofrPR3X7q65f9fojW79TopFIhIbRsgiNkfNRfYxcdfuTfj5U//mo bn/V9atef2Trt0NbRDYcDiMcDlNUWpagHg8f1Xs8qtuf9POh+s9Hdfurrl/1+pNrjxFrC0XD 4TASiUROxXQmqMfDR/Uej+r2J/18qP7zUd3+qutXvf7k2mMUiUQAdESlEdmDejx8VOrxmKUJ UN3+pJ8P1X8+qttfdf2q1x8Z+tva2rBhwwasW7cO69atw4YNG7B//37uPr7C9dnirZ0Rtqq0 2wVbs9XjkbWQrOo9BpH62XIC+jWWqst2Sl3EtzP1mL3cY0HS7wWV6r8Zqttfdf2q1x8Z+rdu 3YrTTjsN48aNw7hx4zB48GDU19ebbqsleEwkEkgkEojH464P2NDQgIaGBl+iVUW/xo4bstXj 8bMeGg8Vewx6ZOivHFAJINVAmrGpRJrtgc7VY9bfY8bGkVVjKUj6vaBi/dejuv1V1696/cn1 HCMg1TgKJ5NJsMZRZ8dtgkcvCRKHDBkiLUFWrCWG6rKdUhM8ytS/fft2nDppkmu7uzkPkfqN x73x6I2O7O/nVYb99fWhumyn1OvrRr9e18JdC9O+X7hrIWItMVT0q8io/0HRL6v+B01/UOtP Z9Gvev0x6hed4HHYsGH43//9X6xfvx7r16/HX//6VwwbNoy7T2j1G/83yRpGXx90NgBK8Ogk waPTZIpGWOWSQWhjCDM2lWjuYCtdfhI8ytQPpHoMdu5UP0nZROo3O/bM5QNRXbZTWpI40fY3 2nLGphK8cl2jsPKNuNHPq6dmtg+afi84qf9+6Ez1xwmq6zeiWv0xwvSzxlK2YAkeBw8oBNCR x8jtIrLt7e346KOP8P7772PNmjV47733pIjNR2iMnI/qY+Sq25/086H6z0cl+5sNx6qk3wzV 648M/WvWrDH93Gy9tGQyiY4URmFEIhFEo1Fu4fv27cNLL72Em266CaNHj8bs2bPRo0cPPPLI I0LEdxZktYTZTU5j5Hxk61fJ/hRV5x6q/3xUsr9Z8IRK+s3fJzNmAAAgAElEQVRQvf7I0s8a QVOmTLFdQBbQhevzPEbf+c53cNFFF2Ht2rX4p3/6J60FNm/ePMRiMQGyOw+yWtzsJj/7omul lM9Qscegh3rMJ7CKqpMJ2Z8P1X8+MvSz4AlATf16VK8/uc5jxAgDsA3Xd9J4Ipwhu8W96Iup UstXtcfAyFaP2UsqByfI0K//YQhKj9mr7YKi3yv5Uv9lQfr5qF5/ch2VxoLRwvo3Vrzyyit4 ++23MWHCBLzzzjuaK+qxxx7D1q1bs6M4T1C9xU36+eh7nDLC9jtLj9/Mm+WEoOj3Sj7VfxmQ fj6q1h+Ws2z2wF8LL1s/jGZHWsOIfcCjd+/euOKKK/D444/j448/xm9/+1u0tbVh3rx5PmV3 LlRvccvQr0/kp6J+Par3OIOmX+/NckLQ9LuF6j8f0s9H1frDOkJlHx+TUr4bEokEwmx4zM0w WTQaxYUXXohFixblVURaPuQxOnXSJOXyGK3YtEJ7VTmP0fbt29PyGMnIZyRTf9DyuNjZMej6 vbyqmIdG1TxAVH+CpT/WEsN917whPI/RmjVrMv7sCK15+9Ukm2NUVHwaAMpjpGIeI6bnyy/f R/fuk7m6gpbHSK/nyy/fz4s8RgzR+Yxk6g9SHhe7a0x5jLwhWr/KeYCCXP+9olr9Yeh/v7KR x2jNmjVpQ2ssj1FJv54nPEahUAjhcNiqDMKAn0m1NMbMR3X9qs9RIP18qP7zUd3+qutXsf7o f08DE5XGGkYUceYcP5NqZbeEVR1jZqiuX/U5CqSfD9V/PqrbX3X9KtYffaBFtqLS7CZih+PJ diQQRzzZ7rrw0tJSlJaWetXWKaEeAx/V9ave48wH/TLSJDCo/p8gHzNHq65fpfpjpHJAZdYy X1vNM2pvb0c8Hj8RlebFY9TQ0ICGhgbX+3VmqMfAR5Z+Fvn20MaHpJTPUL3HqbL+in4VmLGp REqaBIaq9Z8hUn8+Zo5WXb9K9ccMWfrdTL4OhUI0lJZtqMfAR5Z+9hC/HbdLKZ+heo/TiX59 egW3yO5xqm5/Fe/ffMocrbp+FeuPHln62VIgjpcEYf9Qwyg7qNriZqiun4XoykL1HqcT/ayR 6cUzk039fhpwVqhe/4NQf/xA+vmoXn9k6DdrCFk1jlg7SFsrzS7BoxU0x8gdqra4GSro5/0g 1hfV+y6fh+o9zlzpF9WIMcs8LnJoTYX6zyNf648oVNevev3JdVQaaweFWQ4jrw2jfMJpgkc/ CfzYn9v9nLzGWmI4tHat1ASPMvTr9Rxau9bx9lbv9QkjjdvFWmIY/vZwLG5aLEy3/vMZm0qk JngUaX8zO87YVKLNw8pm/Vm4ayFiLTEs3LXQ8XGc2l/kdRBd/xc3Lcbwt4dj2hfTMPzt4Xjh 1dukPR+yVX9If/bqj/HVyfMzaPr11+HQ2rVZSfBoNc+ItYVC76x+KcncR0V9TgfgLsFjaWlp 3k3Atkvw6CdBoizcJD0Mmn63yTKdJv/Tf248Z+M+i5sWY+GuhagcUOl4fS67Yxu/CxpG/Qt3 LdQ8NjzdXpObOtHCyvSS4NH4uQydorGrl0HGLEGi/n3QUV1/vmBmd9ZoEsm2bduwf/9+nHfe eTjppJPSvmMJHvsUdE1NviaPUXaRccH1qD7G/MKrtwktjw3RMKzmGIkadlF5jkLlgEql9QNq 2x8g/XaQfj6qPf+NQ+cy9B85cgR//vOf0dbWhsGDB6O+vh5Hjx413Za1hcLxeBzxeByJREK4 ICIT1cdoRes33hjfLakSWr6xoUNzjDIbi3qCoN/PPKMg6PcD6edD+vmo9vw3pnyQof/jjz9G QUEBRo0ahT59+mDo0KH47LPPTLfV8hg5iUZjiRzN/gh3yGhx63/kVOsxGG8MY49NdGQRRaXx vWKy9LPr+L3/+p7t9bTz2vH2VcH+ZrBhBFX1M/JBv8wEobKe/+yeUu35z2ApH2ToHzlyJAYP HqxFnHXt2hWxmPnvgDbHaPWbq7Q5Rn2KUw0dWkRWnTlGZlpUmmPkZ76Qfp/KAZXaHCGnw2F2 x3Gj3awcN2VlE55GJ/MsnNpKP2/L7Jrw7KbfxmouSEW/irQcOkb9Ks0x8rIgcq5RfY6O1zl2 QUKF5w0PqzojsgFmNdHabBHZnqeEEQ6H1U3w6NZr1dzcjFmzZqGsrAyzZ8/G3r17JSs0R2aP JxtrzeSyx8kbAnI6R0jvMQptDGHaF9Msy/RC0HvMdr1hkfrNrokfj51Ru7FRBATf/naQfj40 x46P6s9/WfpZI8guwaM2xygcDqdmYYfDlhsHEbfLkSxZsgTDhg1DTU0Nhg4diiVLlkhUZz0E JGuMOTkyKW2tGT25HOMXkY/GOMforYNvudrfbmgv6HMU7Gxo1C96KNNsjpdd2WyY1cn1D7r9 7SD9fEg/H9Wf/7nOY6QleGSNItUaRm6pra3FDTfcgO7du2Pu3LmoqamRejwrD4ZZi1vkj4/q PQa3PTanofXMVcs8Fm5dzuwa2Xmmgtrj5Hnb9DD9rD7qz9dNHbXazsxjZ7bmlh69Z2jaF9Mc 6ZdF0Oq/W0g/H9X1q/78l63fjrSGkYpDaW5pa2tDUVER5s6di6KiIrS2tmZutH8/sGoVcPCg doGsXtkNxF6dft93x46M7RfuWqgtfml3XKvjGCuUlS6n+q1ezfT7ebXSb6WTvT6z498AAIu+ mMrdzlj+mbsOOCrfqPPDNx91tD3DrV1l2n9x02JP+lm91G9nLMfquGw7dp3Y9sz+7H3X/9qi bacfypixqSRtaJiVo9/e7LgMq/qVC/vzXu3qfxDqD+kPvn5jg041/Xr7t+/bB6xahZCg/Ij6 YTTjZ0a09tCad6uTbDitR6+USJUmXztNMFleXo7q6moUFxejubkZM2fORG1trem2IiZfGyd+ st73EizBgpELuNu6wTh57dDatejefTK3LD8T9rZv3y7UnWqcfDpjUwleua4x43sGb8ItD1Z+ rCWG+qJ6JEcm0yZbGre10mnEqOc3r/wQcwc9yS3LD27sr/f48NBrnLl8oKvhhKk9puKtg29l JMa0mgStt79dIk6r5I1m3xn1y5p8ne36LxqR+s0mX6usH0ivPzKQVX8YX375vtThKJn1H0j9 fp06aVLOJl8X9jgpNccoGo0CAOLxuDAhQaS8vBzLly/HoUOHsGzZMowdO1ZY2XbDC/ofKP3q 7mbDG36H01QfY64u25kWgmrEbMKtGRX9KrQfav1rfVG99r5yQKXtA5A3BGXWSLth5hNSI0Pc 2F/fKDIOUxltw3A7x4LN0XI6/4vySPEh/XxIPx/Vn/8y9LMJ1+PGjdPeWxGNRlNTi1jGa9WG 0vQRacboNLNItYqKCmzevBnl5eXYsmULFixYkLGNW5xOCrWKytF/fsEpF5iW5Xb+kdElLJps jPGLmGRdOaBSa/iwxkvlgEo0FDQ4blwB1tdWX66eINqfadU3hoy2YcieY+EnKs2qMadH9Tki pJ8P6ecTxOePG2TpP3LkCLZu3YpoNIoDBw6AOYTM0JYEUXGOEYtK0//pvzNSXFyM559/Hps3 b8bzzz+P4uJi3xrc/MAyWI9Z39BJjkziw6Efmm7vdqkK1XsMXnpsTidgA+b63exvR1Dsb9aQ tmoM6bGzv5PGCQ8/HiM3+mUl6Qti/XcD6eejuv6gPH+cYHaPytC/Y8cOfPzxxzj11FMxfPhw 1NXVoaTEvAHM2kJh/QeEfGItsbToJt4PjJeHu+o9Bl6PjRe15HT4yky/m/3tCIr9ndQvM8zs 78TT5JRYS8x0iFMUTL8Ir6MZ5LHg40S/nyjcIOj3g6oeF4ZI/WbPKBn69+/fj1GjRuGMM85A YWEhJkyYYNnA06L0aQHZ7KLvMZvNUQFONIi8PNxV6jGYoe+x6X84nfwYO/mh9arfacMpaPZ3 24A5+6JrTcsws7+XBs2Wi7akDW36aWSZYVZ/RCK7x6y6x8KJfj8LNgdBvx/IY5SJ/v6Xob+s rAwnn3yyo201jxFrGFHjSA7Gh7M+j47VD41VtJSTHpZKPQYz9D02tz+cTrb3o99Jw0t1+y/6 YqrjRqAXT5udfr9eJGP9EY3sHrPqHgvSz4c8Rnxk67cjrWGUSCSQSCRyKigIHD9+HE1NTThw IJVrhVUC4ytr3Fi9139eOaBS+56FKlttf+PRG1HRrwKxlhgW7lqobffTxE8BACs2rbDVc+qk SZblG7dnrw9tfIi7vf51yJAhjrZz86rXU12201a/n1eeft51BFLXJzkyiRuP3mhZvj7UVMZ5 uLG/l+N70W9nN/2rnX79BHkv9jGrPyKvg8j6r69X7H112U4p9V6mfv17J/qN9SIo+q3qT1Dt b2ZP/f0bdP1mdj510iR8/vnnaGpqQktLC3JF6L13XkmyCdiFvU8DoFYeIxk4zWPEW/jRSR4X u/KBE8NtTo4DuMtjVNGvwvWiiezmEIVZHhfmzpYR9s7Tz1sU1qkWlodD1oKaTu3v9fhG/U7K sLObvqyGggapwwn6PEwyFmYVWf/zLQ+QU/1+8kvJ1A+olcfILE+ZSnmMzOwvI4+RE1geo6/3 7Q4ACEciEUQiEZp8nSXcRuW4HQ5wc1N4GWqgORZ8VBzj1yNDv354jOaI8CH9fEj/CcyGYun5 449QKJRK8NhZlgQJCk7yuPDmWegbE2YNC5XGmPNxjkWu7C9qvT0Z+vVzv2TbX/X6Q/r5kP5M 9B1clZ7/ZuR6jhGbUhQGQA2jLOLEY2Q2idgsmWS+9Bj058l6bDIiigD1ezxW+v1E+uhRsf7o UanHbwbp50P6+ah+/+baY8QC0WjydZbxmvmXN+yVTz0G1mOTEVEEqNvjYR4hNlGe5yHy4zVi +mXkGAKox28H6edD+vmo/vzPtceIOYioYZRl9Gt1yUD1HgP12MwxrrWn9xAZG0J+vEZMv4wc Q4B8+7M8TKp6HFWv/6Sfj6rPH4bq+u1IaxhRHqPs4XatLreo3mOgHhsfM4+j1yzXZqhef1ge JlU9jqrXf9LPR/Xnj+r67WBOojDNL5JPNqNyVO8xUI+ND2+OmojM1KrXH9X1q17/ST8fqv98 cu0xam9vx/Hjx2nytR4ZCR6B9MSA27dvF5pQzbjdobVrHSd4tNNv9upHvxMdMzaVSE2wxtNv dx2dlH9o7VpP+zm1V6wlZprYjW3HEoRW9KvgJqJ0ot9Pojarz0XWH6f6RV6HbNR/mfYJgn4/ CR5l6t++fXtOnz8i7O/l/g2Kfnb/5jLBYzweRzweR+j9915NAqnGUc+CgQAowaPoBI+iMEuo 5uXYTvRnA6vzCYoefQI1t3pE21Vvm8oBlabziHJhM0BMYkxZBEWHGU7v56DiVX+u73OjjiDW WycE5TnuFZ5m1ljKFizBY0H3KICOOUZE9pB9wVUfYw7SGL+XuTvZmGNk1igSNdlY9fqjuv4g 1X8vkH4+svW/8OptUstX/f61o0uXLjjppJNo8nW2UX2MtjOO8buZyJuLOUZs2RgR+NXP0ghY QfWfTxDrvxuqy3b6TjLKg+xvDusYfbekSkr5DNXvXzui0Sii0WhqjhGRPUS0uHkPHuox81G9 x2OMShMdgeVXv96bZebFUt3+VP+tqehXgRmbSnwnGeVB9jeHPQNU1c/ItceIoQ2l0eTr7OCn xW2W/doI9Zj5qN7juTR2qdTyRem3arCpbn+q/9ZUDqhUWj+gtv0B9fXn2mMEpNpCtFZalvHT 4nbiGaAeMx/Vezws0kwWqtcf1fWrXv9JPx/SzyfXHiPWFqJw/Syjeoubejx8VLc/6edD9Z8P 6edD+vnk2mOkrZXGGkTUMMoOqre4qcfDR3X7k34+VP/5yNJvXCtQFkb9vDUJvaCq/Rmq3792 aJmvE4kERaV1ICvBo/7zIUOGCEuMZVb+qZMmSU3wKEI/T0d12U6pCdZ4+v3Yhb2eOmmSr/0B 4KGND6X9COjLGzJkiK8EeW70+7mOVtuLrD9O9Yu0Uzbqv9f6kWv927dvd3T/eqm/C3ctRKwl httxe1b1s+Mu3LVQiv0XNy3G8LeHY3HT4qzVn6DWf3b/5jLBIyP07tsvJ8PhMCKRCHr0SrU2 KcFjFQbdPcgyN4yfBI+scnnF7tiH1q5F9+6TTY/tRr8VbvSzBImVAypNbWmWIG7GphJUl+2U lqiMp19Ewr1Da9fi1EmTfCVcMyZzrOhXoX3WUNCA0rZS7XvRdmL63eCmPvmt/3YY9YtOfCdS v1X9f+W6Rkf76fd1ikz9ADBz+UDb+9eLfrZPrCWGLRdt8SLXskyefjOtIu3vpyyv9ccPsusP u39ZYylbsASP/XqfAgCpobRkMol4PJ5VIUFHVsip6mO0bvTzVoC3gsbI01m4a2Haw1M1/UZI Px/V679s/by1AkXA9Nvl4/LKqm6rhA7NGXFifz/Dg6rfv07R5hjRUJo/nFYy2S3hoM6xcNrQ 7Axj5H4eTNu3b09blFg0fuqPk/NRxf5W28jWP2NTidQEidnQ7wa352rM4yUapl9Wx3jFphVS y3dif32H1S1+64/dvZfrOUasHRSmiDR/6HMLOfmxU73FLVN/Rb8KVJftlPKDzwiC/b0+mCr6 VWDIkCGoHFApPLEjw0v9cZJfi6GK/a22yUb9l5kgMWgeI7fnyjxGoidFM/LF4yULv/XH7t7L tcdIC9enPEb+MP442d3oQegx+0Gm/soBlfjyy/el/OAzVLU/awgFUb+b6xVE/W6QXf9leEz1 jQi9fhmNC9keX+Yx8uP14JEt/bLoDB53mYRCIYTD4VTDKBwOIxym1UFEEZS1omS45IPQ4/cD 6edD+vkEzeNiBwuAAFKNCL1+XuPCa6OJPC58VNev+v1rB2sLaR4jahhlh2y0uN0MbbhF9R5D tvW7/WGx257sz0d1/aJ7/MZngFP9Xj0yTL+seVKqe1xU189SFYi6vsZpE7n2GAGa1yjVIKLJ 1/7zGFm9is4DYZfHiC0b4TS/kqw8RkadZnl5zPTnKg+HXs/ipsW+8hhV9KvIyH/i5PqxvCls f+N2ucgDJKJe5lK/3f1qdz4PbXxIyzezqtsq0+295KPJVh4g/faruq3SdNqV4zYf2vbt23H2 Rdda1nuv+vX71RfVe3puebG/lU4/ecTqi+oty3ObH8lrHiOv+h/a+BBux+3a9fWSx8l4fP30 AHb/5jKPEYvSD/3Xuj8lWYLH7j0HAKA8RlVVVXj8+scd5QHSvzdS0a8iY/4F+3HwipM8RnZ5 dMzKsMo1ZMSNflZ2Rb8KrYeh18PTLwuefrPraHYNeej12+W44n2XHJnU0hzoNfitP3Z4tb/+ nHg2y7Z+s2vgJHeM1T3NfpyN94vTfDT6oS2z7VkencoBlZY5wNzkvjGeB9NvxdQeU/HWwbdM P2PXdOGuhWnbec2jpj8HpznPzPQ7yf+jL5+dg9n+zP5m+M1jtLhpMVZsWqHp1z9/rY5ldQ76 3GbZyoMV2hhCrCWGUwediv/3j/8HABn11Ol1tDp+rvMYfb1v99RQWlaPrhhGd6Gb3BayVxe3 cmW6/VFzO+zG9LuZg6C3g932QZsj4nYiuFv9PDuaRZ+pMMbPs5kK+vUYH+7sR41FobI/Pbz7 wu4+Yz/KToeynA5psPOwm+NibBTpP1u4a6Gmh31mtA/P/rznp/F8re4Lr3N09OXzbHr2Rdd6 Kt+pBr1+t8NRTuqEsVEnaoI927++qB4fDv3QUpMxd53bY+d6jpE2tYiG0KwxVkCnjQee58Vv S9iuIeN2jNbtDz/T73YOgtMGmOpzRNzq19tx2hfTbLcPmn6G09xKudLv9OFs/PHW93wr+lWg oaDBUTlukprqMZsjwjRN+2JaRsPCeD9Zbcsa2Ub9Ff0qkByZNPUcVPSr0M79glMuMP3e+Pzg 1R+re9/MTlbPF7M5OlZ29vLDvOiLqZbf+Un6yDSwYXJAzhxQY/3xOlfMaDu2/08TP+Xu4/fY uZ5jFIlEUh4jWieNj9ubyy6/jN8es11Dxq7FbXcexvM1vrfSb2cnvW7edkHzGLnFSr+TemTV C9cTVI+L09xK2dZvl2dscdPitGtj9hDXn5tRP28ogtc4smpImnksjF4aHnbbDhkyJO3YVteL nS87d72XQP+9EWNULK/eO20g6BsjZh4jMzsbo/HMjmvGqZMmpdlHf339RO+xfS+NXepo2oKT 57QZ1WU7LfU4bdgZbaff577R9wEwtyHvOjo9dhA8RgCoYWSHk1avm0zE2e4xGxs47DystPJc o1arW+vL1btQeYjyeLmFZ38RiSXN9BvtYzyevmduN6cpqB4jp2Rbv12eMf3QipMf0O3bMzOP G70q+uupH3LT/1hZNSR5HgseTn94tm/fbnlsu+eYk+ecMSqW9/x0Og9Fj5nHy6x8q+cLO2+r czm0dq2rBKpm58euhZnnjgXG2JXN6g3b38obY4SXudtJw2XaF9Mst6voV6Hdv0Yb2j07nXqN cu0xSiQSSCQSCH3w/mtJ1kpSafJ1c3Mz5s+fj02bNuG8887D0qVL0adPHyHbV1VVofud3bmt dreTckViN4ENsHflW01GFY3eTkzTBadcoE3eM2oJAmaTnv2UY0Yu609nw0n9llEnedff6SKr bEIrqy9uzoW95qKu2el08lyw2sd4fxqHDFmZbB/9NfByXY3XkTdp2g6jB4qdB4CM/92W6UWT 2WR7ABmanDYS9dfReC5m15e32DQArLvvPoz60Y/QjfPbLgI2+Xpg/14AcGLytWqZr5csWYJh w4ahpqYGQ4cOxZIlS4Ruz3oNyZHJtNYwc696fdCI7jFb5YEw06fv0fLKsOKCUy5wnIfDzE56 tzyvxyYTJ/b3s+SGXj8rRw+rU0GpP0aCYH8/eNVvHCqywo1+3jPECr3Hhf0g6esLr5fO7rkP h36Y9qqva9myv9X9bfdcYPYyu29YaDfv/tR7qZj93Hj0zeqPWRl2ZZnNz9J7XPTnrR+y1P9v Nr/LeN2N/9vlMWL2Z/tZDbkaNTF49cd4HfVl6D/n2dBo/7+tXYvq2bNR9+yziB87xj03EbAE j6F1a19PAqk8Rj16pYyqgseovLwc1dXVKC4uRnNzM2bOnIna2loh21dVVeGxxx6TJd03Tj0a bj0fxha/VY/TWK4oD0u+QvbJLXY9c6rHcvFqV7fPOb1HIpvXMGj1xu1zPAgeRiOsAbZi8mTt s1P798fof/93lIwdK/x4zGM0eEAhAKjbMBo6dCi2bNmCf/u3f8OTTz6J4cOH4y9/+Yun7Rsa GlBdXQ0cPQrs2oVNra0oGzwY6NMH2LtX6OvhHTvQbdAg4eVqr3/5CzB0qLTyST/pJ/2kn/ST fmn6Tz8d2LUL6N0bfdet037Du/Xpg1E/+hEGS5igbWwYRVmmR9UoKChAS0sLnnnmGTQ3N6Ow sNDz9qWlpZg/f772vqqqKu29SJqamtCvXz8pZQMANmwARo2SVjzpt4H0cyH9NpB+LqTfhjzT v2LyZERPPhnn33ADzrr8coSjUXnHhi4qTepRJFJeXo7ly5fj0KFDWLZsGcbauNfcbi+LoqIi uQf43/+VWjzpt4H0cyH9NpB+LqTfhjzTP3jSJFz+3HMY+s//LL1RBBjC9VWkoqICmzdvRnl5 ObZs2YIFCxZo35WWlrraPpucdNJJcg/wrW9JLZ7020D6uZB+G0g/F9JvQ57pn3DPPdIj0swI ffD+a1rLSKU5RjJpaGgwbVwRBJGflJaWoqHBWVZrgiDyCzbH6LSSlEcsnEgkQEke08nnRlFd XR2uuuoqDB8+HJMnT8arr74qtPzm5mbMmjULZWVlmD17Nvbu3QsAqK6uxpQpUxCLxfCd73wH GzZsEHpcVSgtLc34E4mV/RlPPPFEXtdvO3Jl/+PHj2Pp0qWYNGkSzjjjjE57Dbw8f9zYysr+ a9aswUUXXYRzzjkHF110Ed5//33P56AypaWl+M1vfgMAeOaZZ4TXQ6v7yu65FBRYW4gyX3cy 5s2bh6uvvhobNmzAihUrUFNTI7R8q3xR7777Lp544gl88skn+N73vod58+YJPa4qNDQ0aJ4J /f+i4OXr2rp1K5YvXy70eKqRK/s//vjjWL16NR5//HH8z//8T6f1TuXq+VNRUYFFixahrq4O d911F26//Xahx1WJlStXIplMYuXKlcLLtrqn3OYRzDVhAIjH42hvb8+1FiILRKNR7Nq1C599 9hn69u2rVVBjC5+9Ly0txXPPPYeRI0dizJgxqbQGHGpra3HDDTege/fumDt3rvbge/TRR3H2 2Wfja1/7GkaNGoXW1laqczqs7P/Xv/4V1157Lc4991xMnToVn376KbccK/sfO3YM8+fPx113 3SXnBPIAs3pudV2ssLJ/dXU1Fi1ahGHDhiEcVjbmxTe8509paSnOOussXHbZZdi4caP2uf57 O6zsX1xcnErcFwohHA7LjawKOL169cLjjz+eFpm9adMmTJ8+HbFYDNOnT8emTZsAuH/+W2F1 XYJGAnHEk+3qTr4mvPHss8+ira0N99xzD8aOHYs333zTdp9IJIKamhr88pe/tG3pt7W1oaio CHPnzkVRURFaW1szvp83bx7mzJmDaBaiDFTn9ttvx9VXX42NGzfizjvvxJ133snd3sr+VVVV OPPMM3H55ZdnQ7aSuKnnVljZv6mpCevWrcPw4cMxadIkvPfeeyKlK4PV84d5Gj777DPcfffd +PGPf6x9rv/eDiv7P/DAA/jxj3+MoUOH4uabb8YvfvELSWcYfGbNmoVf/epXmDVrlvbZXXfd hdmzZ2Pjxo249tpr0zpQMu+LoMEyX0cTiYRyy4EQ3hk0aBDuueceAMDnn3+Of/3Xf8W0adPS tonH42nvv/vd7yISiWDChAlobm7mls/LF1VfX4+bbroJl1xyibQ8UfmA3v6fffYZbrnlFtxy yy0AYOttsLL/smXLkEgk8MYbbwCgycZm2NVz431hhrzf2iQAACAASURBVJX9e/bsiQsvvBC3 3347PvroI8yfPx/f/OY3hZ9D0LF6/rz44ot48sknsXv3brS3t3v+TbKy/2233YaHH34Y5eXl qKmpwW233YbVq1cLOy+VmD59OqZPnw4AuPnmmwGkMk1fdtll6Nq1Ky6//HLcf//92vZunv9W uM07mCvC4XBqjhEAmmPUiZg/fz62bduGo0eP4n/+53+0h33fvn2xfv16HD16FC+88ELaPpFI xHH5VvmiVq1ahUWLFmHp0qVYsGBBpx5OMMPK/rFYDE899RS2bt2KhoYGfPHFF9xyrOz/xRdf ZMyvIdIxq+e8+8IMK/uPGTMm7TnbWTujVs+fBx98EPfddx/q6urwzDPPpNmqa9eu2Llzp6Py rezPPBShUAiRSAQtLS2Cz0xthgwZgj/+8Y84cuQIXn31VQwZMkT7zs3z34qg5BG0g9U7+nXq ZIwbNw7f//73MWLECDzxxBOaa/T222/HrbfeirFjxzrqGVthlS9q4cKF2LRpE6688kptvsDh w4eFnFM+YGX/JUuW4Pe//z3GjBnjaJ5FUPJ15Qtu7wte/X/66adx/vnn46c//WmnHcqxev58 //vfx80334zy8vKMxj/zKjmZY2Rl/wcffBD3338/zj33XPzsZz9DZWXu1zQLEg888ABWrFiB kSNH4vnnn8cDDzzgqRz9M0r/vyrPpWQyiXg8jtA7q1/Smua9+6ZaiZ09jxFBEARBEJ0Dlsdo 4ICeSCQSiAJAIpHIqSiCIAiCIIggEAVACR4JgiAIgujUsHQOWuZrmgxLEARBEERnhUWlRaPR aGqyUSeNkiAIgiAIggiFQgiFQoiyFhI1jAiCIAiC6KxoDaNEIkGTrwmCIAiC6PSEw2FEjx8/ TkNpBEEQBEF0ekKhUKphRJOvCYIgCILozLDo/Gg8HtcWTvODk6ykMpchoLWfCIIgCILwCptv HQVOrB/jmw2c70bxdzVrWLlp6FCjiCAIgiAIr2gNozAiiISiSHpfHksY1LghCIIgCCIXREJR hBBCOBKJIBwOi/EYSaK0tBRXXXUV7rjjDlx88cW466670r6zWlyztLQUv/3tbzFy5EiUl5fj nXfeyaZsgiAIgiAUIplMphpG7C/XjBkzBsOGDcPkyZNRWVmZtvr6nXfeiZUrV6KyshJvvvmm 9nlDQwPX03Tw4EGsX78eixYtwoMPPihVP0EQBEEQasKWRwszb1GuG0YNDQ346KOPsGnTJixb tgytra2YP3++9v25554LACgrK8PBgwcdl3v99dejW7dumDp1Knbu3ClcN0EQBEEQ+UEymUSY 5S8KyiKy0WgUp59+Ou666y7U1tZqn7OGWyQScaW1W7du2n7xeAAmUhEEQRAEETiYxyiaTCYD lfk6mUxiz549eOqppzB69OhcyyEIgiAIohPA2kJawyjXjSM2eTocDqNv376YNGkSfvnLXzre T/8/RbcRBEEQBOGWZDKZWitNGDa5injwGjPsO+Or0/2cbEsQBEEQROdFy3zNlgPxu1YaNToI giAIglCVRCKBSCQCWiCNIAiCIIhOD3MQBS4qjSAIgiAIIleE/Q6hEQRBEARBqE6Gx4ggCIIg CKIzk0wmU3OMqHFEEARBEERnRlsShDWKqHFEEARBEERnJZFIpPIYGb9oaWnBp59+mgtNBEEQ BEEQOUNrGIVCIYRCIezdu5cWWiUIgiAIotMRCoWQSCRSCR412r/EwP69tIYSkGo9MfcS29H4 PftLJBIwDs2x/VgiSZZpOxQKad+Fw2GEw2EkQ4m0ffVi9WWwhJTGYxg1Gf83lqvXnppwFdGW R0kmkwiFQmnHM9pDv00oFEIokkzTY9RvpYERP546P6M+dq7hcFg7vl6D8dxDoRAikQii0ShC oRDi8Tji8TiOHz+edj76Y+uvG11/uv50/en6G8/X6nu6/nT98+n6h0Kh1JIgRpFmBzUr2Cje 7nu9SDMj28GMb9TJyjZebP1J2+ln2xv1sAoZiZy4afRlO9VudXyj7fWfGe3FKqhVRTXekPpj GM9LXz5d/xPb0/Wn60/Xn66/XiNdf2v9xmPwvlfp+kf1H/CEG41udRGcfG/V+jQTzzsZM036 1qu+1egEs+30xmItdrOLmbKReYvX7sJolVD/v8WNof9Obz+zso3XUl+ZjBWIrj9df7r+dP3p +md+Rte/813/qPELo3H0Au0urLGVyQxudmCrY/AwajC7kMwoesM4KdtMk97I+ovIWvbs/FiF S+jO18wWdvBuilAolHZjGPczs4vx2JFIRNvOaB+r41odw0w3XX+6/sZzputP119fvtn/xnKs oOtP1z9b1z+tYdR/4Fm2OxMEQRAEQeQTLS0taD/SBgAI81qJBEEQBEEQ+czevXuxY8cO7X1G HiMA6NGjR9YEEQRBEARB5Apj7sa0PEYEQRAEQRCdEdYO0tZKo4YRQRAEQRCdlbSGEc0vIgiC IAiCwInM19Q4IgiCIAgin1izZo3p51OmTMn4TEsgafyAIAiCOEFpaamwfUpLS7U/giDkM2XK FEyZMgXjxo3T3lvBciBFaW4RQRCEWBoaGrifU8OIILLHkSNH8Je//AXRaBQHDhxANGoakK8R jkQi2sJsBEHIpbS0FNOnT0/7bPr06fRDSRAEIYEdO3bg448/xqmnnorhw4ejrq4OJSUlptuy TOZRtv4LzTEiiOzw5ZdfYvPmzTj33HOxadMmHDp0KNeSCIIg8pL9+/dj1KhROPnkkwEAEyZM sNyWOYjCxtV0CYKQy9VXX40XXngBAPDCCy/g6quv1r6Lx+N47LHHMGnSJJx//vm47bbbcPjw Ye370tJS/PjHP8Y3vvEN/PrXv8Z1112H888/H7///e8BAIcPH8aiRYswZswYjBkzBnfffTe+ +uqrtOOXlpbi6aefxrhx4zB8+HD85je/0b677LLLsH79eu390aNHMXLkSLS0tDg6NzPPl/6z rVu34rrrrsPIkSMxdOhQXH755XjrrbfStv/jH/+Ib37zmzjnnHNw7bXXYufOndp3x48fx/33 348xY8Zg9OjRePbZZ9PKt7OfHUePHsV9992HMWPGYNSoUWm2caLP7xwiv/YnCCKdffv24b// +7+xZs2atD8zIpEIIpFIakkQgiCyxxVXXIE333wTe/bswVtvvYUrrrhC+2758uX48MMP8bvf /Q5r165FNBrF0qVL0/b/7ne/i//8z//EI488grlz52LZsmV46qmnAAC//vWvsXfvXrzxxht4 /fXX8fe//x2PPPJIhoaamhr87ne/w7p167Bu3Trt82uuuQYvvvii9n7NmjUYMWIEioqKhJz7 D3/4Q8ycORPr169HXV0d7r77brzyyitp26xevRrPPPMMPvnkE4wbNw5333239t0TTzyBv/71 r3jttdfw5ptv4sMPP0zb14n9eDz66KPYvXs33njjDbzxxhsZ5dvpa2hosJxf5ATZ9ieIzoab ydcMWiuNILJMnz598H/+z//Bj370I5SXl6N3797ady+++CIeeOABlJSUoGfPnliwYAFWr16d tv/IkSMxYsQIAMDo0aNx7rnnoqmpCQDw1ltv4Y477kCfPn3Qt29f3HnnnXjzzTczNNx///04 7bTTUFhYiOeee077/PLLL0dtba3moXj11VcxY8YMYeceCoWwbds2bNmyBQcOHMA3vvENrVHH eOyxx3D66afj5JNPxpw5c/Dxxx9r373yyitYtGgRiouL0bt3byxcuDBtXyf24/Haa69p9isu LsYdd9yRsQ1Pn19k258gOiNHjhzB1q1bbSdfs5GzaCKRQDKZpHB9gsgi11xzDa677jo8++yz aZ/v3r0bkydPTvvMGBjRpUuXjP/b29sBAHv27MHXv/517fsBAwZgz549Gce3mnx4yimn4OKL L8bLL7+Mq666Ch9//DEefvhh5ydmw29+8xs888wzuP/++7F9+3Z069YN9957Ly655BIAQH19 PRYvXqw1nIw0NzdjwIAB2nv9uQLO7MfDzH567PT5Rbb9CaKzsWPHDvztb39D//79MWjQIO7k ayDVOIqyRhE1jAgie0yYMMF0yKWkpAQrV65Ez549PZXbp08f7N69G6eddhqAVEOhT58+Gdvx 5hRec801mD9/Prp3747Jkydrkxad0LVrV3z11VfaPn//+9/Tvi8tLcWDDz4IIDUf6E9/+hPu uecerWH07//+7/jhD3+IRx99FD169NC8Sozi4mI0NjZqc3h27dqVVr5f+/Xt2zfDfnrs9Dml a9euOHr0KL72ta9lfOfH/gRBpONm8jUjzCZe0+Rrgsg911xzDRYuXIgdO3bg2LFj+PTTT3Hj jTc63n/q1KmorKzE3r17sXfvXlRWVmLq1KmuNJx33nno2rUrHn74YUybNs3Vvueccw6WLVuG w4cPY/fu3XjggQfSvp8zZw4++OADHD58GEePHsXhw4fRq1cv7fsjR46gqKgIJ598Mnbu3Il7 7rknbf+ZM2fi5z//OZqamtDS0oKHHnoo7Xu/9rvssstQWVmJffv2Yc+ePaisrEz73k6fU845 5xy89NJLiMfjGd/5sT9BEOm4mXzNCLNZ2JFIJEsyCYKwYs6cORgzZgzmzp2LESNG4J577sE/ //M/O97/Jz/5CQoLC/Htb38b3/72t1FUVISbb77ZtY6rr74ahw4dwoUXXuhqv5/97Gd4++23 MXLkSHzve9/LyNl0/fXX46mnnsLIkSMxfvx4vP3223j88ce175csWYKHHnoI5557LmbNmoVv fetbafv/8Ic/xJAhQzB9+nRMnToV5eXlac8uv/a76aab0L9/f3z729/GtGnTMGbMmLTv7fTp I9J40Wn33HMPnnvuOZx99tmm23i1P0EQ6YTDYUyaNEmbhD1p0iTL4fVkMplyFK15+9Ukm3xd VHwaAKBHjx5ZlE0QRNB47bXX8Prrr2dMjA4azc3NuOqqq/DBBx/kWopQVLE/QQSdjRs3om/f vtrcwd27d2Pv3r1pQ+Ds+THo6wWphNcsGo2G0giCAIBjx47hD3/4Ay6++OJcSzHl1ltvxfbt 23Hw4EH8+te/zvDaqE7Q7U8QKjF06FDs27cPNTU1qKmpwb59+zB06FDuPmEAFK5PEITGOeec AwAZw2BBYfz48Zg1axamTJmC9vZ23HLLLbmWJJSg258gVOKUU07B+eefjwkTJmDChAk4//zz TfOTAdAi9KPsDTWMCIIArBdADQozZ87EzJkzcy1DGkG3P0HkK6wtFNV/QBAEQRAEkS/YRaAZ SSQSiLK5RTTHiCAIgiCIfMJsCRC7xlIUSIWzkceIIAiCIIjODnmMCIIgCILoNNgtJBulBlEm uz/6CADwdUNyN4IgCIIg1KGtrQ0NDQ04fPgwAKBbt24444wz0jLuG4nGk6nFJ2koLUX7kSP4 sKoKoUgElz37LCK6BTsJgiAIglCHrVu34uyzz0ZhYSEAoKWlBfX19aZZ5dvb25FMJqHlxSbP UYr6F17AP/bswaG//x31f/hDruUQBEEQBJElQqEQDaXpOdjYiC2//732fssf/oAhF12EU/v3 z6EqgiAIgiC8MGzYMGzbtg319fUAUkNpw4YN4+6j5TGiBhKw8YknkDh+XHsfP3oUHz/6KCYb VggnCIIgCCL4FBQUYPTo0Y62DYVCqaE0NreI5hgBk3/xC8x+/33sufRSzH7/fcx+/33hjaKW lhah5RkJrVsntXzSz4f08yH9fEg/H9LPh/Rn0tbWhg0bNmDdunVYt24dNmzYgP3795tuy9pB UWoQmVNQWiql3EOxGAo6XHoyOFZSgi47d0orn/TzIf18SD8f0s+H9PNRXv/ppyN+5ZU48oMf CCvTzeRr1h4Ks7VBqIGUHdolR7klTzpJavmknw/p50P6+ZB+PqSfj/L6AXRbvlzqMXgkEgkk k0mE3n7r/ybZ/KLefYcAAHr06JEzYUGgqqoKP3/88VzLIAiCIIhOw/GTgB73V2L3lVcKK7Ot rQ3btm3LyGNUUFCgbfPBBx8AAHr3+hpCoRDC8Xgc8XgciURCmBBVOX78OJqamnDgwAEUtgLj 62LCX9mfrPKvf71Eavmkn/STftJP+gOmH61oVFl/x+sPXinB32bMQFNTk7D5TGzy9cSJEzFx 4kSMHj06rVGkp729HfF4HKG3/rQyCaRmY/ftdwYA8hhVVVXh8Z+Tx4ggOj2FrUBrYa5VEASX 1o7XfKipyYIktm/fLqw8qwVj9cuCMI9Rz1PCCIfDCIdCIYTDYQrXzxKxxpjU8mfUlkgtn/Tz If18lNJf2IrW1kLtRweFrahrjKXeF7Zqn1nt6wWyPx/Sz2dG2etSy9f0e6zfdsiy/5QpUzL+ eNAcIxPIY0QQBGsYAameuPGnoFD3vdk2+dB7J9RA8xiJ9nCyBhArs6P8VnasDCHp23lBhsfI riHEPEY9uqUcRWHmLQqHw9wdCTFQj4cP6edD+vm41u/Q48PeNer0txoe/MaS0jxL+veFrRnb Mjqd/V1C+vk0vV7m3Zuj389YRke9rasbr33U2lqY8cfqNvO0slL0dZ+nT7b97WBOotB777yS DIVCCIVC6FU4CAB5jMhjRBCdBP1DuuPBrsfY+PF8GIN3CSCPEiGONE+l0csDpHt+zLxATo7B 8Z7yMPW2crYX7TFyAvMY9To1AgCpOUbsr7Ojj0oDTrTuea+tDrdjr+zP6faOXwtbERtfhxll ryM2vi71XsJxZOlvBVDXGMOM2hI59smG/VXWD6TqT2fUH2vU9te/Z71j5iVqbIxpf8bP7V5b WwszPtfrTrt/VbQ/069q/VHZ/h31trExhmO1JZn1WV+vC1tPfM/Om723eWV1X388J691Fp9b 1f/PP/9caFSaG7T20Jp3q5NsOK1Hr5QbizxGzj1GrQhGz89s/oPdGG9QtBvncqhIYGzplY5r oPI5uLoG7P7g9JZFeYssJbDjWHyuEtLmuPjBjRaj51AhzGqwcf5Pa2uh9pn2v0391+8rmrQS DRqSyYKceYwKe5yUWistGk2tIxuPx7MqpLPCWsmyONYxRivroS5Dv16r6mP8qupn1yBrUS2S EGV/q/unMUv3rzAMPzqy7d/0epnU8h3rNxm6dMKMklrX+7ghl89/L/Yw7iO7/su2vx3RaDQV pf/B+68l2ZIgNMcoRT54jAAbXTI9BC57jfkQyaNFaSjW22Qofw3c1mdddA2AtN50trD0GImo R/roIX8lOSLn9UdvM71nxM3+jGzVAd51dlEHvAbOF7Lj8MqWZIugeoz6Fp6MRCKRWiuN5hh5 wOPM/2z1GACk51rRRwN40d6xDxsPljUEESiPi0s7tbYWOtevQv2RQFA9Xk7rbOA8RsZ70eY+ z8f6w6KgtIg/FiHlFJ2dNI+Fxf1pjDTklZX2vkMbm5/jaB+770wQZX8rG6rmMdqwYQOam5sd rwWrzTFau+aPWh4jmmOUwpHHyEuvRCJWt4tprgm2j36s2Q6rckwiG9zaJOe9TQvc9LZdz7HQ j/EHwMsU1GvgFLf62bX12tsWgWePkYl3xArTe1GCtzjD/lms1zwLOFJg9uyy0G97n5tFf5mh v/eNZdl1mmzSRDjF6DFivwfZ8JoaPUb6OVAiPUYHDx7Ezp07sX//fvTv3x8DBgxAF5OFcJnH qF/vU1JzjNgwGuEiKq2j1c+iVnIelaCb7X+stiQzKqBuPFpbCzW97D2LlnETvVNXN/7Eeeuj GwzlutXNXl1FtbBoCxn2L2z1dB4zSmrVi2ox1J/Opt9NdJnXqDSzV7PjH6stsX+uGO47u2ii Ot3+scZYRvksm7dn+xujorJcf7h2dhKdq4tCdKKfF1WllWcR7cX+jJ+b6bHdzqL+uK1/xt+D bNd/vf2bXi8TGpXWo0cPDBs2DKNGjQKQ8iBt3boVBw8eNN0+LSqNvelZMFArrDNj6zEyzEcI Qg/bT8/XSQ/P0iNl8X02PEZcj47PHqvWM3SycWFmBmQn+5DHSByu9Os8JkH0GOm/MyOt3rsc 0tafs/H4nq+7Wf0PiMcIcF4ftLdmeYAMx3IT1WWKR4+RmafPr8com/Pq0o7dgVF/gcQ8RolE Anv37kVjY6PWWAJOeIy+3rf7CY9RIpFAIpGQIiQfyRh/dXFzsFayLGSP8UsfY1Z0jgvDkf2N D6NOVH9U1x+4OUYWWP3Ypc1x8fqjzjlWIOuPi/N0qt9rhFcs1ii1IeLW/m61qFL/rQiHwygu Lk5rFJluBwA0nJY96gfWSy2/y9id4gvVPVgGGvX7fbga9q92q9/l8WXYX/9wkWJ/HUrWHx1B 0O/nhymj/gtG0+8iGzE7n7SlGSxYv/68tH1FNY4YXPvbBG04wUv9cXO9mX5RjRf9tQHS7S8D V/evh2uRtfqfQ0KhEMKRSASRSISi0nzg5iYKZI/ZxQ1CHiM+Tuzv56EbBP1+kKJfV387q8fI a1Sd1+gtK5SpPxbnIlK/sVEE8Nfac1oej85a/0URCoVSQ2m0JEh2CUKP2QynN6mxx2DZ63TR 49Xj1mPk9uESNPurrp+LSR3wpN9FrhXZPc5s9ZhlDbf40e9Ek6X9fTwj9Ait/ya5q/x6XOy8 dmYed5HXOl/qf65gU4rCAKhhlEVU7PFb9XjMvs/4zOXDrzN4jPxA+vm40u/hh1n1HrPq+kXV H6vGyLHaEk+eHKe5k5za32tjieqPP9i0Ipp8rcNpuL5t+CHntX5gvdRw5S5jd7oOm7QLD9Zv P3BgvftwTIfhr42NMVSP3el6Pzd2cmN/p+Xr9XdxoF8r10Pah2zUH2HlmoQXe9Hvpn66sr9u cUynr27qv5dwff3967ReitLv9X5wbH+rsHSf96+j8zcL2ze5/q7s3zGRWrT9M9KhGHTa2d9x /Qto/c/1IrIAEHr/vVeTiUQCyWQSRX1OB0Dh+nbh+m7Da/Wwm1soOnfssdoSz+5Iq3PQny+7 OZyWZxdSb+wZXV9b4ng4jRtmbBEu7NT+bsK/9dseqy1BPxv9+pBfxykfOs5HSv3RaXKi3zEm Yc9e9Nsl2hRhf6e4qf92mIXr6+9f3v3IC/XnwdOflgLAIlTdTJOp/Xkh6Hpvskv7m9UfpzYw hoizkHs39k97LvioP9qz0STZpl395IXru3n+57L+W4XrH6stQfEljdKXBFmzZg2mTJmivdeH 62tLglBUWvZQao6ICa5uCg9DFa6j0lyinP0NNpSiX3cMz/qdXOvCVvXsb0D4HAuD3YKg38+c Fzf6HR9HZ6OM+uNjeSMzgmB/hpfrIMX+OjrDHKNEInFi8nWuWb16NaZOnYrhw4fjiiuuwKef fsrdvrm5GbNmzUJZWRlmz56NvXv3crdvb29HVVUVxo8fjxEjRuDpp58WKd8xzH0okmzmEXEz xmx345l9T3OM+JB+PkGq/17IF/1SJo/rhsP8wNOWL/aXhYr616xZk/FnRXt7O44fPx6cyddv vPEGHn30UXzyySe48cYbMW/ePO72S5YswbBhw1BTU4OhQ4diyZIl3O2ffPJJ1NfXY+XKlfjg gw9sG1KyoB4zH1EeI6uHn+r2D3weJptcNUGyfxB6zEYNgbx/XXhlHOv3mM8oYxhNcANMtfpj JJD1xwUy9E+ZMiXjz4p4PI54PH6iYRSNRoULcsMjjzyCM844A4lEAkePHkW3bt2429fW1uKG G25A9+7dMXfuXNTU1HC3f/nll7Fo0SL0798fPXv2xJ133ilSvmOox8xHusfIuLq14AR3suzP HsKBrj+8VcE7MF1dnFeOS1Sv/0HU76YBkKbfcB1FNGJ81X9Do91MT7btL7phF8T644ZcR6UB QCQSSc0xCgqlpaWIxWK499578fDDD3O3bWtrQ1FREebOnYuioiK0tvIfpk1NTSgpKdGOU1pa mrnR/v3AqlXAwYPaD7TV6zHDq9327PXMxgOutnf6aqxQRn1OXzPKL3s97fviDv2+y7XYjuH2 vJ1+fuaBxvTPS2od6XJrfyfbt7YW2pffoY9tl63647ocZkfda2trYYZ9M+zfUb9Myy1slabf 6/3htf47fWXY1gvDfSlKv+jnj3YdO+pB0+tlaa9Oym9tLdTO98xLFpse18lra2uhdlzj+Tqy f2Gr7+tbbLh/7Z5fTq+TqOd/tuo/T3/7vn3AqlUINTRAFlZeoy5duuCkk05C6L13XtEWke1V OAhAbqPSvvrqK6xevRpPPfUU3nzzTcvtysvLUV1djeLiYjQ3N2PmzJmora213H7SpElYsWIF Bg5MLZRbWlqKBgvD5yQqzceii16jEjIkmH6YHrngJSrBSbQbQ1RUmj7qRE9sfB3q9Wn5ORE4 2iYOdQBwFRWlhxe5xyJnCtFRf85bnxnxI2DBXMCZftPjWiS701+DVgDjndjfsECn2cKZfvT7 8RGKjkozakmLiuJFh5lEVDnBSVSaMfqMvXcTlZZ2/5ks0mobjWmoN2wfY/3x6+81XoM0+xu2 FeFbNkal6XU4PY6oqDQvqBiV1tbWhoaGBhw+fBgA0K1bN5xxxhno1auXtg2LShtQ3COVx0jY 0X1SUVGB3bt3IxKJoEuXLrYeoPLycixfvhyHDh3CsmXLMHbsWO72l1xyCaqqqtDW1oY9e/aI lO6KIM2x8IKwMWaL4RIhc4w4WXbr6wf6L5+DinOM9IjWbxwqUN3+QZlj4XUIRoj+HEZ1Uf3h o5z+LERlbt26FaeddhrGjRuHcePGYfDgwaiv55+HNpSW68nXEyZMwL/8y79gxIgReOKJJ9KG 0syGvSoqKrB582aUl5djy5YtWLBgAbf8efPmoVevXrj44osxY8YM3HHHHcLPwQmBniNigvEB LGqM2erBLn2OUUdSM1MdAuYbybZ/rDGW8ZnIeQrSxvg7bBuLNbpa4sMt+TTHwtYOucjcbXNM J/b3tVag4f4VTT7VHxmort8JoVAoNZQWDocRe8B6cwAAIABJREFUDofRo1dKFCV4lDeUxivT axmipg9nHF/QOj527ne7ba2wHEpjSdOMQz1seEb3WdowjcF971ST8Vzcbs/bJ20YwzCsZtzG K271n9gwfSJrocVk28KObbX3xmESQ5naNfEwlOZEv9jp9t4xG0oz28aInwSVTo7FG0oz02Vm f62+sm0sdNoOpZmco5thJzt410DGUJrVcUUNpamC/plrvL4FBUnhQ2nbtm3LGEorKCjQtmFD aSX9egKgtdKyjhIeI07IdSCj0lz0nFXvcSpRfzik1R9eFJvdjz5vdXTBkYZ6st5jdhDp5wYl evzs2pucrxL6OZD+dIz3uAz9BQUFGD16NCZOnIiJEydi9OjRaY0iPdpaadraINQwyt1aaR0h zH7WuGKvXtZKS9Nf2Gq5Ro/XtXLc2NHTWmkdwzMZazCZrdVVPzDzPDvea2sUdVwPt2tI2a4V 5bf+xBpRr7O/3RpWXuuP67XQLF7NdA4cWJ+5FpThPvBTj7qM3Wm7BpefNZ6yuVZaxnkb7Sp4 rSt2HzSy+0P/PfvcgX6r+mF5XOPzz+b5I/J6Gs+Lt1aaiOvuZK00t89TUc//oNR/0WulmSV4 tEryyDJfh959++VkJBJBJBJB954DANBQWtaj0syGflwgNCrNxv3Nbg7X5Rqw6vt6ikqzi4jS rdlV1xjDeQPrM4dpdEM4foZmREelGYcxxjfGsL4jqsW4jVe8RqXph0syhjuMkUcd71n9Mdrc bFguLZLJ7NhW+i/d1PGFuVVUiUrTbwekR2fB8L9TePr1eozDnWbbMrhRaWwb3lCaybpq+qFZ Y1TseUy/gKFEow15UYEUlSau/pvdu4CcqDS2Lpp+fTSrtdL69zkVQMdQGq2Vlj1UiCpqbS20 fOCokvlaw9BoUj0qxHH98TjsIkq/1aR2O/v7/aHrMnannOUoOshJ/bG4lsIzLztoyPD2ATr0 CxzKNOrQ65dxnZWL6jJA+jOJRqP48ssvEQqF8OWXX+If//gHunTpYrpt2lBaMplEPB4XLojI xGyOSMaPiM3cCx6+xmgd9MAaTfSLRHZUmpV+UQ9Z6XOMnGSO9oEs/cy+mv0NdU2K/dm9JPCH Wnb9t7S/oHPg6fdyDXhzRNh3bhtZvP1yZn9BkH4+MvT3798fn376Kc466yxs3rwZGzduNE/w rCPKGkbkMcoOrnr8ZonvbFCix8AZMhTpMTKLlDLV7zNBoh4p9tfpc5rHxS6Sy1guw5F+H/bi 2V9EQ4npN7v2Ish1j9lvA1JoHjITLV7qv6O62kGu7e8X2c9PJfRzkKH/zDPPxJlnngkAGDBg AHfbZDKJUCiU8hjRxOvskdbjtwhvNsXhQ95Pi9vJQ1dVj1Gax8JiXowIPNvf4fX1FVXn4BgZ UV2CGxfZ7nGKvsa56DGLPAdX+j1ce9n1X0WPhR7Sz0eGfrOJ1laTr1l7SGsYUeMoS1FpLCoK 5lEXra2FJ6IyrKI1FItKMOplUUlm23uOSjN5zfi+Q7/b6J6cRKWZRNk1Nsawfv15llEdtlGO xvpjct5Oorr016+1tZB7PY3HMas/TqOrTM+LE1XEdGlRcDmq/26jcsy2ZxPXs6m/rm68aTSa 9pyy0G9XH9JedeWz6FI7/SKjxHj2F1lvnNjf6XF4uoL4/Hdb/0VHpQGZkWlWhEIhhMNhhD54 /7UkkHIh9SxIuekpKo0TlWYxD8c2UVkH2lpdukgpywRjhugMwNztHPSoBDfRHSKi0ozrMrHP CoFUVNp567lRMjmJSjOJzNF06+ocsz8vKk2zi8lxM/aziMrJiBozKceLL8ms/jiJrtKSBup1 mexnVf9FJQX0Uv+tcBqVJpKg6S80XEu7utDYeCIqTUa2KqN+kckkgfT712tUGm97VepPLqLS eJ/p10oLh8MnPEbhcGCWTQsmfiZx6vZdr18A0SKJmRVOXOqqjzF7mWPkZnJnzsfIfQ5NWeq3 mmhsdTyLz3NRf1wNFdn8cKpe/5XSb1KHlNJvAunno6J+Y6PI6jNGR3so1SCiydccvISxWsDc hm5xelzVx5hzFZUmCjv7+4380fRb5G6yXPvN7Rw1qwYIK8djA8+v/e0maNvWf58N01zXH7+I 1O/J/i7KMoPsz0cZ/bzM9TmEBaO5WhKEhbh99dVXuOKKKzB69GjNBZW36IY03PZs2X56WIvb SVlefkSD2GNwcx7C8xgZEBqVY0K27O90IVk72xu/5+bRETARO9c9zsBEdVkQxPvXDX71212f oOu3g/SnsLrOsvXbkUgkTjSM3PLuu++iS5cu+I//+A889NBDorUFDi8NIuP+bNiMTTh0U7ab 4yvTY7BgRtnrnvbj5T4x9bj4RCvTcL31HhdPw6Q2+7nRry/TqNeqTjH9Gd8LiuSjHjOfzqbf S+ZumWTN/hJSSQCdr/6IRpta5GYIrUuXLojH46irq8O4ceMwevRoNDQ0SJSZXcyi0txEg+ij gsyiNmKxRtOoKK+z+VWLSnASdbFs06We1vgCzKPd9NfPq/0zosRs7O8mijDtOCzqKtaYFp1o ZX+7tcUaDTqc7Ge6VpdJtJ/XKCmRUS1O9bupf9nUr9r969f+ovRnKypNZL3R6xdVP1WvP9mM SnNKJBJJRaWtW/u61jKyWytt2rRp+MUvfoG77roLd9xxB8aPH4/S0tK8ahwB6VFpTtv1Wr/H IpoISEVg1NWN50ZFuTpWBypEJTiNunC7VpcrDR325+k3jUozic6yisxKW6sLyEjExtOcEX1o Ms/HaH9HEV2G9bVY+WYEtv44XBfLLirNbz/dq34zVI9KMyMb+vMiKs3kt4G3Np0R/e+N1Vpv MshG/RcdleYENiVo8ICULcNusl7feuutmDt3LoqKijB27FhpIlWG9/BubS3EwIH1YpPOma1V JJHOOEbuZkKzSP1m9cSvfru5ckG0P+B8yCWo+p1C+vnki35RvwGO5ggKRHX725FIJJBIJE5M vnbCt771LWzcuBErVqxAJBIBgLzzFvmC1vrxjUz9+uEfP2XwyLb9RWd2pvrDh/TzOTHHTk75 ZH8+pD+TtrY2bNiwAevWrcO6deuwYcMG7N+/n7tPlP1Dma8N6BIwikR0HhHqMbhDpH6zRolK +s0g/XxIPx/Sz4f085Ghf+vWrTj77LNRWJh6Xre0tKC+vh4XXnhhxrYsfVGYNYgSiYTtAT75 5BNMmzZNW5ANgO0qtZ2JbK81JjKPiFNU7DHoyZZ+r54cu6Eusj8f0s+H9PPJlX5Rnl+yvxi0 ydfJZBI9eqVEWU2+vvjii3H33Xdjzpw52hBavk6+/rnVkiAWeF0iwQuiJpJmEzeTGJ0+ImSc P+86Ol22wFieHpWuWT4RxHsmm8+MrCPx5LJ5LUVPvnZyLDeTr1WuP1ZVpKAgKXTydVtbG7Zt 24bDhw8DALp164YzzjgDBQUF2jYZk6/dDKHt3r0bF1xwgfb+wIED6NWrlwjtgcAYri8j3JH9 +SmHF9Z5rLZEarimH/1OwlGP1ZbYh+s7DFMXrV9bvJET7m9mf6fh+tmqP3b2D2r98as/aPY3 05PP9hel32zxYJn6RYbr8+wvIlxfpfpjpV90uH5BQQFGjx6NiRMnYuLEiRg9enRao0gPy2MU Wv/BG0kWmWbnMbrxxhtx6aWX4pZbbsHmzZuxePFiHDhwAL/61a+EnEBQII+ReIR6jDpCW6V4 jFx4gxyVl/ZGbNmEc4J4z5DHKHBFmx6LETiPkeLPk2x5jNasWWP6udkisqcP7J3KfO0mweMv fvELvPvuu+jVqxcmTJiAgwcP4t577/UlurPBWsey6DRjzDlaq8uOTmN/j+RUv4Bsw2R/PqSf j0j9NMfUOawRNGXKFO4CsgxXeYx69+6NRx55BBs3bsSGDRvw8MMPCxtKq66uxpQpUxCLxfCd 73wHGzZs4G7f3NyMWbNmoaysDLNnz8bevXu527e3t6Oqqgrjx4/HiBEj8PTTTwvR7RYVZ/Xr yZl+3XpdfnpJdvr99sAs1xoT1Luj+mNBh2l5+sn+pN8O0s9Hdf12sLZQmC2a5sZzJIN3330X TzzxBD755BN873vfw7x587jbL1myBMOGDUNNTQ2GDh2KJUuWcLd/8sknUV9fj5UrV+KDDz6w bUjJQtUWNyPr+gWvKZQL+7tegJgD1R8OhcCxMoX1Q3H7g/TbYavf5/Mu5/p9kuuoNNYWCq15 tzrJJhz1LBgIwHqOUbbYuXMnvvnNb6K+vh7RaNR0m/LyclRXV6O4uBjNzc2YOXMmamtrLcuc MmUKnnnmGQwZMsT2+DTHSDxWKewtt2Up8nXzifTj6qLnAskir+eRBAUH1aCwNXjXIa/rBs0x 8nYsh8+1fKg72ZpjZMaaNWtM5xhpUWkAEI/H0d7ebluYWc4i0XmM2traMG/ePMyZM8eyUcS2 Kyoq0pYoaW3lV5OmpiaUlJRomk11798PrFoFHDyotVydvs5wuF3T6xe4Ktf0tbDV8niHnyvz X74E/TO0/fn6mP4ZJbWp8yypTTtvtr9dOVLt70C/23oRVP1BqT+2rx339uGysrT37FXUdSD7 O9RfUpLy4Kmq33D/Zqv+OH2uqfb8d2P/9n37gFWrEMpBGqAE4ogn2xF6751XkmwYrajP6QCs PUbGnEVffPEFrrjiCtTV1QkRVV9fj5tuugmXXHIJ5s+fr2WhNMOtx2jSpElYsWIFBg4caHou egLvMZIYlSULtxEXrcjMG6SKl0hPPvTsAg95jIKHRLd2PnuM3CxarnrdsXqe5zIqbfDAglRU mpOM13oPC/u/tLQUV155JX7wgx8IkA6sWrUKixYtwtKlS7FgwQJuowhINYyWL1+OQ4cOYdmy ZbaL2l5yySWoqqpCW1sb9uzZI0SzF0SM0fIaB6zlLYtsjjEbz1NEo0j1MXLSb8BQJZiHSBZk fz7HakucZWj1eCvTHEc+eVF/BMMi0fR/VoTDYYTDYYTeWf1SEkglNnLrMRKJ2dDW5s2b0a1b N9PjNjc349Zbb8WmTZtQVlaGpUuXori42LL8I0eOoLKyEn/6058QjUZx/fXXY+7cuabbBt1j lIvj+cWVxyhXniEJRlXtOgUWzY2oe3WzO3mMsoeTdO9eT74wu9eSeecB8hh5xqXbLZdzjE4b pJtjFAQaGhoy/rp166Z9Z6S4uBjPP/88Nm/ejOeff57bKAKArl274t5778VHH32E2tpay0aR bFRscevxrN9hzysn+gshbDXwwNrfIYHRr78mVq8mkMeIjxT9hSf+TKMCC03+rD4v5O/XGCP7 8wicfuO9avOsla2fYeU1SiaTiMfjJzxGANC7bypiK9dRabmGPEZyCKxmiRMJAnvOQUOyg1Cq l8Gj9iB6sVyTC8dulu2WrSjgvPIY2dULjngZc4yMDSErj9HAAT2RSCRSHqNEIgEnc40uu+wy /OEPf8A//vEPYaKDhCprpbld6yco+p2ulZZ1/THDq9tyx8dSvdnxwbY/97Www/5lLvTrztv0 80bd9x2v7M/4uel18PB6rKTE+nvR9mf6PeoVstZVoQ/9bo9jvF4m141rfwGvjbGY0LXLnNhf 5NpsPPuLOK9APH+c3s8W+kWvlQakGkL6PztC76x+KRmPx5FMJtHv62cBsPYY1dbW4oUXXkBN TQ0uuugiXHHFFRg5cqQw8UFBCY8RRWiJw6vHyMr8rSe+V8IrwKtGHubzBBFP1yHIXiync638 zu8J4LXPyj2ls1G2nlt54zFyUmcC6jEaVNLrRFRaMpm0jQIDgLFjx+KRRx7Bu+++i0Qigauu ugqTJ0/GY489hkOHDgk7ESVxOIeGtY4z99f9GT8z+d6qUeR4jNbLA0/Xa7TcnzdnwAHSx8iN +p3qc3heLH+Lq3JdYFl/jGXazeOwOI+MOTqCfxilzxHxM8fIwbkK1W/yzHBV/51cG+McnY7e vOncD7N9XCJ9jpco+1vcI8fKStI/F4yj+9cHgZtj5BJZUWlOPgNSUWnJZDKVxygejyMUCtnO MTp8+DDeeOMNvPjii/jb3/6G6dOnY9y4cXjppZfQ1taG3//+9+LOJod48hj58eC43U1EVyGb PcEs97xMcXq+VgJlzSMRZZQA9uyDhCMvQ7Zt6KSuWXl8OjGuPEZGW3p41mbNY+TiN4Q8RmJh HqPTBxehvb0dUa2FFArZ7jx+/HiMGjUKP/jBDzB58mQtM/XYsWMxfPhwqcKVw+jq7qgIjeNj GLi+XtrD7VhtCcwXMhVTfmMshoH18hYStNRvhdmDjxPW7Um/C9sdKylBl50uF0J0Ub50+3vR 74LA6nd4DaTrLytBF+j0638BBfwaBtb+Dkl5jDzqd3CNTfXrQvb90tgYs1yIVcTUCNfPT5fw 9IvgWFkJ0Ci2TCcJHhlseTTNYwQAfYpTuYSsPEatra0oLMz/7ooXj5FGNsyj2vwA5jGSMS9K 9DkK7qE78hgR0imETd3LxXUwNnioLjjCs8fICyLdMl7nMloUlZceo0KgICnHY7Rt2zbs378f 5513Hk466aS075jHaMhpvVNRacePH8fx48cdrZWW740iYVFpNlEVvqMznEQlFMqLCvGqt65u vHP9guztSb+P8k2jcrKtX1ZUV67qjwD9ra2FQq6DUP364xbmt/1F6ncdvSX7/nUSpVVoYv8s RdVlPSrNbf3X2elYifiotCNHjuDPf/4z2traMHjwYNTX1+Po0aOW24dCIYRee/X5JJt8PWDg OQCsPUaNjY247777sHHjRoRCIYwaNQo//elPtcVZ8wXlPUZBa7/K6trIOE/yGOUngueOCUHv MSIco5zHyEwDeYzSt9d5TEV7jNavX49BgwZh0KBBCIVCOHLkCP7yl79gxIgR2jb6OUYAEGbD aE6i0n7yk5/g/PPPx/r167Fu3TqUlZXhJz/5ibAT6Ayw1rIspEclkH4uykTlWED6+ZB+PqSf jzD9FlHQykaldTSKZNh/5MiRGDx4sDaPumvXrohZXIdkMpXvOlT98m+TkUgEXbp0Qd9+ZwCw 9hgNHz4cH330kbZUx+HDh3HBBRdg8+bNos8lp5DHSDBu50TlsidNHqP8hDxGeYPtfLH0jf3h 1i3jN/o1H/DiMdIh2mPkZPI18xgNHljQsZAsIoiEokjG7Q/w/e9/X1vRnq1qP2fOHCHiOwud 0uPiMqrLaw4VJwSmx+nx/AKj3yOk34BdHinB5IP9ZSa2da3fQW4zPco//2XnMZJUf1gjaMqU KZY5jAAgEooijAiikUgE4XAYkUjEdEOzVe+XLl2a9v62227zITmPcHBjyAyVBYAul8oL1QSy oF9iqC+QA/0CQ30Bsr8djvV7vCSB0e8R0s8nq/VHwkQhmaH6AMSH6hvsKLv+OCGZTCIciUTA /swwW/Xe+Jcv+I5Ky1ZUCC8qIchRLQ7sFwj9AqNauNFQHtYIy4eoopzp93Fdpeg30ZHX9hel PyhRaaLs39mj0gz2l7FWmlOSyWQqr+Pbf3opGY1GEYlE0KNwAADrOUadBc9zjLI1V4DXywjq fAUnPaMgaDfqlDFHIQjn2dkQfV1FQLmLvCFiPUO/xxJRbqBDyzzixi4W9V9WHiMeWlTawN4p jxGbqc1mYxNyYa1jWdAcBT451+/zgZpz/T4JhH4f1yAQ+n1A+vkoX39ojlEGa9asMf0zg3mM wslkEolEAolEQrggIpPAjJF7JO/1+2y45Fy/T0g/H9LPh/TzydAv2GPoeI6R1zlSEpcDAeTY n024HjdunPbeikQikd4wcpL5ulMR5KgojrZO0WPzAennQ/r5kH4+pJ9P4DxGxqg6q78OVPQY Aans11u3bkU0GsWBAwe0dV7NSCaTiJKnKLvkXY9HMKSfD+nnY6vfZ4cn5/p90qn0e7jWmVGl EDoPyJP9XWhw5DEqNLw61QBgIOqlzouSUX927NiBv/3tb+jfvz8GDRqEuro6y9U62JSicDKZ RCgUcpT5mvAP9Xj4kH4+pJ8P6edD+vkEVr/DRozmMXLg/fGCivbfv38/Ro0ahTPOOAOFhYWY MGEChgwZYrptIpFItYeEq1CYbITrD6yvFx/uq3vtsnOn1HBZX/od2C8Q+n2E5ZrqFxDm60q/ j9dA2F+W/qDZ30RPXttflH4JYfpc/bLvX6fnwRaj5WzfZexOITq5+iWG63fZuVN4uH5ZWRlO PvlkR9uyYLTQm6+tTLI3xQP4S4J0FqqqqvDzxx+3D4v3EALMKpdvLLQdKyuR6s72rN9huP6x kgDo56SrtyNDv+AwcWH1x4JA2N8HlvoFLb8hVL9JuHLe2l8QjbEYBq53qN/DtTbVL/Ae1uyv v/Zul3jibH+stiQ1nCZxjqxmf/1voJfjWdT/4sbGnIXrl/TriXA4jChrFBHZoVON8XuA9PMh /XxIPx/Sz0d5/bIzX9fXpzdmREfV5TjzNWsPhYPSMCotLdX+nNDc3IxZs2ahrKwMs2fPxt69 e7nbt7e3o6qqCuPHj8eIESPw9NNPi5DtumIwt6EsAjtG7hDSz4f08yH9fEg/H+X1B3GtTBfI tr8Tkslkao5REBpHbpcXWbJkCYYNG4aamhoMHToUS5Ys4W7/5JNPor6+HitXrsQHH3xg25Di YpzV76JxRD0eE3T2U1K/DtLPh/TrMHluKKXfBNLPJy88RhLJtcdIS/CoTTYKQOPIDbW1tbjh hhvQvXt3zJ07FzU1NdztX375ZSxatAj9+/dHz549ceedd4oV5LBxJLXFXahgj8fQuFROvwHS z4f08yH9fDL0658fAqKvAmd/s/Pg5bEjj5EvtASPOVXhg7a2NhQVFWHu3LkoKipCayt/BltT U5OWu8ByyG7/fmDVKuDgQe0CHSsrSTU49K84cQHdvhZ3RLx53V97rTV/ZfguX7R+o15mT8N2 gdFvoc/uNUO/8bxzZX+v+oNSf/zqJ/vnl/5a3f3p4T71XH981qM0/bzya23Oi/f8F2iPXNSf 9n37gFWrEMrRAvXaIrJAymPUu9/pAHIblVZaWupoSK28vBzV1dUoLi5Gc3MzZs6cidraWsvt J02ahBUrVmDgwIG2x0mLShM8uUxoVJqJtmMlAY5qcWDPQOt3QIZ+HxFuZuRFVBFFpVmSt/YX hBYVJSnqylS/7Kg04zGclK9aVLJDjpXkNiptQHEPJBKJVIJH9qcS5eXlWL58OQ4dOoRly5Zh 7Nix3O0vueQSVFVVoa2tDXv27HF2EAk3n7BKZaEt0GPkDuwZaP0OMNUvsB6pPsZP+vmQfj45 1S8gQSLZn0+u5xiFQqFUgkc2ppbrpUH0w1vGoS6zYa+Kigps3rwZ5eXl2LJlCxYsWMAtf968 eejVqxcuvvhizJgxA3fccYfYE3CI6mO0pJ+PpX5BjSOyPx+ufkEeO5nktf0F0BiLSfMWARb6 BTSIGFLsr5tnRfXHH6xhFHrztZXaUBoleEyhDaURhF8kDMcSHqDrQAQdN8O9gofog0RBMpnT BI9KT75WFdVb3KSfD+nnQx47Pnlrf0H8//buPTaKqu8D+HdmVi5CRajcUd6HVQRKSUm1SQsl RXwIBku9IMRA1SgYow80WCiklJYCFQMCJhjAiNCCEsUnWkh4tK9yEWxV0ootSCNQHl8uhoot CsildGffP9YzzLads9t2r93vJzG4Ozszvz3nzOzpmXPp0PG3tmWqhc8z/dvHWEQ2qFGEmGZr pflrrRk/riXEtZZCLP5eYR6/j/9l/Iyf8TN+Wfy+XiutNUR/a+U/uz82HqX1G/QAAD5K8+ej NFG4/IWjWuQYvxzjl2P8coxfjvHLBXtU2oA+UQDAilFL2MeIiIgo8ILZx0hUjIwlQcJt5utw JZoN/YXPmOUYvxzjl2P8coxfjvHLBbuPkTEq7T+7P3aqqgpFUdBngGtYPFuM2GJEREQUaCEz Ki3cJncMZ+Fe42b8coxfjvHLMX45xi/H+H0jbGe+9geOSmP8jJ/xM37Gz/iDF38wR6WJCa+V Pbs+cor+Rf3vHQqAj9I4Ks0a45dj/HKMX47xyzF+uY4QfzBHpfW7pxsA06g0gBUjgX2MiIiI Ai+YfYxExUjVNA2qqkJVOddjIIhmQ3/hM2Y5xi/H+OUYvxzjl2P8csHuY6QoClRVhfLl5586 RR8jjkpzYYsRERFR4AWzxWhgX1fdRxVD9TmPUWCEe42b8csxfjnGL8f45Ri/HONvH03ToGka lK+++MxoMerdfwgAthixxYiIiCjwgtlidN+AngA4XN9NIIbri//8dfyGQYP8enzGz/gZP+Nn /IzfX/EHc7i+eHKmlOz5t9PpdELXdY5K+xtbjIiIiAIvmC1Ggwf2AvB3i5Gu69B1PaCBRCpR O/YXPmOWY/xyjF+O8csxfjnGLxfsPkaCUrLn305RMRpw34MA2GLEFiMiIqLAC4kWI9ELW9O0 gAYSqcK9xs345Ri/HOOXY/xyjF+O8beP0+l0jdLf97+7jFFp0X3/BwBbjNhiREREFHjBHpWm qqqrjxEAzmMEjkpj/Iyf8TN+xs/4gxl/MEelCcrekmKnqByxxciFLUZERESBF8wWo3v73+1a FgRwPVfjqLTAELVjf+EzZjnGL8f45Ri/HOOXY/xyodDHSNd1qOY3gqm2thYzZ87EqFGjkJ6e josXL/r086Hi3p9+8uvxO50759fjM345xi/H+OUYvxzjl+to8R/Mz8e1AP+267oO1ZjpMch9 jFatWoURI0agtLQUw4YNw6pVq3z6+VAR7jVuxi/H+OUYvxzjl2P8ch0t/v87cADF6emoLCyE o6HBr+c2C5lRaYmJiSguLkbfvn1RW1uLJ598EmVlZT77fGuwjxEREVHgmfsYbR8/3ni/e//+ ePhf/8KgpCSfn1P0MRrUrwd0XYctVFqMLl26hOjoaMyaNQubNm1CfX29Tz8vU1NTg+LiYuDm TeD8eVTV12PpvfdC/esv6N26+fTfy4MeHGmAAAAOUElEQVQG4a5z53x+XPFvY+/esF286Lfj M37Gz/gZP+Nn/P6KX581Czh/HrjnHvQx/U7rjY0BazWyBbtCJPTs2RN1dXXYvHkzamtr0atX L59+XsZutyMzM9N4vWbNGmSYXvvShQsX0K9fP78cGwBQXg489JDfDs/4PWD8UozfA8Yvxfg9 6GDxbx8/HrauXTH6pZcwNC0Nqs3mv3ObqA5nI3Q44HA2BuSEVhITE7F161ZcvXoVW7ZsQZKH 5rLWfj5UREdH+/cE//2vXw/P+D1g/FKM3wPGL8X4Pehg8Q9OSUFaURGGPf10QCpFjY2NcDgc t0elBbvlaOHChTh69CgSExNx7NgxLFiwwNhmt9tb9flQdscdd/j3BP/8p18Pz/g9YPxSjN8D xi/F+D3oYPGPy8vDnb17+/ecTbiWBPmq2Bin3zN6MABO8FhTU9NiZYyIOia73Y6amppgh0FE QSA6X/e7pxsAhE6LUSjpyJWiyspKTJs2DSNHjsT48eOxa9cunx7fan6p4uJiPPLII4iJicFT Tz2F8vJyn543XNjt9mb/+ZKn+b02btzYocu3J8FK/1u3bmHt2rVISUnB/fffH7F50Jb7T2vS yir99+3bh4kTJ2L48OGYOHEi9u/f3+bvEM7sdjvef/99AMDmzZt9Xg6trqtwmXdQ1IOMtdKC PcEjBcacOXMwffp0lJeXY/v27SgtLfXp8a3ml/rqq6+wceNG/PDDD3juuecwZ84cn543XNTU 1BgtE+b/9xXZ/F7Hjx/H1q1bfXq+cBOs9N+wYQNKSkqwYcMGnDhxImJbp4J1/1m4cCFycnJQ WVmJxYsXIysry6fnDSc7d+6E0+nEzp07fX5sq2sqXOYdFPUgVcxhxIpRZLDZbDh//jyqq6vR p08fo4A2reGL13a7HUVFRYiPj0dCQoJrWgOJsrIyvPTSS4iKisKsWbOMG98777yDBx98EJ07 d8ZDDz2E+vp6NDYGt8N/KLFK/19++QUzZsxAbGwsJk2ahCNHjkiPY5X+DQ0NyMzMxOLFi/3z BTqAlsq5Vb5YsUr/4uJi5OTkYMSIEVBVVXqMjkx2/7Hb7Rg6dCimTJmCiooK433zdk+s0r9v 375QVdW1Dpaq+ndkVYi7++67sWHDBreR3FVVVUhNTUVMTAxSU1NRVVUFoPX3fytW+RJqRF2I FaMIU1hYiEuXLiEvLw9JSUn4/PPPPe6jaRpKS0vx1ltveazpm+eXio6Obja/1KVLlzBnzhw8 //zzsAVo6GU4y8rKwvTp01FRUYHs7GxkZ2dLP2+V/mvWrMEDDzyAtLS0QIQdllpTzq1Ypf+F Cxdw8OBBjBw5EikpKdi7d68vQw8bVvcf0dJQXV2NJUuWYO7cucb75u2eWKV/QUEB5s6di2HD hiEjIwNvvPGGn75h6Js5cybefvttzJw503hv8eLFSE9PR0VFBWbMmOH2B5Q/r4tQo+s6nE4n bKwQRZb77rsPeXl5AICff/4ZL7zwAh577DG3zzgcDrfXzz77LDRNw7hx41BbWys9vmx+qZ9+ +gmvvfYaJk+e7DZvFLkzp391dTXmzZuHefPmAYDH1gar9N+yZQt0XceePXsAsLNxSzyV86bX RUus0r9Hjx4YM2YMsrKycPjwYWRmZmLChAk+/w6hzur+8/HHH2PTpk349ddf0djY2OY+r1bp P3/+fKxbtw6JiYkoLS3F/PnzUVJS4rPvFU5SU1ORmpoKAMjIyAAAnD59GlOmTEGXLl2QlpaG 5cuXG59vzf3fii/nHfQn41Gaw+GAw+GArutBDokCITMzE6dOncLNmzdx4sQJ42bfp08fHDp0 CDdv3sRHH33kto+maV4f32p+qU8++QQ5OTlYu3YtFixYENGPE1pilf4xMTF49913cfz4cdTU 1ODkyZPS41il/8mTJ5v1ryF3LZVz2XXREqv0T0hIcGuVj9TBLlb3n5UrVyI/Px+VlZXYvHmz W1p16dIF57xcHNUq/UULhaIo0DQNdXV1Pv5m4W3IkCHYvXs3bty4gV27dmHIkCHGttbc/62E y7yDxjxGkXqBRqqxY8di9uzZiIuLw8aNG42m0aysLLz++utISkry6i9jK1bzSy1atAhVVVV4 5plnjP4C165d88l36gis0n/VqlXYsWMHEhISvOpnEa7ze4Wq1l4XsvL/3nvvYfTo0cjNzY3Y RzlW95/Zs2cjIyMDiYmJzSr/olXJmz5GVum/cuVKLF++HLGxsVi2bBnefPNN33+5MFZQUIDt 27cjPj4eH374IQoKCtp0HPM9yvz/4XJfEt2KlJLPP3GKylHvvq4vEenzGBEREVFkEPMY9eim QlVV11ppbDUiIiKiSCZajGyirwcrR0RERBSpRD3IaDFixYiIiIgiFStGRERERH8zlgQRL1gx IiIiokhlVIzE7MPtGaJNREREFM5sNhtUVb29iCxbjIiIiCiSKYriWhKElSIiIiKKZKJbkc38 Rnt4MytpsJch4PpQ3vEmL4HA5qev8o5lwHvtvabbm9affvopVqxYgT///LPZcWTbKDB8dc9n XgZfOPx+C+ZYfR2ToiiuCR59+ShNtl5uaC4ZR1Y8LX4comsAks8F76pet24dPvjgA4wYMaJV 2yiAfFA8mJehob1Z2dY/hFq7n/ist3/At4YxXN+8WB8RUai4cOGC5Y+lbBuFF+YlhQpjVJrT 6YSu69B1PagBNa39mV/v3bsXkydPxvDhw5stpPn777/j5ZdfxqhRozBp0iQcOXLE2Hb27FlM nToVsbGxWL16tf+/BAEA/vrrL+Tm5iIpKalZfonXQ4cOxZQpU1BRUeHVfgCwbds2xMfHIzEx EV9++aXxPstA6Fq9ejVGjhyJqVOn4uzZs8b7sutd5L2u65blp6VtsnIg9l2/fj1iY2PxxBNP GO/L7i/kP7K8FNtbyi9ZPnsqA+R7VgvHCmfOnMG0adMQGxuLadOm4cyZM17tJ/ut8DcVuL0+ SKjKyspCfn4+jh07hpqaGrdmt2XLluHxxx9HeXk5srOzkZ2dbWzLz89HSkoKvv/+e3Tq1CkY oUekFStW4MqVK/jss8+a5Zd4XV1djSVLlmDu3Lle7QcAly9fxqFDh5CTk4OVK1ca77MMhK6u Xbvi8OHDSElJQX5+vlf7mPPeqvy0tE1WDoTr16/j22+/RWxsrPGe7P5C/iPLS6Gl/JLlszdl gHzLUz4uXboU48aNw3fffYcxY8Zg6dKlXu0n+63wJ0VRoBzYt9spKkY9owcDAO66665WH8xu t3t8Rtmajprm1+np6airq0NSUhLi4uIwYcIEdO3aFQAQHx+PP/74w9hPVVWcPHkSABAXF4dv vvkG3bt3x5UrVxAXF8ebnhfsdrtXfYys0vLhhx9GSUkJerXQEamsrAwFBQWoqanBrVu3oCgK Tp065XE/u92Oo0eP4s4774TD4cDw4cNx4sQJACwD/uL6C05+VXu6pisrK9G9e3dcvXoVycnJ xl/wsutd9p5sm6wciH2qqqrQrVs3t/1k9xeyZrfbPXZM8eZas8pnq/yS5bOnMkAta+/vtzhG S59peg9OTk7Gjz/+6HE/2W+FbL+2+PrrrwEAA/veBV3XXUuChAJN0+BwOKBpGq5du+a2rbCw EGVlZaiqqsKOHTuwbds27Ny5E4Crdid+MCl0WLVAZmVlITc3F+PHj0dDQwNGjRrl1X4AjDwW ZUVgGQgP5nuN7Hpvz/E9lYOmP7KA/P5CwdVSfsnymfeCjsPTb4WqqvD1dEOiS1HILAkycOBA fPHFF7h+/Tq2bt3qtk3TNCQnJ+OVV15BRkYGTp8+bWxLTk7G6tWrcfny5WbHjI+PR1FREa5d u4bCwkJ/fwX626OPPooVK1bgt99+a7btxo0b6NWrFxoaGrB+/Xqv95NhGQhdIu2LiooQHx9v vC+73ttKVg5kZPcXCj2yfG5rGaD2i4qKarF1bvTo0W734NGjR3u1n+y3AgD69+/v1vLkC+Lp Wch0vl60aBGWL1+OsWPHokePHm7bRCesmJgY5Ofnu3Wizc3NRX19PVJSUpp14MrNzcW+ffuQ kJAQ9IpfJMnJyUFUVBTS0tKa5UlOTg5effVVJCcnY8CAAV7vJ8MyELoaGxuRkJCA/fv3Izc3 13hfdr23lawcyMjuLxR6PF3vbSkD1H6zZ8/G008/3SzN8/LycODAASQkJODgwYPIy8vzaj/Z bwUAzJ8/Hy+++KJP81j8Rij79+5y6roOp9OJ6N7/AND2PkaesG9HeAjFCR4p8HhNkwzLR8cR 6Xkp+hgN6BPl6mMkmo7aOyqtIydapGFeEsByQHIsHx0H89JFPD2zif5FoTxcn4iIiMifGhsb 4XA4QqfzNREREVGwOBwO94qRzWbztA8RERFRh6VpmquPEVuLiIiIKJJ16tQJTqeTi8gSERER 2Ww21zxGwQ6EiIiIKFQYj9LE47S6ujquSExEREQRR1GU28P1FUXBxYsXce7cuWDHRURERBRQ ooHIJl4oioI7tOv4x+BoY04jc/8jMdeRmCXb3NJkdN7WFbd9zdsVRTGWHWl6XFVVjfdbWqJE HN/pdEJVVWiaZsTT2NhozD3QuXNnAK7F5VRVNc4ptotzmY9hPq95YVIr5u8ujiPOJY4n4jen V9O4xOfEdxL7OxwOt/fN5zTnlTi2oijQNA2apgHq7fQTxDmbntecxuI7iO/P/Gf+M/+Z/8x/ 5n+k5b+IzWasDWIKvCnzAZpmlNvncDvTzJNGms/RdD/ztqaJZh4xJxLXXIgUxTXNgPhXFJSm Ca9pmvHanGhNP2v+/i19P/NxzLE0jdV8DHNamQtz03OY9xVx6LpuFBpzGpjT2Py+Q3c0u4ha 2s+M+c/8F/8y/5n/LWH+uzD/Iyf//x9a0Fa03cVi8AAAAABJRU5ErkJggg== --------------070809000706090606020605 Content-Type: image/png; name="screenshot2.png" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="screenshot2.png" iVBORw0KGgoAAAANSUhEUgAAAkMAAADGCAYAAADR24TZAAAABmJLR0QA/wD/AP+gvaeTAAAg AElEQVR4nO2df3QU1d3/37OJFVCRbIQIBKTElpCEoCXQJhL5IVZEBfLgzyPI0WKP1uIPKIqI 5alKi0FSf6M+SqAaf2Gfxp+IR6nEBj26yMOGoFXCo4A/1pBs5OELISS73z+SO5mdzMzObHZ2 Z3ffr3P2bHbmM3fuvO/MzZ333HtHevuNjcFgMAgAkCQJkiQhEAjA5XIhLS0NLpcLkiQBAILB IESsQL0sGAzC5XKFLBfbBwIBBINB+bdyn8pPIBAIiRHbuFwuuFyuHnnt6OhAR0dHyP46OjoQ CATk7UWaaWlpct6U+RZpS5IEuELzLfalzLNIV0sDKejqkV9lnDIt9fZq/ZTHJPLX0dHRI0/K fKanp4foInRQ511rHyx/lj/Ln+XP8mf5p1r5p0OF2LFegYn1ysJTfqvROonUaRttoxZMvU91 QYjCUKclCtXlcsnCaBWuHspCFNsqv5Vxyr+VJ6I6z+rjEcvESaO+eJT5NtKwvb0dgUBA/qj3 Z+Y4Wf6hsPxZ/ix/lr9Ik+WffOWfrrdz5Q7UYmitM8qk+iTSE055MMoCVAuhl4YofGVe1GkI QbTyKZbpFZX6RNBCkiRI0G5JK/XROjGUJ72Wpkpd9E5aoPNiEPsWehidgCz/0Dyw/Fn+evlj +bP8Wf4912nFqtNzevmnZw09U0ciQgghhJDkpKmpCe2tfgCAK0wsIYQQQkhS0djYiH379smO kdxnqH///vHMFyGEEEJITNixYweA7sd3dIYIIYQQkrJIksTGECGEEEJSGzaGCCGEEJLS9Jhn iBBCCCEkUdmyZYvm8qlTp/ZYJobX0xkihCQ0OTk5cd+/+BBC4s/UqVMxdepUTJw4Uf6th5iP ic4QIYT0goaGBgDxb5QRQrppbW3F559/jvT0dPz4449ITzdu7tAZIsQBHDhwAEuWLEFxcTFG jRqFCy64AK+99hqAUOchPz8f559/Ph599FG0tbXJ67XgP2dCSCqyb98+fPLJJzj55JNRUFCA nTt3Ijs723AbOkOExJkDBw7gqquuwnXXXYfbbrsNmZmZ2LVrFx555BHMnDkTQLf70Nrain// +9/485//jKamJqxYsSKeWSeEEMfR0tKCoqIi9O3bFwBw7rnnht2GzhAhcebBBx/EvHnzcO21 12LIkCE48cQTMW7cOKxfv75HbJ8+fTB27Fg8+OCDeOutt6Kyfy0HSbls9+7duPbaazFu3Djk 5uZi1qxZePvtt0PiX3vtNZx33nkYPXo0rr76ahw4cEBed/z4cdx7772YMGECxo8fj/Xr14ek 39HRgcceewyTJ0/G2WefjT/84Q84cuSIpWN4++23MW3aNOTl5WHWrFnYvXu3vO7IkSNYvnw5 JkyYgAkTJuDuu+/G0aNHTR+/+P1f//VfmDhxIgoKCvDMM8+YztvMmTPxwQcfyL+PHTuGcePG oampycohEkJMcvDgQXz44YfYsmVLyEcPTrpIiAP417/+hQsvvNDSNuq3YdvJjTfeiLKyMnzw wQfYuXMn7r77bvzjH/8Iidm8eTOefvppfPrpp5g4cSLuvvtued3atWvx1Vdf4fXXX8emTZvw 0UcfhWxbWVmJjz76CM899xzef/99pKeno6KiwlIe33jjDWzYsAEejwfTpk0LccweeughNDY2 4s0338Qbb7yB7777Dg8//LBlHWpra/Hcc8+hpqYGNTU1pre78sor8dJLL8m/t2zZgrPOOguZ mZmW80AICY/VDtQAnSFC4k5zczMGDRpkKra1tRU7d+7ErbfearkBFSmSJGHPnj3YtWsXfvzx R/ziF7/Ak08+GRLz2GOP4ac//Sn69u2L+fPn45NPPpHX/eMf/8Dy5cuRlZWF0047DUuXLg3Z 9qWXXsLKlSuRnZ2NU089FUuWLMHmzZst5XH58uUYOnQo+vXrhwULFuCzzz6T17399tu48847 MXDgQAwaNAjLli3Dpk2bLOtw7733YsSIEXC73diwYYPp7WbNmoVt27bJTtCrr76K2bNnW94/ IcQ8ra2t2L17d9gO1KIxxD5DhMQZt9uNH374AcOGDdONEY9t+vTpg8GDB+Piiy/GjTfeCABw ubTvafSWW+WZZ57B008/jXvvvRd79+5Fv3798J//+Z+46KKLAAD19fW4//775caSGp/Ph6FD h8q/hwwZErL+22+/xZQpU3qV99NPP13+u2/fviGPwX744YeQfQ4dOhQ//PCDpfQBhO2AqcdJ J52ECy64AP/93/+Nyy+/HJ988gn++te/RpQWISQ8+/btw9dff43Bgwdj+PDh7EBNSCJwzjnn YNOmTfjtb3+rGyM6UGtx8skno729PeTOp62tDSeffLKp/ffp0wdHjx6VOxt+9913IetzcnLw l7/8BUBn/5633noLK1askBtDv//973HjjTfi0UcfRf/+/WX3SJCVlYX9+/fLDbpvvvkmJP3s 7Gy8/PLLOPXUU03l1yoDBw7Et99+ixEjRgDobHwNHDhQXh/u+AWSJBnup0+fPjh27BhOPPHE HuuuvPJKLF68GKeccgqmTJki74sQEn3YgZqQBOTWW2/Fhg0bUFlZie+++w5tbW3YuXMnrrvu OlPbFxcXY/Xq1fjmm2/Q3t6OH374AQ888ACKi4tNbT969GisW7cOR44cwbfffouVK1eGrJ8/ fz62bt2KI0eO4NixYzhy5AgGDBggr29tbUVmZib69u2LAwcO9BjhVlZWhvvuuw/ff/89mpqa sHr16pD1V155JZYuXYp9+/ahra0NO3bsMGwYWmX69OlYtWoVGhsb0djYiFWrVmH69Ommj98s o0ePxt///nd0dHT0WDd27Fj06dMHf/3rX2P2eJOQVMVKB+pgMMgO1IQ4gWHDhuHFF1/Erl27 MHv2bBQUFOD222833a/k/vvvRyAQwJVXXokxY8Zgzpw5CAaDuP/++01tf8899+Cdd97BuHHj cM011+CSSy4JWX/dddfhySefxLhx41BaWop33nkHjz/+uLy+vLwcq1evxpgxYzB37lycf/75 IdvfeOONGDlyJC655BJMnz4dxcXFSEtLk9fPnz8fEyZMwIIFC3DWWWdhxYoVmDNnjqm8m+HW W2+F2+3GjBkzMGPGDGRmZuKWW24xffxmWbFiBTZs2IBRo0ZpjlC74oorcPjwYZxzzjkRHwsh JDwulwuTJ0+WO1JPnjw57KN3aceOHUEA6N+/f0wySQhJbXw+Hy6//HJs3bo13lmJKa+//jre eOONHp3PCSHRZfv27Rg0aJDcV/Dbb79FY2NjyON7Uf+cftpJAPiYjBASAxYtWoS9e/fi0KFD eOihh3q4R8lOW1sbXnjhBVxwwQXxzgohSU9ubi4OHjyI2tpa1NbW4uDBg8jNzdWMTU9PR1pa GjtQE0Lsp7S0FHPnzkVbWxumTp2K2267Ld5ZiimjR4/GL3/5y4gfwRFCzHPSSSfh7LPPDlm2 ZcsWzfmGXC4XgsEgG0OEEPspKytDWVlZvLMRN4xGAxJC4gcbQ4QQQghJOoxevaFGjCZjY4gQ QgghSYPW47BwQ+vZGCKEEEJIShIIBDjPECGEEEKSn3Ava6Uz1MW3H38MABgyYUKcc0IIIYSQ SPH7/WhoaMCRI0cAAP369cOZZ54ZMnO+QJIkSJLExhAAtLe24qM1ayClpWHm+vVI+8lP4p0l QgghhETA7t27MWrUKLjdbgBAU1MT6uvrNWd/d7lckCSJj8kAoP7FF/H/fvgBh7/7DvUvvBDv 7BBCCCEkBkiSBJfLRWfo0P792PX88/LvXS+8gJG//jVOHjw4jrkihBBCSCTk5eVhz549qK+v B9D5mCwvL08zNhgMAkiw13Hk5OTIHyU+nw9z585FYWEh5s2bh8bGRsPlSravXYvA8ePy745j x/DJo4/aeyCEEEIIsYWMjAyMHz8ekyZNwqRJkzB+/HhkZGRoxkqSBCDBGkMNDQ2aM7mWl5cj Ly8PtbW1yM3NRXl5ueFyJVP+/GfM++c/8cPFF2PeP/+Jef/8J6asXCmvb2pqspRHu+Olmhpb 00/0eOpjDPUxhvoYQ32MoT7GWNUnUvx+PzweD2pqalBTUwOPx4OWlhbDbZLiMdm2bdtQXV2N U045BQsWLJCn/ddbboXjCtfICfHBfv1sTT/R46mPMdTHGOpjDPUxhvoYY1WfSLHSgVqSpOSZ Z8jv9yMzMxMLFixAZmYmmpubDZeH0NICbNwIHDqEQfv2YeTIkSHfmZmZmsv1vm2P79vXWflx Wjz1oT7Uh/pQH2fG6+gzvH9/YONGSHF6h58kSZB27NgRBID+/fvHJRORkJOTE/K4rLi4GNXV 1cjKyoLP50NZWRm2bdumu1yLNWvW4LHHHovVIRBCCCGki71790YtLb/fjz179vSYZ0jZb2jr 1q0AgJHDByIQCCSHM1RcXIzKykocPnwY69atQ0lJieFyK1gtILvjD7//vq3pJ3o89TGG+hhD fYyhPsZQH2Os6hMpVjpQAwnoDKlHkQGdnap9Ph8WLVoEr9eLwsJCVFRUyG6Q1nIt6AwRQggh 8SGazpDeS1mVr+RQOkNAgo4mU34AICsrC1VVVairq0NVVZXc4NFbboVEb1k7Lf/UJ77x1McY 6mMM9TGG+hgTK2cI6Gz4qD9aBNCBjmB7YjlDdkJniBBCCIkP0XaGjF7MCnQ7QyOGu5NnNFlv OH78OL7//nv8+OOPALoLRHyLDtfq5XrfdsfvfvVVR+XHafHUh/pQH+pDfeIX75cky/r8+9// xvfff2953qJokXB9huyEzhAhhBDSO/yShIyuV1xYQTSMYoVwhs4YlsEXtZrBagHZHc9n0sZQ H2OojzHUxxjqYwz1MSaWfYbMEgwGEQwG6QwJ6AyZx9/1LhcAEd0BEEIISU4SzRkaNvRUOkNm SPSWtdPyT33iG099jKE+xlAfY6iPMU50hgDQGVJCZ8g8dIYIIYRokWjOUPaQzrZPyjtDHE0W WXxp/k640Ux9Eiye+lAf6kN9nKZPNEeTeTwe+Hw+BE02yDo6OtDR0UFnSEBnyDyS5Jf/Dgb1 pzgnhBCSWsTbGTp06BAOHDiAlpYWDB48GEOHDsVPfvKTHnHCGRpy+skA6AyFxWoB2R3PZ9LG UB9jqI8x1McY6mNMquuj7EKhRSz6DPXv3x95eXkoKioC0OkU7d69G4cOHdKMlySJ8wwpoTNk HjpDhBBC1IjGkJP6DAUCATQ2NmL//v1yAwlgnyHLOK0lnup3HuGgPsZQH2OojzHUxxjqY0w8 RpO5XC5kZWWFNITU610uF50hAZ0h89AZIoQQosaJzpAewhkanj2A8wwBHE0WaXx+/n7qk4Dx 1If6UB/qY1f8/vz8iPThu8kcBJ0h89AZIoQQoiZRnSGXy0VnKBxWC8jueEc8k/a45Q/1Sax4 6mMM9TGG+hhDfYxx4gzU6enpSE9PTw5naMuWLVi1ahX279+PYcOG4c4778SUKVPg8/mwePFi eL1ejB07FhUVFRg4cKBmGnSGzCNt7x4+GRzHGagJIYQkpjP00zMyk8cZuuOOO7B8+XLs3LkT d911F26//XYAQHl5OfLy8lBbW4vc3FyUl5dbTjvRW9ZOyz/1iW889TGG+hhDfYyhPtqIRlI8 naEtW7ZoLk+qt9ZffPHFWLp0KX75y1/io48+Qnl5OV5//XUUFxejuroaWVlZ8Pl8KCsrkzt8 qaEzZB46Q4QQQtToOUNmZqW22xnasmULpk6dKv8WztCI4e7kcYZWrlyJm2++Gbm5ubjlllvw 5z//GQDg9/uRmZmJBQsWIDMzE83NzT03bmkBNm4EDh2SW63K771792ou1/u2O1584pmf2d5s +Zv6JFY89aE+1If62Bnflp1tWZ/2gweBjRshNTQgGmzZsqXHx4ikcYbOP/98LF++HMXFxait rcWqVauwefNmOkM2QWeIEEKImkR1hoAkmYFaOD6SJCEtLU2eq6C4uBiVlZU4fPgw1q1bh5KS EstpO+WZq0C0pu1KP9HjqY8x1McY6mMM9TGG+hhjVZ9YIElS8jhD77zzDsrLy3HgwAFkZ2dj 6dKlmDZtGnw+HxYtWgSv14vCwkJUVFQgKytLMw06Q+ahM0QIIUSNk50hNcrRZIFAIDmcoV// +td499138fnnn+Pdd9/FtGnTAABZWVmoqqpCXV0dqqqqdBtCRiR6y9pp+ac+8Y2nPsZQH2Oo jzHUx5hYOUN+vx8ejwc1NTWoqamBx+NBS0uLZqx4opQUzlA0oDNkHjpDhBBC1DjFGaqtrcWo UaPgdnf2B2pqasIXX3yBc845R44RztDIEafx3WQA300WaXx+Uz71ScB46kN9qA/1sSte791k YrlT300GgM6QgM6QeegMEUIIUeMUZ8jv92PPnj04cuQIAKBfv34488wzkZHR/S5NZZ8hOkMm sFpAdsfzmbQx1McY6mMM9TGG+hhDfYyJVZ+hjIwMjB8/HpMmTcKkSZMwfvz4kIaQmqQZTRYN 6AyZh84QIYQQNeGcISOHKJrOkN4ki1rzDP30jEwEg0E6Q+FI9Ja10/JPfeIbT32MoT7GUB9j qE9PRAMJ6J6dOhaIhs/UqVNDGkFaSJJEZ0hAZ8g8dIYIIcS59Obt8dHer3JZrJyhmpoanH32 2fB4PCgqKoLL5cKOHTswceJEOUY5mozOEDiaLNJ4jiZLzHjqQ32oT3Lr45ckefSW00aT/e+k SZrroz2abPDgwdixYwd+/vOfo66uDtu3b0dOTo7hNnSGuqAzZB46Q4QQ4kyUj6VS1RkyA/sM WcRqAdkdz2fSxlAfY6iPMdTHGOpjDPUxJlZ9hrQ6UOt1qg4GgxxNpoTOkHnoDBFCiDOhM2Rt NNkZwzqH3NMZCoPTWta88zCG+hhDfYyhPsZQH2OcoI8bzfLHafrEejSZ8qMHnSEVdIbMQ2eI EEKciST55b+DQf2JBu3AKc6QGYQzNDx7AGegBjiaLNJ4jiZLzHjqQ32oT/Lrk5+/Py76GI0m 80tSzEaTWUHqaqzRGeqCzpB56AwRQogzcaozJHCaM3TGsAw6Q2awWkB2xzvhmbST46mPMdTH GOpjDPUxhvoYE8s+Q2YJBAIIBALJ4QwdP34cjzzyCF577TUcOHAAwWAQDQ0N8Pl8WLx4Mbxe L8aOHYuKigoMHDhQMw06Q+ahM0QIIc6EzpA5lH2GAIPRZDk5OT0++fn5uPTSS/HVV1/FJLNm efzxx7F582Y8/vjj+OKLL9DQ0AAAKC8vR15eHmpra5Gbm4vy8nLLaTutZc07D2OojzHUxxjq Ywz1McYx+njcgMftOH2c6Ay5XC7r7yYLBAJ45pln8Mknn+Cpp56yPZNmmTJlCu655x6UlpaG LC8uLkZ1dTWysrLg8/lQVlYmd/hSQ2fIPHSGCCHEmcSzfk5EZyiiGahdLhcuu+wyfPLJJ7Zk LlK+//571NTUoKCgAJMnT8Z7770HAPD7/cjMzMSCBQuQmZmJ5ubmnhu3tAAbNwKHDsmteuX3 3r17NZfrfdsdLz7xzM9sb7b8TX3iE++XJOrD84f6OCzeCfrEon42qn+E+6P+3ZadjbbsbM30 2g8eBDZuhNT1VKe3+P1+eDwe1NTUoKamBh6PBy0tLZqxkiRF9tb6lpYWTJkyBTt27IhKpqPB r371K6xatQrnnHMOPv74YyxevBgfffQRnSGboDMUf4zm6yCEpC6xqJ/16h+nOEO1tbUYNWoU 3G43AKCpqQlffPEFzjnnHDlG+dZ6wOIM1IFAAK+88grGjx8frTxHhQkTJiCoEFjMG1BcXIzK ykocPnwY69atQ0lJieW0nfbMVbSm7Uo/0eOpjzHUxxjqYwz1MSaV9PFLUlL0GQJg7Axpve6+ T58+yMvLw+rVqzFixIiYZNIM3377Lf7whz/A6/UiKysLy5cvx5QpU+Dz+bBo0SJ4vV4UFhai oqICWVlZmmnQGTIPnaH44pckuNEc85EihBDnEytnCOg5q7RTnCG/3489e/bgyJEjAIB+/frh zDPPREZGd52pdIZcLpe+M9TQ0NDjU19fj40bNzqqIQQAQ4YMwfPPP49du3bhvffew5QpUwAA WVlZqKqqQl1dHaqqqnQbQkY4qSUOpNadRyTx1McY6mMM9TGG+hhDfYyJlTOUkZGB8ePHY9Kk SZg0aRLGjx8f0hBScsIJJ3SOKNNzhg4cOIAlS5Zg165dKCgowJo1azBkyBD7jyJO0BkyD52h 2KH1bJ7OECFED7vrZ1H/NMOt6wwBCFkm4pXL1UTTGbLy1vpRPxtsPJrsvvvuQ2FhIbZt24b8 /HysXLkyahl1Enw3WWTxfDdZbOL13vWTv7lUjlc+u/cWFKSUPjx/qI+T4p2iT7TqZ736RLz7 bMuMGSHLRX0l3kWmjo/lu8lEwyfcW+sFus5QUVERNm/ejMzMTDQ2NmLGjBmOG1IfTegMmYfO UGxQ32WJOzC/JMHtAZqLQuPVMRxtRkjqEc36WVmfAN11jNLpEctFvBKt+Fg5Q1OnTpW/lcsE whn6WU5n9xldZ0jM0QMAAwcO1J6jJwWwWkB2x/OZtDHJqo+oZNSVjbgTM9pGSbLqE6146mMM 9TEmmfTRqnPkvz2dDRtR//glSbO+Ucc7cTRZMBhEMBg0Hk3WoJgASf072aAzZB46Q7FB3FEB QDPc8t2VG82Ax43mIoSsV6J8pk8ISR2iUT+LRkxInQNo1j+i7ukRg+56Selkx2sGaj1nKOen ne8rNZxnSPleMq3fqYDTWu7JdOdhR3wy6SMqHeXf6mXiWbwbzSEfsUx9t5ZM+tgRT32MoT7G JKM+cp3j6W7kCPLz94fUOeoYdbwTnSFBUry1PhrQGTIPnaHYoNQ5LEXdlZFyGd0hQlKLaDlD yhsvLZqLOh0fFDX3qHfk38p6KUx9FO0+Q1pE5Ay98MILuPnmm3ssX7hwIV588cVeZdRJcDRZ fEcrJKs+0YoXOut9X/jVhZ2/N5cCHnfoeo+7x2i0ZNOH5w/1cVK8U/Tpbf1cmr+zZ32i+naj ubP+0ah35N+qeilWo8nECDLlR4+w7yYrKSnBSy+9hGHDhoUs//rrr3HVVVfpvuMrUaEzZB46 Q7HBkjOkQ3OR/jN6QkjyEY362VTdo3aETGBUH4mGkV0YjSYznGdIOZpMycCBA+H3+23IqjOx WkB2xyfjM+loxqeaPuKOyyyppg/Pn+jGUx9jUkofheNjlnj2GQrrDuk5Q7NmzcLSpUtRXFwc svzDDz/E/fffj+rqahuyGz/oDJmHzlBsoDNECLFKzJyhSCjSnzk/ms6Q2gXSWiacoZ+febqx M/S73/0Oy5Ytw9atW3Ho0CEcOnQIW7duxZ133okbb7wxapl2Oo5qiSPF7jwiiE81fcLdmalH c6SaPjx/ohtPfYxJNX2sOkOzs2PXvWbLli0hHz3CzjMEAO+++y6eeOIJfPbZZwCA3Nxc3HDD DTj//PNtyn78oDNkHjpDsSFad2csI0JSB0c7Q9DPU7ycIVPzDE2bNg2vvPIK6uvrUV9fj7// /e9J1xDiaLL4jFZIdn2iFW96NFmYuGTVh+cP9XFSvFP06W39HK4+sVr/iO/fbI/daDIzywAg EAggEAhwniEBnSHz0BmKDXSGCCFWoTNkDuEMjRjeOSLO0Blqb2/Hc889J/+uqKhAYWEhVq9e bWMWnYXVArI7PtWeSVMfYziaLLrx1McY6mNMquljuc+QNzajydT9hYz6DUlds/SHdYb+9re/ YcSIETj33HMxZswYPPzww7j55ptRV1dn02FEztq1a/HAAw/I71Dz+XxYvHgxvF4vxo4di4qK CgwcOFBzWzpD5qEzFBvoDBFCrEJnqJs9e/agpaUFY8eOxQknnBCyTukMGY4mE1xzzTXYtGkT GhsbMXfuXNx8882YO3du1DPdW3bv3o3KysqQZeXl5cjLy0NtbS1yc3NRXl5uOV2ntcRT7c6D +hhDZyi68dTHGOpjTKrp41RnqLW1Ff/zP/8Dv9+PM844A/X19Th27JhmbNg+Q2vWrMGGDRsw b948XH/99bjvvvtQXl4Olyts+ynmtLW1YdasWbjhhhuwaNEi2RkqLi5GdXU1srKy4PP5UFZW pjtzNp0h89AZig10hgghVqEzBHzwwQcYPnw4hg8fDkmS0Nrais8//xxnnXWWHCOcoeHZA4yd ofXr1+Ohhx7C3/72NwwYMACXXXYZnnzyyahlNpqsWbMGP/vZzzBr1qyQ5WIW7QULFiAzMxPN zc09N25pATZuBA4dklv1yu+9e/dqLtf7tjtefOKZH9G6n+3Npj42xit11vvOb8oPG5es+vD8 oT5Oi3eCPtGqn6NV/yi/Z3uzNffbfvAgsHEjpC4jo7eMGzcOZ5xxhtwfqE+fPsjPN3axdJ2h +++/H8899xzmzp2LO+64AwDw1ltvYfr06Y5zh372s58hEAiELGtoaKAzZBN0hmIDnSFCiFXo DFl7a/0ZwzIQCAT0naE77rgDdXV1ckMIAGbMmOG4hhAAfPnll2hoaJAfjykfk1VWVuLw4cNY t24dSkpKLKfttGe0ojVtV/qJHp9q+rDPUHTjqY8x1MeYVNPHqX2GgO6GT7i31guSbp6hnJyc kNFkixYtgtfrRWFhISoqKpCVlaW5HZ0h89AZig10hgghVqEz1D3btHLWab0ZqEcMd6OjoyP8 aLJEo0HxzDErKwtVVVWoq6tDVVWVbkPICKe1xFPtzoP6GENnKLrx1McY6mNMqunjZGfILJIk cQZqJXSGzENnKDbQGSKEWIXOkDmEM3TmyEE4fvx48jlDVkmWd5N5CwrglyR4Cwpikh++myw2 8Xw3Gc8fJ8VTn8TQJ9XfTWZ1BmqXy0VnSJDozpBf6m7JZwTtdQLoDNmPX5Lg9kQnLZZRJ35J sv3aICTe0Bnqpq2tDf/617969B8CQt9a397eTmcoHFYLyO74VHsmTX2MYdsKtmYAABq7SURB VJ+h6MZTH2OojzGppo+T+wy1trZi9+7dSE9Px48//oj09HTNuOPHj+P48eN0hgR0hsxDZ8h+ 6AxFHzpDJBWgMwTs27cPX3/9NQYPHgy3241du3YhOzsbI0eOlGOEMzQ46yQ6Q2ZwWks81e48 UlUfNzRmS9eAzlB046mPMdTHmFTTx6nOUEtLC4qKinDmmWfC7Xbj3HPPDWkIKWlvb0dHRwed IQGdIfPQGbKfaN6ZsYw6oTNEUgE6Q+YQztBp7p8Yz0CdKiTLaLL9Xe9dEd9OHq3gl6SY6xOv 8nLKaI5k04fnD/VxYrxT9En10WRWCHYALqTRGRLQGTJPb+88eIceHjpD0YfnHUkF6AyZQ3aG BpyItLQ0OkPhsFpAdsen2jNp6mMM+wxFN576GEN9jEk1fZzaZ8gKaWlpSEujMyST6M6QJPnl v4PBDHv3RWfIdugMRR+edyQVoDNkDnk02cCT6QyZwWkt8VS786A+xtAZim489TGG+hiTavok gzMkSVLnh85QJ3SGLOyLzpDt0BmKPjzvSCpAZ8gcwhkaMugUuFwuOkPJMposP39/yLfTRyvE Wp945ccpozmSTR+eP9THifFO0YejyawRDAbpDAkS3hmK4dw/dIa6sWsUH52h6JNM5x0hetAZ MoeyzxAAOkPhsFpAdsen2jNp6mMM+wyFjxcNVjOzeqeiPlagPsakmj7J0Geoo6MjeZyh6upq PPzww/D5fBg1ahSWLVuGoqIi+Hw+LF68GF6vF2PHjkVFRQUGDhyomQadodjtK5nu0OkMOR9x vkmS3/b+dITEm97Uz/K1kkLOUFZmP0iSlBzO0Lvvvou1a9fi008/xTXXXIOFCxcCAMrLy5GX l4fa2lrk5uaivLzcctpOa4mbvfMQ/6Sdln+n6BNp+k6LpzNkId7jDhuf0vqYgPoYk8j6+CVJ foOBWZLBGXK5XMk5muzAgQM477zzUF9fj9LSUlRXVyMrKws+nw9lZWVyBzE1yeYM2em+0Bnq xq5RfHSGoofybjfVtSDJT6T1s7iBTjVnaMigUwAkWZ8hv9+PhQsXYv78+UhPT4ff70dmZiYW LFiAzMxMNDdr9BloaQE2bgQOHZJb9crvvXv3ai7X+7Y7Xny01otW9+H334cbzbblR+xntjc7 ovTbsrMtxUdLH7vKa3b2NlvSV+qs953flB82Lt76xDtenG9aelCf6FxffklKiPybjRfHkwj1 j/o70vq5LTsb+/PzNdPR+zZT/yi/Z3uzNffffvAgsHEjpIYGxJqkm2eovr4eN910Ey666CIs XrwYLpcLxcXFKesM2dk/gs5QN4ngDDUX2f++OidDZ8h+kq0/ViLXUXSGzCGcoezTT0UwGEwO Z2jjxo1Yvnw5KioqsGTJErhcnYdVXFyMyspKHD58GOvWrUNJSYnltJ30TBfobk2bQfl2eLvy 47R4K/pEkn6PeI+7+2NH+mFgn6HoxlMfY7T0UQ4iiHV+nBaf6OdPKvYZApA8zlBOTk6PZXV1 dfi///s/LFq0CF6vF4WFhaioqEBWVpZmGknnDG2XbHMEeusMJdNdpF2j+OgMRQ86Q/bilyS4 PcnVN43OUGo5Q0CS9BlqaGjo8enXrx+ysrJQVVWFuro6VFVV6TaEjHBayz3R7zyoT3Tj6QxF N576GNNbfYxcpEjy47T4RD5/3GhOWWcIQHI4Q9EgWZ0hIPquQK+doSS6Q08EZwhFzUnjxEUC nSF7kSQ/4HGb0jZRHJdEyacWkdZJkuRHM9y2O0N6TnW8nKGhWZ1tn6RwhnpD0rybTOddMDsu vNCW/PDdZNq6Ryv9qL4byOOOmz6JFk99elf/hEtfOA9OP16Rz0Q9fyKqnz1u+f9FKr2bLBgM IhAI0BkS0BmKfF+RbJ8sd+gJ4QwhufpzWEV+HYeH/acixcgpoTPkLCJ2hhT9TFPJGRoy6JTk GU1mJ1YLyO54q8+krT4Ddtrx2q2P0/JvNZ59hqIbT320Ee9109RHYySl0Qz4iTD6TBxvqp0/ 7DMEOkN0hiLfVyTbJ4tTQWfI+dAZ6j1GI0DFuao8x/ScFeWIpd5gt3PTmxGv8XaV6AyZg86Q RZzWcrd651Gav9NSvNOON9XuzOgMRT9e+bb6cG+uT1R9YvUuwt7qo6e/496l2OV2pVr9k6rO UNK8tT4aJJMz1FzUfRdsdv4PK3czkdx5KNNPBWeot3eHdIb0saqtGCXj9nT+TkZ3yHanxOCa teIMiRnb1a6LVSfGarxVR6o3dVQiOkNirqhUdIYGDzyZzhCQ/KPJRG//cOkKB2nLjBmm4q2O Voh0dIbTR3PojSbr7fFGezRHvPRxRLzHLZeH3qgnJ+ujnEk+WteXt6DAVLyIE7qZHU0m6pMe 18vm0pDRjfLy/P2W8m81Xi8/0arfhE5mzx+z+kd6/kRaP6fqaDKAfYZk1M6QunUf79Z+OHrr DIk7LTPHSWeom2g5Q+r4ZHeGenM9WXaGuvpCJJIzZPV6sctZMTNHk5YzpJe+VqxYbrV/i9U5 dADz7w+0mr6yvEzVoRE4W3Y79+J/BtB9rdiB05yh0087CQDnGdIl0tEEZuPFxWk13swzaeVz eeUzXcPZX7uekdvVx8iqnlb7EFjRpzfp23W8kZ5vZp/Zi+O1q0+DVX3sTj/SPih26WM1P3bX P2bj1X18jPTxS1J3HRNhnxvb4rveHxhJ+uFmzQbCjLYLk74SvX3F6nyIRZ+hcP324gUbQ2EY OXKkqThRWZuNt5q+4OTJk8MHKYa61mfWAzBuCCnfvF6/Yaypi19gOv9deTIbLy4Yq/ocnzLF 1vTtOl7L6Xchyjcc4nhNnT+w/3zujf5mKlOj9LXuesU5b5c+8T7felzTingzN0kCtT5a2yqX 9fZ4w9VFGTk5luur3tZvmtt36WT2/NErL11Hxqb6RB5p2XV+Dqs3V58IzNY/gurCA5biY0Ew GGSfIUMivLMRz4J7m75e5WX1zkO03A3/gXSl7Zckyy39aNypGFUulu78ALRlmxytoJO+bkVp 951uhOmbLq8w54/ecduW/97Ea8xr09v0o3VnH638RPt807v+ja5HrXPi8PvvhyxXj9QTn3D5 0UPEh7hMBhg5GVrbm65vw8TrkWijyeTGV9f5tj8/35JzE9FoMjPXbxxgYygMomUd9gTpKmCr LWtbnCEF9Zn1nRe6iRPQjWZTTpKSaNypKLVV79fsnY3g9Oqedx5Gx2K3c+MUZ0jQ2zvXcFiN z8jJiUr6emVs9fp1mj6Rxiv1MLr+M3Jy4EZzjwaOnp4nT54cqqUy3a5HUcqbKz0nRjR21PlU Oj1iP1px4ntYfb1u2Wott+I8ifxYaRxYrZ+NnCqlY6+MN0Kr/rTihA2rr7fUWEkGZ0iSJEiS xMaQ1mgySfL36I2fv7lUXq/1LeKvvvA9wzit3v5ao0XU7/AR8Z92XWziWbZ6fbje/ob58bhx 4VcXhry9ONzoEuVoBb8koaDAG/Z4lel4CwpCnuWXbs6XHSpJ8mPGjC2a6Yn9lG7uPl6/JPUY reAtKIDbY360hd5x68Ur9dYqD7OjgXqcbzaN5tAb7SJ0jHQ0SqTxV79ibrRjuPT1rhdZz67R R3rXQTh9zObH7PkfSfpa55G3oCDE0d0yYwb8kiRfR96CArjR3OO4xfV19SsXhoy2E9djaf5O uD3d57X4/nTyZMDjRkGBN2S/Wt9uNMujU9X1Rmn+TrmeUaYjRjOV5u/szMfmfLg9CDkeZfoi /2o9lPWKWh9lfsT+xbdmfa4x+k2rvIQ+bjTL+VDny2x9opd/dX0ijkPEqUdNqtNXxmuVW6xH k4n8xHM0WXp6OtLS0jiaTKAcTaYc8SB6/Stn59RCjD4Q3+F6/ytniNYa8RUymkTx7h/lm4XV I056UNTc2cpXfIsRDGa21RwN0pUXgXKWa72RIkp9lMeu3MZwfgtVvt0IPS6hu/gt6yri0HME g7KcRP6VumiN7tIq/5D8KvKpLF9l+noo9REjO6KNyL/WuRlyPqJbV7NEOhrI7Hbq88foetGK V2M0msnsiMpw57mZ5XrHoL5G1Oeq+m5fOXpU0GNkUNf5qXd+ydeRmqLm3j/aKGoOmetJiZzv Ih3HyUQ+QuoPRZzuMSnSU+9f1PfKdJTXjhay/oo6Rxwz0LlcWW+LOl/UZ1r1RkiM4n+E3rmq nk9L5BtAjxhlvkM0ikZZm0CZf5E/q48Ae4sYTTZ8SAb7DGmhtmPzN5fKdqXb03kiCttWkvwh Fq5WnxvlOhEvcKNZHr2lfk7v9ihsUoXtPHtdIdye7n+Y6jRD8Lhlx0f8lo9FxwqX8+9xy3kW +9ay20s393zG7JckWSf1sYk7ZzUhx6vOj+L43R6EHI8yfXjcnaMVPAiN60pfWW5iO3GHKsd1 /a2VH+EwaeVT7F95PrjRHKKPJPnlSkeS/D3KTuhjpSFk5Zm9G80oG/am7uMHN5p76OYtKAg5 XhEv8q9GL175WER9vWjFq/9Wpq9EpKf3qLXgHe0+fFr5AQDfsGGy/np5UeZffbzq9WoKCrwh 5a7Mg9Y1qT5e5bWv/Ih1yvNBuQ6AfH4qlynjdRsNivPB6gzCyvpE93GWot+KmIcoXD7U6Qtd 1HFqPUOul664kH4zijpSrj/ReW1Ikr9T067rRP4o6ozZ6wpD0lemLeoUpf6lm/M16w3lOSj+ liS/XH/K54yn+39LyP5E+l0OnzpGvQ9Zn64nIGaJdAbqkGON8+gyl8sFl8uV3M6Qz+fD4sWL 4fV6MXbsWFRUVGDgwIGascIZisr8Lqo7DvUdW7jtwi6LUv5Mp9+bPEQr/zG4Y9ErJ/UcNRG5 Nqq7TDvn8bCaH6vxuvlXxQCKO27lvjT2rYxX662Vjl6ZKP82dAVMHmOPY9XRTXkHrz4utZMZ Dq18O+KcIVHF8vmZTKiuheC4bmeo5k9/QtHvfod+Ov+ro4Vwhs4Y6k5+Z6i8vBx5eXmora1F bm4uysvLDeN1nQkLKJ0M5R2bYbxAq6JULYv4zkwvTVX6YePNpG+wbUR6hsmDkt7oo1dOPe7k Iklf5VKZyY+l9E0i62NST0v5V95JK10J9b4Uv7Xilemr01HGa+VH+be03eLoyC5nUZlOj2M1 OJ+1dFE7mWbyo6yHtI7XiJidPzaln+jxVvRR9k21Kz9OizdT/3z9/vuonjcPO9evR0dbm6X0 I0EMrU9qZ6i4uBjV1dXIysqCz+dDWVmZ3KFMzZo1a/D4dY/HOIeEEEJIaqN0hp5VzBV38uDB GP/73yO7pCTq+xTOUPbppyIYDCI96ntwEH6/H5mZmViwYAGeeOIJNDd3P5tsaGhAdXU1cOwY 8M038DY34xffDcNX7v+HEc0nyd/HfzIAJ7S19Fiu9213fP/WE3Coz3HH5Mdp8dSH+lAf6kN9 nBmvp89N7gXAN98Ap52GQYr/4YH29pi5Q0ndGMrIyEBTUxOefvpp+Hw+uN3d1lxOTg4WL14s /16zZk3Ib8H333+P008/3fQ+7Y6HxwMUFYWPi1F+nBZPfcJAfYyhPsZQH2OojzEm9Hl2yhSk 9+2Ls3/zG/x81iy40u1tpqTEPEPFxcWorKzE4cOHsW7dOpREYLVlZmY6Kh7/+7+2pp/o8dQn DNTHGOpjDPUxhvoYY0KfMyZPxqwNG5A7Z47tDSGgczRZ0s8z5PP5sGjRIni9XhQWFqKiogJZ WVmasXrOkONoaQEGDIh3LpwL9TGG+hhDfYyhPsZQH2McpI/oMzQiu7NBl9SNISs0NDQgx+Kr AQghhBCSeCiH1gN8N5lMNBpCO3fuxOWXX46CggJMmTIFr3ZN7R8tcnJy5I+S6upqTJ06Ffn5 +fiP//gPeDzG43B9Ph/mzp2LwsJCzJs3D42NjYbLSWzIycnBM888AwB4+umno944j1a58/xx JnafP6x/khtl+WqVc29xav0jdU0AycZQFFm4cCGuuOIKeDwePPvss6itrY1q+g0NDWhoaOix /N1338XatWvx6aef4pprrsHChQsN09Gbf8nqvEwk+rz88ssIBoN4+eWXo552tMqd549zsfP8 Yf2T3CjLV6+se4PT6x82hqJIeno6vvnmG3z22WcYNGiQXBjqFrb4nZOTgw0bNmDcuHGYMGFC 51D/CHj00UcxatQonHjiiSgqKkJzczPa29s19w10vrzvN7/5DU455RQsWLBAbrTpLSexY8CA AXj88cdDRj56vV5ccsklyM/PxyWXXAKvt3NKfqvnTyTlzvMnsbDz/NGD9U9yo/f/66uvvsLV V1+NMWPGYPr06dixY4dhOk6tf+gM2cD69evh9/uxYsUKlJSUYNOmTWG3SUtLQ21tLR544IFe 3wn5/X4sXLgQ8+fPR3pXL3yt1r1y/qXMzEx5/iW95SR2zJ07Fw8++CDmzp0rL7vrrrswb948 bN++HVdffTXuuusueZ2V8yeScuf5k1jYef6Eg/VPanH77bfjiiuuwPbt27Fs2TIsW7bMMN7J 9U/SD62PNcOHD8eKFSvw2muvoaqqCvfcc0+PmI6OjpDfV111Ffr06YNzzz0XPp8v4n3X19ej rKwMJSUlWLp0qWGscv6lpqYm+S5SbzmJHZdccgm+/PJLXHzxxfKyvXv3YubMmejTpw9mzZoV 8nZnK+dPtMqd549zsfP8MYL1T2qg/P/12Wef4bbbbsPo0aNx7bXXYs+ePYbbOrX+oTNkA4sX L8aePXtw7NgxfPHFF/KJM2jQIHzwwQc4duwYXnzxxZBt0tLSer3fjRs3Yvny5aioqMCSJUvg chkXq978S9GYl4lEn5EjR+K1115Da2srXn31VYwcOVJeZ+X8iVa58/xJLKJ1/ujB+ie50fv/ lZ+fjyeffBK7d+9GQ0MDvvzyS8N0nFz/0BmKMhMnTsT111+Ps846C2vXrpVt59tvvx2LFi1C SUlJD2fICsoe/sq/ly5dCq/Xi8suu0xefuTIETlOzR133IG6ujoUFxdj165dWLJkieFyEl9W rlyJZ599FuPGjUNVVRVWrlwZUTqRlDvPn8QnWucP65/URO//V3l5OZ5//nlMmDDB1OgzJ9c/ kiRxniFCCCGEpBbypIvD3QgGg3SGCCGEEJLasDFECCGEkJSFfYYIIYQQkrIEg0EAgP2vhCWE EEIIcSBsDBFCCCEkpRGNIT4mI4QQQkjKEgwG6QwRQgghJDXhYzJCCCGEpDRiIsleNYbCzTgJ aL9ojRDiTHhNE0LiXQ/k5OTErJ4R7ybrvTPkMVhX1OvUCSExx+htz/F5eWYsK0dCSHxrAb1r 3Y56IHqNIUIIIYSQBMQRb61XW3HK3++99x4uuugijB49usdL4A4ePIjf/va3KCwsxPTp07Fj x44e6TzyyCMYM2YMZs+ebe9BEEJk9K7b5uZmjBs3Tn6B57Zt2zBz5syw2+n9LTCqC3JycnD5 5ZfjzjvvxAUXXIC77rrL1mMnJNXx+/0YP348jh49GrL86NGjmDBhAvx+v3wd//znP8fMmTOx fft2OU6s02obGNUDvcHlcnV+opZilLn99tvxpz/9Cbt27UJDQ0OINXbPPffg4osvhsfjwbJl y7Bs2bIe2x89ehQffvghxowZE8tsE5LS6F23brcbRUVFePPNNwEAL774Iq666qqw2+n9LQhX Fyxbtgwvv/wyVq1ahU2bNtl23IQQICMjA8XFxT2utbfeegu/+tWvkJGRIV/Hn332Ge6++27c fPPNcpzWNa5erhfTGyRJcu5jsry8PPzxj39ESUkJzjrrLJx33nno27cvAKC2thZvvvkmbrvt NgCdLTs1N910E0466STce++9Mc03IamM0XU7Z84cPPXUU5g2bRpqa2vxl7/8xdR2RoSrC8TN UGFhIQ4dOhStwySE6HDFFVfgwQcfxLnnnovi4mJ8/PHHeOGFF+RrdNu2bVi5ciUaGhpw/Phx +TFVPIn7u8nS0tLkYW3CPhesX78ed955JzIyMvD8889j/vz58jpJklBXVye3EL/88sseaZ90 0kn2Zp4Q0gOj63bq1Kn46quvsGrVKvz6178OuUaNtjMiXF2QlpYmf4v5RAgh9lFSUoLGxkas X78egUAAlZWVOHjwIEpKSgB0usC33HIL6urq4PV6HXNdxrUxNHToULz99ts4evQoKisrQ9al paWhtLQUN9xwA2655Rbs3btXXldaWorVq1fzTo8Qh2F03aanp2PmzJl45ZVXcOmll5reDgBO OeUUzZse1gWEOAtJknDZZZfhqaeewsyZM/HEE0/g8ssvlx2g1tZWuN1utLW14ZFHHrGUtl49 0BvS09ORnp4e38bQ0qVLce+992LixIk49dRTQ9aJTlL5+fn405/+hNWrV8vr/vjHP6K5uRmT J0+OemcqQkjkGF23ADBx4kQMHz4c48aNs7Td9ddfjzlz5vS41lkXEOI8Lr30Upx44olYvnw5 +vTpgzlz5sjrli9fjt/97ncoLS3FkCFDQrYL11Farx7oDcFgEMFgENKOHTuCANC/f3/LicR7 YiZCSHSx85pubW3F4sWLkZeXh5tuuimiNAgh9pMK/9u3bt0KABgx3A2Xy9W7DtSJLgYhJBQ7 r+mCggKMHz/edH8gQkh8SMX/7Y4dTUYISS727NkT7ywQQkgPgsGgc+cZIoQQQgixGzaGCCGE EJKySJLExhAhhBBCCBtDhBBCCElJJElCWloaG0OEEEIISV3kd5M1NTX1ePM7IYQQQkgqkN7Y 2IgDBw7EOx+EEEIIITFFvBstXeo4jGGDBwCA/O6Qjo4OBINBBAIBeapqsVEwGMQJJ5wAl8sl f4LBICRJkj/K7cXfkiQhPT0daWlpcu9tJWJbl8sFKS00k+q/Ozo65HhlPkU6yr9FHsWxqY9F edwinTQpPST9jo4OBAKBkDSV+25vb0cgEJC3ES+HVKatJb4WIg1lXCAQkPMm/lamrYwDgOPH jyMtLQ0nnHBCSF4AhGihzh/Ln+Uv9GX5s/xZ/iz/VCh/8TtdZFQsUO5AuRMlysJQCqGVcSGW WjzxrfyIZQiGpqN38igPWuxfFIb6gNWFLo5BmZb8QbcO6o8Q2Gze1Ps22k6pp/q3y+VCIBAI efu2eh/ixG9vbw85udRaaZW58m+WP8uf5c/yZ/mz/FOp/P8/qbpzL2Wu9XMAAAAASUVORK5C YII= --------------070809000706090606020605-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx160.postini.com [74.125.245.160]) by kanga.kvack.org (Postfix) with SMTP id D65976B0074 for ; Tue, 11 Dec 2012 16:56:27 -0500 (EST) Date: Tue, 11 Dec 2012 22:56:22 +0100 From: Zlatko Calusic MIME-Version: 1.0 References: <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> <20121210180141.GK1009@suse.de> <50C62AE6.3030000@iskon.hr> <50C6477A.4090005@iskon.hr> <50C67C13.6090702@iskon.hr> In-Reply-To: <50C67C13.6090702@iskon.hr> Message-ID: <50C7AC06.8030502@iskon.hr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: kswapd craziness in 3.7 Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Andrew Morton , Mel Gorman , Johannes Weiner , Rik van Riel , linux-mm , Linux Kernel Mailing List , Hugh Dickins On 11.12.2012 01:19, Zlatko Calusic wrote: > > I will now make one last attempt, I've just reverted 2 Johannes' commits > that were also applied in attempt to fix breakage that removing > gfp_no_kswapd introduced, namely ed23ec4 & c702418. For various reasons > the results of this test will be available tommorow, so it's your call > Linus. > To be honest, I don't see any difference with those two commits reverted. Like those lines never did much anyway, so it's probably good we got rid of them. :P -- Zlatko -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx140.postini.com [74.125.245.140]) by kanga.kvack.org (Postfix) with SMTP id 73D996B0068 for ; Wed, 19 Dec 2012 17:24:06 -0500 (EST) Date: Wed, 19 Dec 2012 23:24:00 +0100 From: Zlatko Calusic MIME-Version: 1.0 References: <20121203194208.GZ24381@cmpxchg.org> <20121204214210.GB20253@cmpxchg.org> <20121205030133.GA17438@wolff.to> <20121206173742.GA27297@wolff.to> <50C32D32.6040800@iskon.hr> <50C3AF80.8040700@iskon.hr> <20121210110337.GH1009@suse.de> <20121210163904.GA22101@cmpxchg.org> <20121210180141.GK1009@suse.de> <50C62AE6.3030000@iskon.hr> <50C6477A.4090005@iskon.hr> <50C67C13.6090702@iskon.hr> In-Reply-To: <50C67C13.6090702@iskon.hr> Message-ID: <50D23E80.3010408@iskon.hr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: kswapd craziness in 3.7 Sender: owner-linux-mm@kvack.org List-ID: To: Linus Torvalds Cc: Andrew Morton , Mel Gorman , Johannes Weiner , Rik van Riel , linux-mm , Linux Kernel Mailing List , Hugh Dickins On 11.12.2012 01:19, Zlatko Calusic wrote: >> On 10.12.2012 20:13, Linus Torvalds wrote: >>> >>> It's worth giving this as much testing as is at all possible, but at >>> the same time I really don't think I can delay 3.7 any more without >>> messing up the holiday season too much. So unless something obvious >>> pops up, I will do the release tonight. So testing will be minimal - >>> but it's not like we haven't gone back-and-forth on this several times >>> already, and we revert to *mostly* the same old state as 3.6 anyway, >>> so it should be fairly safe. >>> > > So, here's what I found. In short: close, but no cigar! > > Kswapd is certainly no more CPU pig, and memory seems to be utilized > properly (the kernel still likes to keep 400MB free, somebody else can > confirm if that's to be expected on a 4GB THP-enabled machine). So it > looks very decent, and much better than anything I run in last 10 days, > barring !THP kernel. > > What remains a mystery is that kswapd occassionaly still likes to get > stuck in a D state, only now it recovers faster than before (sometimes > in a matter of seconds, but sometimes it takes a few minutes). Now, I > admit it's a small, maybe even cosmetic issue. But, it could also be a > warning sign of a bigger problem that will reveal itself on a more > loaded machine. > Ha, I nailed it! The cigar aka the explanation together with a patch will follow shortly in a separate topic. It's a genuine bug that has been with us for a long long time. -- Zlatko -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org