[PATCH 0/3 v2] mm: Batch page reclamation under shink_page_list

[PATCH 0/3 v2] mm: Batch page reclamation under shink_page_list

[Tim Chen]

This is the second version of the patch series. Thanks to Matthew Wilcox 
for many valuable suggestions on improving the patches.

To do page reclamation in shrink_page_list function, there are two
locks taken on a page by page basis.  One is the tree lock protecting
the radix tree of the page mapping and the other is the
mapping->i_mmap_mutex protecting the mapped
pages.  I try to batch the operations on pages sharing the same lock
to reduce lock contentions.  The first patch batch the operations protected by
tree lock while the second and third patch batch the operations protected by 
the i_mmap_mutex.

I managed to get 14% throughput improvement when with a workload putting
heavy pressure of page cache by reading many large mmaped files
simultaneously on a 8 socket Westmere server.

Tim

Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
---
Diffstat 

 include/linux/rmap.h |    8 +++-
 mm/rmap.c            |  110 ++++++++++++++++++++++++++++++++++---------------
 mm/vmscan.c          |  113 +++++++++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 185 insertions(+), 46 deletions(-)
















--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3 v2] mm: Batch page reclamation under shink_page_list

[Minchan Kim]

Hi Tim,

On Mon, Sep 10, 2012 at 09:19:20AM -0700, Tim Chen wrote:
> This is the second version of the patch series. Thanks to Matthew Wilcox 
> for many valuable suggestions on improving the patches.
> 
> To do page reclamation in shrink_page_list function, there are two
> locks taken on a page by page basis.  One is the tree lock protecting
> the radix tree of the page mapping and the other is the
> mapping->i_mmap_mutex protecting the mapped
> pages.  I try to batch the operations on pages sharing the same lock
> to reduce lock contentions.  The first patch batch the operations protected by
> tree lock while the second and third patch batch the operations protected by 
> the i_mmap_mutex.
> 
> I managed to get 14% throughput improvement when with a workload putting
> heavy pressure of page cache by reading many large mmaped files
> simultaneously on a 8 socket Westmere server.
> 
> Tim
> 
> Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>

If you send next versions, please use git-format-patch --thread style.
Quote from man
"       If given --thread, git-format-patch will generate In-Reply-To and References
       headers to make the second and subsequent patch mails appear as replies to the
       first mail; this also generates a Message-Id header to reference.
"

It helps making your all patches in this series thread type in mailbox
so reviewers can find all patches related to a subject easily.

Thanks.

-- 
Kind regards,
Minchan Kim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3 v2] mm: Batch page reclamation under shink_page_list

[Tim Chen]

On Tue, 2012-09-11 at 14:36 +0900, Minchan Kim wrote:

> 
> If you send next versions, please use git-format-patch --thread style.
> Quote from man
> "       If given --thread, git-format-patch will generate In-Reply-To and References
>        headers to make the second and subsequent patch mails appear as replies to the
>        first mail; this also generates a Message-Id header to reference.
> "
> 
> It helps making your all patches in this series thread type in mailbox
> so reviewers can find all patches related to a subject easily.
> 

Thanks for your detailed review of the patch series.  Will incorporate
your suggestions on the next version.

Tim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3 v2] mm: Batch page reclamation under shink_page_list

[Andrew Morton]

On Mon, 10 Sep 2012 09:19:20 -0700
Tim Chen <tim.c.chen@linux.intel.com> wrote:

> This is the second version of the patch series. Thanks to Matthew Wilcox 
> for many valuable suggestions on improving the patches.
> 
> To do page reclamation in shrink_page_list function, there are two
> locks taken on a page by page basis.  One is the tree lock protecting
> the radix tree of the page mapping and the other is the
> mapping->i_mmap_mutex protecting the mapped
> pages.  I try to batch the operations on pages sharing the same lock
> to reduce lock contentions.  The first patch batch the operations protected by
> tree lock while the second and third patch batch the operations protected by 
> the i_mmap_mutex.
> 
> I managed to get 14% throughput improvement when with a workload putting
> heavy pressure of page cache by reading many large mmaped files
> simultaneously on a 8 socket Westmere server.

That sounds good, although more details on the performance changes
would be appreciated - after all, that's the entire point of the
patchset.

And we shouldn't only test for improvements - we should also test for
degradation.  What workloads might be harmed by this change?  I'd suggest

- a single process which opens N files and reads one page from each
  one, then repeats.  So there are no contiguous LRU pages which share
  the same ->mapping.  Get some page reclaim happening, measure the
  impact.

- The batching means that we now do multiple passes over pageframes
  where we used to do things in a single pass.  Walking all those new
  page lists will be expensive if they are lengthy enough to cause L1
  cache evictions.

  What would be a test for this?  A simple, single-threaded walk
  through a file, I guess?

Mel's review comments were useful, thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3 v2] mm: Batch page reclamation under shink_page_list

[Tim Chen]

On Wed, 2012-09-12 at 12:27 -0700, Andrew Morton wrote:

> 
> That sounds good, although more details on the performance changes
> would be appreciated - after all, that's the entire point of the
> patchset.
> 
> And we shouldn't only test for improvements - we should also test for
> degradation.  What workloads might be harmed by this change?  I'd suggest
> 
> - a single process which opens N files and reads one page from each
>   one, then repeats.  So there are no contiguous LRU pages which share
>   the same ->mapping.  Get some page reclaim happening, measure the
>   impact.
> 
> - The batching means that we now do multiple passes over pageframes
>   where we used to do things in a single pass.  Walking all those new
>   page lists will be expensive if they are lengthy enough to cause L1
>   cache evictions.

I need to address both your concerns and Mel's concerns about the
downside of prolonging the holding page locks for the pages to be
unmmaped for patch 1 in the series.  I'll try to do some testing to see
what kind of benefit I get by only batching operations under the
i_mmap_mutex (i.e. patch 2 and 3 only) and not do batch unmap. Those
other changes don't have the downsides of prolonged page locking and we
can incorporate them with less risks.

> 
>   What would be a test for this?  A simple, single-threaded walk
>   through a file, I guess?

Thanks for your test suggestions.  I will do tests along your
suggestions when I generate the next iterations of the patch. 

I've been playing with these patches for a while and they are based on
3.4 kernel.  I'll move them to 3.6 kernel in my next iteration.

> 
> Mel's review comments were useful, thanks.

Very much appreciate comments from you, Mel and Minchan. I'll try to
incorporate them in my changes. 

Tim

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>