[patch] vm early reclaim orphaned pages

[patch] vm early reclaim orphaned pages

[Nick Piggin]

[-- Attachment #1: Type: text/plain, Size: 199 bytes --]

This any good?

The workload in question is bonnie, with ext3 data=ordered.
Chris has some ReiserFS workloads where orphaned pages build
up that I think were improved.


-- 
SUSE Labs, Novell Inc.



[-- Attachment #2: vm-early-reclaim-orphaned.patch --]
[-- Type: text/x-patch, Size: 1323 bytes --]

We have workloads where orphaned pages build up and appear to slow
the system down when it starts reclaiming memory.

Stripping the referenced bit from orphaned pages and putting them
on the end of the inactive list should help improve reclaim.

Signed-off-by: Nick Piggin <npiggin@suse.de>

Index: linux-2.6/mm/truncate.c
===================================================================
--- linux-2.6.orig/mm/truncate.c	2005-06-01 16:09:34.000000000 +1000
+++ linux-2.6/mm/truncate.c	2005-06-17 13:01:01.090334444 +1000
@@ -45,11 +45,30 @@
 static void
 truncate_complete_page(struct address_space *mapping, struct page *page)
 {
+	int orphaned = 0;
+	
 	if (page->mapping != mapping)
 		return;
 
 	if (PagePrivate(page))
-		do_invalidatepage(page, 0);
+		orphaned = !(do_invalidatepage(page, 0));
+
+	if (orphaned) {
+		/*
+		 * Put orphaned pagecache on the end of the inactive
+		 * list so it can get reclaimed quickly.
+		 */
+		unsigned long flags;
+		struct zone *zone = page_zone(page);
+		spin_lock_irqsave(&zone->lru_lock, flags);
+		ClearPageReferenced(page);
+		if (PageLRU(page)) {
+			list_move_tail(&page->lru, &zone->inactive_list);
+			if (PageActive(page))
+				ClearPageActive(page);
+		}
+		spin_unlock_irqrestore(&zone->lru_lock, flags);
+	}
 
 	clear_page_dirty(page);
 	ClearPageUptodate(page);

Re: [patch] vm early reclaim orphaned pages

[Andrew Morton]

Nick Piggin <nickpiggin@yahoo.com.au> wrote:
>
>  We have workloads where orphaned pages build up and appear to slow
>  the system down when it starts reclaiming memory.
> 
>  Stripping the referenced bit from orphaned pages and putting them
>  on the end of the inactive list should help improve reclaim.

Presumably if do_invalidatepage() failed, there's some reason why this page
is not reclaimable (eg, JBD is still dinking with it).  Hence there's a
very good chance that kswapd won't be able to reclaim it either.

Adding some instrumentation would be useful: set some new page flag on
these pages and then accumulate the success/failure stats in vmscan.c, see
what they say.

>  Signed-off-by: Nick Piggin <npiggin@suse.de>
> 
>  Index: linux-2.6/mm/truncate.c
>  ===================================================================
>  --- linux-2.6.orig/mm/truncate.c	2005-06-01 16:09:34.000000000 +1000
>  +++ linux-2.6/mm/truncate.c	2005-06-17 13:01:01.090334444 +1000
>  @@ -45,11 +45,30 @@
>   static void
>   truncate_complete_page(struct address_space *mapping, struct page *page)
>   {
>  +	int orphaned = 0;
>  +	
>   	if (page->mapping != mapping)
>   		return;
>   
>   	if (PagePrivate(page))
>  -		do_invalidatepage(page, 0);
>  +		orphaned = !(do_invalidatepage(page, 0));
>  +
>  +	if (orphaned) {
>  +		/*
>  +		 * Put orphaned pagecache on the end of the inactive
>  +		 * list so it can get reclaimed quickly.
>  +		 */
>  +		unsigned long flags;
>  +		struct zone *zone = page_zone(page);
>  +		spin_lock_irqsave(&zone->lru_lock, flags);
>  +		ClearPageReferenced(page);
>  +		if (PageLRU(page)) {
>  +			list_move_tail(&page->lru, &zone->inactive_list);
>  +			if (PageActive(page))
>  +				ClearPageActive(page);
>  +		}
>  +		spin_unlock_irqrestore(&zone->lru_lock, flags);
>  +	}

A standalone function in swap.c would be nicer.

Re: [patch] vm early reclaim orphaned pages

[Nick Piggin]

On Thu, 2005-06-16 at 20:34 -0700, Andrew Morton wrote:
> Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> >
> >  We have workloads where orphaned pages build up and appear to slow
> >  the system down when it starts reclaiming memory.
> > 
> >  Stripping the referenced bit from orphaned pages and putting them
> >  on the end of the inactive list should help improve reclaim.
> 
> Presumably if do_invalidatepage() failed, there's some reason why this page
> is not reclaimable (eg, JBD is still dinking with it).  Hence there's a
> very good chance that kswapd won't be able to reclaim it either.
> 

Yeah that is a problem I was worried about. Perhaps just stripping
PageReferenced and putting it on the *front* of the inactive list
would be better?

> Adding some instrumentation would be useful: set some new page flag on
> these pages and then accumulate the success/failure stats in vmscan.c, see
> what they say.
> 

OK.

[snip patch]

> A standalone function in swap.c would be nicer.

Will do. Thanks.


-- 
SUSE Labs, Novell Inc.



Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: [patch] vm early reclaim orphaned pages

[Andrea Arcangeli]

Hello everyone,

On Fri, Jun 17, 2005 at 01:42:29PM +1000, Nick Piggin wrote:
> Yeah that is a problem I was worried about. Perhaps just stripping
> PageReferenced and putting it on the *front* of the inactive list
> would be better?

I thought about putting in font instead of the tail too.

But then I also thought this after all is truncate that will literally
free some ram, and in turn it'll delay the VM shrinking a bit (i.e.
kswapd will stop shortly after truncate started). Plus there's no reason
to assume kswapd was running when truncate was invoked.

So overall I think the basic idea is to move it in the inactive list and
to strip the referenced bit, either at the head or tail probably doesn't
make much difference, they have opposite pros/cons.

The front sounds conceptually safer even if tail probably works better
in practice.

> Will do. Thanks.

Thanks!

[patch 1/2] vm early reclaim orphaned pages (take 2)

[Nick Piggin]

[-- Attachment #1: Type: text/plain, Size: 46 bytes --]

How about this?

-- 
SUSE Labs, Novell Inc.



[-- Attachment #2: vm-early-reclaim-orphaned.patch --]
[-- Type: text/x-patch, Size: 3137 bytes --]

We have workloads where orphaned pages build up and appear to slow
the system down when it starts reclaiming memory.

Stripping the referenced bit from orphaned pages and putting them
on the end of the inactive list should help improve reclaim.

Signed-off-by: Nick Piggin <npiggin@suse.de>

Index: linux-2.6/mm/truncate.c
===================================================================
--- linux-2.6.orig/mm/truncate.c	2005-06-01 16:09:34.000000000 +1000
+++ linux-2.6/mm/truncate.c	2005-06-20 17:05:41.011026426 +1000
@@ -12,6 +12,7 @@
 #include <linux/module.h>
 #include <linux/pagemap.h>
 #include <linux/pagevec.h>
+#include <linux/swap.h>
 #include <linux/buffer_head.h>	/* grr. try_to_release_page,
 				   block_invalidatepage */
 
@@ -48,9 +49,11 @@
 	if (page->mapping != mapping)
 		return;
 
-	if (PagePrivate(page))
-		do_invalidatepage(page, 0);
-
+	if (PagePrivate(page)) {
+		if (!(do_invalidatepage(page, 0)))
+			rotate_orphaned_page(page);
+	}
+				
 	clear_page_dirty(page);
 	ClearPageUptodate(page);
 	ClearPageMappedToDisk(page);
Index: linux-2.6/include/linux/swap.h
===================================================================
--- linux-2.6.orig/include/linux/swap.h	2005-06-01 16:09:26.000000000 +1000
+++ linux-2.6/include/linux/swap.h	2005-06-20 17:05:01.632921946 +1000
@@ -169,6 +169,7 @@
 extern void FASTCALL(mark_page_accessed(struct page *));
 extern void lru_add_drain(void);
 extern int rotate_reclaimable_page(struct page *page);
+extern void rotate_orphaned_page(struct page *page);
 extern void swap_setup(void);
 
 /* linux/mm/vmscan.c */
Index: linux-2.6/mm/swap.c
===================================================================
--- linux-2.6.orig/mm/swap.c	2004-12-25 08:34:31.000000000 +1100
+++ linux-2.6/mm/swap.c	2005-06-20 17:20:28.216728238 +1000
@@ -87,7 +87,7 @@
 	spin_lock_irqsave(&zone->lru_lock, flags);
 	if (PageLRU(page) && !PageActive(page)) {
 		list_del(&page->lru);
-		list_add_tail(&page->lru, &zone->inactive_list);
+		list_move_tail(&page->lru, &zone->inactive_list);
 		inc_page_state(pgrotated);
 	}
 	if (!test_clear_page_writeback(page))
@@ -97,6 +97,32 @@
 }
 
 /*
+ * A page has been truncated, but is being orphaned on the LRU list due to
+ * a filesystem dependancy.
+ *
+ * Strip the referenced bit from this page, and if it is on the active list
+ * then put it on the head of the inactive list to aid page reclaim.
+ *
+ * We don't put it on the tail of the inactive list because the page is
+ * not able to be immediately freed due to filesystem dependancy (however
+ * in general, putting the page on the tail would probably be a win, but
+ * slightly more prone to introducing a regression).
+ */
+void rotate_orphaned_page(struct page *page)
+{
+	unsigned long flags;
+	struct zone *zone = page_zone(page);
+
+	spin_lock_irqsave(&zone->lru_lock, flags);
+	ClearPageReferenced(page);
+	if (PageLRU(page) && PageActive(page)) {
+		list_move(&page->lru, &zone->inactive_list);
+		ClearPageActive(page);
+	}
+	spin_unlock_irqrestore(&zone->lru_lock, flags);
+}
+
+/*
  * FIXME: speed this up?
  */
 void fastcall activate_page(struct page *page)

[patch 2/2] stats for orphaned pages (-mm only)

[Nick Piggin]

[-- Attachment #1: Type: text/plain, Size: 40 bytes --]

And this.

-- 
SUSE Labs, Novell Inc.



[-- Attachment #2: vm-orphaned-debug.patch --]
[-- Type: text/x-patch, Size: 3300 bytes --]

Index: linux-2.6/mm/swap.c
===================================================================
--- linux-2.6.orig/mm/swap.c	2005-06-20 17:20:28.216728238 +1000
+++ linux-2.6/mm/swap.c	2005-06-20 17:21:38.253021265 +1000
@@ -114,8 +114,16 @@
 	struct zone *zone = page_zone(page);
 
 	spin_lock_irqsave(&zone->lru_lock, flags);
-	ClearPageReferenced(page);
+	if (PageLRU(page))
+		SetPageOrphaned(page);
+	if (PageReferenced(page)) {
+		if (PageLRU(page))
+			inc_page_state(pg_orph_stripped);
+		ClearPageReferenced(page);
+	}
+	
 	if (PageLRU(page) && PageActive(page)) {
+		inc_page_state(pg_orph_rotated);
 		list_move(&page->lru, &zone->inactive_list);
 		ClearPageActive(page);
 	}
Index: linux-2.6/include/linux/page-flags.h
===================================================================
--- linux-2.6.orig/include/linux/page-flags.h	2005-06-20 17:20:28.216728238 +1000
+++ linux-2.6/include/linux/page-flags.h	2005-06-20 17:20:56.005273542 +1000
@@ -77,6 +77,8 @@
 #define PG_nosave_free		19	/* Free, should not be written */
 #define PG_uncached		20	/* Page has been mapped as uncached */
 
+#define PG_orphaned		21
+
 /*
  * Global page accounting.  One instance per CPU.  Only unsigned longs are
  * allowed.
@@ -132,6 +134,11 @@
 
 	unsigned long pgrotated;	/* pages rotated to tail of the LRU */
 	unsigned long nr_bounce;	/* pages for bounce buffers */
+	
+	unsigned long pg_orph_stripped;	/* Removed ref bit from orphaned page */
+	unsigned long pg_orph_rotated;	/* Deactivated orphaned page */
+	unsigned long pg_orph_busy;	/* Found orphans still busy */
+	unsigned long pg_orph_reclaim;	/* Reclaimed orphan at first sight */
 };
 
 extern void get_page_state(struct page_state *ret);
@@ -306,6 +313,10 @@
 #define SetPageUncached(page)	set_bit(PG_uncached, &(page)->flags)
 #define ClearPageUncached(page)	clear_bit(PG_uncached, &(page)->flags)
 
+#define PageOrphaned(page)	test_bit(PG_orphaned, &(page)->flags)
+#define SetPageOrphaned(page)	set_bit(PG_orphaned, &(page)->flags)
+#define ClearPageOrphaned(page)	clear_bit(PG_orphaned, &(page)->flags)
+
 struct page;	/* forward declaration */
 
 int test_clear_page_dirty(struct page *page);
Index: linux-2.6/mm/page_alloc.c
===================================================================
--- linux-2.6.orig/mm/page_alloc.c	2005-06-20 17:20:28.216728238 +1000
+++ linux-2.6/mm/page_alloc.c	2005-06-20 17:20:56.006273417 +1000
@@ -1899,6 +1899,11 @@
 
 	"pgrotated",
 	"nr_bounce",
+
+	"pg_orph_stripped",
+	"pg_orph_rotated",
+	"pg_orph_busy",
+	"pg_orph_reclaim",
 };
 
 static void *vmstat_start(struct seq_file *m, loff_t *pos)
Index: linux-2.6/mm/vmscan.c
===================================================================
--- linux-2.6.orig/mm/vmscan.c	2005-06-20 17:20:28.216728238 +1000
+++ linux-2.6/mm/vmscan.c	2005-06-20 17:21:26.171523247 +1000
@@ -522,6 +522,11 @@
 		__put_page(page);
 
 free_it:
+		if (PageOrphaned(page)) {
+			inc_page_state(pg_orph_reclaim);
+			ClearPageOrphaned(page);
+		}
+		
 		unlock_page(page);
 		reclaimed++;
 		if (!pagevec_add(&freed_pvec, page))
@@ -534,6 +539,11 @@
 keep_locked:
 		unlock_page(page);
 keep:
+		if (PageOrphaned(page)) {
+			inc_page_state(pg_orph_busy);
+			ClearPageOrphaned(page);
+		}
+
 		list_add(&page->lru, &ret_pages);
 		BUG_ON(PageLRU(page));
 	}

Re: [patch 1/2] vm early reclaim orphaned pages (take 2)

[Andrew Morton]

Nick Piggin <nickpiggin@yahoo.com.au> wrote:
>
>  How about this?
>

It might be good, it's hard to tell.

Performance testing is needed.

>  --- linux-2.6.orig/mm/swap.c	2004-12-25 08:34:31.000000000 +1100
>  +++ linux-2.6/mm/swap.c	2005-06-20 17:20:28.216728238 +1000
>  @@ -87,7 +87,7 @@
>   	spin_lock_irqsave(&zone->lru_lock, flags);
>   	if (PageLRU(page) && !PageActive(page)) {
>   		list_del(&page->lru);
>  -		list_add_tail(&page->lru, &zone->inactive_list);
>  +		list_move_tail(&page->lru, &zone->inactive_list);
>   		inc_page_state(pgrotated);
>   	}
>   	if (!test_clear_page_writeback(page))
>  @@ -97,6 +97,32 @@
>   }

Correctness testing is needed too ;)

Re: [patch 1/2] vm early reclaim orphaned pages (take 2)

[Nick Piggin]

[-- Attachment #1: Type: text/plain, Size: 170 bytes --]

On Mon, 2005-06-20 at 17:23 +1000, Nick Piggin wrote:
> How about this?
> 

Sorry, something else leaked into that. Updated patch attached.

-- 
SUSE Labs, Novell Inc.



[-- Attachment #2: vm-early-reclaim-orphaned.patch --]
[-- Type: text/x-patch, Size: 2826 bytes --]

We have workloads where orphaned pages build up and appear to slow
the system down when it starts reclaiming memory.

Stripping the referenced bit from orphaned pages and putting them
on the end of the inactive list should help improve reclaim.

Signed-off-by: Nick Piggin <npiggin@suse.de>

Index: linux-2.6/mm/truncate.c
===================================================================
--- linux-2.6.orig/mm/truncate.c	2005-06-01 16:09:34.000000000 +1000
+++ linux-2.6/mm/truncate.c	2005-06-20 17:05:41.011026426 +1000
@@ -12,6 +12,7 @@
 #include <linux/module.h>
 #include <linux/pagemap.h>
 #include <linux/pagevec.h>
+#include <linux/swap.h>
 #include <linux/buffer_head.h>	/* grr. try_to_release_page,
 				   block_invalidatepage */
 
@@ -48,9 +49,11 @@
 	if (page->mapping != mapping)
 		return;
 
-	if (PagePrivate(page))
-		do_invalidatepage(page, 0);
-
+	if (PagePrivate(page)) {
+		if (!(do_invalidatepage(page, 0)))
+			rotate_orphaned_page(page);
+	}
+				
 	clear_page_dirty(page);
 	ClearPageUptodate(page);
 	ClearPageMappedToDisk(page);
Index: linux-2.6/include/linux/swap.h
===================================================================
--- linux-2.6.orig/include/linux/swap.h	2005-06-01 16:09:26.000000000 +1000
+++ linux-2.6/include/linux/swap.h	2005-06-20 17:05:01.632921946 +1000
@@ -169,6 +169,7 @@
 extern void FASTCALL(mark_page_accessed(struct page *));
 extern void lru_add_drain(void);
 extern int rotate_reclaimable_page(struct page *page);
+extern void rotate_orphaned_page(struct page *page);
 extern void swap_setup(void);
 
 /* linux/mm/vmscan.c */
Index: linux-2.6/mm/swap.c
===================================================================
--- linux-2.6.orig/mm/swap.c	2004-12-25 08:34:31.000000000 +1100
+++ linux-2.6/mm/swap.c	2005-06-20 17:31:43.755744637 +1000
@@ -97,6 +97,32 @@
 }
 
 /*
+ * A page has been truncated, but is being orphaned on the LRU list due to
+ * a filesystem dependancy.
+ *
+ * Strip the referenced bit from this page, and if it is on the active list
+ * then put it on the head of the inactive list to aid page reclaim.
+ *
+ * We don't put it on the tail of the inactive list because the page is
+ * not able to be immediately freed due to filesystem dependancy (however
+ * in general, putting the page on the tail would probably be a win, but
+ * slightly more prone to introducing a regression).
+ */
+void rotate_orphaned_page(struct page *page)
+{
+	unsigned long flags;
+	struct zone *zone = page_zone(page);
+
+	spin_lock_irqsave(&zone->lru_lock, flags);
+	ClearPageReferenced(page);
+	if (PageLRU(page) && PageActive(page)) {
+		list_move(&page->lru, &zone->inactive_list);
+		ClearPageActive(page);
+	}
+	spin_unlock_irqrestore(&zone->lru_lock, flags);
+}
+
+/*
  * FIXME: speed this up?
  */
 void fastcall activate_page(struct page *page)

Re: [patch 2/2] stats for orphaned pages (-mm only)

[Andrew Morton]

Nick Piggin <nickpiggin@yahoo.com.au> wrote:
>
> And this.
> 
> ...
> 
> [vm-orphaned-debug.patch  text/x-patch (3472 bytes)]

yup.  Observing the chnages in these numbers across various workloads would
go a long way toward validating the need for some patch and toward
validating a particular patch's effectiveness too.

Re: [patch 2/2] stats for orphaned pages (-mm only)

[Nick Piggin]

On Mon, 2005-06-20 at 00:36 -0700, Andrew Morton wrote:
> Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> >
> > And this.
> > 
> > ...
> > 
> > [vm-orphaned-debug.patch  text/x-patch (3472 bytes)]
> 
> yup.  Observing the chnages in these numbers across various workloads would
> go a long way toward validating the need for some patch and toward
> validating a particular patch's effectiveness too.

I'll try getting some numbers shortly.

-- 
SUSE Labs, Novell Inc.



Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: [patch 1/2] vm early reclaim orphaned pages (take 2)

[Rik Van Riel]

On Mon, 20 Jun 2005, Nick Piggin wrote:

> +       if (PageLRU(page) && PageActive(page)) {
> +               list_move(&page->lru, &zone->inactive_list);
> +               ClearPageActive(page);
> +       }

Unless I'm missing something subtle, you might want to
update zone->nr_active and zone->nr_inactive ...

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics

Re: [patch 1/2] vm early reclaim orphaned pages (take 2)

[Nick Piggin]

Rik Van Riel wrote:
> On Mon, 20 Jun 2005, Nick Piggin wrote:
> 
> 
>>+       if (PageLRU(page) && PageActive(page)) {
>>+               list_move(&page->lru, &zone->inactive_list);
>>+               ClearPageActive(page);
>>+       }
> 
> 
> Unless I'm missing something subtle, you might want to
> update zone->nr_active and zone->nr_inactive ...
> 

You're right, thanks very much Rik.

Send instant messages to your online friends http://au.messenger.yahoo.com