[PATCH 0/3] writeback: bug fixes for 2.6.35

[PATCH 0/3] writeback: bug fixes for 2.6.35

[Dave Chinner]

Hi Linus,

This is a resend of my current writeback bug fixes without the preceding
tracing patches.  Please consider these for 2.6.35 as they are fixes for
regressions introduced in previous kernel releases.

Cheers,

Dave.

[PATCH 1/3] writeback: pay attention to wbc->nr_to_write in write_cache_pages

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

If a filesystem writes more than one page in ->writepage, write_cache_pages
fails to notice this and continues to attempt writeback when wbc->nr_to_write
has gone negative - this trace was captured from XFS:


    wbc_writeback_start: towrt=1024
    wbc_writepage: towrt=1024
    wbc_writepage: towrt=0
    wbc_writepage: towrt=-1
    wbc_writepage: towrt=-5
    wbc_writepage: towrt=-21
    wbc_writepage: towrt=-85

This has adverse effects on filesystem writeback behaviour. write_cache_pages()
needs to terminate after a certain number of pages are written, not after a
certain number of calls to ->writepage are made.  This is a regression
introduced by 17bc6c30cf6bfffd816bdc53682dd46fc34a2cf4 ("vfs: Add
no_nrwrite_index_update writeback control flag"), but cannot be reverted
directly due to subsequent bug fixes that have gone in on top of it.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
---
 include/linux/writeback.h   |    9 ---------
 include/trace/events/ext4.h |    5 +----
 mm/page-writeback.c         |   15 +++++----------
 3 files changed, 6 insertions(+), 23 deletions(-)

diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index f641346..d63ef8f 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -56,15 +56,6 @@ struct writeback_control {
 	unsigned for_reclaim:1;		/* Invoked from the page allocator */
 	unsigned range_cyclic:1;	/* range_start is cyclic */
 	unsigned more_io:1;		/* more io to be dispatched */
-	/*
-	 * write_cache_pages() won't update wbc->nr_to_write and
-	 * mapping->writeback_index if no_nrwrite_index_update
-	 * is set.  write_cache_pages() may write more than we
-	 * requested and we want to make sure nr_to_write and
-	 * writeback_index are updated in a consistent manner
-	 * so we use a single control to update them
-	 */
-	unsigned no_nrwrite_index_update:1;
 };
 
 /*
diff --git a/include/trace/events/ext4.h b/include/trace/events/ext4.h
index f5b1ba9..f3865c7 100644
--- a/include/trace/events/ext4.h
+++ b/include/trace/events/ext4.h
@@ -306,7 +306,6 @@ TRACE_EVENT(ext4_da_writepages_result,
 		__field(	int,	pages_written		)
 		__field(	long,	pages_skipped		)
 		__field(	char,	more_io			)	
-		__field(	char,	no_nrwrite_index_update )
 		__field(       pgoff_t,	writeback_index		)
 	),
 
@@ -317,16 +316,14 @@ TRACE_EVENT(ext4_da_writepages_result,
 		__entry->pages_written	= pages_written;
 		__entry->pages_skipped	= wbc->pages_skipped;
 		__entry->more_io	= wbc->more_io;
-		__entry->no_nrwrite_index_update = wbc->no_nrwrite_index_update;
 		__entry->writeback_index = inode->i_mapping->writeback_index;
 	),
 
-	TP_printk("dev %s ino %lu ret %d pages_written %d pages_skipped %ld more_io %d no_nrwrite_index_update %d writeback_index %lu",
+	TP_printk("dev %s ino %lu ret %d pages_written %d pages_skipped %ld more_io %d writeback_index %lu",
 		  jbd2_dev_to_name(__entry->dev),
 		  (unsigned long) __entry->ino, __entry->ret,
 		  __entry->pages_written, __entry->pages_skipped,
 		  __entry->more_io,
-		  __entry->no_nrwrite_index_update,
 		  (unsigned long) __entry->writeback_index)
 );
 
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 5fa63bd..b3dbb80 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -835,7 +835,6 @@ int write_cache_pages(struct address_space *mapping,
 	pgoff_t done_index;
 	int cycled;
 	int range_whole = 0;
-	long nr_to_write = wbc->nr_to_write;
 
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {
@@ -935,11 +934,10 @@ continue_unlock:
 					done = 1;
 					break;
 				}
- 			}
+			}
 
-			if (nr_to_write > 0) {
-				nr_to_write--;
-				if (nr_to_write == 0 &&
+			if (wbc->nr_to_write > 0) {
+				if (--wbc->nr_to_write == 0 &&
 				    wbc->sync_mode == WB_SYNC_NONE) {
 					/*
 					 * We stop writing back only if we are
@@ -970,11 +968,8 @@ continue_unlock:
 		end = writeback_index - 1;
 		goto retry;
 	}
-	if (!wbc->no_nrwrite_index_update) {
-		if (wbc->range_cyclic || (range_whole && nr_to_write > 0))
-			mapping->writeback_index = done_index;
-		wbc->nr_to_write = nr_to_write;
-	}
+	if (wbc->range_cyclic || (range_whole && wbc->nr_to_write > 0))
+		mapping->writeback_index = done_index;
 
 	return ret;
 }
-- 
1.7.1

Re: [PATCH 1/3] writeback: pay attention to wbc->nr_to_write in write_cache_pages

[Andrew Morton]

On Wed,  9 Jun 2010 10:37:18 +1000
Dave Chinner <david@fromorbit.com> wrote:

> From: Dave Chinner <dchinner@redhat.com>
> 
> If a filesystem writes more than one page in ->writepage, write_cache_pages
> fails to notice this and continues to attempt writeback when wbc->nr_to_write
> has gone negative - this trace was captured from XFS:
> 
> 
>     wbc_writeback_start: towrt=1024
>     wbc_writepage: towrt=1024
>     wbc_writepage: towrt=0
>     wbc_writepage: towrt=-1
>     wbc_writepage: towrt=-5
>     wbc_writepage: towrt=-21
>     wbc_writepage: towrt=-85
> 
> This has adverse effects on filesystem writeback behaviour. write_cache_pages()
> needs to terminate after a certain number of pages are written, not after a
> certain number of calls to ->writepage are made.  This is a regression
> introduced by 17bc6c30cf6bfffd816bdc53682dd46fc34a2cf4 ("vfs: Add
> no_nrwrite_index_update writeback control flag"), but cannot be reverted
> directly due to subsequent bug fixes that have gone in on top of it.

Might be needed in -stable.  Unfortunately the most important piece of
information which is needed to make that decision was cunningly hidden
from us behind the vague-to-the-point-of-uselessness term "adverse
effects".

_what_ "adverse effects"??

Re: [PATCH 1/3] writeback: pay attention to wbc->nr_to_write in write_cache_pages

[Dave Chinner]

On Wed, Jun 09, 2010 at 02:09:42PM -0700, Andrew Morton wrote:
> On Wed,  9 Jun 2010 10:37:18 +1000
> Dave Chinner <david@fromorbit.com> wrote:
> 
> > From: Dave Chinner <dchinner@redhat.com>
> > 
> > If a filesystem writes more than one page in ->writepage, write_cache_pages
> > fails to notice this and continues to attempt writeback when wbc->nr_to_write
> > has gone negative - this trace was captured from XFS:
> > 
> > 
> >     wbc_writeback_start: towrt=1024
> >     wbc_writepage: towrt=1024
> >     wbc_writepage: towrt=0
> >     wbc_writepage: towrt=-1
> >     wbc_writepage: towrt=-5
> >     wbc_writepage: towrt=-21
> >     wbc_writepage: towrt=-85
> > 
> > This has adverse effects on filesystem writeback behaviour. write_cache_pages()
> > needs to terminate after a certain number of pages are written, not after a
> > certain number of calls to ->writepage are made.  This is a regression
> > introduced by 17bc6c30cf6bfffd816bdc53682dd46fc34a2cf4 ("vfs: Add
> > no_nrwrite_index_update writeback control flag"), but cannot be reverted
> > directly due to subsequent bug fixes that have gone in on top of it.
> 
> Might be needed in -stable.  Unfortunately the most important piece of
> information which is needed to make that decision was cunningly hidden
> from us behind the vague-to-the-point-of-uselessness term "adverse
> effects".
> 
> _what_ "adverse effects"??

Depends on how the specific filesystem handles a negative
nr_to_write, doesn't it? I can't speak for the exact effect on
anything other than XFS except to say that most ->write_page
implemetnations don't handle the wbc->nr_to_write < 0 specifically...

For XFS, it results in increased CPU usage because it triggers
page-at-a-time allocation (i.e no clustering), which increases
overhead in the elveator due to merging requirements of single page
bios and increased fragmentation due to small interleaved
allocations on concurrent writeback workloads. Effectively it causes
accelerated aging of XFS filesystems...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH 1/3] writeback: pay attention to wbc->nr_to_write in write_cache_pages

[Dave Chinner]

On Thu, Jun 10, 2010 at 08:58:04AM +1000, Dave Chinner wrote:
> On Wed, Jun 09, 2010 at 02:09:42PM -0700, Andrew Morton wrote:
> > On Wed,  9 Jun 2010 10:37:18 +1000
> > Dave Chinner <david@fromorbit.com> wrote:
> > 
> > > From: Dave Chinner <dchinner@redhat.com>
> > > 
> > > If a filesystem writes more than one page in ->writepage, write_cache_pages
> > > fails to notice this and continues to attempt writeback when wbc->nr_to_write
> > > has gone negative - this trace was captured from XFS:
> > > 
> > > 
> > >     wbc_writeback_start: towrt=1024
> > >     wbc_writepage: towrt=1024
> > >     wbc_writepage: towrt=0
> > >     wbc_writepage: towrt=-1
> > >     wbc_writepage: towrt=-5
> > >     wbc_writepage: towrt=-21
> > >     wbc_writepage: towrt=-85
> > > 
> > > This has adverse effects on filesystem writeback behaviour. write_cache_pages()
> > > needs to terminate after a certain number of pages are written, not after a
> > > certain number of calls to ->writepage are made.  This is a regression
> > > introduced by 17bc6c30cf6bfffd816bdc53682dd46fc34a2cf4 ("vfs: Add
> > > no_nrwrite_index_update writeback control flag"), but cannot be reverted
> > > directly due to subsequent bug fixes that have gone in on top of it.
> > 
> > Might be needed in -stable.  Unfortunately the most important piece of
> > information which is needed to make that decision was cunningly hidden
> > from us behind the vague-to-the-point-of-uselessness term "adverse
> > effects".
> > 
> > _what_ "adverse effects"??
> 
> Depends on how the specific filesystem handles a negative
> nr_to_write, doesn't it? I can't speak for the exact effect on
> anything other than XFS except to say that most ->write_page
> implemetnations don't handle the wbc->nr_to_write < 0 specifically...
> 
> For XFS, it results in increased CPU usage because it triggers
> page-at-a-time allocation (i.e no clustering), which increases
> overhead in the elveator due to merging requirements of single page
> bios and increased fragmentation due to small interleaved
> allocations on concurrent writeback workloads. Effectively it causes
> accelerated aging of XFS filesystems...

Sorry, forgot to address the -stable part of the question.

This series is dependent on the ext4 change to use it's own
writepage going into -stable first.  (i.e.
8e48dcfbd7c0892b4cfd064d682cc4c95a29df32 "ext4: Use our own
write_cache_pages()").

I'd suggest that all 4 patches (the ext4 patch and the three in this
series) should go back to 2.6.34-stable due to the long term affect
this writeback bug could have on XFS filesystems, and the sync
taking too long problem has been fairly widely reported since at
least .32...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [stable] [PATCH 1/3] writeback: pay attention to wbc->nr_to_write in write_cache_pages

[Greg KH]

On Thu, Jun 10, 2010 at 10:08:11AM +1000, Dave Chinner wrote:
> On Thu, Jun 10, 2010 at 08:58:04AM +1000, Dave Chinner wrote:
> > On Wed, Jun 09, 2010 at 02:09:42PM -0700, Andrew Morton wrote:
> > > On Wed,  9 Jun 2010 10:37:18 +1000
> > > Dave Chinner <david@fromorbit.com> wrote:
> > > 
> > > > From: Dave Chinner <dchinner@redhat.com>
> > > > 
> > > > If a filesystem writes more than one page in ->writepage, write_cache_pages
> > > > fails to notice this and continues to attempt writeback when wbc->nr_to_write
> > > > has gone negative - this trace was captured from XFS:
> > > > 
> > > > 
> > > >     wbc_writeback_start: towrt=1024
> > > >     wbc_writepage: towrt=1024
> > > >     wbc_writepage: towrt=0
> > > >     wbc_writepage: towrt=-1
> > > >     wbc_writepage: towrt=-5
> > > >     wbc_writepage: towrt=-21
> > > >     wbc_writepage: towrt=-85
> > > > 
> > > > This has adverse effects on filesystem writeback behaviour. write_cache_pages()
> > > > needs to terminate after a certain number of pages are written, not after a
> > > > certain number of calls to ->writepage are made.  This is a regression
> > > > introduced by 17bc6c30cf6bfffd816bdc53682dd46fc34a2cf4 ("vfs: Add
> > > > no_nrwrite_index_update writeback control flag"), but cannot be reverted
> > > > directly due to subsequent bug fixes that have gone in on top of it.
> > > 
> > > Might be needed in -stable.  Unfortunately the most important piece of
> > > information which is needed to make that decision was cunningly hidden
> > > from us behind the vague-to-the-point-of-uselessness term "adverse
> > > effects".
> > > 
> > > _what_ "adverse effects"??
> > 
> > Depends on how the specific filesystem handles a negative
> > nr_to_write, doesn't it? I can't speak for the exact effect on
> > anything other than XFS except to say that most ->write_page
> > implemetnations don't handle the wbc->nr_to_write < 0 specifically...
> > 
> > For XFS, it results in increased CPU usage because it triggers
> > page-at-a-time allocation (i.e no clustering), which increases
> > overhead in the elveator due to merging requirements of single page
> > bios and increased fragmentation due to small interleaved
> > allocations on concurrent writeback workloads. Effectively it causes
> > accelerated aging of XFS filesystems...
> 
> Sorry, forgot to address the -stable part of the question.
> 
> This series is dependent on the ext4 change to use it's own
> writepage going into -stable first.  (i.e.
> 8e48dcfbd7c0892b4cfd064d682cc4c95a29df32 "ext4: Use our own
> write_cache_pages()").
> 
> I'd suggest that all 4 patches (the ext4 patch and the three in this
> series) should go back to 2.6.34-stable due to the long term affect
> this writeback bug could have on XFS filesystems, and the sync
> taking too long problem has been fairly widely reported since at
> least .32...

Ok, can someone please tell me the git commit ids that need to be
applied to the -stable trees?

thanks,

greg k-h

[PATCH 2/3] xfs: remove nr_to_write writeback windup.

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

Now that the background flush code has been fixed, we shouldn't need to
silently multiply the wbc->nr_to_write to get good writeback. Remove
that code.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
---
 fs/xfs/linux-2.6/xfs_aops.c |    8 --------
 1 files changed, 0 insertions(+), 8 deletions(-)

diff --git a/fs/xfs/linux-2.6/xfs_aops.c b/fs/xfs/linux-2.6/xfs_aops.c
index a0fa3bf..34640d6 100644
--- a/fs/xfs/linux-2.6/xfs_aops.c
+++ b/fs/xfs/linux-2.6/xfs_aops.c
@@ -1381,14 +1381,6 @@ xfs_vm_writepage(
 	if (!page_has_buffers(page))
 		create_empty_buffers(page, 1 << inode->i_blkbits, 0);
 
-
-	/*
-	 *  VM calculation for nr_to_write seems off.  Bump it way
-	 *  up, this gets simple streaming writes zippy again.
-	 *  To be reviewed again after Jens' writeback changes.
-	 */
-	wbc->nr_to_write *= 4;
-
 	/*
 	 * Convert delayed allocate, unwritten or unmapped space
 	 * to real space and flush out to disk.
-- 
1.7.1

Re: [PATCH 2/3] xfs: remove nr_to_write writeback windup.

[Andrew Morton]

On Wed,  9 Jun 2010 10:37:19 +1000
Dave Chinner <david@fromorbit.com> wrote:

> From: Dave Chinner <dchinner@redhat.com>
> 
> Now that the background flush code has been fixed, we shouldn't need to
> silently multiply the wbc->nr_to_write to get good writeback. Remove
> that code.
> 

And anyone who merges [1/3] will want to merge this.  Otherwise they
might suffer adverse effects ;)

[PATCH 3/3] writeback: limit write_cache_pages integrity scanning to current EOF

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

sync can currently take a really long time if a concurrent writer is
extending a file. The problem is that the dirty pages on the address
space grow in the same direction as write_cache_pages scans, so if
the writer keeps ahead of writeback, the writeback will not
terminate until the writer stops adding dirty pages.

For a data integrity sync, we only need to write the pages dirty at
the time we start the writeback, so we can stop scanning once we get
to the page that was at the end of the file at the time the scan
started.

This will prevent operations like copying a large file preventing
sync from completing as it will not write back pages that were
dirtied after the sync was started. This does not impact the
existing integrity guarantees, as any dirty page (old or new)
within the EOF range at the start of the scan will still be
captured.

This patch will not prevent sync from blocking on large writes into
holes. That requires more complex intervention while this patch only
addresses the common append-case of this sync holdoff.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
---
 mm/page-writeback.c |   15 +++++++++++++++
 1 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index b3dbb80..bbd396a 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -851,7 +851,22 @@ int write_cache_pages(struct address_space *mapping,
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
 		cycled = 1; /* ignore range_cyclic tests */
+
+		/*
+		 * If this is a data integrity sync, cap the writeback to the
+		 * current end of file. Any extension to the file that occurs
+		 * after this is a new write and we don't need to write those
+		 * pages out to fulfil our data integrity requirements. If we
+		 * try to write them out, we can get stuck in this scan until
+		 * the concurrent writer stops adding dirty pages and extending
+		 * EOF.
+		 */
+		if (wbc->sync_mode == WB_SYNC_ALL &&
+		    wbc->range_end == LLONG_MAX) {
+			end = i_size_read(mapping->host) >> PAGE_CACHE_SHIFT;
+		}
 	}
+
 retry:
 	done_index = index;
 	while (!done && (index <= end)) {
-- 
1.7.1

Re: [PATCH 3/3] writeback: limit write_cache_pages integrity scanning to current EOF

[Andrew Morton]

On Wed,  9 Jun 2010 10:37:20 +1000
Dave Chinner <david@fromorbit.com> wrote:

> From: Dave Chinner <dchinner@redhat.com>
> 
> sync can currently take a really long time if a concurrent writer is
> extending a file. The problem is that the dirty pages on the address
> space grow in the same direction as write_cache_pages scans, so if
> the writer keeps ahead of writeback, the writeback will not
> terminate until the writer stops adding dirty pages.
> 
> For a data integrity sync, we only need to write the pages dirty at
> the time we start the writeback, so we can stop scanning once we get
> to the page that was at the end of the file at the time the scan
> started.
> 
> This will prevent operations like copying a large file preventing
> sync from completing as it will not write back pages that were
> dirtied after the sync was started. This does not impact the
> existing integrity guarantees, as any dirty page (old or new)
> within the EOF range at the start of the scan will still be
> captured.
> 
> This patch will not prevent sync from blocking on large writes into
> holes. That requires more complex intervention while this patch only
> addresses the common append-case of this sync holdoff.
> 

I don't know if this regression sucks enough to warrant backporting,
but the fix is simple and is independent of [1/3] and [2/3].