[patch] __block_write_full_page bug

[patch] __block_write_full_page bug

[Nick Piggin]

[-- Attachment #1: Type: text/plain, Size: 50 bytes --]

Another buffer bug :(

-- 
SUSE Labs, Novell Inc.

[-- Attachment #2: __block_write_full_page-bug.patch --]
[-- Type: text/plain, Size: 3057 bytes --]

When running
	fsstress -v -d $DIR/tmp -n 1000 -p 1000 -l 2
on an ext2 filesystem with 1024 byte block size, on SMP i386 with 4096 byte
page size over loopback to an image file on a tmpfs filesystem, I would
very quickly hit
	BUG_ON(!buffer_async_write(bh));
in fs/buffer.c:end_buffer_async_write

It seems that more than one request would be submitted for a given bh
at a time. __block_write_full_page looks like the culprit - with the
following patch things are very stable.

Signed-off-by: Nick Piggin <nickpiggin@yahoo.com.au>

Index: linux-2.6/fs/buffer.c
===================================================================
--- linux-2.6.orig/fs/buffer.c	2005-04-25 13:13:37.000000000 +1000
+++ linux-2.6/fs/buffer.c	2005-04-25 13:14:52.000000000 +1000
@@ -1750,8 +1750,9 @@ static int __block_write_full_page(struc
 	int err;
 	sector_t block;
 	sector_t last_block;
-	struct buffer_head *bh, *head;
-	int nr_underway = 0;
+	struct buffer_head *bh, *head, *arr[MAX_BUF_PER_PAGE];
+	int idx = 0;
+	int nr_underway;
 
 	BUG_ON(!PageLocked(page));
 
@@ -1826,6 +1827,7 @@ static int __block_write_full_page(struc
 		}
 		if (test_clear_buffer_dirty(bh)) {
 			mark_buffer_async_write(bh);
+			arr[idx++] = bh;
 		} else {
 			unlock_buffer(bh);
 		}
@@ -1839,15 +1841,12 @@ static int __block_write_full_page(struc
 	set_page_writeback(page);
 	unlock_page(page);
 
-	do {
-		struct buffer_head *next = bh->b_this_page;
-		if (buffer_async_write(bh)) {
+	for (nr_underway = 0; nr_underway < idx; nr_underway++) {
+		bh = arr[nr_underway];
+		if (buffer_async_write(bh))
 			submit_bh(WRITE, bh);
-			nr_underway++;
-		}
 		put_bh(bh);
-		bh = next;
-	} while (bh != head);
+	}
 
 	err = 0;
 done:
@@ -1890,6 +1889,7 @@ recover:
 		if (buffer_mapped(bh) && buffer_dirty(bh)) {
 			lock_buffer(bh);
 			mark_buffer_async_write(bh);
+			arr[idx++] = bh;
 		} else {
 			/*
 			 * The buffer may have been set dirty during
@@ -1902,16 +1902,14 @@ recover:
 	BUG_ON(PageWriteback(page));
 	set_page_writeback(page);
 	unlock_page(page);
-	do {
-		struct buffer_head *next = bh->b_this_page;
+	for (nr_underway = 0; nr_underway < idx; nr_underway++) {
+		bh = arr[nr_underway];
 		if (buffer_async_write(bh)) {
 			clear_buffer_dirty(bh);
 			submit_bh(WRITE, bh);
-			nr_underway++;
 		}
 		put_bh(bh);
-		bh = next;
-	} while (bh != head);
+	}
 	goto done;
 }
 
@@ -2741,6 +2739,7 @@ sector_t generic_block_bmap(struct addre
 static int end_bio_bh_io_sync(struct bio *bio, unsigned int bytes_done, int err)
 {
 	struct buffer_head *bh = bio->bi_private;
+	bh_end_io_t *end_fn;
 
 	if (bio->bi_size)
 		return 1;
@@ -2750,7 +2749,16 @@ static int end_bio_bh_io_sync(struct bio
 		set_bit(BH_Eopnotsupp, &bh->b_state);
 	}
 
-	bh->b_end_io(bh, test_bit(BIO_UPTODATE, &bio->bi_flags));
+	end_fn = bh->b_end_io;
+
+	/* 
+	 * These two lines are debugging only - make sure b_end_io
+	 * isn't run twice for the same io request.
+	 */
+	BUG_ON(!end_fn);
+	bh->b_end_io = NULL;
+	
+	end_fn(bh, test_bit(BIO_UPTODATE, &bio->bi_flags));
 	bio_put(bio);
 	return 0;
 }

Re: [patch] __block_write_full_page bug

[Andrew Morton]

Nick Piggin <nickpiggin@yahoo.com.au> wrote:
>
>  When running
>  	fsstress -v -d $DIR/tmp -n 1000 -p 1000 -l 2
>  on an ext2 filesystem with 1024 byte block size, on SMP i386 with 4096 byte
>  page size over loopback to an image file on a tmpfs filesystem, I would
>  very quickly hit
>  	BUG_ON(!buffer_async_write(bh));
>  in fs/buffer.c:end_buffer_async_write
> 
>  It seems that more than one request would be submitted for a given bh
>  at a time. __block_write_full_page looks like the culprit - with the
>  following patch things are very stable.

What's the bug?  I don't see it.

Was an ENOSPC involved?

Those tests for buffer_async_write(bh) are redundant now, aren't they?

Re: [patch] __block_write_full_page bug

[Nick Piggin]

On Tue, 2005-04-26 at 04:50 -0700, Andrew Morton wrote:
> Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> >
> >  When running
> >  	fsstress -v -d $DIR/tmp -n 1000 -p 1000 -l 2
> >  on an ext2 filesystem with 1024 byte block size, on SMP i386 with 4096 byte
> >  page size over loopback to an image file on a tmpfs filesystem, I would
> >  very quickly hit
> >  	BUG_ON(!buffer_async_write(bh));
> >  in fs/buffer.c:end_buffer_async_write
> > 
> >  It seems that more than one request would be submitted for a given bh
> >  at a time. __block_write_full_page looks like the culprit - with the
> >  following patch things are very stable.
> 
> What's the bug?  I don't see it.
> 

Ah, the bug is that end_buffer_async_write first does
	BUG_ON(!buffer_async_write(bh));
then a bit later does
	clear_buffer_async_write(bh);

That's where it was blowing up for me, because end_buffer_async_write
was being run twice for that buffer.

Or did you mean *how* is it being run twice? I didn't exactly find
the stack traces involved, but I imagine that simply testing
buffer_async_write catches other requests in flight - ie. we've
lost track of exactly which ones we own.


> Was an ENOSPC involved?
> 

No.

> Those tests for buffer_async_write(bh) are redundant now, aren't they?

They are, yes. Sorry I noticed that earlier too - they should probably
be BUG_ON(!buffer_async_write(bh)) instead. Would make things clearer.

Nick

Re: [patch] __block_write_full_page bug

[Andrew Morton]

Nick Piggin <nickpiggin@yahoo.com.au> wrote:
>
> On Tue, 2005-04-26 at 04:50 -0700, Andrew Morton wrote:
>  > Nick Piggin <nickpiggin@yahoo.com.au> wrote:
>  > >
>  > >  When running
>  > >  	fsstress -v -d $DIR/tmp -n 1000 -p 1000 -l 2
>  > >  on an ext2 filesystem with 1024 byte block size, on SMP i386 with 4096 byte
>  > >  page size over loopback to an image file on a tmpfs filesystem, I would
>  > >  very quickly hit
>  > >  	BUG_ON(!buffer_async_write(bh));
>  > >  in fs/buffer.c:end_buffer_async_write
>  > > 
>  > >  It seems that more than one request would be submitted for a given bh
>  > >  at a time. __block_write_full_page looks like the culprit - with the
>  > >  following patch things are very stable.
>  > 
>  > What's the bug?  I don't see it.
>  > 
> 
>  Ah, the bug is that end_buffer_async_write first does
>  	BUG_ON(!buffer_async_write(bh));
>  then a bit later does
>  	clear_buffer_async_write(bh);
> 
>  That's where it was blowing up for me, because end_buffer_async_write
>  was being run twice for that buffer.
> 
>  Or did you mean *how* is it being run twice? I didn't exactly find
>  the stack traces involved, but I imagine that simply testing
>  buffer_async_write catches other requests in flight - ie. we've
>  lost track of exactly which ones we own.
> 

How can such a thing come about?  Both PageLocked() and PageWriteback() are
supposed to stop new writeback being started against the page.

<looks>

Were you using nobh?  I guess not.  What's to stop the new
mpage_writepage() from trying to write a page which is already under
PageWriteback()?

I don't think we understand this bug yet.

Re: [patch] __block_write_full_page bug

[Nick Piggin]

Andrew Morton wrote:
> Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> 
>>On Tue, 2005-04-26 at 04:50 -0700, Andrew Morton wrote:
>> > Nick Piggin <nickpiggin@yahoo.com.au> wrote:
>> > >
>> > >  When running
>> > >  	fsstress -v -d $DIR/tmp -n 1000 -p 1000 -l 2
>> > >  on an ext2 filesystem with 1024 byte block size, on SMP i386 with 4096 byte
>> > >  page size over loopback to an image file on a tmpfs filesystem, I would
>> > >  very quickly hit
>> > >  	BUG_ON(!buffer_async_write(bh));
>> > >  in fs/buffer.c:end_buffer_async_write
>> > > 
>> > >  It seems that more than one request would be submitted for a given bh
>> > >  at a time. __block_write_full_page looks like the culprit - with the
>> > >  following patch things are very stable.
>> > 
>> > What's the bug?  I don't see it.
>> > 
>>
>> Ah, the bug is that end_buffer_async_write first does
>> 	BUG_ON(!buffer_async_write(bh));
>> then a bit later does
>> 	clear_buffer_async_write(bh);
>>
>> That's where it was blowing up for me, because end_buffer_async_write
>> was being run twice for that buffer.
>>
>> Or did you mean *how* is it being run twice? I didn't exactly find
>> the stack traces involved, but I imagine that simply testing
>> buffer_async_write catches other requests in flight - ie. we've
>> lost track of exactly which ones we own.
>>
> 
> 
> How can such a thing come about?  Both PageLocked() and PageWriteback() are
> supposed to stop new writeback being started against the page.
> 

You have a point.

> <looks>
> 
> Were you using nobh?  I guess not.  What's to stop the new

No

> mpage_writepage() from trying to write a page which is already under
> PageWriteback()?
> 

Don't know... I didn't think mapping->writepage should be called
for a PageWriteback page?

> I don't think we understand this bug yet.
> 

It appears not. I'll look into it further.

-- 
SUSE Labs, Novell Inc.

Re: [patch] __block_write_full_page bug

[Nick Piggin]

[-- Attachment #1: Type: text/plain, Size: 138 bytes --]

Andrew Morton wrote:

> Those tests for buffer_async_write(bh) are redundant now, aren't they?
> 

Like this?

-- 
SUSE Labs, Novell Inc.

[-- Attachment #2: __block_write_full_page-cleanup.patch --]
[-- Type: text/plain, Size: 1011 bytes --]

Previous patch made buffer_async_write tests superfluous.
As suggested by Andrew Morton.

Signed-off-by: Nick Piggin <nickpiggin@yahoo.com.au>

Index: linux-2.6/fs/buffer.c
===================================================================
--- linux-2.6.orig/fs/buffer.c	2005-04-25 13:14:52.000000000 +1000
+++ linux-2.6/fs/buffer.c	2005-04-27 11:00:01.000000000 +1000
@@ -1843,8 +1843,8 @@ static int __block_write_full_page(struc
 
 	for (nr_underway = 0; nr_underway < idx; nr_underway++) {
 		bh = arr[nr_underway];
-		if (buffer_async_write(bh))
-			submit_bh(WRITE, bh);
+		BUG_ON(!buffer_async_write(bh));
+		submit_bh(WRITE, bh);
 		put_bh(bh);
 	}
 
@@ -1904,10 +1904,9 @@ recover:
 	unlock_page(page);
 	for (nr_underway = 0; nr_underway < idx; nr_underway++) {
 		bh = arr[nr_underway];
-		if (buffer_async_write(bh)) {
-			clear_buffer_dirty(bh);
-			submit_bh(WRITE, bh);
-		}
+		BUG_ON(!buffer_async_write(bh));
+		clear_buffer_dirty(bh);
+		submit_bh(WRITE, bh);
 		put_bh(bh);
 	}
 	goto done;