From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nick Piggin Subject: Re: [patch 0/9] writeback data integrity and other fixes (take 3) Date: Wed, 29 Oct 2008 01:04:24 +0100 Message-ID: <20081029000424.GD15599@wotan.suse.de> References: <20081028144715.683011000@suse.de> <20081028153953.GB3082@wotan.suse.de> <20081028222746.GB4985@disturbed> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii To: akpm@linux-foundation.org, xfs@oss.sgi.com, linux-fsdevel@vger.kernel.org, Chris Mason Return-path: Received: from ns.suse.de ([195.135.220.2]:59206 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751771AbYJ2AE3 (ORCPT ); Tue, 28 Oct 2008 20:04:29 -0400 Content-Disposition: inline In-Reply-To: <20081028222746.GB4985@disturbed> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On Wed, Oct 29, 2008 at 09:27:46AM +1100, Dave Chinner wrote: > On Tue, Oct 28, 2008 at 04:39:53PM +0100, Nick Piggin wrote: > > On Wed, Oct 29, 2008 at 01:47:15AM +1100, npiggin@suse.de wrote: > > I haven't seen any -EIO failures from XFS... maybe I'm just not doing the > > right thing, or there is a caveat I'm not aware of. > > > > All fault injections I noticed had a trace like this: [...] > > XFS reports bio errors through the I/O completion path, not the > submission path. Right, that's just to give you an indication of where it's failing... > > And the kernel would sometimes say this: > > Buffer I/O error on device ram0, logical block 279 > > lost page write due to I/O error on ram0 > > Buffer I/O error on device ram0, logical block 379 > > lost page write due to I/O error on ram0 > > Buffer I/O error on device ram0, logical block 389 > > lost page write due to I/O error on ram0 > > Yes - that's coming from end_buffer_async_write() when an error is > reported in bio completion. This does: > > 465 set_bit(AS_EIO, &page->mapping->flags); > 466 set_buffer_write_io_error(bh); > 467 clear_buffer_uptodate(bh); > 468 SetPageError(page); > > Hmmmm - do_fsync() calls filemap_fdatawait() which ends up in > wait_on_page_writeback_range() which is appears to be checking the > mapping flags for errors. I wonder why that error is not being > propagated then? AFAICT both XFS and the fsync code are doing the > right thing but somewhere the error has gone missing... Yeah, I couldn't immediately see why nothing comes out. I'll do a bit more digging. > > I think I also saw a slab bug when running dbench with fault injection on. > > Running latest Linus kernel. [...] > > Now that is interesting. > > We've got a rolling transaction in progress, and the commit of the > first part of the transaction has got the I/O error. That frees the > transaction structure used during that commit, as well as the > ticket. > > However, before we committed the initial transaction, we duplicated > the transaction structure to allow the transaction to continue to > track all the dirty objects in the first commit. That included > duplicating the pointer to the ticket. > > Then the EIO is returned to mkdir code with the duplicated > transaction, which is then cancelled, and that frees the transaction > and the ticket it holds. However, we'd already freed the ticket. > > Ok, we're only seeing this problem now because I recently modified > the ticket allocation to use a slab instead of a roll-your-own free > list structure that wouldn't have been poisoned. Nice to know that > this change did more than just remove code. ;) > > This might take a little while to fix - a lot of code needs > auditing - but thanks for reporting the problem. No problem, hope it helps. Thanks, Nick