review: use correct buffer flags when reading superblock

review: use correct buffer flags when reading superblock

[Lachlan McIlroy]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

When reading the superblock during log recovery we are not setting
the correct buffer flags.  Specifically we are not turning off flags
we do not need such as XBF_ASYNC that is causing the synchronous
xfs_iowait() to hang.  We should also turn off XBF_WRITE and remove
the buffer from the delay write queue just to be safe.

Lachlan

[-- Attachment #2: xfs_getsb.diff --]
[-- Type: text/x-patch, Size: 433 bytes --]

--- fs/xfs/xfs_log_recover.c_1.329	2007-10-10 15:59:18.000000000 +1000
+++ fs/xfs/xfs_log_recover.c	2007-10-10 16:03:08.000000000 +1000
@@ -3824,7 +3824,10 @@ xlog_do_recover(
 	 */
 	bp = xfs_getsb(log->l_mp, 0);
 	XFS_BUF_UNDONE(bp);
+	XFS_BUF_UNWRITE(bp);
+	XFS_BUF_UNDELAYWRITE(bp);
 	XFS_BUF_READ(bp);
+	XFS_BUF_UNASYNC(bp);
 	xfsbdstrat(log->l_mp, bp);
 	if ((error = xfs_iowait(bp))) {
 		xfs_ioerror_alert("xlog_do_recover",

Re: review: use correct buffer flags when reading superblock

[Christoph Hellwig]

On Wed, Oct 10, 2007 at 06:37:47PM +1000, Lachlan McIlroy wrote:
> When reading the superblock during log recovery we are not setting
> the correct buffer flags.  Specifically we are not turning off flags
> we do not need such as XBF_ASYNC that is causing the synchronous
> xfs_iowait() to hang.  We should also turn off XBF_WRITE and remove
> the buffer from the delay write queue just to be safe.

Where are these set up in the first time?  AFAICS the buffer only written
out by xfs_unmountfs_writesb, xfs_syncsub and xfs_trans_log_buf, and all
these should only ever happen after log recovery has finished.

Re: review: use correct buffer flags when reading superblock

[David Chinner]

On Wed, Oct 10, 2007 at 10:34:51AM +0100, Christoph Hellwig wrote:
> On Wed, Oct 10, 2007 at 06:37:47PM +1000, Lachlan McIlroy wrote:
> > When reading the superblock during log recovery we are not setting
> > the correct buffer flags.  Specifically we are not turning off flags
> > we do not need such as XBF_ASYNC that is causing the synchronous
> > xfs_iowait() to hang.  We should also turn off XBF_WRITE and remove
> > the buffer from the delay write queue just to be safe.
> 
> Where are these set up in the first time?  AFAICS the buffer only written
> out by xfs_unmountfs_writesb, xfs_syncsub and xfs_trans_log_buf, and all
> these should only ever happen after log recovery has finished.

It can also be written by xfsbufd when it has been bdwrite()
or as a result of log tail pushing.

And in the case of log recovery, if the superblock buffer is in a
transaction that is recovered, xlog_recover_do_buffer_trans() will
bdwrite() it after it has been recovered. Hence it will be on the
delwri list.

Once recovery is complete, xlog_do_recover() then calls XFS_bflush()
to write them all out and wait for them. This is down by
xfs_flush_buftarg() with the wait flag set, so the async flag on the
buffer should be cleared. This is normally the case.

The issue here is that when we openteh buftarg, we start up the
xfsbufd which will periodically flush the delwri list
asynchronously. If recovery takes long enough, it may flush the
delwri list before recovery completes.In this case, instead of doing
a sync write we'll get an async write being done and that leaves the
XBF_ASYNC flag hanging around on the buffer at I/O completion. 

Because the superblock buffer is XBF_FS_MANAGED, it does not get
torn down when it is clean and has no references, so the XBF_ASYNC
flag never gets cleared unless the fs specifically clears it. If the
superblock is then not recovered out of any further transactions
during recovery after xfsbufd flushed it, the XBF_ASYNC flag remains
set for the re-read that is issued in xlog_do_recover() and we
hang.....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: review: use correct buffer flags when reading superblock

[David Chinner]

On Wed, Oct 10, 2007 at 06:37:47PM +1000, Lachlan McIlroy wrote:
> When reading the superblock during log recovery we are not setting
> the correct buffer flags.  Specifically we are not turning off flags
> we do not need such as XBF_ASYNC that is causing the synchronous
> xfs_iowait() to hang.  We should also turn off XBF_WRITE and remove
> the buffer from the delay write queue just to be safe.

We really don't need the removal of the write flags - the XFS_bflush()
call above the xfs_getsb() call guarantees that they won't be set....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: review: use correct buffer flags when reading superblock

[Lachlan McIlroy]

David Chinner wrote:
> On Wed, Oct 10, 2007 at 06:37:47PM +1000, Lachlan McIlroy wrote:
>> When reading the superblock during log recovery we are not setting
>> the correct buffer flags.  Specifically we are not turning off flags
>> we do not need such as XBF_ASYNC that is causing the synchronous
>> xfs_iowait() to hang.  We should also turn off XBF_WRITE and remove
>> the buffer from the delay write queue just to be safe.
> 
> We really don't need the removal of the write flags - the XFS_bflush()
> call above the xfs_getsb() call guarantees that they won't be set....
> 

It's not obvious though.  It wasn't obvious that ASYNC was still set
and look where that got us.

Re: review: use correct buffer flags when reading superblock

[David Chinner]

On Thu, Oct 11, 2007 at 12:42:25PM +1000, Lachlan McIlroy wrote:
> David Chinner wrote:
> >On Wed, Oct 10, 2007 at 06:37:47PM +1000, Lachlan McIlroy wrote:
> >>When reading the superblock during log recovery we are not setting
> >>the correct buffer flags.  Specifically we are not turning off flags
> >>we do not need such as XBF_ASYNC that is causing the synchronous
> >>xfs_iowait() to hang.  We should also turn off XBF_WRITE and remove
> >>the buffer from the delay write queue just to be safe.
> >
> >We really don't need the removal of the write flags - the XFS_bflush()
> >call above the xfs_getsb() call guarantees that they won't be set....
> 
> It's not obvious though.  It wasn't obvious that ASYNC was still set
> and look where that got us.

Sure, but it is incorrect to have them set at that point - incorrect
enough that it might cause corruption. That is, if they are set
then the changes made during log replay have not been written to
disk and we've got bigger problems than a hanging I/O to worry
about....

So rather than clearing them, we should be asserting that the
write flags are not set at all.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: review: use correct buffer flags when reading superblock

[Lachlan McIlroy]

[-- Attachment #1: Type: text/plain, Size: 1191 bytes --]

David Chinner wrote:
> On Thu, Oct 11, 2007 at 12:42:25PM +1000, Lachlan McIlroy wrote:
>> David Chinner wrote:
>>> On Wed, Oct 10, 2007 at 06:37:47PM +1000, Lachlan McIlroy wrote:
>>>> When reading the superblock during log recovery we are not setting
>>>> the correct buffer flags.  Specifically we are not turning off flags
>>>> we do not need such as XBF_ASYNC that is causing the synchronous
>>>> xfs_iowait() to hang.  We should also turn off XBF_WRITE and remove
>>>> the buffer from the delay write queue just to be safe.
>>> We really don't need the removal of the write flags - the XFS_bflush()
>>> call above the xfs_getsb() call guarantees that they won't be set....
>> It's not obvious though.  It wasn't obvious that ASYNC was still set
>> and look where that got us.
> 
> Sure, but it is incorrect to have them set at that point - incorrect
> enough that it might cause corruption. That is, if they are set
> then the changes made during log replay have not been written to
> disk and we've got bigger problems than a hanging I/O to worry
> about....
> 
> So rather than clearing them, we should be asserting that the
> write flags are not set at all.
> 

New patch attached.

[-- Attachment #2: xfs_getsb.diff --]
[-- Type: text/x-patch, Size: 455 bytes --]

--- fs/xfs/xfs_log_recover.c_1.329	2007-10-10 15:59:18.000000000 +1000
+++ fs/xfs/xfs_log_recover.c	2007-10-11 15:03:00.000000000 +1000
@@ -3824,7 +3824,10 @@ xlog_do_recover(
 	 */
 	bp = xfs_getsb(log->l_mp, 0);
 	XFS_BUF_UNDONE(bp);
+	ASSERT(!(XFS_BUF_ISWRITE(bp)));
+	ASSERT(!(XFS_BUF_ISDELAYWRITE(bp)));
 	XFS_BUF_READ(bp);
+	XFS_BUF_UNASYNC(bp);
 	xfsbdstrat(log->l_mp, bp);
 	if ((error = xfs_iowait(bp))) {
 		xfs_ioerror_alert("xlog_do_recover",

Re: review: use correct buffer flags when reading superblock

[Christoph Hellwig]

On Wed, Oct 10, 2007 at 09:25:06PM +1000, David Chinner wrote:
> > Where are these set up in the first time?  AFAICS the buffer only written
> > out by xfs_unmountfs_writesb, xfs_syncsub and xfs_trans_log_buf, and all
> > these should only ever happen after log recovery has finished.
> 
> It can also be written by xfsbufd when it has been bdwrite()
> or as a result of log tail pushing.

Hmm, true.

> Because the superblock buffer is XBF_FS_MANAGED, it does not get
> torn down when it is clean and has no references, so the XBF_ASYNC
> flag never gets cleared unless the fs specifically clears it. If the
> superblock is then not recovered out of any further transactions
> during recovery after xfsbufd flushed it, the XBF_ASYNC flag remains
> set for the re-read that is issued in xlog_do_recover() and we
> hang.....

Makes sense as an explanation.  I still don't really like patch, maybe
we should always clear the ASYNC flag in the b_iodone callback?

Re: review: use correct buffer flags when reading superblock

[Lachlan McIlroy]

Christoph Hellwig wrote:
> On Wed, Oct 10, 2007 at 09:25:06PM +1000, David Chinner wrote:
>>> Where are these set up in the first time?  AFAICS the buffer only written
>>> out by xfs_unmountfs_writesb, xfs_syncsub and xfs_trans_log_buf, and all
>>> these should only ever happen after log recovery has finished.
>> It can also be written by xfsbufd when it has been bdwrite()
>> or as a result of log tail pushing.
> 
> Hmm, true.
> 
>> Because the superblock buffer is XBF_FS_MANAGED, it does not get
>> torn down when it is clean and has no references, so the XBF_ASYNC
>> flag never gets cleared unless the fs specifically clears it. If the
>> superblock is then not recovered out of any further transactions
>> during recovery after xfsbufd flushed it, the XBF_ASYNC flag remains
>> set for the re-read that is issued in xlog_do_recover() and we
>> hang.....
> 
> Makes sense as an explanation.  I still don't really like patch, maybe
> we should always clear the ASYNC flag in the b_iodone callback?
> 

That sounds like a good idea.  Or get rid of the XBF_FS_MANAGED special
case and get a new fresh buffer each time.

Re: review: use correct buffer flags when reading superblock

[David Chinner]

On Fri, Oct 19, 2007 at 11:32:18AM +1000, Lachlan McIlroy wrote:
> Christoph Hellwig wrote:
> >On Wed, Oct 10, 2007 at 09:25:06PM +1000, David Chinner wrote:
> >>Because the superblock buffer is XBF_FS_MANAGED, it does not get
> >>torn down when it is clean and has no references, so the XBF_ASYNC
> >>flag never gets cleared unless the fs specifically clears it. If the
> >>superblock is then not recovered out of any further transactions
> >>during recovery after xfsbufd flushed it, the XBF_ASYNC flag remains
> >>set for the re-read that is issued in xlog_do_recover() and we
> >>hang.....
> >
> >Makes sense as an explanation.  I still don't really like patch, maybe
> >we should always clear the ASYNC flag in the b_iodone callback?
> 
> That sounds like a good idea. 

<shrug> 

Makes no real difference - you just have to be careful where the
ASYNC flag is cleared because it is used throughout the io
completion code....

> Or get rid of the XBF_FS_MANAGED special
> case and get a new fresh buffer each time.

I don't think we want to do that. It will add substantial overhead
because the superblock is the single most used buffer in the
filesysem. It's typically gained during transaction commit, at which
time we really, really want to get it quickly and this is known to
be a performance limiting bottleneck.....

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group