[PATCH 1/2] ext4: call ext4_error_inode() if jbd2_journal_dirty_metadata() fails

[PATCH 1/2] ext4: call ext4_error_inode() if jbd2_journal_dirty_metadata() fails

[Theodore Ts'o]

While it's true that errors can only happen if there is a bug in
jbd2_journal_dirty_metadata(), if a bug does happen, we need to halt
the kernel or remount the file system read-only in order to avoid
further data loss.  The ext4_journal_abort_handle() function doesn't
do any of this, and while it's likely that this call (since it doesn't
adjust refcounts) will likely result in the file system eventually
deadlocking since the current transaction will never be able to close,
it's much cleaner to call let ext4's error handling system deal with
this situation.

There's a separate bug here which is that if certain jbd2 errors
errors occur and file system is mounted errors=continue, the file
system will probably eventually end grind to a halt as described
above.  But things have been this way in a long time, and usually when
we have these sorts of errors it's pretty much a disaster --- and
that's why the jbd2 layer aggressively retries memory allocations,
which is the most likely cause of these jbd2 errors.

Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Cc: stable@vger.kernel.org
---
 fs/ext4/ext4_jbd2.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/fs/ext4/ext4_jbd2.c b/fs/ext4/ext4_jbd2.c
index 17ac112..3fe29de 100644
--- a/fs/ext4/ext4_jbd2.c
+++ b/fs/ext4/ext4_jbd2.c
@@ -259,6 +259,15 @@ int __ext4_handle_dirty_metadata(const char *where, unsigned int line,
 		if (WARN_ON_ONCE(err)) {
 			ext4_journal_abort_handle(where, line, __func__, bh,
 						  handle, err);
+			ext4_error_inode(inode, where, line,
+					 bh->b_blocknr,
+					 "journal_dirty_metadata failed: "
+					 "handle type %u started at line %u, "
+					 "credits %u/%u, errcode %d",
+					 handle->h_type,
+					 handle->h_line_no,
+					 handle->h_requested_credits,
+					 handle->h_buffer_credits, err);
 		}
 	} else {
 		if (inode)
-- 
1.8.5.rc3.362.gdf10213

[PATCH 2/2] jbd2: return ENOSPC in journal_dirty_metadata if a handle runs out of space

[Theodore Ts'o]

If a handle runs out of space, we currently stop the kernel with a
BUG.  This makes it hard to figure out what might be going on.  So
return an error of ENOSPC, so we can let the file system layer figure
out what is going on, to make it more likely we can get useful
debugging information).  This should make it easier to debug problems
such as the one which was reported by:

    https://bugzilla.kernel.org/show_bug.cgi?id=44731

The only two callers of this function are ext4_handle_dirty_metadata()
and ocfs2_journal_dirty().  The ocfs2 function will trigger a
BUG_ON(), which means there will be no change in behavior.  The ext4
function will call ext4_error_inode() which will print the useful
debugging information and then handle the situation using ext4's error
handling mechanisms (i.e., which might mean halting the kernel or
remounting the file system read-only).

Also, since both file systems already call WARN_ON(), drop the WARN_ON
from jbd2_journal_dirty_metadata() to avoid two stack traces from
being displayed.

Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Cc: ocfs2-devel@oss.oracle.com
---
 fs/jbd2/transaction.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/jbd2/transaction.c b/fs/jbd2/transaction.c
index 7aa9a32..b0b74e5 100644
--- a/fs/jbd2/transaction.c
+++ b/fs/jbd2/transaction.c
@@ -1290,7 +1290,10 @@ int jbd2_journal_dirty_metadata(handle_t *handle, struct buffer_head *bh)
 		 * once a transaction -bzzz
 		 */
 		jh->b_modified = 1;
-		J_ASSERT_JH(jh, handle->h_buffer_credits > 0);
+		if (handle->h_buffer_credits <= 0) {
+			ret = -ENOSPC;
+			goto out_unlock_bh;
+		}
 		handle->h_buffer_credits--;
 	}
 
@@ -1373,7 +1376,6 @@ out_unlock_bh:
 	jbd2_journal_put_journal_head(jh);
 out:
 	JBUFFER_TRACE(jh, "exit");
-	WARN_ON(ret);	/* All errors are bugs, so dump the stack */
 	return ret;
 }
 
-- 
1.8.5.rc3.362.gdf10213

Re: [PATCH 2/2] jbd2: return ENOSPC in journal_dirty_metadata if a handle runs out of space

[Jan Kara]

On Mon 02-12-13 09:45:02, Ted Tso wrote:
> If a handle runs out of space, we currently stop the kernel with a
> BUG.  This makes it hard to figure out what might be going on.  So
> return an error of ENOSPC, so we can let the file system layer figure
> out what is going on, to make it more likely we can get useful
> debugging information).  This should make it easier to debug problems
> such as the one which was reported by:
> 
>     https://bugzilla.kernel.org/show_bug.cgi?id=44731
> 
> The only two callers of this function are ext4_handle_dirty_metadata()
> and ocfs2_journal_dirty().  The ocfs2 function will trigger a
> BUG_ON(), which means there will be no change in behavior.  The ext4
> function will call ext4_error_inode() which will print the useful
> debugging information and then handle the situation using ext4's error
> handling mechanisms (i.e., which might mean halting the kernel or
> remounting the file system read-only).
> 
> Also, since both file systems already call WARN_ON(), drop the WARN_ON
> from jbd2_journal_dirty_metadata() to avoid two stack traces from
> being displayed.
> 
> Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
> Cc: ocfs2-devel@oss.oracle.com
  The patch looks good. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>

								Honza

> ---
>  fs/jbd2/transaction.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/jbd2/transaction.c b/fs/jbd2/transaction.c
> index 7aa9a32..b0b74e5 100644
> --- a/fs/jbd2/transaction.c
> +++ b/fs/jbd2/transaction.c
> @@ -1290,7 +1290,10 @@ int jbd2_journal_dirty_metadata(handle_t *handle, struct buffer_head *bh)
>  		 * once a transaction -bzzz
>  		 */
>  		jh->b_modified = 1;
> -		J_ASSERT_JH(jh, handle->h_buffer_credits > 0);
> +		if (handle->h_buffer_credits <= 0) {
> +			ret = -ENOSPC;
> +			goto out_unlock_bh;
> +		}
>  		handle->h_buffer_credits--;
>  	}
>  
> @@ -1373,7 +1376,6 @@ out_unlock_bh:
>  	jbd2_journal_put_journal_head(jh);
>  out:
>  	JBUFFER_TRACE(jh, "exit");
> -	WARN_ON(ret);	/* All errors are bugs, so dump the stack */
>  	return ret;
>  }
>  
> -- 
> 1.8.5.rc3.362.gdf10213
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [PATCH 2/2] jbd2: return ENOSPC in journal_dirty_metadata if a handle runs out of space

[Joel Becker]

On Mon, Dec 02, 2013 at 09:45:02AM -0500, Theodore Ts'o wrote:
> If a handle runs out of space, we currently stop the kernel with a
> BUG.  This makes it hard to figure out what might be going on.  So
> return an error of ENOSPC, so we can let the file system layer figure
> out what is going on, to make it more likely we can get useful
> debugging information).  This should make it easier to debug problems
> such as the one which was reported by:
> 
>     https://bugzilla.kernel.org/show_bug.cgi?id=44731
> 
> The only two callers of this function are ext4_handle_dirty_metadata()
> and ocfs2_journal_dirty().  The ocfs2 function will trigger a
> BUG_ON(), which means there will be no change in behavior.  The ext4
> function will call ext4_error_inode() which will print the useful
> debugging information and then handle the situation using ext4's error
> handling mechanisms (i.e., which might mean halting the kernel or
> remounting the file system read-only).
> 
> Also, since both file systems already call WARN_ON(), drop the WARN_ON
> from jbd2_journal_dirty_metadata() to avoid two stack traces from
> being displayed.
> 
> Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
> Cc: ocfs2-devel@oss.oracle.com

Acked-by: Joel Becker <jlbec@evilplan.org>

> ---
>  fs/jbd2/transaction.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/jbd2/transaction.c b/fs/jbd2/transaction.c
> index 7aa9a32..b0b74e5 100644
> --- a/fs/jbd2/transaction.c
> +++ b/fs/jbd2/transaction.c
> @@ -1290,7 +1290,10 @@ int jbd2_journal_dirty_metadata(handle_t *handle, struct buffer_head *bh)
>  		 * once a transaction -bzzz
>  		 */
>  		jh->b_modified = 1;
> -		J_ASSERT_JH(jh, handle->h_buffer_credits > 0);
> +		if (handle->h_buffer_credits <= 0) {
> +			ret = -ENOSPC;
> +			goto out_unlock_bh;
> +		}
>  		handle->h_buffer_credits--;
>  	}
>  
> @@ -1373,7 +1376,6 @@ out_unlock_bh:
>  	jbd2_journal_put_journal_head(jh);
>  out:
>  	JBUFFER_TRACE(jh, "exit");
> -	WARN_ON(ret);	/* All errors are bugs, so dump the stack */
>  	return ret;
>  }
>  
> -- 
> 1.8.5.rc3.362.gdf10213
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 

Re: [PATCH 1/2] ext4: call ext4_error_inode() if jbd2_journal_dirty_metadata() fails

[Jan Kara]

On Mon 02-12-13 09:45:01, Ted Tso wrote:
> While it's true that errors can only happen if there is a bug in
> jbd2_journal_dirty_metadata(), if a bug does happen, we need to halt
> the kernel or remount the file system read-only in order to avoid
> further data loss.  The ext4_journal_abort_handle() function doesn't
> do any of this, and while it's likely that this call (since it doesn't
> adjust refcounts) will likely result in the file system eventually
> deadlocking since the current transaction will never be able to close,
> it's much cleaner to call let ext4's error handling system deal with
> this situation.
> 
> There's a separate bug here which is that if certain jbd2 errors
> errors occur and file system is mounted errors=continue, the file
> system will probably eventually end grind to a halt as described
> above.  But things have been this way in a long time, and usually when
> we have these sorts of errors it's pretty much a disaster --- and
> that's why the jbd2 layer aggressively retries memory allocations,
> which is the most likely cause of these jbd2 errors.
> 
> Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
> Cc: stable@vger.kernel.org
  The patch looks good. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>

								Honza

> ---
>  fs/ext4/ext4_jbd2.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/fs/ext4/ext4_jbd2.c b/fs/ext4/ext4_jbd2.c
> index 17ac112..3fe29de 100644
> --- a/fs/ext4/ext4_jbd2.c
> +++ b/fs/ext4/ext4_jbd2.c
> @@ -259,6 +259,15 @@ int __ext4_handle_dirty_metadata(const char *where, unsigned int line,
>  		if (WARN_ON_ONCE(err)) {
>  			ext4_journal_abort_handle(where, line, __func__, bh,
>  						  handle, err);
> +			ext4_error_inode(inode, where, line,
> +					 bh->b_blocknr,
> +					 "journal_dirty_metadata failed: "
> +					 "handle type %u started at line %u, "
> +					 "credits %u/%u, errcode %d",
> +					 handle->h_type,
> +					 handle->h_line_no,
> +					 handle->h_requested_credits,
> +					 handle->h_buffer_credits, err);
>  		}
>  	} else {
>  		if (inode)
> -- 
> 1.8.5.rc3.362.gdf10213
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR