[PATCH] jbd: abort instead of waiting for nonexistent transactions

[PATCH] jbd: abort instead of waiting for nonexistent transactions

[Duane Griffin]

The __log_wait_for_space function sits in a loop checkpointing transactions
until there is sufficient space free in the journal. However, if there are
no transactions to be processed (e.g. because the free space calculation is
wrong due to a corrupted filesystem) it will never progress.

Check for space being required when no transactions are outstanding and
abort the journal instead of endlessly looping.

This patch fixes the bug reported by Sami Liedes at:
http://bugzilla.kernel.org/show_bug.cgi?id=10976

Signed-off-by: Duane Griffin <duaneg@dghda.com>
Tested-by: Sami Liedes <sliedes@cc.hut.fi>
---
diff --git a/fs/jbd/checkpoint.c b/fs/jbd/checkpoint.c
index a5432bb..9fac177 100644
--- a/fs/jbd/checkpoint.c
+++ b/fs/jbd/checkpoint.c
@@ -126,14 +126,29 @@ void __log_wait_for_space(journal_t *journal)
 
 		/*
 		 * Test again, another process may have checkpointed while we
-		 * were waiting for the checkpoint lock
+		 * were waiting for the checkpoint lock. If there are no
+		 * outstanding transactions there is nothing to checkpoint and
+		 * we can't make progress. Abort the journal in this case.
 		 */
 		spin_lock(&journal->j_state_lock);
+		spin_lock(&journal->j_list_lock);
 		nblocks = jbd_space_needed(journal);
 		if (__log_space_left(journal) < nblocks) {
+			int chkpt = journal->j_checkpoint_transactions != NULL;
+
+			spin_unlock(&journal->j_list_lock);
 			spin_unlock(&journal->j_state_lock);
-			log_do_checkpoint(journal);
+			if (chkpt) {
+				log_do_checkpoint(journal);
+			} else {
+				printk(KERN_ERR "%s: no transactions\n",
+				       __func__);
+				journal_abort(journal, 0);
+			}
+
 			spin_lock(&journal->j_state_lock);
+		} else {
+			spin_unlock(&journal->j_list_lock);
 		}
 		mutex_unlock(&journal->j_checkpoint_mutex);
 	}

[PATCH] jbd2: abort instead of waiting for nonexistent transactions

[Duane Griffin]

The __jbd2_log_wait_for_space function sits in a loop checkpointing
transactions until there is sufficient space free in the journal. However,
if there are no transactions to be processed (e.g. because the free space
calculation is wrong due to a corrupted filesystem) it will never progress.

Check for space being required when no transactions are outstanding and
abort the journal instead of endlessly looping.

This patch fixes the bug reported by Sami Liedes at:
http://bugzilla.kernel.org/show_bug.cgi?id=10976

Signed-off-by: Duane Griffin <duaneg@dghda.com>
---
diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c
index 91389c8..af4651b 100644
--- a/fs/jbd2/checkpoint.c
+++ b/fs/jbd2/checkpoint.c
@@ -126,14 +126,29 @@ void __jbd2_log_wait_for_space(journal_t *journal)
 
 		/*
 		 * Test again, another process may have checkpointed while we
-		 * were waiting for the checkpoint lock
+		 * were waiting for the checkpoint lock. If there are no
+		 * outstanding transactions there is nothing to checkpoint and
+		 * we can't make progress. Abort the journal in this case.
 		 */
 		spin_lock(&journal->j_state_lock);
+		spin_lock(&journal->j_list_lock);
 		nblocks = jbd_space_needed(journal);
 		if (__jbd2_log_space_left(journal) < nblocks) {
+			int chkpt = journal->j_checkpoint_transactions != NULL;
+
+			spin_unlock(&journal->j_list_lock);
 			spin_unlock(&journal->j_state_lock);
-			jbd2_log_do_checkpoint(journal);
+			if (chkpt) {
+				jbd2_log_do_checkpoint(journal);
+			} else {
+				printk(KERN_ERR "%s: no transactions\n",
+				       __func__);
+				jbd2_journal_abort(journal, 0);
+			}
+
 			spin_lock(&journal->j_state_lock);
+		} else {
+			spin_unlock(&journal->j_list_lock);
 		}
 		mutex_unlock(&journal->j_checkpoint_mutex);
 	}

Re: [PATCH] jbd: abort instead of waiting for nonexistent transactions

[Andrew Morton]

On Tue,  5 Aug 2008 00:51:34 +0100 "Duane Griffin" <duaneg@dghda.com> wrote:

> The __log_wait_for_space function sits in a loop checkpointing transactions
> until there is sufficient space free in the journal. However, if there are
> no transactions to be processed (e.g. because the free space calculation is
> wrong due to a corrupted filesystem) it will never progress.
> 
> Check for space being required when no transactions are outstanding and
> abort the journal instead of endlessly looping.
> 
> This patch fixes the bug reported by Sami Liedes at:
> http://bugzilla.kernel.org/show_bug.cgi?id=10976
> 
> Signed-off-by: Duane Griffin <duaneg@dghda.com>
> Tested-by: Sami Liedes <sliedes@cc.hut.fi>
> ---
> diff --git a/fs/jbd/checkpoint.c b/fs/jbd/checkpoint.c
> index a5432bb..9fac177 100644
> --- a/fs/jbd/checkpoint.c
> +++ b/fs/jbd/checkpoint.c
> @@ -126,14 +126,29 @@ void __log_wait_for_space(journal_t *journal)
>  
>  		/*
>  		 * Test again, another process may have checkpointed while we
> -		 * were waiting for the checkpoint lock
> +		 * were waiting for the checkpoint lock. If there are no
> +		 * outstanding transactions there is nothing to checkpoint and
> +		 * we can't make progress. Abort the journal in this case.
>  		 */
>  		spin_lock(&journal->j_state_lock);
> +		spin_lock(&journal->j_list_lock);
>  		nblocks = jbd_space_needed(journal);
>  		if (__log_space_left(journal) < nblocks) {
> +			int chkpt = journal->j_checkpoint_transactions != NULL;
> +
> +			spin_unlock(&journal->j_list_lock);
>  			spin_unlock(&journal->j_state_lock);
> -			log_do_checkpoint(journal);
> +			if (chkpt) {
> +				log_do_checkpoint(journal);
> +			} else {
> +				printk(KERN_ERR "%s: no transactions\n",
> +				       __func__);
> +				journal_abort(journal, 0);
> +			}
> +
>  			spin_lock(&journal->j_state_lock);
> +		} else {
> +			spin_unlock(&journal->j_list_lock);
>  		}
>  		mutex_unlock(&journal->j_checkpoint_mutex);
>  	}

I don't expect that the additional taking of j_list_lock in here does
anything useful.

Plus..  after j_list_lock has been dropped, new transactions could
theoretically appear at journal->j_checkpoint_transactions, so we
_could_ reclaim more journal space.  But a) that probably couldn't
happen due to ->j_state_lock and lots of other things and b) it's
hopelessly theoretical even if it _could_ happen, methinks.  Just
sayin'..

Re: [PATCH] jbd: abort instead of waiting for nonexistent transactions

[Duane Griffin]

2008/8/5 Andrew Morton <akpm@linux-foundation.org>:
> On Tue,  5 Aug 2008 00:51:34 +0100 "Duane Griffin" <duaneg@dghda.com> wrote:
>
>> The __log_wait_for_space function sits in a loop checkpointing transactions
>> until there is sufficient space free in the journal. However, if there are
>> no transactions to be processed (e.g. because the free space calculation is
>> wrong due to a corrupted filesystem) it will never progress.
>>
>> Check for space being required when no transactions are outstanding and
>> abort the journal instead of endlessly looping.
>>
>> This patch fixes the bug reported by Sami Liedes at:
>> http://bugzilla.kernel.org/show_bug.cgi?id=10976
>>
>> Signed-off-by: Duane Griffin <duaneg@dghda.com>
>> Tested-by: Sami Liedes <sliedes@cc.hut.fi>
>> ---
>> diff --git a/fs/jbd/checkpoint.c b/fs/jbd/checkpoint.c
>> index a5432bb..9fac177 100644
>> --- a/fs/jbd/checkpoint.c
>> +++ b/fs/jbd/checkpoint.c
>> @@ -126,14 +126,29 @@ void __log_wait_for_space(journal_t *journal)
>>
>>               /*
>>                * Test again, another process may have checkpointed while we
>> -              * were waiting for the checkpoint lock
>> +              * were waiting for the checkpoint lock. If there are no
>> +              * outstanding transactions there is nothing to checkpoint and
>> +              * we can't make progress. Abort the journal in this case.
>>                */
>>               spin_lock(&journal->j_state_lock);
>> +             spin_lock(&journal->j_list_lock);
>>               nblocks = jbd_space_needed(journal);
>>               if (__log_space_left(journal) < nblocks) {
>> +                     int chkpt = journal->j_checkpoint_transactions != NULL;
>> +
>> +                     spin_unlock(&journal->j_list_lock);
>>                       spin_unlock(&journal->j_state_lock);
>> -                     log_do_checkpoint(journal);
>> +                     if (chkpt) {
>> +                             log_do_checkpoint(journal);
>> +                     } else {
>> +                             printk(KERN_ERR "%s: no transactions\n",
>> +                                    __func__);
>> +                             journal_abort(journal, 0);
>> +                     }
>> +
>>                       spin_lock(&journal->j_state_lock);
>> +             } else {
>> +                     spin_unlock(&journal->j_list_lock);
>>               }
>>               mutex_unlock(&journal->j_checkpoint_mutex);
>>       }
>
> I don't expect that the additional taking of j_list_lock in here does
> anything useful.
>
> Plus..  after j_list_lock has been dropped, new transactions could
> theoretically appear at journal->j_checkpoint_transactions, so we
> _could_ reclaim more journal space.  But a) that probably couldn't
> happen due to ->j_state_lock and lots of other things and b) it's
> hopelessly theoretical even if it _could_ happen, methinks.  Just
> sayin'..

Fair enough. I was just trying to be extra careful in taking the lock,
so I'm happy to drop it if you think it is safe. It will simplify the
patch significantly.

Cheers,
Duane.

-- 
"I never could learn to drink that blood and call it wine" - Bob Dylan

Re: [PATCH] jbd: abort instead of waiting for nonexistent transactions

[Stephen C. Tweedie]

Hi,

On Tue, 2008-08-05 at 00:51 +0100, Duane Griffin wrote:
> The __log_wait_for_space function sits in a loop checkpointing transactions
> until there is sufficient space free in the journal. However, if there are
> no transactions to be processed (e.g. because the free space calculation is
> wrong due to a corrupted filesystem) it will never progress.
> 
> Check for space being required when no transactions are outstanding and
> abort the journal instead of endlessly looping.

I'm not sure this is the right fix --- it seems like we're fixing the
symptoms, not the problem.

The journal free space fields are reset in journal_reset() when we load
the journal, so we can't get this situation of j_free being insufficient
on an idle filesystem unless the main journal start/end pointers are
corrupt.

Surely we'd be better off detecting this in the first place at mount
time, not later on during checkpoint?

Cheers,
 Stephen

Re: [PATCH] jbd: abort instead of waiting for nonexistent transactions

[Duane Griffin]

2008/8/5 Stephen C. Tweedie <sct@redhat.com>:
> On Tue, 2008-08-05 at 00:51 +0100, Duane Griffin wrote:
>> The __log_wait_for_space function sits in a loop checkpointing transactions
>> until there is sufficient space free in the journal. However, if there are
>> no transactions to be processed (e.g. because the free space calculation is
>> wrong due to a corrupted filesystem) it will never progress.
>>
>> Check for space being required when no transactions are outstanding and
>> abort the journal instead of endlessly looping.
>
> I'm not sure this is the right fix --- it seems like we're fixing the
> symptoms, not the problem.
>
> The journal free space fields are reset in journal_reset() when we load
> the journal, so we can't get this situation of j_free being insufficient
> on an idle filesystem unless the main journal start/end pointers are
> corrupt.
>
> Surely we'd be better off detecting this in the first place at mount
> time, not later on during checkpoint?

Sounds sensible. In fact I've got another patch, waiting for feedback
from the reporter, that adds some very basic validation there (i.e.
first > 0 && last >= first). Not enough, I suspect. I guess we could
do much better?

> Cheers,
>  Stephen

Cheers,
Duane.

-- 
"I never could learn to drink that blood and call it wine" - Bob Dylan

Re: [PATCH] jbd: abort instead of waiting for nonexistent transactions

[Stephen C. Tweedie]

Hi,

On Thu, 2008-08-07 at 01:47 +0100, Duane Griffin wrote:

> > Surely we'd be better off detecting this in the first place at mount
> > time, not later on during checkpoint?
> 
> Sounds sensible. In fact I've got another patch, waiting for feedback
> from the reporter, that adds some very basic validation there (i.e.
> first > 0 && last >= first). Not enough, I suspect. I guess we could
> do much better?

Right: in journal.c we initialise the maximum size of a transaction to

	journal->j_max_transaction_buffers = journal->j_maxlen / 4;

(the logic being that we need the journal to be able to hold an absolute
minimum of one full transaction being checkpointed, one being committed,
and one being live concurrently for the transaction engine to work
correctly, which gives three outstanding transactions; we up that to
four to protect against rounding errors and to ensure space for the
sequence and commit blocks that take up log space in addition to the
journaled buffers themselves.)

If, during journal load, that's not enough for a minimum-sized single
update, we'll never be able to start some transactions, so that would be
a good place to check that we're starting off with a large enough
journal.

--Stephen

[PATCH] jbd: abort instead of waiting for nonexistent transactions

[Duane Griffin]

The __log_wait_for_space function sits in a loop checkpointing transactions
until there is sufficient space free in the journal. However, if there are
no transactions to be processed (e.g. because the free space calculation is
wrong due to a corrupted filesystem) it will never progress.

Check for space being required when no transactions are outstanding and
abort the journal instead of endlessly looping.

This patch fixes the bug reported by Sami Liedes at:
http://bugzilla.kernel.org/show_bug.cgi?id=10976

Signed-off-by: Duane Griffin <duaneg@dghda.com>
---
diff --git a/fs/jbd/checkpoint.c b/fs/jbd/checkpoint.c
index a5432bb..af2b554 100644
--- a/fs/jbd/checkpoint.c
+++ b/fs/jbd/checkpoint.c
@@ -126,13 +126,24 @@ void __log_wait_for_space(journal_t *journal)
 
 		/*
 		 * Test again, another process may have checkpointed while we
-		 * were waiting for the checkpoint lock
+		 * were waiting for the checkpoint lock. If there are no
+		 * outstanding transactions there is nothing to checkpoint and
+		 * we can't make progress. Abort the journal in this case.
 		 */
 		spin_lock(&journal->j_state_lock);
 		nblocks = jbd_space_needed(journal);
 		if (__log_space_left(journal) < nblocks) {
+			int chkpt = journal->j_checkpoint_transactions != NULL;
+
 			spin_unlock(&journal->j_state_lock);
-			log_do_checkpoint(journal);
+			if (chkpt) {
+				log_do_checkpoint(journal);
+			} else {
+				printk(KERN_ERR "%s: no transactions\n",
+				       __func__);
+				journal_abort(journal, 0);
+			}
+
 			spin_lock(&journal->j_state_lock);
 		}
 		mutex_unlock(&journal->j_checkpoint_mutex);

Re: [PATCH] jbd: abort instead of waiting for nonexistent transactions

[Andrew Morton]

On Tue,  5 Aug 2008 02:05:20 +0100 "Duane Griffin" <duaneg@dghda.com> wrote:

> The __log_wait_for_space function sits in a loop checkpointing transactions
> until there is sufficient space free in the journal. However, if there are
> no transactions to be processed (e.g. because the free space calculation is
> wrong due to a corrupted filesystem) it will never progress.
> 
> Check for space being required when no transactions are outstanding and
> abort the journal instead of endlessly looping.
> 
> This patch fixes the bug reported by Sami Liedes at:
> http://bugzilla.kernel.org/show_bug.cgi?id=10976
> 
> Signed-off-by: Duane Griffin <duaneg@dghda.com>
> ---
> diff --git a/fs/jbd/checkpoint.c b/fs/jbd/checkpoint.c
> index a5432bb..af2b554 100644
> --- a/fs/jbd/checkpoint.c
> +++ b/fs/jbd/checkpoint.c
> @@ -126,13 +126,24 @@ void __log_wait_for_space(journal_t *journal)
>  
>  		/*
>  		 * Test again, another process may have checkpointed while we
> -		 * were waiting for the checkpoint lock
> +		 * were waiting for the checkpoint lock. If there are no
> +		 * outstanding transactions there is nothing to checkpoint and
> +		 * we can't make progress. Abort the journal in this case.
>  		 */
>  		spin_lock(&journal->j_state_lock);
>  		nblocks = jbd_space_needed(journal);
>  		if (__log_space_left(journal) < nblocks) {
> +			int chkpt = journal->j_checkpoint_transactions != NULL;
> +
>  			spin_unlock(&journal->j_state_lock);
> -			log_do_checkpoint(journal);
> +			if (chkpt) {
> +				log_do_checkpoint(journal);
> +			} else {
> +				printk(KERN_ERR "%s: no transactions\n",
> +				       __func__);
> +				journal_abort(journal, 0);
> +			}
> +
>  			spin_lock(&journal->j_state_lock);
>  		}
>  		mutex_unlock(&journal->j_checkpoint_mutex);

umm, OK, but...

There's not really a lot of point in testing
journal->j_checkpoint_transactions inside j_state_lock, is there? 
Hence local variable chkpt isn't really needed.

But log_do_checkpoint() already checks to see if there are any
checkpointing transactions upon which to operate, so rather than doing
log_do_checkpoint()'s work for it, perhaps it would be cleaner to teach
log_do_checkpoint() to tell the caller whether it manage to do any
work?

The nice thing about that is that even if
journal->j_checkpoint_transactions is NULL, log_do_checkpoint() might
still be able to do some useful work in cleanup_journal_tail().

otoh, two existing callers of log_do_checkpoint() already test
journal->j_checkpoint_transactions before calling log_do_checkpoint(),
so maybe that's pretty pointless.

otoh2, those existing callers do the seemingly-unneeded
spin_lock(j_list_lock).  hrm.  So if we're playing match-the-existing
code, we should go with your first patches.

ho hum, I guess I'll do "otoh2".