an infinite loop in ext4 in 3.14

an infinite loop in ext4 in 3.14

[Mikulas Patocka]

Hi

I hit a bug in ext4 - jbd2 was stuck in an infinite loop when remounting 
the root filesystem read-only during shutdown.

The filesystem is ext3, but it uses the ext4 driver with the following 
options: rw,relatime,discard,errors=remount-ro,data=ordered

The machine was stuck, but it was possible to obtain a stacktrace with 
Alt-SysRQ-P. I put several stacktraces here:

http://people.redhat.com/~mpatocka/crashes/ext4/

The stacktraces change in jbd2_log_do_checkpoint and 
jbd2_cleanup_journal_tail. jbd2_log_do_checkpoint doesn't call 
jbd2_cleanup_journal_tail from a loop, so the the probable location of the 
infinite loop was this piece of code in jbd2_journal_flush:

while (!err && journal->j_checkpoint_transactions != NULL) {

Mikulas

Re: an infinite loop in ext4 in 3.14

[Theodore Ts'o]

On Thu, Apr 17, 2014 at 03:23:13PM -0400, Mikulas Patocka wrote:
> 
> I hit a bug in ext4 - jbd2 was stuck in an infinite loop when remounting 
> the root filesystem read-only during shutdown.

Is this at all repeatable?  I suspect what happened is that we're not
checking the error return from jbd2_log_do_checkpoint(), and if it ran
into an error doing the jbd2_log_do_checkpoint --- for example, if it
wasn't able to write to the journal --- say, because __wait_cp_io()
returned -EIO, we might be spinning in the while loop in jbd2_journal_flush:

> while (!err && journal->j_checkpoint_transactions != NULL) {


(as you suspected).

I can add some error checking, but it would be interesting to know if
you can easily reproduce the problem so we can confirm if that's what
was really going on.

Regards,

						- Ted

Re: an infinite loop in ext4 in 3.14

[Mikulas Patocka]

On Thu, 17 Apr 2014, Theodore Ts'o wrote:

> On Thu, Apr 17, 2014 at 03:23:13PM -0400, Mikulas Patocka wrote:
> > 
> > I hit a bug in ext4 - jbd2 was stuck in an infinite loop when remounting 
> > the root filesystem read-only during shutdown.
> 
> Is this at all repeatable?

No - it happened just once.

> I suspect what happened is that we're not
> checking the error return from jbd2_log_do_checkpoint(), and if it ran
> into an error doing the jbd2_log_do_checkpoint --- for example, if it

There were no I/O errors on the console when the lockup happened.

> wasn't able to write to the journal --- say, because __wait_cp_io()
> returned -EIO, we might be spinning in the while loop in jbd2_journal_flush:
> 
> > while (!err && journal->j_checkpoint_transactions != NULL) {
> 
> 
> (as you suspected).
> 
> I can add some error checking, but it would be interesting to know if
> you can easily reproduce the problem so we can confirm if that's what
> was really going on.

I can write a script that reboots the machine and run it overnight...

> Regards,
> 
> 						- Ted

Mikulas

Re: an infinite loop in ext4 in 3.14

[Mikulas Patocka]

On Thu, 17 Apr 2014, Theodore Ts'o wrote:

> On Thu, Apr 17, 2014 at 03:23:13PM -0400, Mikulas Patocka wrote:
> > 
> > I hit a bug in ext4 - jbd2 was stuck in an infinite loop when remounting 
> > the root filesystem read-only during shutdown.
> 
> Is this at all repeatable?  I suspect what happened is that we're not
> checking the error return from jbd2_log_do_checkpoint(), and if it ran
> into an error doing the jbd2_log_do_checkpoint --- for example, if it
> wasn't able to write to the journal --- say, because __wait_cp_io()
> returned -EIO, we might be spinning in the while loop in jbd2_journal_flush:
> 
> > while (!err && journal->j_checkpoint_transactions != NULL) {
> 
> 
> (as you suspected).
> 
> I can add some error checking, but it would be interesting to know if
> you can easily reproduce the problem so we can confirm if that's what
> was really going on.
> 
> Regards,
> 
> 						- Ted

Hi

It turned out that the computer had bad memory, there were other stability 
issues - so you can ignore this bug report.

Mikulas