Re: [BUG] ext3: cannot unfreeze a filesystem due to a deadlock

[Jan Kara <jack@suse.cz>]

On Wed 07-09-11 13:56:08, Greg Freemyer wrote:
> On Wed, Sep 7, 2011 at 1:34 PM, Jan Kara <jack@suse.cz> wrote:
> >  Hello,
> >
> >  Thanks for report!
> >
> > On Wed 07-09-11 12:29:30, Masayoshi MIZUMA wrote:
> >> When I checked the freeze feature for ext3 filesystem using fsfreeze
> >> command at 3.1.0-rc4, I think the following deadlock problem happened.
> >>
> >> How to reproduce:
> >>  # mkfs -t ext3 /dev/sdd1
> >>  # mount /dev/sdd1 /MNT
> >>  # ./fsstress -d /MNT/tmp -n 10 -p 1000 > /dev/null 2>&1 &
> >>  # fsfreeze -f /MNT
> >>  # fsfreeze -u /MNT
> >>
> >>  If this deadlock is reproduced, "fsfreeze -u /MNT" does not return.
> >>
> >> The detail of deadlock:
> >> o [flush-8:16:1523]
> >>   wb_do_writeback
> >>    wb_writeback
> >>    ...
> >>      ext3_journalled_writepage
> >>       journal_start
> >>        start_this_handle
> >>        # waiting until journal->j_barrier_count turns 0...
> >>        # j_barrier_count was incremented by journal_lock_updates()
> >>        # via ext3_freeze().
> >>
> >> o [fsstress:2673]
> >>   sys_sync
> >>    sync_filesystems
> >>     iterate_supers
> >>      down_read(sb->s_umount)
> >>      sync_one_sb
> >>       __sync_filesystem
> >>        writeback_inodes_sb
> >>         writeback_inodes_sb_nr
> >>          wait_for_completion
> >>           wait_for_common
> >>           # waiting for completion of [flush-8:16:1523]...
> >>
> >> o [fsfreeze:2749]
> >>   sys_ioctl
> >>    do_vfs_ioctl
> >>     thaw_super
> >>     # waiting for down_write(sb->s_umount)...
> >>     # [fsfreeze:2673] did down_read(sb->s_umount).
> >  Yes, this is a classical deadlock that can happen for any filesystem. The
> > problem is flusher thread holds s_umount semaphore (either directly, or as
> > in your case, indirectly via blocked sync) and tries to do some IO which
> > blocks on frozen filesystem. It's particularly easy to hit for ext3 because
> > it doesn't do vfs_check_frozen() checks but all other filesystems have the
> > race window as well. Val Henson is working on fixing the problem - she even
> > has some first version of patches I believe.
> >
> >                                                                Honza
> 
> xfstests test 068 has been around since kernel 2.4 days and should
> have caught it if xfs is impacted.
> 
> I know I ran the 2002 version many times to prove to myself that
> fsfreeze for xfs was stable when teamed with LVM.  (It wasn't when I
> first wrote 068 way back then).
> 
> 068 has been greatly simplified since 2002, but it still looks like it
> should do a good job.
> 
> Is there a problem with 068?  Does it need extra test coverage even for xfs?
  I believe at least mmapped writes can trigger the deadlock even for xfs
and fsstress (slightly surprisingly) does not test that. It's a narrow race
window but it is there and it has been triggered in practice (for ext4 but
it's a race in VFS code used by both XFS and ext4). So maybe extending
fsstress would be a way to go?

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html