[Bug 61631] New: kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[Bug 61631] New: kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

            Bug ID: 61631
           Summary: kernel BUG at fs/ext4/super.c:818 umounting md raid6
                    volume
           Product: File System
           Version: 2.5
    Kernel Version: 3.11.1
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: ext4
          Assignee: fs_ext4@kernel-bugs.osdl.org
          Reporter: jfaulkne@ccs.neu.edu
        Regression: No

Under recent kernels, including 3.11.1, I've been seeing kernel BUGs when
umounting a large md raid6 volume.  umount itself gives me a segfault.  I will
attach my config.gz and dmesg, which includes the BUG message.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #1 from Jim Faulkner <jfaulkne@ccs.neu.edu> ---
Created attachment 108831
  --> https://bugzilla.kernel.org/attachment.cgi?id=108831&action=edit
3.11.1 .config

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #2 from Jim Faulkner <jfaulkne@ccs.neu.edu> ---
Created attachment 108841
  --> https://bugzilla.kernel.org/attachment.cgi?id=108841&action=edit
dmesg including BUG message

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #3 from Jim Faulkner <jfaulkne@ccs.neu.edu> ---
I've done a full fsck -f -v -y on this filesystem using e2fsck 1.42.7
(21-Jan-2013).  umounts still result in a BUG.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #4 from Jim Faulkner <jfaulkne@ccs.neu.edu> ---
FYI, this bug only appears after some period of usage.  After a reboot, I can
umount without error.  After a reboot and an rsync of the gentoo portage tree
to the filesystem, I can still umount without error.  It seems the filesystem
only fails to unmount after the filesystem has been in use for a while. 
However, it is quite consistent in failing to umount after several days of
usage.

Some more details on this filesystem:

thud ~ # df -h  | grep md1
/dev/md1        7.2T  6.7T  488G  94% /nfs
thud ~ # df -i | grep md1
/dev/md1       484597760 41868439 442729321    9% /nfs
thud ~ # tune2fs -l /dev/md1
tune2fs 1.42.7 (21-Jan-2013)
Filesystem volume name:   <none>
Last mounted on:          /nfs
Filesystem UUID:          a9f907fb-4d04-4a28-8164-a2ac4d5a26f4
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype
needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg
dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Remount read-only
Filesystem OS type:       Linux
Inode count:              484597760
Block count:              1938377664
Reserved block count:     0
Free blocks:              127846725
Free inodes:              442729321
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      561
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Sat Oct  9 08:53:46 2010
Last mount time:          Wed Sep 18 18:25:38 2013
Last write time:          Wed Sep 18 18:25:38 2013
Mount count:              4
Maximum mount count:      23
Last checked:             Wed Sep 11 04:36:58 2013
Check interval:           15552000 (6 months)
Next check after:         Mon Mar 10 04:36:58 2014
Lifetime writes:          18 TB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      1584912d-08c0-46a6-a652-60800c55c4df
Journal backup:           inode blocks
thud ~ #

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

Re: [Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[Theodore Ts'o]

On Wed, Sep 18, 2013 at 10:27:02PM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> FYI, this bug only appears after some period of usage.  After a reboot, I can
> umount without error.  After a reboot and an rsync of the gentoo portage tree
> to the filesystem, I can still umount without error.  It seems the filesystem
> only fails to unmount after the filesystem has been in use for a while. 
> However, it is quite consistent in failing to umount after several days of
> usage.

Hmm, can you say a bit more about what sort of files you store on the
file system and how the file system gets used?  What looks like is
going on is that there is a whole series of inodes that have been left
stalled on the orpaned inode list.  By the time we reach that point in
the unmount, the in-memory orphan list should have been cleared.

So here are a couple of things that would be really useful to try.

First of all, if you could try to reproduce the crash, and then before
you do the umount, run "dumpe2fs -h /dev/md1 > ~/dumpe2fs.md1.save;
sync".  Then if the system crashes with the same BUG_ON, send us the
dumpe2fs.md1.save, along with the console output.

The thing which I am trying to determine is whether the on-disk
orphaned inode list is set at the time of the umount.  If it is set,
it would be interesting if you could run sync, wait for things to
settle, check to see if dumpe2fs shows that the orphaned list is
empty, and then see if you can trigger the crash.

The other thing that would be useful is to grab one of the inodes
listed in the console, i.e.:

[642473.269223]   inode md1:20289506 at ffff88000499aed0: mode 100644, nlink 0, next 20367069

... and then run the command: "debugfs -R 'stat <20289506>' /dev/md1"

What I am interested in is the inode's atime/ctime/dtime.  It would be
interesting to see if the file was deleted right before the umount was
attempted.

Thanks for the bug report!  Hopefully we'll be able to figure out why
you are seeing this.

Cheers,

					- Ted

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

Jim Faulkner <jfaulkne@ccs.neu.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Regression|No                          |Yes

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #5 from Theodore Tso <tytso@mit.edu> ---
On Wed, Sep 18, 2013 at 10:27:02PM +0000, bugzilla-daemon@bugzilla.kernel.org
wrote:
> FYI, this bug only appears after some period of usage.  After a reboot, I can
> umount without error.  After a reboot and an rsync of the gentoo portage tree
> to the filesystem, I can still umount without error.  It seems the filesystem
> only fails to unmount after the filesystem has been in use for a while. 
> However, it is quite consistent in failing to umount after several days of
> usage.

Hmm, can you say a bit more about what sort of files you store on the
file system and how the file system gets used?  What looks like is
going on is that there is a whole series of inodes that have been left
stalled on the orpaned inode list.  By the time we reach that point in
the unmount, the in-memory orphan list should have been cleared.

So here are a couple of things that would be really useful to try.

First of all, if you could try to reproduce the crash, and then before
you do the umount, run "dumpe2fs -h /dev/md1 > ~/dumpe2fs.md1.save;
sync".  Then if the system crashes with the same BUG_ON, send us the
dumpe2fs.md1.save, along with the console output.

The thing which I am trying to determine is whether the on-disk
orphaned inode list is set at the time of the umount.  If it is set,
it would be interesting if you could run sync, wait for things to
settle, check to see if dumpe2fs shows that the orphaned list is
empty, and then see if you can trigger the crash.

The other thing that would be useful is to grab one of the inodes
listed in the console, i.e.:

[642473.269223]   inode md1:20289506 at ffff88000499aed0: mode 100644, nlink 0,
next 20367069

... and then run the command: "debugfs -R 'stat <20289506>' /dev/md1"

What I am interested in is the inode's atime/ctime/dtime.  It would be
interesting to see if the file was deleted right before the umount was
attempted.

Thanks for the bug report!  Hopefully we'll be able to figure out why
you are seeing this.

Cheers,

                    - Ted

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

Jan Kara <jack@suse.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jack@suse.cz

--- Comment #6 from Jan Kara <jack@suse.cz> ---
Since the mountpoint is named /nfs, I suppose you are exporting the filesystem
via NFS, right? There has been a bug in NFS server code leading to exactly the
problems you are describing. Commit bf7bd3e98be5c74813bee6ad496139fb0a011b3b
should fix the issue (in 3.12-rc1).

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #7 from Jim Faulkner <jfaulkne@ccs.neu.edu> ---
(In reply to Theodore Tso from comment #5)

> Hmm, can you say a bit more about what sort of files you store on the
> file system and how the file system gets used?  What looks like is
> going on is that there is a whole series of inodes that have been left
> stalled on the orpaned inode list.  By the time we reach that point in
> the unmount, the in-memory orphan list should have been cleared.

This filesystem holds a wide variety of files.  A fair amount (and the majority
of the disk usage), is large mp3 (~5 mb) and video (500 mb to 4 gb) files. 
However, most I/O happens on small files.  It hosts the gentoo portage tree
which I update regularly, as well as regular rdiff-backups of other servers on
my network.  Both of these are involve a lot of reading and modifying of lots
of small files.

(In reply to Jan Kara from comment #6)
> Since the mountpoint is named /nfs, I suppose you are exporting the
> filesystem via NFS, right? There has been a bug in NFS server code leading
> to exactly the problems you are describing. Commit
> bf7bd3e98be5c74813bee6ad496139fb0a011b3b should fix the issue (in 3.12-rc1).

Yes, I serve large video and mp3 files over NFS.  But in addition, the gentoo
portage tree on this filesystem is NFS exported to all gentoo hosts on my
network.  Any gentoo portage operations (emerge --metadata as well as updates)
results in lots of NFS I/O on lots of small files.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #8 from Jim Faulkner <jfaulkne@ccs.neu.edu> ---
I suspect commit bf7bd3e98be5c74813bee6ad496139fb0a011b3b will indeed fix this
issue since it sounds exactly like my issue.

However, before applying any patch, I'll do the debug tests that Ted suggests
so we can be sure.  I need some uptime to be sure I can reproduce this bug, so
I'll report back a few days from now.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #9 from Jim Faulkner <jfaulkne@ccs.neu.edu> ---
I ran dumpe2fs -h followed by sync a few times, but was not able to trigger the
bug, nor did the dumpe2fs output differ between syncs.

umount did trigger the bug, however, and the dumpe2fs output was different
after the umount.

After triggering the bug, I was able to capture the debugfs output of one of
the listed inodes.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #10 from Jim Faulkner <jfaulkne@ccs.neu.edu> ---
Created attachment 110761
  --> https://bugzilla.kernel.org/attachment.cgi?id=110761&action=edit
dumpe2fs output before the umount and bug_on

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #11 from Jim Faulkner <jfaulkne@ccs.neu.edu> ---
Created attachment 110771
  --> https://bugzilla.kernel.org/attachment.cgi?id=110771&action=edit
dumpe2fs output after the umount and bug_on

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #12 from Jim Faulkner <jfaulkne@ccs.neu.edu> ---
Created attachment 110781
  --> https://bugzilla.kernel.org/attachment.cgi?id=110781&action=edit
dmesg output after umount and bug_on

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #13 from Jim Faulkner <jfaulkne@ccs.neu.edu> ---
Created attachment 110791
  --> https://bugzilla.kernel.org/attachment.cgi?id=110791&action=edit
debugfs -R 'stat <20035337>' /dev/md1 > debugfs.1 after umount/bug_on

Please let me know if there's anymore information I can provide.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

Jan Kara <jack@suse.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #110761|application/octet-stream    |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

Jan Kara <jack@suse.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #110771|application/octet-stream    |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

Jan Kara <jack@suse.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #110781|application/octet-stream    |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

Jan Kara <jack@suse.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #110791|application/octet-stream    |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #14 from Jan Kara <jack@suse.cz> ---
Umm, no, it really looks like a ton of inodes remains in the orphan list and I
really things this is the NFS bug. So just try a kernel with that fix
(bf7bd3e98be5c74813bee6ad496139fb0a011b3b) applied. Thanks!

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

--- Comment #15 from Jim Faulkner <jfaulkne@ccs.neu.edu> ---
Looks like bf7bd3e98be5c74813bee6ad496139fb0a011b3b indeed fixed it.  I ran
3.11.5 (which I believe includes that patch) for a few days, doing normal
activities such as nfs i/o and rdiff-backups.  I was able to umount without
problems.  Thanks for fixing this!

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 61631] kernel BUG at fs/ext4/super.c:818 umounting md raid6 volume

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=61631

Jan Kara <jack@suse.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |CODE_FIX

--- Comment #16 from Jan Kara <jack@suse.cz> ---
Thanks for confirmation.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.