[Bug 12543] New: ext4_da_writepages error in 2.6.28.1 after a disk error

[Bug 12543] New: ext4_da_writepages error in 2.6.28.1 after a disk error

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=12543

           Summary: ext4_da_writepages error in 2.6.28.1 after a disk error
           Product: File System
           Version: 2.5
     KernelVersion: 2.6.28.1
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: ext4
        AssignedTo: fs_ext4@kernel-bugs.osdl.org
        ReportedBy: abaptist@cleversafe.com


Distribution: 
custom stripped down kernel parameters

Hardware Environment: 
4 disk SATA system -

Software Environment:
Problem Description:
While running tests over the weekend one of the drives reported errors "Buffer
I/O error on device". After this error, a series of "emerg" level stack traces
were printed with the following stack:
Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: Pid: 13486, comm:
pdflush Not tainted 2.6.28.1.cs.1 #1

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: Call Trace:

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c10e9168>]
ext4_da_writepages+0x2d8/0x310

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c12d5b0b>]
schedule+0x22b/0x880

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c10626ab>]
do_writepages+0x2b/0x50

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c109ccb9>]
__writeback_single_inode+0x89/0x300

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c109d2ee>]
generic_sync_sb_inodes+0x1ce/0x2b0

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c109d738>]
writeback_inodes+0x88/0xb0

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c10630f4>]
wb_kupdate+0x84/0xf0

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c1063520>]
pdflush+0x0/0x1a0

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c106360b>]
pdflush+0xeb/0x1a0

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c1063070>]
wb_kupdate+0x0/0xf0

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c1039b42>]
kthread+0x42/0x70

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c1039b00>]
kthread+0x0/0x70

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c1004aab>]
kernel_thread_helper+0x7/0x1c

Steps to reproduce:
The syslog before these errors started was the following:
Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: Buffer I/O error on
device sdd1, logical block 121667584

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: lost page write due
to I/O error on sdd1

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: ext4_abort called.

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: EXT4-fs error
(device sdd1): ext4_journal_start_sb: Detected aborted journal

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: Remounting
filesystem read-only

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: EXT4-fs error
(device sdd1) in ext4_delete_inode: Journal has aborted

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: EXT4-fs error
(device sdd1) in ext4_create: IO failure

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: JBD2: I/O error
detected when updating journal superblock for sdd1:8.

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: lost page write due
to I/O error on sdd1

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: sd 3:0:0:0: [sdd]
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: end_request: I/O
error, dev sdd, sector 26623


One note is that the filesystem was NOT remounted read-only even though it
claimed to be (at least by looking at mount).


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 12543] ext4_da_writepages error in 2.6.28.1 after a disk error

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=12543


DiegoCG@teleline.es changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |DiegoCG@teleline.es




------- Comment #1 from DiegoCG@teleline.es  2009-01-27 10:19 -------
IMHO I'd say there's a broken sector, have you passed a badblocks test? (but
ext4 should have handled the failure gracefully)


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 12543] ext4_da_writepages error in 2.6.28.1 after a disk error

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=12543





------- Comment #2 from abaptist@cleversafe.com  2009-01-27 10:26 -------
The disk was failing, really the purpose of this bug was just to have ext4
handle this gracefully. The continuous emerg level messages printed to console
(about 2/sec) make it very hard to work with (at least with default of emerg
going to *)


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 12543] ext4_da_writepages error in 2.6.28.1 after a disk error

[bugme-daemon]

http://bugzilla.kernel.org/show_bug.cgi?id=12543


tytso@mit.edu changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |CLOSED
         Resolution|                            |PATCH_ALREADY_AVAILABLE




------- Comment #3 from tytso@mit.edu  2009-01-27 11:08 -------
Having ext4 handling this gracefully is already in mainline.   See commit:

2a21e37e4: ext4: tone down ext4_da_writepages warnings


    ext4: tone down ext4_da_writepages warnings

    If the filesystem has errors, ext4_da_writepages() will return a *lot*
    of errors, including lots and lots of stack dumps.  While it's true
    that we are dropping user data on the floor, which is unfortunate, the
    stack dumps aren't helpful, and they tend to obscure the true original
    root cause of the problem.  So in the case where the filesystem has
    aborted, return an EROFS right away.

    Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>

It is in 2.6.29-rc1 and I plan to send it for the 2.6.28.y release, just
because it is *so* annoying.


-- 
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.