From: Nathaniel W Filardo <nwf@cs.jhu.edu>
To: "Theodore Ts'o" <tytso@mit.edu>
Cc: Mike Rubin <mrubin@google.com>, Frank Mayhar <fmayhar@google.com>,
admins@acm.jhu.edu, linux-ext4@vger.kernel.org
Subject: Re: ext4 metadata corruption bug?
Date: Sun, 20 Apr 2014 12:32:12 -0400 [thread overview]
Message-ID: <20140420163211.GT10985@gradx.cs.jhu.edu> (raw)
In-Reply-To: <20140410221702.GD31614@thunk.org>
[-- Attachment #1: Type: text/plain, Size: 2060 bytes --]
We just got
> [817576.492013] EXT4-fs (vdd): pa ffff88000dea9b90: logic 0, phys. 1934464544, len 32
> [817576.492468] EXT4-fs error (device vdd): ext4_mb_release_inode_pa:3729: group 59035, free 14, pa_free 12
> [817576.492987] Aborting journal on device vdd-8.
> [817576.493919] EXT4-fs (vdd): Remounting filesystem read-only
Upon unmount, further
> [825457.072206] EXT4-fs error (device vdd): ext4_put_super:791: Couldn't clean up the journal
fscking generated
> fsck from util-linux 2.20.1
> e2fsck 1.42.9 (4-Feb-2014)
> /dev/vdd: recovering journal
> /dev/vdd contains a file system with errors, check forced.
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> Block bitmap differences: +(1934464544--1934464545)
> Fix<y>? yes
> Free blocks count wrong (1379876836, counted=1386563079).
> Fix<y>? yes
> Free inodes count wrong (331897442, counted=331912336).
> Fix<y>? yes
>
> /dev/vdd: ***** FILE SYSTEM WAS MODIFIED *****
> /dev/vdd: 3631984/335544320 files (1.6% non-contiguous), 1297791481/2684354560 blocks
The particular error reported by the kernel seems to be the first of the
three, but the other two look like leaks? A huge number of inodes (14894)
and blocks (6686243, or 3.2Gi of storage!) were marked busy in a way that
fsck didn't believe, if I am reading that right?
/dev/vdd is virtio on Ceph RBD, using write-through caching. We have had a
crash on one of the Ceph OSDs recently in a way that seems to have generated
inconsistent data in Ceph, but subsequent repair commands seem to have made
everything happy again, at least so far as Ceph tells us.
The guest `uname -a` sayeth
> Linux afsscratch-kvm 3.13-1-amd64 #1 SMP Debian 3.13.7-1 (2014-03-25) x86_64 GNU/Linux
And in case it's relevant, host QEMU emulator is version 1.7.0 (Debian
1.7.0+dfsg-3) [modified locally to include rbd]; guest ceph, librbd, etc.
are Debian package 0.72.2-1~bpo70+1 .
Cheers,
--nwf;
[-- Attachment #2: Type: application/pgp-signature, Size: 181 bytes --]
next prev parent reply other threads:[~2014-04-20 16:32 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20140409223820.GU10985@gradx.cs.jhu.edu>
[not found] ` <CAGagf4eEzY4+3cfNWSEENTo1PKe40nq1Ne6ZzOLGm-O78W7RcA@mail.gmail.com>
2014-04-10 5:04 ` ext4 metadata corruption bug? Nathaniel W Filardo
2014-04-10 14:03 ` Theodore Ts'o
2014-04-10 16:33 ` Nathaniel W Filardo
2014-04-10 22:17 ` Theodore Ts'o
2014-04-20 16:32 ` Nathaniel W Filardo [this message]
2014-04-20 17:57 ` Theodore Ts'o
2014-04-23 7:23 ` Sander Smeenk
2014-04-23 14:36 ` Theodore Ts'o
2014-04-23 15:30 ` Nathaniel W Filardo
2014-04-23 18:05 ` Sander Smeenk
2014-04-29 15:22 ` Nathaniel W Filardo
2014-05-01 16:25 ` Nathaniel W Filardo
2014-05-06 15:42 ` Theodore Ts'o
2014-05-06 15:51 ` Nathaniel W Filardo
2014-07-31 2:37 ` Theodore Ts'o
2014-08-06 8:53 ` Sander Smeenk
2014-05-01 17:02 ` Sander Smeenk
2014-05-06 14:22 ` Sander Smeenk
2014-05-26 14:59 ` Sander Smeenk
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140420163211.GT10985@gradx.cs.jhu.edu \
--to=nwf@cs.jhu.edu \
--cc=admins@acm.jhu.edu \
--cc=fmayhar@google.com \
--cc=linux-ext4@vger.kernel.org \
--cc=mrubin@google.com \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).