* Fwd: e2fsck -fD corruption of large htree/extent directory
[not found] <D261A180.118DB8%andreas.dilger@intel.com>
@ 2015-11-06 8:23 ` Andreas Dilger
2015-11-11 10:13 ` Dilger, Andreas
1 sibling, 0 replies; 2+ messages in thread
From: Andreas Dilger @ 2015-11-06 8:23 UTC (permalink / raw)
To: Theodore Ts'o; +Cc: linux-ext4
[-- Attachment #1: Type: text/plain, Size: 6582 bytes --]
Ted, per our discussion this morning, here are the details of the
e2fsck -fD corruption problem we saw.
Running e2fsck -fD on a large extent+htree directory (> 300k entries,
1600+ filesystem blocks) showed corruption on a large number of dirs.
This is definitely caused by a bug in the code rather than hardware, as
this corrupted multiple large directories on 11 different systems.
Sometimes, similar directories on the same systems did not have errors.
As yet the reason and mechanism has not been determined, but it may
relate to the filesystem history (the directories may have originally
been block mapped, an in any case the blocks are mostly discontiguous
on disk). These dirs undergo continuous insertion and deletion of
entries with ~10-character filenames, so the leaf blocks may have become
quite fragmented over time.
Running e2fsck on the filesystem showed:
e2fsck 1.42.12.wc1 (15-Sep-2014)
MMP interval is 5 seconds and total wait time is 22 seconds. Please wait.
Pass 1: Checking inodes, blocks, and sizes
Interior extent node level 1 of inode 39321606:
Logical start 1430 does not match logical start 1875 at next level.
Fix? yes
Inode 39321606, end of extent exceeds allowed value
(logical block 1875, physical block 1258402260, len 1)
Clear? yes
Failed to iterate extents in inode 39321606
(op EXT2_EXTENT_UP, blk 1258402260, lblk 1875): No 'up' extent
Clear inode? yes
Inode 39321606 is a zero-length directory. Clear? yes
Update quota info for quota type 0? yes
Update quota info for quota type 1? yes
Restarting e2fsck from the beginning...
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'd2' in /O/0 (39321602) has deleted/unused inode 39321606.
Clear? yes
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Unattached inode 147
Connect to /lost+found? yes
Inode 147 ref count is 2, should be 1. Fix? yes
Unattached inode 173
Connect to /lost+found? yes
Inode 173 ref count is 2, should be 1. Fix? yes
:
:
Unattached inode 92016391
Connect to /lost+found? yes
Inode 92016391 ref count is 2, should be 1. Fix? yes
Pass 5: Checking group summary information
Block bitmap differences: -1258308100
Update quota info for quota type 0? yesm
Update quota info for quota type 1? yes
scratch-OST0049: ***** FILE SYSTEM WAS MODIFIED *****
Stat data for the corrupted directory inode:
debugfs -c -R "stat <39321606>"
Inode: 39321606 Type: directory Mode: 0700 Flags: 0x81000
Generation: 2310511783 Version: 0x00000000:00000000
User: 0 Group: 0 Size: 6750208
File ACL: 0 Directory ACL: 0
Links: 2 Blockcount: 13232
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x563111cf:15fb2694 -- Wed Oct 28 14:19:59 2015
atime: 0x52f30c97:9fe5c3ac -- Wed Feb 5 23:16:23 2014
mtime: 0x563111cf:15fb2694 -- Wed Oct 28 14:19:59 2015
crtime: 0x52f30c97:9fe5c3ac -- Wed Feb 5 23:16:23 2014
Size of extra inode fields: 28
Extended attributes stored in inode body:
invalid EA entry in inode
EXTENTS:
[shown below]
The debugfs dump_extents command shows that the extent tree is mostly OK.
In all observed cases, the extent tree was 5 blocks long (possibly a
result of 4 extent blocks being moved out of the in-inode i_block[]
array and into an external second-level index block), or because the
number of entries in each directory is roughly the same, not sure.
Level Entries Logical Physical Length Flags
0/ 2 1/ 1 0 - 1647 1258392344 1648
1/ 2 1/ 5 0 - 353 1258308301 354
2/ 2 1/340 0 - 0 1258308100 - 1258308100 1
2/ 2 2/340 1 - 2 1258308174 - 1258308175 2
2/ 2 3/340 3 - 3 1258308213 - 1258308213 1
2/ 2 4/340 4 - 4 1258308241 - 1258308241 1
:
:
2/ 2 339/340 352 - 352 1258319291 - 1258319291 1
2/ 2 340/340 353 - 353 1258319375 - 1258319375 1
1/ 2 2/ 5 354 - 704 1258319416 351
2/ 2 1/340 354 - 354 1258319415 - 1258319415 1
2/ 2 2/340 355 - 355 1258319470 - 1258319470 1
:
:
2/ 2 339/340 703 - 703 1258350886 - 1258350886 1
2/ 2 340/340 704 - 704 1258350895 - 1258350895 1
1/ 2 3/ 5 705 - 1055 1258350929 351
2/ 2 1/339 705 - 705 1258350928 - 1258350928 1
2/ 2 2/339 706 - 706 1258343948 - 1258343948 1
:
:
2/ 2 336/339 1052 - 1052 1258365348 - 1258365348 1
2/ 2 337/339 1053 - 1053 1258365355 - 1258365355 1
2/ 2 338/339 1054 - 1054 1258365417 - 1258365417 1
2/ 2 339/339 1055 - 1055 1258365432 - 1258365432 1
1/ 2 4/ 5 1056 - 1874 1258324458 819
2/ 2 1/340 1056 - 1056 1258365435 - 1258365435 1
2/ 2 2/340 1057 - 1057 1258366983 - 1258366983 1
2/ 2 3/340 1058 - 1059 1258366993 - 1258366994 2
:
:
2/ 2 338/340 1427 - 1427 1258379312 - 1258379312 1
2/ 2 339/340 1428 - 1428 1258379117 - 1258379117 1
2/ 2 340/340 1429 - 1429 1258379133 - 1258379133 1
1/ 2 5/ 5 1875 - 4294968943 1258406330 4294967069
2/ 2 1/ 1 1875 - 1875 1258402260 - 1258402260 1
The 4/5 extent index block shows an extent length of 1874 - 1056 = 819
blocks, but the extent block only has 1429 - 1056 = 373 blocks in the
extent. The extent root block reports 1648 blocks, which matches both
i_size and i_blocks. There appears to be one block missing from the
extent tree, or it was clobbered by 5/5 during an update, and/or the
starting offset of block 5/5 is just wrong.
There doesn't appear to be any other data corruption in the filesystem
besides the directory extent blocks, but this resulted in several
hundred leaf blocks being lost per directory, resulting in millions of
files in lost+found (see my other recent email on that topic).
In some cases, it appears that 100% of files were readable from the
corrupted directory using debugfs _before_ the e2fsck was run:
debugfs -c -R "ls -l $DIR" $DEV
even though e2fsck was unhappy with the extent structure and cleared
part of the extent tree and dumped the files into lost+found. This
implies that the directory entries were all moved into the first blocks
of the directory (i.e. leaf blocks under extent indices 1/5..4/5, and
the blocks in the corrupt part of the directory were somehow "extra" and
the bug lies in the extent handling when shrinking the directory.
Cheers, Andreas
[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 2+ messages in thread* Re: e2fsck -fD corruption of large htree/extent directory
[not found] <D261A180.118DB8%andreas.dilger@intel.com>
2015-11-06 8:23 ` Fwd: e2fsck -fD corruption of large htree/extent directory Andreas Dilger
@ 2015-11-11 10:13 ` Dilger, Andreas
1 sibling, 0 replies; 2+ messages in thread
From: Dilger, Andreas @ 2015-11-11 10:13 UTC (permalink / raw)
To: Theodore Ts'o; +Cc: linux-ext4@vger.kernel.org
[-- Attachment #1: Type: text/plain, Size: 4109 bytes --]
On 2015/11/06, 00:12, "Dilger, Andreas" <andreas.dilger@intel.com> wrote:
>Running e2fsck -fD on a large extent+htree directory (> 300k entries,
>1600+ filesystem blocks) may result in the directory becoming corrupted.
>This is definitely caused by a bug in the code rather than hardware, as
>this corrupted multiple large directories on different systems.
Thanks to a suggestion from Darrick, I was able to reproduce this problem
with an e2fsck test script (attached) when shrinking an htree extent
directory with only 3 index blocks referenced directly by the inode. The
problem is not present on block-mapped directories but looks to be a
danger for any user of the "-fD" option with extent-mapped directories.
It looks like the problem is if the inode shrinks enough that one of the
index blocks is dropped from the end of the file (blocks after logical
block 114 were freed), but the write_directory() write_dir_block()
iterator doesn't free the index block 800:
\x01
:
write_dir_block 113:583 - write
write_dir_block 114:587 - write
write_dir_block 115:591 - free
write_dir_block 116:595 - free
:
:
write_dir_block 165:791 - free
write_dir_block -1:800 - skip
write_dir_block 166:795 - free
write_dir_block 167:799 - free
write_dir_block 168:804 - free
write_dir_block 169:808 - free
write_dir_block 170:812 - free
write_dir_block 171:813 - free
write_dir_block 172:814 - free
write_dir_block -1:800 - skip
Pass 4: Checking reference counts
Pass 5: Checking group summary information
The extent tree now has a bogus index block at the end, but somehow is
also missing the valid extent block that was holding the rest of the
file, as shown by debugfs (after "e2fsck -fD" but before the second
e2fsck that detects the corruption) and logical blocks 83-114 are lost:
debugfs: stat subdir
Inode: 12 Type: directory Mode: 0755 Flags: 0x81000
Generation: 0 Version: 0x00000000
User: 0 Group: 0 Size: 117760
File ACL: 0 Directory ACL: 0
Links: 2 Blockcount: 238
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x5642e764 -- Tue Nov 10 23:59:48 2015
atime: 0x5642e764 -- Tue Nov 10 23:59:48 2015
mtime: 0x5642e764 -- Tue Nov 10 23:59:48 2015
EXTENTS:
(ETB0):146, (0):129, (1):133, (2):137, (3):141, (4):145, (5):150,
(6):154, (7):158, (8):162, (9):166, (10):170, (11):174, (12):178,
(13):182, (14):186, (15):190, (16):194, (17):198, (18):202,
(19):206, (20):210, (21):214, (22):218, (23):222, (24):226,
(25):230, (26):234, (27):238, (28):242, (29):246, (30):250,
(31):254, (32):258, (33):262, (34):266, (35):270, (36):274,
(37):278, (38):282, (39):286, (40):290, (41):294, (42):298,
(43):302, (44):306, (45):310, (46):314, (47):318, (48):322,
(49):326, (50):330, (51):334, (52):338, (53):342, (54):346,
(55):350, (56):354, (57):358, (58):362, (59):366, (60):370,
(61):374, (62):378, (63):382, (64):386, (65):390, (66):394,
(67):398, (68):402, (69):406, (70):410, (71):414, (72):418,
(73):422, (74):426, (75):430, (76):434, (77):438, (78):442,
(79):446, (80):450, (81):454, (82):458, (ETB0):800, (172):814
debugfs: extents subdir
:
:
1/ 1 82/ 83 81 - 81 454 - 454 1
1/ 1 83/ 83 82 - 82 458 - 458 1
0/ 1 2/ 2 170 - 4294967410 800 4294967241
1/ 1 1/ 1 172 - 172 814 - 814 1
The i_size is correct for 115 data blocks written, and i_blocks would
be correct if the second index block wouldn't have been lost. It seems
the bug is in the extent handling code, but I haven't yet dug into why
the last extent is kept. I tried deleting it like the other blocks,
but the iteration immediately stops with an error that the index block
is corrupted, and I'm not sure how to catch it the second time.
Cheers, Andreas
--
Andreas Dilger
Lustre Principal Engineer
Intel High Performance Data Division
[-- Attachment #2: script --]
[-- Type: application/octet-stream, Size: 1978 bytes --]
#!/bin/bash
TMP=${TMP:-"/tmp"}
test_name=${test_name:-$(basename $(dirname $0))}
test_dir=${test_dir:-$test_name}
cmd_dir=${cmd_dir:-"."}
OUT=$test_name.log
MKFS=${MKFS:-../misc/mke2fs}
FSCK=${FSCK:-../e2fsck/e2fsck}
DEBUGFS=${DEBUGFS:-../debugfs/debugfs}
# parameters for run_e2fsck
SKIP_GUNZIP="true"
FSCK_OPT="-fyvD"
NAMELEN=250
SRC=$TMP/$test_name.tmp
SUB=subdir
BASE=$SRC/$SUB/$(yes | tr -d '\n' | dd bs=$NAMELEN count=1 2> /dev/null)
TMPFILE=${TMPFILE:-"$TMP/image"}
BSIZE=1024
> $OUT
mkdir -p $SRC/$SUB
# calculate the number of files needed to create the directory extent tree
# deep enough to exceed the in-inode index and spill into an index block.
#
# dirents per block * extents per block * (index blocks > i_blocks)
NUM=$(((BSIZE / (NAMELEN + 8)) * (BSIZE / 12) * 2))
# Create source files. Unfortunately hard links will be copied as links,
# and blocks with only NULs will be turned into holes.
if [ ! -f $BASE.1 ]; then
for N in $(seq $NUM); do
echo "foo" > $BASE.$N
done >> $OUT
fi
# make filesystem with enough inodes and blocks to hold all the test files
> $TMPFILE
NUM=$((NUM * 5 / 3))
echo "mke2fs -b $BSIZE -O dir_index,extent -d$SRC -N$NUM $TMPFILE $NUM" >> $OUT
$MKFS -b $BSIZE -O dir_index,extent -d$SRC -N$NUM $TMPFILE $NUM >> $OUT 2>&1
rm -r $SRC
# Run e2fsck to convert dir to htree before deleting the files, as mke2fs
# doesn't do this. Run second e2fsck to verify there is no corruption yet.
(
EXP1=$test_dir/expect.pre.1
EXP2=$test_dir/expect.pre.2
OUT1=$test_name.pre.1.log
OUT2=$test_name.pre.2.log
DESCRIPTION="$(cat $test_dir/name) setup"
. $cmd_dir/run_e2fsck
)
# generate a list of filenames for debugfs to delete, one from each leaf block
DELETE_LIST=$TMP/delete.$$
$DEBUGFS -c -R "htree subdir" $TMPFILE 2>> $OUT |
grep -A2 "Reading directory block" |
awk '/yyyyy/ { print "rm '$SUB'/"$4 }' > $DELETE_LIST
$DEBUGFS -w -f $DELETE_LIST $TMPFILE >> $OUT 2>&1
rm $DELETE_LIST
cp $TMPFILE $TMPFILE.sav
. $cmd_dir/run_e2fsck
^ permalink raw reply [flat|nested] 2+ messages in thread