ext4 > 16T woes

ext4 > 16T woes

[Eric Sandeen]

I'll try to look into this but just wanted to put it out there.

After creating a 32T fs and filling it with 1T preallocated files,
I get quite a lot of corruption; this wasn't really the test
I wanted to do, I wanted to then remove a file with mostly
high blocks, run fsstress, and test recovery :(

# uname -r
3.4.0-rc4+
# truncate --size 32t imagefile2
# mkfs.ext4 -F imagefile2
mke2fs 1.42.3 (14-May-2012)
...
# mount -o loop imagefile2 /mnt/scratch
# for I in `seq 1 32`; do fallocate -l 1t /mnt/scratch/1tfile$I; done
# umount /mnt/scratch
# time e2fsck -fn imagefile2
e2fsck 1.42.3 (14-May-2012)
<many many many lines snipped>
Free blocks count wrong for group #131072 (0, counted=32768).
Fix? no

Free blocks count wrong for group #131073 (0, counted=32768).
Fix? no

Free blocks count wrong for group #131074 (0, counted=4164).
Fix? no

Free blocks count wrong (3, counted=4294967299).
Fix? no


imagefile2: ********** WARNING: Filesystem still has errors **********

imagefile2: 43/536870912 files (0.0% non-contiguous), 8589934589/8589934592 blocks

real	27m40.133s
user	27m12.886s
sys	0m6.238s

Re: ext4 > 16T woes

[Andreas Dilger]

On 2012-05-23, at 12:48 PM, Eric Sandeen wrote:
> I'll try to look into this but just wanted to put it out there.
> 
> After creating a 32T fs and filling it with 1T preallocated files,
> I get quite a lot of corruption; this wasn't really the test
> I wanted to do, I wanted to then remove a file with mostly
> high blocks, run fsstress, and test recovery :(

Hmm, we've done full-filesystem write + read +verify + e2fsck up to
128TB (took several days to write and read so much data) with RHEL6
without similar problems (up to 32TB on RHEL5) using real disks.

We don't use fallocate() or loop devices, so it may be that some
problems are therein, or possibly with newer kernels.

Presumably this is using XFS for the backing loop file?  I don't
anticipate problems with that, but there may be strange interactions
between fallocate(), sparse files, and TRIM/DISCARD?

> # uname -r
> 3.4.0-rc4+
> # truncate --size 32t imagefile2
> # mkfs.ext4 -F imagefile2
> mke2fs 1.42.3 (14-May-2012)
> ...
> # mount -o loop imagefile2 /mnt/scratch
> # for I in `seq 1 32`; do fallocate -l 1t /mnt/scratch/1tfile$I; done
> # umount /mnt/scratch

Did you try sync here, or running e2fsck on the loop device instead
of the backing file?  It looks like nothing was written to disk...

> # time e2fsck -fn imagefile2
> e2fsck 1.42.3 (14-May-2012)
> <many many many lines snipped>
> Free blocks count wrong for group #131072 (0, counted=32768).
> Fix? no
> 
> Free blocks count wrong for group #131073 (0, counted=32768).
> Fix? no
> 
> Free blocks count wrong for group #131074 (0, counted=4164).
> Fix? no
> 
> Free blocks count wrong (3, counted=4294967299).
> Fix? no
> 
> 
> imagefile2: ********** WARNING: Filesystem still has errors **********
> 
> imagefile2: 43/536870912 files (0.0% non-contiguous), 8589934589/8589934592 blocks
> 
> real	27m40.133s
> user	27m12.886s
> sys	0m6.238s
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Cheers, Andreas

Re: ext4 > 16T woes

[Eric Sandeen]

On 5/23/12 2:33 PM, Andreas Dilger wrote:
> On 2012-05-23, at 12:48 PM, Eric Sandeen wrote:
>> I'll try to look into this but just wanted to put it out there.
>>
>> After creating a 32T fs and filling it with 1T preallocated files,
>> I get quite a lot of corruption; this wasn't really the test
>> I wanted to do, I wanted to then remove a file with mostly
>> high blocks, run fsstress, and test recovery :(
> 
> Hmm, we've done full-filesystem write + read +verify + e2fsck up to
> 128TB (took several days to write and read so much data) with RHEL6
> without similar problems (up to 32TB on RHEL5) using real disks.
> 
> We don't use fallocate() or loop devices, so it may be that some
> problems are therein, or possibly with newer kernels.
> 
> Presumably this is using XFS for the backing loop file?  I don't
> anticipate problems with that, but there may be strange interactions
> between fallocate(), sparse files, and TRIM/DISCARD?

It is backed by XFS, yes.  I don't think I'm freeing any blocks during
the test, so I wouldn't expect discard to come into it.  Really, all it
should be doing is writing fairly small amounts of metadata, right?

I'll keep investigating.

-Eric

>> # uname -r
>> 3.4.0-rc4+
>> # truncate --size 32t imagefile2
>> # mkfs.ext4 -F imagefile2
>> mke2fs 1.42.3 (14-May-2012)
>> ...
>> # mount -o loop imagefile2 /mnt/scratch
>> # for I in `seq 1 32`; do fallocate -l 1t /mnt/scratch/1tfile$I; done
>> # umount /mnt/scratch
> 
> Did you try sync here, or running e2fsck on the loop device instead
> of the backing file?  It looks like nothing was written to disk...
> 
>> # time e2fsck -fn imagefile2
>> e2fsck 1.42.3 (14-May-2012)
>> <many many many lines snipped>
>> Free blocks count wrong for group #131072 (0, counted=32768).
>> Fix? no
>>
>> Free blocks count wrong for group #131073 (0, counted=32768).
>> Fix? no
>>
>> Free blocks count wrong for group #131074 (0, counted=4164).
>> Fix? no
>>
>> Free blocks count wrong (3, counted=4294967299).
>> Fix? no
>>
>>
>> imagefile2: ********** WARNING: Filesystem still has errors **********
>>
>> imagefile2: 43/536870912 files (0.0% non-contiguous), 8589934589/8589934592 blocks
>>
>> real	27m40.133s
>> user	27m12.886s
>> sys	0m6.238s
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> Cheers, Andreas
> 
> 
> 
> 
> 

Re: ext4 > 16T woes

[Eric Sandeen]

On 5/23/12 1:48 PM, Eric Sandeen wrote:
> I'll try to look into this but just wanted to put it out there.
> 
> After creating a 32T fs and filling it with 1T preallocated files,
> I get quite a lot of corruption; this wasn't really the test
> I wanted to do, I wanted to then remove a file with mostly
> high blocks, run fsstress, and test recovery :(

FWIW I tracked this down to the rbtree bitmap backend and sent
a patch,

[PATCH] libext2fs: Fix rbtree backend for extent lengths greater than 2^32

-Eric

> # uname -r
> 3.4.0-rc4+
> # truncate --size 32t imagefile2
> # mkfs.ext4 -F imagefile2
> mke2fs 1.42.3 (14-May-2012)
> ...
> # mount -o loop imagefile2 /mnt/scratch
> # for I in `seq 1 32`; do fallocate -l 1t /mnt/scratch/1tfile$I; done
> # umount /mnt/scratch
> # time e2fsck -fn imagefile2
> e2fsck 1.42.3 (14-May-2012)
> <many many many lines snipped>
> Free blocks count wrong for group #131072 (0, counted=32768).
> Fix? no
> 
> Free blocks count wrong for group #131073 (0, counted=32768).
> Fix? no
> 
> Free blocks count wrong for group #131074 (0, counted=4164).
> Fix? no
> 
> Free blocks count wrong (3, counted=4294967299).
> Fix? no
> 
> 
> imagefile2: ********** WARNING: Filesystem still has errors **********
> 
> imagefile2: 43/536870912 files (0.0% non-contiguous), 8589934589/8589934592 blocks
> 
> real	27m40.133s
> user	27m12.886s
> sys	0m6.238s
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html