Re: large fs testing

[Ric Wheeler <rwheeler@redhat.com>]

On 05/28/2009 02:30 AM, Andreas Dilger wrote:
> On May 26, 2009  18:17 -0400, Ric Wheeler wrote:
>> What I did get was the following from the fsck run:
>>
>> root@l82bi250:/home/redhat\aYou have new mail in /var/spool/mail/root
>> [root@l82bi250 redhat]# time /sbin/fsck.ext4 -tt -y /dev/mapper/Big_boy-Big_boy
>> e2fsck 1.41.4 (27-Jan-2009)
>> Pass 1: Checking inodes, blocks, and sizes
>> Pass 1: Memory used: 1596k/1177752k (1447k/150k), time: 1184.73/514.16/344.38
>> Pass 1: I/O read: 50655MB, write: 0MB, rate: 42.76MB/s
>> Pass 2: Checking directory structure
>> Entry '4a1590dc~~~~~~~~O4A0SMJ1VC34YQ1PD3B5DL9Q' in /da (188378)
>> references inode 196988 in group 30 where _INODE_UNINIT is set.
>> Fix? yes
>>
>> Restarting e2fsck from the beginning...
>> Group descriptor 15 checksum is invalid.  Fix? yes
>>
>> Pass 1: Checking inodes, blocks, and sizes
>> Pass 1: Memory used: 120396k/-1389015k (120134k/263k), time: 1134.71/522.48/323.65
>> Pass 1: I/O read: 50656MB, write: 0MB, rate: 44.64MB/s
>> Pass 2: Checking directory structure
>> Entry '4a15910c~~~~~~~~H8099TRM701Q29CSTCWBVIHJ' in /0b (404925)
>> references inode 413100 in group 62 where _INODE_UNINIT is set.
>> Fix? yes
>>
>> Restarting e2fsck from the beginning...
>> Group descriptor 31 checksum is invalid.  Fix? yes
>
> This looks like there is a patch of ours missing from the upstream e2fsprogs.
> We have a patch that will restart e2fsck only a single time for inodes
> beyond the high waterwark.  On a large filesystem like yours this would
> have cut 30 minutes off the e2fsck time.  I'll submit that separately.
>
>> Pass 1: Checking inodes, blocks, and sizes
>> Pass 1: Memory used: 231360k/246272k (231083k/278k), time: 1140.48/521.00/334.74
>> Pass 1: I/O read: 50658MB, write: 0MB, rate: 44.42MB/s
>> Pass 2: Checking directory structure
>> Pass 2: Memory used: 231360k/1290436k (231083k/278k), time: 538.22/264.56/83.49
>> Pass 2: I/O read: 13749MB, write: 0MB, rate: 25.55MB/s
>> Pass 3: Checking directory connectivity
>> Peak memory: Memory used: 231360k/1789000k (231083k/278k), time:
>> 4221.57/1947.37/1116.21
>> Pass 3A: Memory used: 231360k/1789000k (231083k/278k), time:  0.00/ 0.00/ 0.00
>> Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
>> Pass 3: Memory used: 231360k/1290436k (231083k/278k), time:  9.99/ 0.26/ 1.37
>> Pass 3: I/O read: 1MB, write: 0MB, rate: 0.10MB/s
>> Pass 4: Checking reference counts
>> Pass 4: Memory used: 231360k/-1481575k (231082k/279k), time: 147.16/139.87/ 1.94
>
> Sign overflow here...  Looks like we exceed 2.5GB of memory here.   Still,
> not too bad considering this is a 80TB filesystem.

We fsck had a virtual size of around 10GB (5.4 resident in the 6GB of DRAM) when 
I checked...  I wonder if it would have been significantly faster without the 
excessive swap use (i.e., on a box with more memory)?

>
>> Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
>> Pass 5: Checking group summary information
>> Inode bitmap differences:  -(98404--98405)
>>
>> Note that it got truncated in Pass 5 - just after writing out some values
>> that look like they sign wrapped?
>>
>> -(103650--103655) -(103659--103660) -103663 -103665 -103667
>> -(103669--103670) -(103673--103676) -103679 -103684 -103687 -10
>
> No, this is what gets printed when there are inodes (or blocks) marked
> in the bitmap that are not in use.  It shouldn't be truncated however.
> You said the node crashed at this point?
>
> Cheers, Andreas

Yes - unfortunately, no logs or other indication of why it crashed and we did 
not have a serial console setup either so we don't have anything to go on.

I am going to push harder to get some large storage configurations that we can 
use for testing internally, so hopefully, we will have something to test on in a 
couple of months....

Ric