Extent Depth Histogram Fsck

Extent Depth Histogram Fsck

[Nelson, John R]

What does the extent depth histogram mean? Is it a measure of something?

like mine is 


Extent depth histogram: 36010/81

Re: Extent Depth Histogram Fsck

[Andreas Dilger]

On 2012-06-26, at 8:34 AM, Nelson, John R wrote:
> What does the extent depth histogram mean? Is it a measure of something?
> 
> like mine is 
> 
> 
> Extent depth histogram: 36010/81

This means that of all the extent-mapped files in the filesystem,
36010 files have an extent tree of depth 0 (i.e. they fit inside the inode)
81 files have an extent tree of depth 1 (i.e. there is a single index block)

Typically, files larger than 4 * 128MB = 512MB need an index block, but if
the maximum-sized extents cannot be allocated then an index block will be needed for smaller files.  Only if you have very large files (> 40GB),
or a very fragmented free space would you need more than a single level
of index blocks.

Cheers, Andreas

RE: Extent Depth Histogram Fsck

[Nelson, John R]

ok i see!
So when there are like
3/3/4 that means double index blocks?? How many extents can a single extent index hold in a block?
________________________________________
From: Andreas Dilger [adilger@dilger.ca]
Sent: Tuesday, June 26, 2012 11:40 AM
To: Nelson, John R
Cc: linux-ext4@vger.kernel.org
Subject: Re: Extent Depth Histogram Fsck

On 2012-06-26, at 8:34 AM, Nelson, John R wrote:
> What does the extent depth histogram mean? Is it a measure of something?
>
> like mine is
>
>
> Extent depth histogram: 36010/81

This means that of all the extent-mapped files in the filesystem,
36010 files have an extent tree of depth 0 (i.e. they fit inside the inode)
81 files have an extent tree of depth 1 (i.e. there is a single index block)

Typically, files larger than 4 * 128MB = 512MB need an index block, but if
the maximum-sized extents cannot be allocated then an index block will be needed for smaller files.  Only if you have very large files (> 40GB),
or a very fragmented free space would you need more than a single level
of index blocks.

Cheers, Andreas

Re: Extent Depth Histogram Fsck

[Ted Ts'o]

On Tue, Jun 26, 2012 at 06:07:10PM +0000, Nelson, John R wrote:
> ok i see!
> So when there are like
> 3/3/4 that means double index blocks?? How many extents can a single extent index hold in a block?

You can put 4 entries in the inode, and then there can be 340 entries
in each 4k extent tree block.  Each entry can either be a leaf entry
or an index entry (i.e., pointing to another extent tree block).

So it's pretty rare for there extent tree to have a depth greater than
1.  In order to actually test the code, what I do is use a 1k block
size (so you can only fit 84 entries in each ETB), and then use a
small file system, and then do something like this:

touch /tmp/test.img
mke2fs -t ext4 -O ^has_journal -F -b 1024 -N 131084 /tmp/test.img 150M
mount -o loop /tmp/test.img /mnt
cd /mnt
seq 1 131072 | xargs -n 1 fallocate -l 1024
seq 1 2 131072 | xargs rm
fallocate -l 64M test-file
cd ..
umount /mnt

i.e., create a maximally fragmented file system, and then create a
large file.  But this is very rare in real-life workloads.

      	     	      	      - Ted

Re: Extent Depth Histogram Fsck

[Andreas Dilger]

On 2012-06-26, at 12:07, "Nelson, John R" <John_Nelson@student.uml.edu> wrote:

> ok i see!
> So when there are like
> 3/3/4 that means double index blocks?? How many extents can a single extent index hold in a block?

The header takes 12 bytes, and each extent or index pointer takes 12 bytes, so for 4kB blocksize there can be (4096 / 12) - 1 = 340 extents per block.

The maximum extent size for 4kB blocks is 2^15*4kB = 128MB, so each index block can map up to ~42GB, so a two-level tree can map just over 14TB under ideal conditions. 

Cheers, Andreas

> ________________________________________
> From: Andreas Dilger [adilger@dilger.ca]
> Sent: Tuesday, June 26, 2012 11:40 AM
> To: Nelson, John R
> Cc: linux-ext4@vger.kernel.org
> Subject: Re: Extent Depth Histogram Fsck
> 
> On 2012-06-26, at 8:34 AM, Nelson, John R wrote:
>> What does the extent depth histogram mean? Is it a measure of something?
>> 
>> like mine is
>> 
>> 
>> Extent depth histogram: 36010/81
> 
> This means that of all the extent-mapped files in the filesystem,
> 36010 files have an extent tree of depth 0 (i.e. they fit inside the inode)
> 81 files have an extent tree of depth 1 (i.e. there is a single index block)
> 
> Typically, files larger than 4 * 128MB = 512MB need an index block, but if
> the maximum-sized extents cannot be allocated then an index block will be needed for smaller files.  Only if you have very large files (> 40GB),
> or a very fragmented free space would you need more than a single level
> of index blocks.
> 
> Cheers, Andreas
> 
> 
> 
> 
> 
> 
>