Too large value in inode.i_blocks[1]

Too large value in inode.i_blocks[1]

[Felipe Monteiro de Carvalho]

Hello,

Continuing in my adventures with ext4, now I am having
a problem that when reading the root
inode everything is fine. 
When reading the inodes of directories placed on the file system root,
my reading fails, because inode.i_blocks has the following data 
(for 1 particular subdir for example):

127754,
4,
0,
0,
1,
1366,
0,...

The root works because in its inode i_blocks[1] has a smaller, 
in-bounds value, but this one has
the first value completely out of bounds of the filesystem =(

My ext4 filesystem has 3 incompat flags set: EXT4_FEATURE_INCOMPAT_FILETYPE,
EXT4_FEATURE_INCOMPAT_EXTENTS, EXT4_FEATURE_INCOMPAT_FLEX_BG

I suppose it must be related to either EXTENTS or FLEX_BG ...

any ideas?

Also, where can I read about the exact effect of ext4 new features?

thanks,

Felipe Monteiro de Carvalho

Re: Too large value in inode.i_blocks[1]

[Theodore Ts'o]

On Wed, Jun 12, 2013 at 05:08:21AM +0000, Felipe Monteiro de Carvalho wrote:
> 
> Continuing in my adventures with ext4, now I am having
> a problem that when reading the root
> inode everything is fine. 
> When reading the inodes of directories placed on the file system root,
> my reading fails, because inode.i_blocks has the following data 
> (for 1 particular subdir for example):
> 
> 127754,
> 4,
> 0,
> 0,
> 1,
> 1366,
> 0,...
> 
> The root works because in its inode i_blocks[1] has a smaller, 
> in-bounds value, but this one has
> the first value completely out of bounds of the filesystem =(
> 
> My ext4 filesystem has 3 incompat flags set: EXT4_FEATURE_INCOMPAT_FILETYPE,
> EXT4_FEATURE_INCOMPAT_EXTENTS, EXT4_FEATURE_INCOMPAT_FLEX_BG

The inode in question has the EXTENTS_FL flag set, which means the
i_blocks[] array needs to get interpreted as the root node of the
extent tree. 127754 is 0x0001F30A, where F30A is the magic number for
the extents header, and 0x0001 means there is a single entry in the
extent node.

I **strongly** discourage people from trying to access the file system
directly.  It's much better to use the libext2fs library, since it
will deal with these sorts of issues for you automatically.  A
program, even one written years ago, which uses the block iterator, or
the directory iterator, or the namei function, or the
ext2fs_file_{open,read,write,llseek}() functions provided by
libext2fs, would have continued working when ext4 appeared, since we
added ext4 support to the libext2fs library in such a way that older
programs would continue working just fine.

A good example of such a user of libext2fs is the e2tools userspace
package.

Regards,

						- Ted

Re: Too large value in inode.i_blocks[1]

[Felipe Monteiro de Carvalho]

Hello,

Theodore Ts'o <tytso <at> mit.edu> writes:
> The inode in question has the EXTENTS_FL flag set, which means the
> i_blocks[] array needs to get interpreted as the root node of the
> extent tree. 127754 is 0x0001F30A, where F30A is the magic number for
> the extents header, and 0x0001 means there is a single entry in the
> extent node.

Thanks, great =) So reading 
https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#The_Contents_of_inod
e.i_block

I see that i_block has 15*4=60 bytes, and with EXTENTS_FL the first 12 bytes 
of i_block are filled with a ext4_extent_header

What about the rest of it? I couldn't understand from the wiki page what
the rest holds exactly, it is not 100% clear ...

My first guess would be that it has in this order:

ext4_extent_header, 12 bytes long
ext4_extent_idx, 12 bytes long:
ext4_extent, 12 bytes long
ext4_extent_tail 4 bytes long

Which would sum 40 bytes ... still unsure about the other 20 however.

Any help here is greatly appreciated =)

> I **strongly** discourage people from trying to access the file system
> directly.  It's much better to use the libext2fs library, since it
> will deal with these sorts of issues for you automatically.  A

Please don't worry that I am not frivolously attempting to access the
file system directly. My project explictly requires this, it is a
recovery tool to repair damaged file systems, and it must work in
Windows and Mac OS X. Maybe libext2fs works outside Linux, not sure, 
but anyway we already have source code for reading/recovering ext2/3, 
I am just expanding it to ext4. External dependencies are always
not ideal.

I am new in the project, that's why I have so many beginner questions,
I appologise for it, but I'm thankful that I experienced developers 
like you answer them so that I can learn about the Linux Ext4
file system.

thanks =)

Felipe Monteiro de Carvalho

Re: Too large value in inode.i_blocks[1]

[Theodore Ts'o]

On Wed, Jun 12, 2013 at 01:34:37PM +0000, Felipe Monteiro de Carvalho wrote:
> I see that i_block has 15*4=60 bytes, and with EXTENTS_FL the first 12 bytes 
> of i_block are filled with a ext4_extent_header
> 
> What about the rest of it? I couldn't understand from the wiki page what
> the rest holds exactly, it is not 100% clear ...
> 
> My first guess would be that it has in this order:
> 
> ext4_extent_header, 12 bytes long
> ext4_extent_idx, 12 bytes long:
> ext4_extent, 12 bytes long
> ext4_extent_tail 4 bytes long

The ext4_extent_tail is not used in i_blocks[].  There will be up to 4
ext4_extent structures, or ext4_extent_idx structures, depending on
the depth of the tree.  The ext4_extent_idx structures are used for
the interior inodes of the tree.  The ext4_extent structures are used
for the leaf nodes of the tree.  The ext4_extent_tail is used in
extent tree blocks so we can checksum the metadata if the
metadata_csum feature is enabled.  The entire inode is checksummed, so
we don't need the ext4_extent_tail in i_blocks[].

> Please don't worry that I am not frivolously attempting to access the
> file system directly. My project explictly requires this, it is a
> recovery tool to repair damaged file systems, and it must work in
> Windows and Mac OS X. Maybe libext2fs works outside Linux, not sure, 
> but anyway we already have source code for reading/recovering ext2/3, 
> I am just expanding it to ext4. External dependencies are always
> not ideal.

Libext2fs is designed to be cross OS portable.  It's used for the FUSE
extensions that allow Mac and Windows machines to access ext2/3/4 file
systems.  And libext2fs is designed to work with corrupted file
systems, since after e2fsck is the normal tool most people use to
repair damaged file systems, and debugfs is the command line tool
which is used by people to examine file systems, corrupt fie systems
for e2fsck test cases, and in a few cases, attempt to do manual repair
of file systems in some extreme circumstances.

> I am new in the project, that's why I have so many beginner questions,
> I appologise for it, but I'm thankful that I experienced developers 
> like you answer them so that I can learn about the Linux Ext4
> file system.

Is this a student project, or a commmercial project, out of curiosity?

   	   	  	  	      	- Ted

Re: Too large value in inode.i_blocks[1]

[Eric Sandeen]

On 6/12/13 8:52 AM, Theodore Ts'o wrote:
> On Wed, Jun 12, 2013 at 01:34:37PM +0000, Felipe Monteiro de Carvalho wrote:
>> I see that i_block has 15*4=60 bytes, and with EXTENTS_FL the first 12 bytes 
>> of i_block are filled with a ext4_extent_header
>>
>> What about the rest of it? I couldn't understand from the wiki page what
>> the rest holds exactly, it is not 100% clear ...
>>
>> My first guess would be that it has in this order:
>>
>> ext4_extent_header, 12 bytes long
>> ext4_extent_idx, 12 bytes long:
>> ext4_extent, 12 bytes long
>> ext4_extent_tail 4 bytes long
> 
> The ext4_extent_tail is not used in i_blocks[].  There will be up to 4
> ext4_extent structures, or ext4_extent_idx structures, depending on
> the depth of the tree.  The ext4_extent_idx structures are used for
> the interior inodes of the tree.  The ext4_extent structures are used
> for the leaf nodes of the tree.  The ext4_extent_tail is used in
> extent tree blocks so we can checksum the metadata if the
> metadata_csum feature is enabled.  The entire inode is checksummed, so
> we don't need the ext4_extent_tail in i_blocks[].
> 
>> Please don't worry that I am not frivolously attempting to access the
>> file system directly. My project explictly requires this, it is a
>> recovery tool to repair damaged file systems, and it must work in
>> Windows and Mac OS X. Maybe libext2fs works outside Linux, not sure, 
>> but anyway we already have source code for reading/recovering ext2/3, 
>> I am just expanding it to ext4. External dependencies are always
>> not ideal.
> 
> Libext2fs is designed to be cross OS portable.  It's used for the FUSE
> extensions that allow Mac and Windows machines to access ext2/3/4 file
> systems.  And libext2fs is designed to work with corrupted file
> systems, since after e2fsck is the normal tool most people use to
> repair damaged file systems, and debugfs is the command line tool
> which is used by people to examine file systems, corrupt fie systems
> for e2fsck test cases, and in a few cases, attempt to do manual repair
> of file systems in some extreme circumstances.

Indeed, I know of at least one "clean room" ext3 driver for windows
which did a fine job of corrupting proper ext3 filesystems, because it
was written incorrectly and didn't use the standard body of code.
Wasted a lot of my time getting to the bottom of that one.  :(

-Eric

Re: Too large value in inode.i_blocks[1]

[Felipe Monteiro de Carvalho]

On Wed, Jun 12, 2013 at 3:52 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> The ext4_extent_tail is not used in i_blocks[].  There will be up to 4
> ext4_extent structures, or ext4_extent_idx structures, depending on
> the depth of the tree.  The ext4_extent_idx structures are used for
> the interior inodes of the tree.  The ext4_extent structures are used
> for the leaf nodes of the tree.  The ext4_extent_tail is used in
> extent tree blocks so we can checksum the metadata if the
> metadata_csum feature is enabled.  The entire inode is checksummed, so
> we don't need the ext4_extent_tail in i_blocks[].

Hello,

thanks, this part I understood fairly well, but what if eh_entries is
8, for example .... where are the other 4 structures?


-- 
Felipe Monteiro de Carvalho

Re: Too large value in inode.i_blocks[1]

[Theodore Ts'o]

On Mon, Jul 01, 2013 at 05:15:21PM +0200, Felipe Monteiro de Carvalho wrote:
> On Wed, Jun 12, 2013 at 3:52 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> > The ext4_extent_tail is not used in i_blocks[].  There will be up to 4
> > ext4_extent structures, or ext4_extent_idx structures, depending on
> > the depth of the tree.  The ext4_extent_idx structures are used for
> > the interior inodes of the tree.  The ext4_extent structures are used
> > for the leaf nodes of the tree.  The ext4_extent_tail is used in
> > extent tree blocks so we can checksum the metadata if the
> > metadata_csum feature is enabled.  The entire inode is checksummed, so
> > we don't need the ext4_extent_tail in i_blocks[].
> 
> Hello,
> 
> thanks, this part I understood fairly well, but what if eh_entries is
> 8, for example .... where are the other 4 structures?

For the root node which is located in the inode, that would be
illegal.  There is only room for four entries in the root node.  If an
inode needs more extents than that, then we have to have an extent
leaf tree block in an external 4k block and the root inode will point
to up to four extent tree leaf blocks.  If there are more than four
leaf blocks, then the tree would have to grow in depth again, and the
root node would contain a pointer to up to four index blocks, and each
index node block can point to up to 340 leaf tree blocks.  (Assuming a 4k block size)

Basically, this is a standard tree data structure.

       	   	       	   	      	  	 - Ted