Which features should I implement in my ext4 reader?

Which features should I implement in my ext4 reader?

[Felipe Monteiro de Carvalho]

Hello,

Thanks to the help here I have essentially finished my ext4 reader =)
At least it can read ext4 filesystems with these features, which is
what I found when using Ubuntu and Mageia LInux:

   EXT4_FEATURE_INCOMPAT_JOURNAL_DEV	= $0008; // Journal device */
   EXT4_FEATURE_INCOMPAT_META_BG	= $0010;
   EXT4_FEATURE_INCOMPAT_EXTENTS	= $0040; // extents support */
   EXT4_FEATURE_INCOMPAT_FLEX_BG	= $0200;
   EXT4_FEATURE_INCOMPAT_64BIT		= $0080;

Some features seam to not require any extra care for my reader:

   EXT4_FEATURE_INCOMPAT_COMPRESSION	= $0001;
   EXT4_FEATURE_INCOMPAT_FILETYPE	= $0002;
   EXT4_FEATURE_INCOMPAT_RECOVER	= $0004; // Needs recovery */
   EXT4_FEATURE_INCOMPAT_BG_USE_META_CSUM= $2000; // use crc32c for bg */

But I now wonder about these features:

   EXT4_FEATURE_INCOMPAT_MMP            = $0100;
   EXT4_FEATURE_INCOMPAT_EA_INODE	= $0400; // EA in inode */
   EXT4_FEATURE_INCOMPAT_DIRDATA	= $1000; // data in dirent */
   EXT4_FEATURE_INCOMPAT_LARGEDIR	= $4000; // >2GB or 3-lvl htree */
   EXT4_FEATURE_INCOMPAT_INLINEDATA	= $8000; // data in inode */

Could anyone comment how common is the use of those features and if
they are already stable and in production usage (I read that they were
under development in 2011)?

And also which ones require an active work for a reader application to
implement. As I already found out that FLEX_BG does not require active
work to be supported despite being in the INCOMPAT list...

>From the description I'm pretty sure that at least DIRDATA and
INLINEDATA would require adjustments in my software. But I wonder if
they are already being used and stable, etc, and if it is a good idea
to work on supporting them.

thanks, =)
-- 
Felipe Monteiro de Carvalho

Re: Which features should I implement in my ext4 reader?

[Felipe Monteiro de Carvalho]

Hello,

Did my last message go through? Just wondering if it was lost ...

thanks,

Felipe Monteiro de Carvalho

On Mon, Aug 12, 2013 at 5:38 PM, Felipe Monteiro de Carvalho
<felipemonteiro.carvalho@gmail.com> wrote:
> Hello,
>
> Thanks to the help here I have essentially finished my ext4 reader =)
> At least it can read ext4 filesystems with these features, which is
> what I found when using Ubuntu and Mageia LInux:
>
>    EXT4_FEATURE_INCOMPAT_JOURNAL_DEV    = $0008; // Journal device */
>    EXT4_FEATURE_INCOMPAT_META_BG        = $0010;
>    EXT4_FEATURE_INCOMPAT_EXTENTS        = $0040; // extents support */
>    EXT4_FEATURE_INCOMPAT_FLEX_BG        = $0200;
>    EXT4_FEATURE_INCOMPAT_64BIT          = $0080;
>
> Some features seam to not require any extra care for my reader:
>
>    EXT4_FEATURE_INCOMPAT_COMPRESSION    = $0001;
>    EXT4_FEATURE_INCOMPAT_FILETYPE       = $0002;
>    EXT4_FEATURE_INCOMPAT_RECOVER        = $0004; // Needs recovery */
>    EXT4_FEATURE_INCOMPAT_BG_USE_META_CSUM= $2000; // use crc32c for bg */
>
> But I now wonder about these features:
>
>    EXT4_FEATURE_INCOMPAT_MMP            = $0100;
>    EXT4_FEATURE_INCOMPAT_EA_INODE       = $0400; // EA in inode */
>    EXT4_FEATURE_INCOMPAT_DIRDATA        = $1000; // data in dirent */
>    EXT4_FEATURE_INCOMPAT_LARGEDIR       = $4000; // >2GB or 3-lvl htree */
>    EXT4_FEATURE_INCOMPAT_INLINEDATA     = $8000; // data in inode */
>
> Could anyone comment how common is the use of those features and if
> they are already stable and in production usage (I read that they were
> under development in 2011)?
>
> And also which ones require an active work for a reader application to
> implement. As I already found out that FLEX_BG does not require active
> work to be supported despite being in the INCOMPAT list...
>
> From the description I'm pretty sure that at least DIRDATA and
> INLINEDATA would require adjustments in my software. But I wonder if
> they are already being used and stable, etc, and if it is a good idea
> to work on supporting them.
>
> thanks, =)
> --
> Felipe Monteiro de Carvalho



-- 
Felipe Monteiro de Carvalho

Re: Which features should I implement in my ext4 reader?

[Felipe Monteiro de Carvalho]

Hello,

I wonder if when    EXT4_FEATURE_INCOMPAT_DIRDATA and
EXT4_FEATURE_INCOMPAT_INLINEDATA are active, will all inodes have the
first part of the data inlined? Or only some? How to know which ones?

Any ideas on this and also about which features are the most important
to implement, how often they are used, how stable their implementation
is, etc would be greatly appreciated =)

thanks,

Felipe Monteiro de Carvalho


On Wed, Aug 14, 2013 at 7:06 AM, Felipe Monteiro de Carvalho
<felipemonteiro.carvalho@gmail.com> wrote:
> Hello,
>
> Did my last message go through? Just wondering if it was lost ...
>
> thanks,
>
> Felipe Monteiro de Carvalho
>
> On Mon, Aug 12, 2013 at 5:38 PM, Felipe Monteiro de Carvalho
> <felipemonteiro.carvalho@gmail.com> wrote:
>> Hello,
>>
>> Thanks to the help here I have essentially finished my ext4 reader =)
>> At least it can read ext4 filesystems with these features, which is
>> what I found when using Ubuntu and Mageia LInux:
>>
>>    EXT4_FEATURE_INCOMPAT_JOURNAL_DEV    = $0008; // Journal device */
>>    EXT4_FEATURE_INCOMPAT_META_BG        = $0010;
>>    EXT4_FEATURE_INCOMPAT_EXTENTS        = $0040; // extents support */
>>    EXT4_FEATURE_INCOMPAT_FLEX_BG        = $0200;
>>    EXT4_FEATURE_INCOMPAT_64BIT          = $0080;
>>
>> Some features seam to not require any extra care for my reader:
>>
>>    EXT4_FEATURE_INCOMPAT_COMPRESSION    = $0001;
>>    EXT4_FEATURE_INCOMPAT_FILETYPE       = $0002;
>>    EXT4_FEATURE_INCOMPAT_RECOVER        = $0004; // Needs recovery */
>>    EXT4_FEATURE_INCOMPAT_BG_USE_META_CSUM= $2000; // use crc32c for bg */
>>
>> But I now wonder about these features:
>>
>>    EXT4_FEATURE_INCOMPAT_MMP            = $0100;
>>    EXT4_FEATURE_INCOMPAT_EA_INODE       = $0400; // EA in inode */
>>    EXT4_FEATURE_INCOMPAT_DIRDATA        = $1000; // data in dirent */
>>    EXT4_FEATURE_INCOMPAT_LARGEDIR       = $4000; // >2GB or 3-lvl htree */
>>    EXT4_FEATURE_INCOMPAT_INLINEDATA     = $8000; // data in inode */
>>
>> Could anyone comment how common is the use of those features and if
>> they are already stable and in production usage (I read that they were
>> under development in 2011)?
>>
>> And also which ones require an active work for a reader application to
>> implement. As I already found out that FLEX_BG does not require active
>> work to be supported despite being in the INCOMPAT list...
>>
>> From the description I'm pretty sure that at least DIRDATA and
>> INLINEDATA would require adjustments in my software. But I wonder if
>> they are already being used and stable, etc, and if it is a good idea
>> to work on supporting them.
>>
>> thanks, =)
>> --
>> Felipe Monteiro de Carvalho
>
>
>
> --
> Felipe Monteiro de Carvalho



-- 
Felipe Monteiro de Carvalho

Re: Which features should I implement in my ext4 reader?

[Theodore Ts'o]

On Wed, Aug 14, 2013 at 07:06:44AM +0200, Felipe Monteiro de Carvalho wrote:
> 
> Did my last message go through? Just wondering if it was lost ...

I think part of the problem (besids lots of people being busy and on
vacation) is it's not clear what your ext4 reader is going to be used
for.  As a result it's hard to give advice, not knowing what the
requirements are for said reader.

> >    EXT4_FEATURE_INCOMPAT_COMPRESSION    = $0001;

Compression isn't implemeneted, and so I'd suggest not trying to
assume anything the file system with this feature.

> >    EXT4_FEATURE_INCOMPAT_FILETYPE       = $0002;

This would be true only if you are properly handling the name_len in
the direct structure.  (It is a 16 bit Little Endian element, but only
the low 8 bits are the name length; the high 8 bits are for the file
type).

> >    EXT4_FEATURE_INCOMPAT_RECOVER        = $0004; // Needs recovery */

If you are not doing journal replay, you should *not* try to handle a
file system with the needs recovery bit set.

> >    EXT4_FEATURE_INCOMPAT_BG_USE_META_CSUM= $2000; // use crc32c for bg */

If you're going to ignore potential checksum errors in the
metadata....

> > But I now wonder about these features:
> >
> >    EXT4_FEATURE_INCOMPAT_MMP            = $0100;

This is used to protect against multiple systems trying to access a
shared block device.  If you are not supporting the MMP protocol, it's
possible for the metadata blocks on a shared block device to be
changing out from under you, with potentially hilarious (if you find
kernel oops or file system reader crashes funny) results.

> >    EXT4_FEATURE_INCOMPAT_EA_INODE       = $0400; // EA in inode */
> >    EXT4_FEATURE_INCOMPAT_DIRDATA        = $1000; // data in dirent */
> >    EXT4_FEATURE_INCOMPAT_LARGEDIR       = $4000; // >2GB or 3-lvl htree */
> >    EXT4_FEATURE_INCOMPAT_INLINEDATA     = $8000; // data in inode */

These features aren't quite fully supported yet.  The inline data
feature is probably closest to be being fully supported, when we
release an the next major version of e2fsprogs.

> > And also which ones require an active work for a reader application to
> > implement. As I already found out that FLEX_BG does not require active
> > work to be supported despite being in the INCOMPAT list...

A lot of whether changes are needed to a file system implementation
depends on how carefully and robustly it had been written in the first
place.  Flex_bg had to be an incompat feature because the Linux kernel
implementation was checking certain invariants which flex_bg would
have violated.  If you have an implementation that isn't checking for
various metadata inconsistenties or errors, it might not matter.

By the way, depending on what your propietary reader is going to be
used for, you might want to consider using an fsfuzzer to make sure it
handles subtly corrupted file systems in a sane way.  If not (again
depending on how oyu use it), you could end up crashing your system
(if this mysterious proprietary reader is part of some OS) or
potentially causing a security breach (buffer overruns, anyway?  :-)

Cheers,

						- Ted

Re: Which features should I implement in my ext4 reader?

[Felipe Monteiro de Carvalho]

On Sat, Aug 17, 2013 at 3:02 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> I think part of the problem (besids lots of people being busy and on
> vacation) is it's not clear what your ext4 reader is going to be used
> for.  As a result it's hard to give advice, not knowing what the
> requirements are for said reader.

I think that I formulated my question incorrectly. I did not mean
"please give me advice over my specific application" this time, but
rather what I wanted was "please tell me a little about these various
features and also tell me if they are finished or still work in
progress, etc". Which is precisely what you answered already, so
thanks a lot =)

> These features aren't quite fully supported yet.  The inline data
> feature is probably closest to be being fully supported, when we
> release an the next major version of e2fsprogs.

Just for completeness, which version number will that by?

So, I understand that currently mkfs.ext4 cannot generate ext4 file
systems with this feature, correct?

Is there any other way to generate a file system with this feature
with a 3.8 kernel which will work?

Just wondering if I can already start work on this or if I need to wait.

thanks,
-- 
Felipe Monteiro de Carvalho

Re: Which features should I implement in my ext4 reader?

[Zheng Liu]

Hello,

On Sun, Aug 18, 2013 at 08:04:20AM +0200, Felipe Monteiro de Carvalho wrote:
> On Sat, Aug 17, 2013 at 3:02 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> > I think part of the problem (besids lots of people being busy and on
> > vacation) is it's not clear what your ext4 reader is going to be used
> > for.  As a result it's hard to give advice, not knowing what the
> > requirements are for said reader.
> 
> I think that I formulated my question incorrectly. I did not mean
> "please give me advice over my specific application" this time, but
> rather what I wanted was "please tell me a little about these various
> features and also tell me if they are finished or still work in
> progress, etc". Which is precisely what you answered already, so
> thanks a lot =)
> 
> > These features aren't quite fully supported yet.  The inline data
> > feature is probably closest to be being fully supported, when we
> > release an the next major version of e2fsprogs.
> 
> Just for completeness, which version number will that by?
> 
> So, I understand that currently mkfs.ext4 cannot generate ext4 file
> systems with this feature, correct?

You need to checkout to e2fsprogs/pu branch.  All patches for inline data
are on this branch.  BTW, the latest refinement of patches had been sent
to the mailing list [1].  It would be great if you could give it a try.

1. http://comments.gmane.org/gmane.comp.file-systems.ext4/39825

Regards,
                                                - Zheng

> 
> Is there any other way to generate a file system with this feature
> with a 3.8 kernel which will work?
> 
> Just wondering if I can already start work on this or if I need to wait.
> 
> thanks,
> -- 
> Felipe Monteiro de Carvalho
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Which features should I implement in my ext4 reader?

[Felipe Monteiro de Carvalho]

On Mon, Aug 19, 2013 at 7:08 AM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> You need to checkout to e2fsprogs/pu branch.  All patches for inline data
> are on this branch.  BTW, the latest refinement of patches had been sent
> to the mailing list [1].  It would be great if you could give it a try.
>
> 1. http://comments.gmane.org/gmane.comp.file-systems.ext4/39825

Ok I can try this branch, but I wonder if other tools will work too. I
need also to be able to add some files and directories to the
partition.

Will "mount" from a linux with kernel 3.8 work with a partition
generated with this new e2fsprogs?

-- 
Felipe Monteiro de Carvalho

Re: Which features should I implement in my ext4 reader?

[Zheng Liu]

On Tue, Aug 20, 2013 at 12:16:09PM +0200, Felipe Monteiro de Carvalho wrote:
> On Mon, Aug 19, 2013 at 7:08 AM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> > You need to checkout to e2fsprogs/pu branch.  All patches for inline data
> > are on this branch.  BTW, the latest refinement of patches had been sent
> > to the mailing list [1].  It would be great if you could give it a try.
> >
> > 1. http://comments.gmane.org/gmane.comp.file-systems.ext4/39825
> 
> Ok I can try this branch, but I wonder if other tools will work too. I
> need also to be able to add some files and directories to the
> partition.

I add some new interfaces in libext2fs to manipulate a file system
with inline data feature.  You can use libext2fs to add some files and
directories.  But these interfaces might be changed because it isn't
exported as an ABI.  After it is released within a new e2fsprogs
version, they will never be changed.

> 
> Will "mount" from a linux with kernel 3.8 work with a partition
> generated with this new e2fsprogs?

Yes, you can mount a file system with inline data in a 3.8 kernel using
this branch.

Regards,
                                                - Zheng

Re: Which features should I implement in my ext4 reader?

[Felipe Monteiro de Carvalho]

Hello,

Ok, thanks a lot. I cloned the git repo, built everything, created my
partition, and started studying inline_data =)

So, the wiki says that files up to 60 bytes are stucked into i_block
and that goes OK was easy to implement

But I am having problems with " If the rest of the file would fit
inside the extended attribute space, then it might be found as the
extended attribute "system.data". This of course constrains the amount
of extended attributes one can attach to an inode."

I have some questions about this =)

1> Up to how many bytes exactly will the extended attribute be used?

2> Are the extended attribute space always used up to this limit of bytes?

3> There is no per-inode setting of the inlining, right? For example,
for extents I check for ext4_inode.i_flags if there is EXT4_EXTENTS_FL
inside it. I couldn't find an equivalent EXT4_*_FL for inlining or
does it exist?

4> Do all inodes contain extended attributes? If one didn't previously
contain any the disk saving would not happen I guess.

thanks a lot,
-- 
Felipe Monteiro de Carvalho

Re: Which features should I implement in my ext4 reader?

[Felipe Monteiro de Carvalho]

Hello,

I went forward trying to implement this myself, but I am having trouble.

Files up to 60 bytes work great, and large files too. But a file with
61 bytes does not.

Reading ignoring the inline results in zero on all bytes, which can't
be right, so I guess it is inlined.

After the first 60 bytes I try to get the extended attribute, but
i_file_acl_lo == FData.l_i_file_acl_high == 0

And I think that it cannot possibly have the extended attribute data
in block zero =/ (it contains DEADBEEF or something like that)

Any ideas where I got this wrong or any tips about how to figure out
in debugfs where the data is?

thanks,
-- 
Felipe Monteiro de Carvalho

Re: Which features should I implement in my ext4 reader?

[Felipe Monteiro de Carvalho]

Hello,

Any idea where to find the extra data over 60 bytes in a very small
file (for example 120 bytes) would be greatly appreciated ... I
already tried the extra space after the inode but there was nothing
useful there! =/ Or at least nothing that matched the description of
the extended attributes. It should start with a magic number, but
started with many zeroes =/

thanks,

Felipe Monteiro de Carvalho

On Fri, Aug 23, 2013 at 2:54 PM, Felipe Monteiro de Carvalho
<felipemonteiro.carvalho@gmail.com> wrote:
> Hello,
>
> I went forward trying to implement this myself, but I am having trouble.
>
> Files up to 60 bytes work great, and large files too. But a file with
> 61 bytes does not.
>
> Reading ignoring the inline results in zero on all bytes, which can't
> be right, so I guess it is inlined.
>
> After the first 60 bytes I try to get the extended attribute, but
> i_file_acl_lo == FData.l_i_file_acl_high == 0
>
> And I think that it cannot possibly have the extended attribute data
> in block zero =/ (it contains DEADBEEF or something like that)
>
> Any ideas where I got this wrong or any tips about how to figure out
> in debugfs where the data is?
>
> thanks,
> --
> Felipe Monteiro de Carvalho



-- 
Felipe Monteiro de Carvalho

Re: Which features should I implement in my ext4 reader?

[Felipe Monteiro de Carvalho]

Hello,

Ok, I found the extra data over 60 bytes ... it is in an extended
attribute called "data" in the space after the inode. The wiki says it
should be called "system.data" =(

Also the wiki says here
https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Extended_Attributes
that "Extended attributes, when >>>not<<<< stored after the inode,
have a header ext4_xattr_ibody_header that is 4 bytes long:"

Which in my oppinion is false, it should be: "Extended attributes,
when stored after the inode, have a header ext4_xattr_ibody_header
that is 4 bytes long:"  without the "not"

But I still am not sure of one thing: How can I know how many entries
of extended attributes there are? Nothing in the wiki explains this =(

Otherwise I cannot differentiate between yet another entry and simply
data from an extended attribute...

-- 
Felipe Monteiro de Carvalho

Re: Which features should I implement in my ext4 reader?

[Zheng Liu]

Hello,

Sorry for my late reply.

On Mon, Sep 02, 2013 at 11:40:15AM +0200, Felipe Monteiro de Carvalho wrote:
> Hello,
> 
> Ok, I found the extra data over 60 bytes ... it is in an extended
> attribute called "data" in the space after the inode. The wiki says it
> should be called "system.data" =(
> 
> Also the wiki says here
> https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Extended_Attributes
> that "Extended attributes, when >>>not<<<< stored after the inode,
> have a header ext4_xattr_ibody_header that is 4 bytes long:"
> 
> Which in my oppinion is false, it should be: "Extended attributes,
> when stored after the inode, have a header ext4_xattr_ibody_header
> that is 4 bytes long:"  without the "not"
> 
> But I still am not sure of one thing: How can I know how many entries
> of extended attributes there are? Nothing in the wiki explains this =(

You need to traverse the all entries to get this value.

> 
> Otherwise I cannot differentiate between yet another entry and simply
> data from an extended attribute...

You can use attribute name and name index to differentiate these
entries.  You can read the comment at the front of fs/ext4/xattr.c and
it might help you to understand extended attribute.

Regards,
                                                - Zheng

Re: Which features should I implement in my ext4 reader?

[Felipe Monteiro de Carvalho]

Hello,

Ok, I think I got it almost finished =) Just 1 last wierd thing. I
dumped my inode and I found that when the extended attribute is in the
space after the inode, then the attribute data starts 3 bytes before
the end of this area! But in the wiki nothing is said about this. It
was supposed to start at the end of the section ... not 3 bytes before
the end of the section

Here are the 3 zeroes in the end:

7760 4444 4444 4444 4444 4444 4444 0a00 0000 DDDDDDDDDDDD....

Any idea what are those 3 zeroes in the end? I simply hardcoded in my
application that there exists 3 zeroes in the end, but its really
strange as this does not make any sense to waste 3 bytes and I didn't
see any such a thing in the wiki nor the xattr.h header...

Also wierd: This 3 last bytes do no appear when extended attributes
are stored in a separate block.

Full set of dumps:

debugfs: stat smallfile120.txt
Inode: 16 Type: regular Mode: 0644 Flags: 0x10000000
Generation: 230499867 Version: 0x00000000:00000001
User: 0 Group: 0 Size: 121
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 0
Fragment: Address: 0 Number: 0 Size: 0
 ctime: 0x52149464:04e2df1c -- Wed Aug 21 06:20:20 2013
 atime: 0x52177658:c0a4e548 -- Fri Aug 23 10:48:56 2013
 mtime: 0x52149464:04e2df1c -- Wed Aug 21 06:20:20 2013
crtime: 0x52149464:04e2df1c -- Wed Aug 21 06:20:20 2013
Size of extra inode fields: 28
Extended attributes stored in inode body:
  data = "CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD\012" (61)
INLINE DATA:
  The size of inline data: 121

More detailed information about the file with extended attribute:

debugfs: imap smallfile120.txt
Inode 16 is part of block group 0
        located at block 65, offset 0x0f00

So we know that it is located in the block 0000065 with offset 0x0f00
= 0740 octal

debugfs: block_dump 65
.....
*
7400 a481 0000 7900 0000 5876 1752 6494 1452 ....y...Xv.Rd..R
7420 6494 1452 0000 0000 0000 0100 0000 0000 d..R............
7440 0000 0010 0100 0000 4141 4141 4141 4141 ........AAAAAAAA
7460 4141 4141 4141 4141 4141 4141 4141 4141 AAAAAAAAAAAAAAAA
7500 4141 4141 4141 4242 4242 4242 4242 4242 AAAAAABBBBBBBBBB
7520 4242 4242 4242 4242 4242 4242 4242 4242 BBBBBBBBBBBBBBBB
7540 4242 4242 1b26 bd0d 0000 0000 0000 0000 BBBB.&..........
7560 0000 0000 0000 0000 0000 0000 0000 0000 ................
7600 1c00 0000 1cdf e204 1cdf e204 48e5 a4c0 ............H...
7620 6494 1452 1cdf e204 0000 0000 0000 02ea d..R............
7640 0407 2000 0000 0000 3d00 0000 0000 0000 .. .....=.......
7660 6461 7461 0000 0000 0000 0000 0000 0000 data............
7700 4343 4343 4343 4343 4343 4343 4343 4343 CCCCCCCCCCCCCCCC
7720 4343 4343 4343 4343 4343 4343 4343 4444 CCCCCCCCCCCCCCDD
7740 4444 4444 4444 4444 4444 4444 4444 4444 DDDDDDDDDDDDDDDD
7760 4444 4444 4444 4444 4444 4444 0a00 0000 DDDDDDDDDDDD....

And thats how I got to this place.

thanks,

Felipe Monteiro de Carvalho

On Mon, Sep 2, 2013 at 2:07 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> Hello,
>
> Sorry for my late reply.
>
> On Mon, Sep 02, 2013 at 11:40:15AM +0200, Felipe Monteiro de Carvalho wrote:
>> Hello,
>>
>> Ok, I found the extra data over 60 bytes ... it is in an extended
>> attribute called "data" in the space after the inode. The wiki says it
>> should be called "system.data" =(
>>
>> Also the wiki says here
>> https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Extended_Attributes
>> that "Extended attributes, when >>>not<<<< stored after the inode,
>> have a header ext4_xattr_ibody_header that is 4 bytes long:"
>>
>> Which in my oppinion is false, it should be: "Extended attributes,
>> when stored after the inode, have a header ext4_xattr_ibody_header
>> that is 4 bytes long:"  without the "not"
>>
>> But I still am not sure of one thing: How can I know how many entries
>> of extended attributes there are? Nothing in the wiki explains this =(
>
> You need to traverse the all entries to get this value.
>
>>
>> Otherwise I cannot differentiate between yet another entry and simply
>> data from an extended attribute...
>
> You can use attribute name and name index to differentiate these
> entries.  You can read the comment at the front of fs/ext4/xattr.c and
> it might help you to understand extended attribute.
>
> Regards,
>                                                 - Zheng



-- 
Felipe Monteiro de Carvalho

Re: Which features should I implement in my ext4 reader?

[Zheng Liu]

On Mon, Sep 02, 2013 at 02:35:31PM +0200, Felipe Monteiro de Carvalho wrote:
> Hello,
> 
> Ok, I think I got it almost finished =) Just 1 last wierd thing. I
> dumped my inode and I found that when the extended attribute is in the
> space after the inode, then the attribute data starts 3 bytes before
> the end of this area! But in the wiki nothing is said about this. It
> was supposed to start at the end of the section ... not 3 bytes before
> the end of the section
> 
> Here are the 3 zeroes in the end:
> 
> 7760 4444 4444 4444 4444 4444 4444 0a00 0000 DDDDDDDDDDDD....
> 
> Any idea what are those 3 zeroes in the end? I simply hardcoded in my
> application that there exists 3 zeroes in the end, but its really
> strange as this does not make any sense to waste 3 bytes and I didn't
> see any such a thing in the wiki nor the xattr.h header...
> 
> Also wierd: This 3 last bytes do no appear when extended attributes
> are stored in a separate block.

Could you please tell me how to reproduce this problem?  I try to create
a file with inline data, but I couldn't reproduce your problem.

Thanks,
                                                - Zheng

> 
> Full set of dumps:
> 
> debugfs: stat smallfile120.txt
> Inode: 16 Type: regular Mode: 0644 Flags: 0x10000000
> Generation: 230499867 Version: 0x00000000:00000001
> User: 0 Group: 0 Size: 121
> File ACL: 0 Directory ACL: 0
> Links: 1 Blockcount: 0
> Fragment: Address: 0 Number: 0 Size: 0
>  ctime: 0x52149464:04e2df1c -- Wed Aug 21 06:20:20 2013
>  atime: 0x52177658:c0a4e548 -- Fri Aug 23 10:48:56 2013
>  mtime: 0x52149464:04e2df1c -- Wed Aug 21 06:20:20 2013
> crtime: 0x52149464:04e2df1c -- Wed Aug 21 06:20:20 2013
> Size of extra inode fields: 28
> Extended attributes stored in inode body:
>   data = "CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD\012" (61)
> INLINE DATA:
>   The size of inline data: 121
> 
> More detailed information about the file with extended attribute:
> 
> debugfs: imap smallfile120.txt
> Inode 16 is part of block group 0
>         located at block 65, offset 0x0f00
> 
> So we know that it is located in the block 0000065 with offset 0x0f00
> = 0740 octal
> 
> debugfs: block_dump 65
> .....
> *
> 7400 a481 0000 7900 0000 5876 1752 6494 1452 ....y...Xv.Rd..R
> 7420 6494 1452 0000 0000 0000 0100 0000 0000 d..R............
> 7440 0000 0010 0100 0000 4141 4141 4141 4141 ........AAAAAAAA
> 7460 4141 4141 4141 4141 4141 4141 4141 4141 AAAAAAAAAAAAAAAA
> 7500 4141 4141 4141 4242 4242 4242 4242 4242 AAAAAABBBBBBBBBB
> 7520 4242 4242 4242 4242 4242 4242 4242 4242 BBBBBBBBBBBBBBBB
> 7540 4242 4242 1b26 bd0d 0000 0000 0000 0000 BBBB.&..........
> 7560 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 7600 1c00 0000 1cdf e204 1cdf e204 48e5 a4c0 ............H...
> 7620 6494 1452 1cdf e204 0000 0000 0000 02ea d..R............
> 7640 0407 2000 0000 0000 3d00 0000 0000 0000 .. .....=.......
> 7660 6461 7461 0000 0000 0000 0000 0000 0000 data............
> 7700 4343 4343 4343 4343 4343 4343 4343 4343 CCCCCCCCCCCCCCCC
> 7720 4343 4343 4343 4343 4343 4343 4343 4444 CCCCCCCCCCCCCCDD
> 7740 4444 4444 4444 4444 4444 4444 4444 4444 DDDDDDDDDDDDDDDD
> 7760 4444 4444 4444 4444 4444 4444 0a00 0000 DDDDDDDDDDDD....
> 
> And thats how I got to this place.
> 
> thanks,
> 
> Felipe Monteiro de Carvalho
> 
> On Mon, Sep 2, 2013 at 2:07 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> > Hello,
> >
> > Sorry for my late reply.
> >
> > On Mon, Sep 02, 2013 at 11:40:15AM +0200, Felipe Monteiro de Carvalho wrote:
> >> Hello,
> >>
> >> Ok, I found the extra data over 60 bytes ... it is in an extended
> >> attribute called "data" in the space after the inode. The wiki says it
> >> should be called "system.data" =(
> >>
> >> Also the wiki says here
> >> https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Extended_Attributes
> >> that "Extended attributes, when >>>not<<<< stored after the inode,
> >> have a header ext4_xattr_ibody_header that is 4 bytes long:"
> >>
> >> Which in my oppinion is false, it should be: "Extended attributes,
> >> when stored after the inode, have a header ext4_xattr_ibody_header
> >> that is 4 bytes long:"  without the "not"
> >>
> >> But I still am not sure of one thing: How can I know how many entries
> >> of extended attributes there are? Nothing in the wiki explains this =(
> >
> > You need to traverse the all entries to get this value.
> >
> >>
> >> Otherwise I cannot differentiate between yet another entry and simply
> >> data from an extended attribute...
> >
> > You can use attribute name and name index to differentiate these
> > entries.  You can read the comment at the front of fs/ext4/xattr.c and
> > it might help you to understand extended attribute.
> >
> > Regards,
> >                                                 - Zheng
> 
> 
> 
> -- 
> Felipe Monteiro de Carvalho
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Which features should I implement in my ext4 reader?

[jon ernst]

On Wed, Sep 4, 2013 at 1:35 AM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
>
> On Mon, Sep 02, 2013 at 02:35:31PM +0200, Felipe Monteiro de Carvalho wrote:
> > Hello,
> >
> > Ok, I think I got it almost finished =) Just 1 last wierd thing. I
> > dumped my inode and I found that when the extended attribute is in the
> > space after the inode, then the attribute data starts 3 bytes before
> > the end of this area! But in the wiki nothing is said about this. It
> > was supposed to start at the end of the section ... not 3 bytes before
> > the end of the section
> >
> > Here are the 3 zeroes in the end:
> >
> > 7760 4444 4444 4444 4444 4444 4444 0a00 0000 DDDDDDDDDDDD....
> >
> > Any idea what are those 3 zeroes in the end? I simply hardcoded in my
> > application that there exists 3 zeroes in the end, but its really
> > strange as this does not make any sense to waste 3 bytes and I didn't
> > see any such a thing in the wiki nor the xattr.h header...
> >
> > Also wierd: This 3 last bytes do no appear when extended attributes
> > are stored in a separate block.
>
> Could you please tell me how to reproduce this problem?  I try to create
> a file with inline data, but I couldn't reproduce your problem.
>
> Thanks,
>                                                 - Zheng
>
I did reproduce this one. what I did is: use pu branch e2fsprogs to
make ext4 image, mount it, cat
"AAAAAAAAAABBBBBBBBBBCCCCCCCCCCDDDDDDDDDD" to a file.
Then use debugfs to check corresponding block.
I got almost same result as Felipe's.
kernel 3.8.0-27

But I don't quite understand why does Felipe suspect those padding 0s?
What's wrong with it? Pardon my silly question.

Jon
>
> >
> > Full set of dumps:
> >
> > debugfs: stat smallfile120.txt
> > Inode: 16 Type: regular Mode: 0644 Flags: 0x10000000
> > Generation: 230499867 Version: 0x00000000:00000001
> > User: 0 Group: 0 Size: 121
> > File ACL: 0 Directory ACL: 0
> > Links: 1 Blockcount: 0
> > Fragment: Address: 0 Number: 0 Size: 0
> >  ctime: 0x52149464:04e2df1c -- Wed Aug 21 06:20:20 2013
> >  atime: 0x52177658:c0a4e548 -- Fri Aug 23 10:48:56 2013
> >  mtime: 0x52149464:04e2df1c -- Wed Aug 21 06:20:20 2013
> > crtime: 0x52149464:04e2df1c -- Wed Aug 21 06:20:20 2013
> > Size of extra inode fields: 28
> > Extended attributes stored in inode body:
> >   data = "CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD\012" (61)
> > INLINE DATA:
> >   The size of inline data: 121
> >
> > More detailed information about the file with extended attribute:
> >
> > debugfs: imap smallfile120.txt
> > Inode 16 is part of block group 0
> >         located at block 65, offset 0x0f00
> >
> > So we know that it is located in the block 0000065 with offset 0x0f00
> > = 0740 octal
> >
> > debugfs: block_dump 65
> > .....
> > *
> > 7400 a481 0000 7900 0000 5876 1752 6494 1452 ....y...Xv.Rd..R
> > 7420 6494 1452 0000 0000 0000 0100 0000 0000 d..R............
> > 7440 0000 0010 0100 0000 4141 4141 4141 4141 ........AAAAAAAA
> > 7460 4141 4141 4141 4141 4141 4141 4141 4141 AAAAAAAAAAAAAAAA
> > 7500 4141 4141 4141 4242 4242 4242 4242 4242 AAAAAABBBBBBBBBB
> > 7520 4242 4242 4242 4242 4242 4242 4242 4242 BBBBBBBBBBBBBBBB
> > 7540 4242 4242 1b26 bd0d 0000 0000 0000 0000 BBBB.&..........
> > 7560 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > 7600 1c00 0000 1cdf e204 1cdf e204 48e5 a4c0 ............H...
> > 7620 6494 1452 1cdf e204 0000 0000 0000 02ea d..R............
> > 7640 0407 2000 0000 0000 3d00 0000 0000 0000 .. .....=.......
> > 7660 6461 7461 0000 0000 0000 0000 0000 0000 data............
> > 7700 4343 4343 4343 4343 4343 4343 4343 4343 CCCCCCCCCCCCCCCC
> > 7720 4343 4343 4343 4343 4343 4343 4343 4444 CCCCCCCCCCCCCCDD
> > 7740 4444 4444 4444 4444 4444 4444 4444 4444 DDDDDDDDDDDDDDDD
> > 7760 4444 4444 4444 4444 4444 4444 0a00 0000 DDDDDDDDDDDD....
> >
> > And thats how I got to this place.
> >
> > thanks,
> >
> > Felipe Monteiro de Carvalho
> >
> > On Mon, Sep 2, 2013 at 2:07 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> > > Hello,
> > >
> > > Sorry for my late reply.
> > >
> > > On Mon, Sep 02, 2013 at 11:40:15AM +0200, Felipe Monteiro de Carvalho wrote:
> > >> Hello,
> > >>
> > >> Ok, I found the extra data over 60 bytes ... it is in an extended
> > >> attribute called "data" in the space after the inode. The wiki says it
> > >> should be called "system.data" =(
> > >>
> > >> Also the wiki says here
> > >> https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Extended_Attributes
> > >> that "Extended attributes, when >>>not<<<< stored after the inode,
> > >> have a header ext4_xattr_ibody_header that is 4 bytes long:"
> > >>
> > >> Which in my oppinion is false, it should be: "Extended attributes,
> > >> when stored after the inode, have a header ext4_xattr_ibody_header
> > >> that is 4 bytes long:"  without the "not"
> > >>
> > >> But I still am not sure of one thing: How can I know how many entries
> > >> of extended attributes there are? Nothing in the wiki explains this =(
> > >
> > > You need to traverse the all entries to get this value.
> > >
> > >>
> > >> Otherwise I cannot differentiate between yet another entry and simply
> > >> data from an extended attribute...
> > >
> > > You can use attribute name and name index to differentiate these
> > > entries.  You can read the comment at the front of fs/ext4/xattr.c and
> > > it might help you to understand extended attribute.
> > >
> > > Regards,
> > >                                                 - Zheng
> >
> >
> >
> > --
> > Felipe Monteiro de Carvalho
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Which features should I implement in my ext4 reader?

[Felipe Monteiro de Carvalho]

Hello,

On Thu, Sep 5, 2013 at 6:50 AM, jon ernst <jonernst07@gmail.com> wrote:
> But I don't quite understand why does Felipe suspect those padding 0s?
> What's wrong with it? Pardon my silly question.

My point is why there are exactly 3 zeroes of padding? Can sometimes
it be 2 zeroes? Or 1? Or none? How is this decided? Any tips to where
in the source code this padding is decided?

I am implementing my own ext4 reader library so I am not using the
standard library, so this kind of detail is very important to me to
make sure that my implementation is correct. For now I hard coded the
existence of 3 zeroes of padding, but maybe the amount could be
different?

Is it to make sure that the block of data starts aligned to 4-bytes?

thanks,
-- 
Felipe Monteiro de Carvalho

Re: Which features should I implement in my ext4 reader?

[Felipe Monteiro de Carvalho]

On Thu, Sep 5, 2013 at 9:04 AM, Felipe Monteiro de Carvalho
<felipemonteiro.carvalho@gmail.com> wrote:
> Is it to make sure that the block of data starts aligned to 4-bytes?

Hello,

Ok, I just checked here experimentally and it is indeed padding to 4-bytes...

Does the same padding exist if the attribute is in a block as opposed
to being in the space after the inode?

thanks,
-- 
Felipe Monteiro de Carvalho

Re: Which features should I implement in my ext4 reader?

[Darrick J. Wong]

On Mon, Sep 02, 2013 at 11:40:15AM +0200, Felipe Monteiro de Carvalho wrote:
> Hello,
> 
> Ok, I found the extra data over 60 bytes ... it is in an extended
> attribute called "data" in the space after the inode. The wiki says it
> should be called "system.data" =(
> 
> Also the wiki says here
> https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Extended_Attributes
> that "Extended attributes, when >>>not<<<< stored after the inode,
> have a header ext4_xattr_ibody_header that is 4 bytes long:"

Oops, that's right.  I corrected the wiki page.

--D
> 
> Which in my oppinion is false, it should be: "Extended attributes,
> when stored after the inode, have a header ext4_xattr_ibody_header
> that is 4 bytes long:"  without the "not"
> 
> But I still am not sure of one thing: How can I know how many entries
> of extended attributes there are? Nothing in the wiki explains this =(
> 
> Otherwise I cannot differentiate between yet another entry and simply
> data from an extended attribute...
> 
> -- 
> Felipe Monteiro de Carvalho
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Which features should I implement in my ext4 reader?

[Theodore Ts'o]

On Sun, Aug 18, 2013 at 08:04:20AM +0200, Felipe Monteiro de Carvalho wrote:
> > These features aren't quite fully supported yet.  The inline data
> > feature is probably closest to be being fully supported, when we
> > release an the next major version of e2fsprogs.
> 
> Just for completeness, which version number will that by?

When we release 1.43.x.  We are currently releasing maintenance
releases on 1.42.x, tracked on the maint branch.  What will eventually
become e2fsprogs 1.43.x is on the master branch in the git tree.  (As
well as the next and pu -- proposed updates -- branches, which are
supersets of the master branch.)

> So, I understand that currently mkfs.ext4 cannot generate ext4 file
> systems with this feature, correct?

Well, if you are building with the e2fsprogs with the pu branch, it
will work.  This is a rewinding branch, so for example, when I process
Zheng's latest set of changes, the pu branch will get rewound and the
new patches applied.

BTW, for any new ABI's exported via libext2fs, there is no guarantee
of stability until 1.43.x gets released.  If we find ABI
incompatibilities for old interfaces in between 1.43.x and 1.42.x on
the master branch, that is a bug that we will fix.

> Is there any other way to generate a file system with this feature
> with a 3.8 kernel which will work?
> 
> Just wondering if I can already start work on this or if I need to wait.

It should be sufficient to use debugfs to manually set the inline_data
feature.  However, you won't be able to run e2fsck on the file system
after you do that.  So using the pu branch is probably your best bet.

The git tree can be found here:

    git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git

Regards,

						- Ted