inline_data feature

inline_data feature

[Ian Nartowicz]

I have been trying to test the inline_data feature of EXT4.  I am using
kernel 3.9.5 and have compiled e2fsprogs from the pu branch.  I am able to
set the feature and apparently to use the filesystem, but I get a lot of
warnings from the kernel and fsck reports several errors and then crashes.

Typical kernel warnings from dmesg:
EXT4-fs warning (device sda10): ext4_rmdir:2714: empty directory has too
many links (9)
EXT4-fs error (device sda10): empty_inline_dir:1650: inode #26263: block
1924: comm pool: bad entry in directory: directory entry across range -
offset=40(40), inode=26270, rec_len=8020, name_len=2
EXT4-fs warning (device sda10): empty_inline_dir:1657: bad inline directory
(dir #26263) - inode 26270, rec_len 8020, name_len 2inline size 60

Data does appear to be correctly written and read from the filesystem
without corruption.  Small directories appear with size 60 bytes or 132
bytes rather than 4.1kB so everything seems to work.

Re: inline_data feature

[Ian Nartowicz]

Ian Nartowicz <claws <at> nartowicz.co.uk> writes:

> 
> I have been trying to test the inline_data feature of EXT4.  I am using
> kernel 3.9.5 and have compiled e2fsprogs from the pu branch.  I am able to
> set the feature and apparently to use the filesystem, but I get a lot of
> warnings from the kernel and fsck reports several errors and then crashes.
> 



Seems like fsck is the culprit.  No errors until I try to check the filesystem.

Re: inline_data feature

[Theodore Ts'o]

On Fri, Jan 24, 2014 at 04:05:48PM +0000, Ian Nartowicz wrote:
> > I have been trying to test the inline_data feature of EXT4.  I am using
> > kernel 3.9.5 and have compiled e2fsprogs from the pu branch.  I am able to
> > set the feature and apparently to use the filesystem, but I get a lot of
> > warnings from the kernel and fsck reports several errors and then crashes.
> 
> Seems like fsck is the culprit.  No errors until I try to check the filesystem.

Can you send us the fsck transcript?

Thanks,

					- Ted

Re: inline_data feature

[Ian Nartowicz]

Theodore Ts'o <tytso <at> mit.edu> writes:


> 
> Can you send us the fsck transcript?
> 
> Thanks,
> 
> 					- Ted

>From command:
sudo ./e2fsck -yf /dev/sda10 > fsck.out

e2fsck 1.43-WIP (8-Jul-2013)
*** glibc detected *** ./e2fsck: corrupted double-linked list: 0x095989a8 ***
======= Backtrace: =========
[0x80c1ea8]
[0x80c35ba]
[0x80c62b9]
[0x808473e]
======= Memory map: ========
08048000-08148000 r-xp 00000000 08:09 298149    
/data/src/e2fsprogs/e2fsck/e2fsck
08148000-0814d000 rw-p 000ff000 08:09 298149    
/data/src/e2fsprogs/e2fsck/e2fsck
0814d000-0814f000 rw-p 00000000 00:00 0 
09567000-09645000 rw-p 00000000 00:00 0          [heap]
a7400000-a7421000 rw-p 00000000 00:00 0 
a7421000-a7500000 ---p 00000000 00:00 0 
a75c8000-a7609000 rw-p 00000000 00:00 0 
a7609000-a7610000 r--s 00000000 08:05 2226      
/usr/lib/i386-linux-gnu/gconv/gconv-modules.cache
a7610000-a7787000 r--p 00000000 08:05 1909       /usr/lib/locale/locale-
archive
a7787000-a7788000 r-xp 00000000 00:00 0          [vdso]
af7eb000-af80c000 rw-p 00000000 00:00 0          [stack]
Signal (6) SIGABRT si_code=SI_TKILL 
[0x806c375]
[0xa778740c]
[0x80dfd21]
[0x80a91f3]
[0x80bb0b5]
[0x80c1ea8]
[0x80c35ba]
[0x80c62b9]
[0x808473e]


fsck.out:

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Directory inode 16, block #0, offset 60: directory corrupted
Salvage? yes

Entry
've^@^@E^A^@^@^L^@^C^Brss^@F^A^@^@^P^@^F^Bshares^@^@G^A^@^@^L^@^C^Bdht^@M^A
^@^@M-,^A^H^Btorrents^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@'
in ??? (16) has rec_len of 25441, should be 28637.
Fix? yes

Directory inode 28, block #0, offset 60: directory corrupted
Salvage? yes

Entry '' in ??? (28) has rec_len of 0, should be 3196.
Fix? yes


--iann

Re: inline_data feature

[Ian Nartowicz]

Ian Nartowicz <claws <at> nartowicz.co.uk> writes:

> 
> I have been trying to test the inline_data feature of EXT4.  I am using
> kernel 3.9.5 and have compiled e2fsprogs from the pu branch.  I am able to
> set the feature and apparently to use the filesystem, but I get a lot of
> warnings from the kernel and fsck reports several errors and then crashes.
> 
I wonder if I might have compiled against the wrong (clib?) headers?  Are
you aware of any version dependencies in the pu branch tools?

--ian

Re: inline_data feature

[Theodore Ts'o]

On Sat, Jan 25, 2014 at 08:22:56PM +0000, Ian Nartowicz wrote:
> Ian Nartowicz <claws <at> nartowicz.co.uk> writes:
> 
> > 
> > I have been trying to test the inline_data feature of EXT4.  I am using
> > kernel 3.9.5 and have compiled e2fsprogs from the pu branch.  I am able to
> > set the feature and apparently to use the filesystem, but I get a lot of
> > warnings from the kernel and fsck reports several errors and then crashes.
> > 
> I wonder if I might have compiled against the wrong (clib?) headers?  Are
> you aware of any version dependencies in the pu branch tools?

I doubt it; this looks like a straightforward memory bug of some kind
(i.e., use after free, etc.)

Could you try running e2fsck under valgrind?  And could you also
confirm the git commit ID of the e2fsprogs pu branch that you used to
compile it?

A compressed raw e2image file would be useful as well, but one problem
with inline data is that a raw e2image file will have some amount of
file data (for the small files that are using inline data) in the
e2image file.  We should probably look at teaching e2image to can zero
out the contents of inline data files....

Many thanks,

					- Ted

Re: inline_data feature

[Zheng Liu]

On Sat, Jan 25, 2014 at 10:29:09PM -0500, Theodore Ts'o wrote:
> On Sat, Jan 25, 2014 at 08:22:56PM +0000, Ian Nartowicz wrote:
> > Ian Nartowicz <claws <at> nartowicz.co.uk> writes:
> > 
> > > 
> > > I have been trying to test the inline_data feature of EXT4.  I am using
> > > kernel 3.9.5 and have compiled e2fsprogs from the pu branch.  I am able to
> > > set the feature and apparently to use the filesystem, but I get a lot of
> > > warnings from the kernel and fsck reports several errors and then crashes.
> > > 
> > I wonder if I might have compiled against the wrong (clib?) headers?  Are
> > you aware of any version dependencies in the pu branch tools?
> 
> I doubt it; this looks like a straightforward memory bug of some kind
> (i.e., use after free, etc.)
> 
> Could you try running e2fsck under valgrind?  And could you also
> confirm the git commit ID of the e2fsprogs pu branch that you used to
> compile it?
> 
> A compressed raw e2image file would be useful as well, but one problem
> with inline data is that a raw e2image file will have some amount of
> file data (for the small files that are using inline data) in the
> e2image file.  We should probably look at teaching e2image to can zero
> out the contents of inline data files....

Ian,
Thanks for reporting the problem.  That would be great if you can run
e2fsck under valgrind as Ted suggested because it would help us to find
the root cause.

Further, Here is the latest patch set for making e2fsprogs support
inline_data feature [1], which isn't applied into e2fsprogs/pu branch.
This patch set is against e2fsprogs/next branch.  Could you please give
it a try?  Unfortunately the patch set contains a lot of patches.  I am
happy to provide a tar file to you.  Please let me know if you need any
help.

1. http://www.spinics.net/lists/linux-ext4/msg40852.html

Ted,
It seems that we should teach e2image to zero out the inline data files.
I will put this in my TODO list but I am afraid that it doesn't have a
higher priority. :(

Regards,
                                                - Zheng

Re: inline_data feature

[Ian Nartowicz]

Zheng Liu <gnehzuil.liu <at> gmail.com> writes:

> 
> On Sat, Jan 25, 2014 at 10:29:09PM -0500, Theodore Ts'o wrote:
> > On Sat, Jan 25, 2014 at 08:22:56PM +0000, Ian Nartowicz wrote:
> > > Ian Nartowicz <claws <at> nartowicz.co.uk> writes:
> > > 
> > > > 
> > > > I have been trying to test the inline_data feature of EXT4.
> > 
> > Could you try running e2fsck under valgrind?  And could you also
> > confirm the git commit ID of the e2fsprogs pu branch that you used to
> > compile it?
> > 
> 
> Ian,
> Thanks for reporting the problem.  That would be great if you can run
> e2fsck under valgrind as Ted suggested because it would help us to find
> the root cause.
> 
> Further, Here is the latest patch set for making e2fsprogs support
> inline_data feature [1], which isn't applied into e2fsprogs/pu branch.
> This patch set is against e2fsprogs/next branch.

> Regards,
>                                                 - Zheng
> --


I was using the pu commit ID 11266daa7cc2af6601929f041684f1e2a2c4f773 from
October 14th.  I'll try the latest patches against next.  Are those patches
available as a tarball?

I ran valgrind and got a lot of output.  Mostly uninitialised stuff, moves
and one or two others.  Do you want that or just wait and see how the new
build goes?

--ian

Re: inline_data feature

[Ian Nartowicz]

Ian Nartowicz <claws <at> nartowicz.co.uk> writes:

> 
> I have been trying to test the inline_data feature of EXT4.  I am using
> kernel 3.9.5 and have compiled e2fsprogs from the pu branch.  I am able to
> set the feature and apparently to use the filesystem, but I get a lot of
> warnings from the kernel and fsck reports several errors and then crashes.
> 
> Typical kernel warnings from dmesg:
> EXT4-fs warning (device sda10): ext4_rmdir:2714: empty directory has too
> many links (9)
> EXT4-fs error (device sda10): empty_inline_dir:1650: inode #26263: block
> 1924: comm pool: bad entry in directory: directory entry across range -
> offset=40(40), inode=26270, rec_len=8020, name_len=2
> EXT4-fs warning (device sda10): empty_inline_dir:1657: bad inline directory
> (dir #26263) - inode 26270, rec_len 8020, name_len 2inline size 60
> 
> Data does appear to be correctly written and read from the filesystem
> without corruption.  Small directories appear with size 60 bytes or 132
> bytes rather than 4.1kB so everything seems to work.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

I have built with the latest patchset and fsck is now a happy bunny, at
least on my first pass.

However there is a different problem, possibly not with fsck.  For my tests
I copied the contents of /home onto the new partition.  fsck reports, and
other utilities confirm, that symlinks with targets of 60 characters or
longer were corrupted by the copy.  For example, a truncated symlink:
All That You Can't Leave Behind.m3u -> /data/cd/U2/All That You Can't Leave
Behind/playlist.flac.m

If I create the same symlink with ln, it appears OK until I unmount and
mount the partition, then it shows truncated.  fsck doesn't like it but is
unable to correct it.

--ian

Re: inline_data feature

[Zheng Liu]

On Sun, Feb 09, 2014 at 03:45:12PM +0000, Ian Nartowicz wrote:
> Ian Nartowicz <claws <at> nartowicz.co.uk> writes:
> 
> > 
> > I have been trying to test the inline_data feature of EXT4.  I am using
> > kernel 3.9.5 and have compiled e2fsprogs from the pu branch.  I am able to
> > set the feature and apparently to use the filesystem, but I get a lot of
> > warnings from the kernel and fsck reports several errors and then crashes.
> > 
> > Typical kernel warnings from dmesg:
> > EXT4-fs warning (device sda10): ext4_rmdir:2714: empty directory has too
> > many links (9)
> > EXT4-fs error (device sda10): empty_inline_dir:1650: inode #26263: block
> > 1924: comm pool: bad entry in directory: directory entry across range -
> > offset=40(40), inode=26270, rec_len=8020, name_len=2
> > EXT4-fs warning (device sda10): empty_inline_dir:1657: bad inline directory
> > (dir #26263) - inode 26270, rec_len 8020, name_len 2inline size 60
> > 
> > Data does appear to be correctly written and read from the filesystem
> > without corruption.  Small directories appear with size 60 bytes or 132
> > bytes rather than 4.1kB so everything seems to work.
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> > the body of a message to majordomo <at> vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> 
> I have built with the latest patchset and fsck is now a happy bunny, at
> least on my first pass.

I'd appreciate your testing.

> 
> However there is a different problem, possibly not with fsck.  For my tests
> I copied the contents of /home onto the new partition.  fsck reports, and
> other utilities confirm, that symlinks with targets of 60 characters or
> longer were corrupted by the copy.  For example, a truncated symlink:
> All That You Can't Leave Behind.m3u -> /data/cd/U2/All That You Can't Leave
> Behind/playlist.flac.m
> 
> If I create the same symlink with ln, it appears OK until I unmount and
> mount the partition, then it shows truncated.  fsck doesn't like it but is
> unable to correct it.

That would be great if you can provide some steps to reproduce this
issue.  I write a simple script to try to reproduce it, but I couldn't
hit the problem.  Am I missing something?

  #!/bin/bash
  
  mkdir test
  cd test
  filename="ALL-That-You-Can't-Leave-Behind.m3u"
  
  echo "hello" > $filename
  ln -s $filename symlinkfile
  readlink symlinkfile
  
  newdir="data/cd"
  mkdir -p $newdir
  cp -d symlinkfile $newdir
  readlink $newdir/symlinkfile
  
Thanks,
                                                - Zheng

Re: inline_data feature

[Ian Nartowicz]

Zheng Liu <gnehzuil.liu <at> gmail.com> writes:

> 
> On Sun, Feb 09, 2014 at 03:45:12PM +0000, Ian Nartowicz wrote:
> > Ian Nartowicz <claws <at> nartowicz.co.uk> writes:
> > 
> > I copied the contents of /home onto the new partition.  fsck reports, and
> > other utilities confirm, that symlinks with targets of 60 characters or
> > longer were corrupted by the copy.  For example, a truncated symlink:
> > All That You Can't Leave Behind.m3u -> /data/cd/U2/All That You Can't Leave
> > Behind/playlist.flac.m
> > 
> > If I create the same symlink with ln, it appears OK until I unmount and
> > mount the partition, then it shows truncated.  fsck doesn't like it but is
> > unable to correct it.
> 
> That would be great if you can provide some steps to reproduce this
> issue.  I write a simple script to try to reproduce it, but I couldn't
> hit the problem.  Am I missing something?
> 
>   #!/bin/bash
> 
>   mkdir test
>   cd test
>   filename="ALL-That-You-Can't-Leave-Behind.m3u"
> 
>   echo "hello" > $filename
>   ln -s $filename symlinkfile
>   readlink symlinkfile
> 
>   newdir="data/cd"
>   mkdir -p $newdir
>   cp -d symlinkfile $newdir
>   readlink $newdir/symlinkfile
> 
> Thanks,
>                                                 - Zheng
> --

To reproduce this on my system, the full path for $filename needs to be
longer than 60 characters.  That seems to be the only requirement.  Create
the symlink, unmount, mount, and the target path of the synlink is truncated
to 59 characters and hence the link is broken.

fsck reports those symlinks as follows:
Symlink /iann/test.m3u (inode #139) is invalid.
Clear<y>? no
Entry 'test.m3u' in /iann (8001) has an incorrect filetype (was 7, should be 0).
Fix<y>? no

If I try to clear or fix, it just leaves behind damaged inodes which fsck
can't fix.

--ian

Re: inline_data feature

[Zheng Liu]

On Mon, Feb 10, 2014 at 12:06:05PM +0000, Ian Nartowicz wrote:
> Zheng Liu <gnehzuil.liu <at> gmail.com> writes:
> 
> > 
> > On Sun, Feb 09, 2014 at 03:45:12PM +0000, Ian Nartowicz wrote:
> > > Ian Nartowicz <claws <at> nartowicz.co.uk> writes:
> > > 
> > > I copied the contents of /home onto the new partition.  fsck reports, and
> > > other utilities confirm, that symlinks with targets of 60 characters or
> > > longer were corrupted by the copy.  For example, a truncated symlink:
> > > All That You Can't Leave Behind.m3u -> /data/cd/U2/All That You Can't Leave
> > > Behind/playlist.flac.m
> > > 
> > > If I create the same symlink with ln, it appears OK until I unmount and
> > > mount the partition, then it shows truncated.  fsck doesn't like it but is
> > > unable to correct it.
> > 
> > That would be great if you can provide some steps to reproduce this
> > issue.  I write a simple script to try to reproduce it, but I couldn't
> > hit the problem.  Am I missing something?
> > 
> >   #!/bin/bash
> > 
> >   mkdir test
> >   cd test
> >   filename="ALL-That-You-Can't-Leave-Behind.m3u"
> > 
> >   echo "hello" > $filename
> >   ln -s $filename symlinkfile
> >   readlink symlinkfile
> > 
> >   newdir="data/cd"
> >   mkdir -p $newdir
> >   cp -d symlinkfile $newdir
> >   readlink $newdir/symlinkfile
> > 
> > Thanks,
> >                                                 - Zheng
> > --
> 
> To reproduce this on my system, the full path for $filename needs to be
> longer than 60 characters.  That seems to be the only requirement.  Create
> the symlink, unmount, mount, and the target path of the synlink is truncated
> to 59 characters and hence the link is broken.
> 
> fsck reports those symlinks as follows:
> Symlink /iann/test.m3u (inode #139) is invalid.
> Clear<y>? no
> Entry 'test.m3u' in /iann (8001) has an incorrect filetype (was 7, should be 0).
> Fix<y>? no
> 
> If I try to clear or fix, it just leaves behind damaged inodes which fsck
> can't fix.

I believe that this patch can fix the bug that corrupts the fast
symlink [1].  Meanwhile e2fsck should fix this issue but it doesn't.  So
there is a bug in e2fsprogs.  I will try to fix it soon.

1. http://permalink.gmane.org/gmane.comp.file-systems.ext4/42518

Thanks,
                                                - Zheng