Status of ext4 tools

Status of ext4 tools

[J.A. Magallón]

Hi all...

I'm following the threads on ext4, but I have a couple questions, and
perhaps this can be a thread to bookmark with all the answers people
need... ;)

I have migrated some partitions to ext4 in several boxes (even my /home
on the One...and works fast), but...

- Where is the 'official' source for e4defrag ?
- I've read that e4defrag needs in-kernel support.
  Is it already in mainline ?
  From what kernel version ?
- I have seen that some distros refer to a package called 'e4fsprogs'.
  Does that thing exist ? Where ?
- Booting:
  - There seem to be patches for grub
    http://lists.openwall.net/linux-ext4/2008/11/19/8
    to boot ext4
  - and a Google SoC called 'grub4ext4'.
   -and there is also grub2 (ext4 support in svn).
  What would you recommend ?

That's what I miss to move completely to ext4 ;)

Apart from that, it works pretty fine.
The thing that most surprised me was that after the needed fsck, it had
converted files with contiguous blocks to big extents ;). Nice.

TIA

-- 
J.A. Magallon <jamagallon()ono!com>     \               Software is like sex:
                                         \         It's better when it's free
Mandriva Linux release 2009.1 (Cooker) for x86_64
Linux 2.6.28.2-desktop-1mnb (gcc 4.3.2 (GCC) #1 Wed Jan

Re: Status of ext4 tools

[Ralf Hildebrandt]

* J.A. Magallón <jamagallon@ono.com>:

> - I have seen that some distros refer to a package called 'e4fsprogs'.
>   Does that thing exist ? Where ?

RedHat seems to have that, but here on Debian/Ubuntu that doesn't exist.

-- 
Ralf Hildebrandt                                Ralf.Hildebrandt@charite.de
Charite - Universitätsmedizin Berlin            Tel.  +49 (0)30-450 570-155
Geschäftsbereich IT | Abt. Netzwerk             Fax.  +49 (0)30-450 570-962
Hindenburgdamm 30 | 12200 Berlin

Re: Status of ext4 tools

[Kay Diederichs]

Ralf Hildebrandt schrieb:
> * J.A. Magallón <jamagallon@ono.com>:
> 
>> - I have seen that some distros refer to a package called 'e4fsprogs'.
>>   Does that thing exist ? Where ?
> 
> RedHat seems to have that, but here on Debian/Ubuntu that doesn't exist.
> 

The release notes say
"Please note that "e4fsprogs" simply contains renamed static binaries
from the equivalent upstream e2fsprogs release; it is packaged this
way for Red Hat Enterprise Linux 5 to ensure that the many changes
included for ext4 do not destabilize the core e2fsprogs in RHEL5."

Re: Status of ext4 tools

[Theodore Tso]

On Thu, Feb 19, 2009 at 11:48:38PM +0100, J.A. Magallón wrote:
> 
> - Where is the 'official' source for e4defrag ?
> - I've read that e4defrag needs in-kernel support.
>   Is it already in mainline ?
>   From what kernel version ?

Ext4's defragmentation code is still under development.  There are
multiple versions of e4defrag, paired with kernel patches that have
been submitted to linux-ext4@vger.kernel.org.  The patches are not yet
in in mainline, and the kernel/userspace interface is still subject to
change (which is why it's not in mainline yet).

> - I have seen that some distros refer to a package called 'e4fsprogs'.
>   Does that thing exist ? Where ?

e4fsprogs was a snapshot of something between e2fprogs 1.41.1 and
e2fsprogs 1.41.2, if I remember correctly, and it was created for
RHEL5's technology preview of ext4; the concern was that Red Hat
didn't want to risk making a major change to e2fsprogs in an
enterprise distribution; so e4fsprogs is the e2fsprogs's various
binaries with ext4 support, statically linked.

> - Booting:
>   - There seem to be patches for grub
>     http://lists.openwall.net/linux-ext4/2008/11/19/8
>     to boot ext4
>   - and a Google SoC called 'grub4ext4'.
>    -and there is also grub2 (ext4 support in svn).
>   What would you recommend ?

I can't speak to that; I'm still using an ext3 boot partition since
until very recently I was still using an somewhat oldish (Ubuntu
Hardy) userspace.  It's exciting enough to be on the bleeding edge of
the kernel, I don't need to be on the bleeding edge of userspace too.  :-)

(For example, after my laptop got stolen and I decided to finally move
to Ubuntu Intrepid, with pulseaudio sound is only sporadically
working.  And I don't have time to debug how and why the new
complexities in the !@#!@ sound system is failing.  I'll probably have
to carve out a few hours to figure out how to rip out pulseaudio by
its roots, since it clearly isn't working for me.  Lovely...)

> Apart from that, it works pretty fine.
> The thing that most surprised me was that after the needed fsck, it had
> converted files with contiguous blocks to big extents ;). Nice.

Um, e2fsck won't convert files to use extents.  Once you set the
extents flag, new files that are created will be created with extents,
but existing files are left untouched.  Also, some of ext4's
performance improvements will only come after doing a
dump/mke2fs/restore pass.  People who are thinking about doing that
would be advised to wait until after 2.6.30 or 2.6.31, though, since
we have a patch that changes how the block/inode allocator works
currently pending in development.

Eventually, I hope the online defragger will be able to fully convert
an ext3 filesystem to use extents, with files defragged and placed in
locations that is as close as possible to a dump/mke2fs/restore pass
--- but we're not there yet.  Obviously you'll get a some of the
performance improvements simply going to ext4, and more by enabling
various ext4 features, such as extents et. al.  But for example, the
defragger still tries to keep blocks in the same block group, and it's
not aware of the new allocation policies that tries to keep inodes and
blocks within the same flex_bg (which is a collection of block
groups).  So that's an example of how the defragger needs to be made
more intelligent, and that work hasn't happened yet.

							- Ted

Re: Status of ext4 tools

[J.A. Magallón]

Sorry for the late answer...

On Fri, 20 Feb 2009 07:46:18 -0500, Theodore Tso <tytso@mit.edu> wrote:

> On Thu, Feb 19, 2009 at 11:48:38PM +0100, J.A. Magallón wrote:
> > 
> > - Where is the 'official' source for e4defrag ?
> > - I've read that e4defrag needs in-kernel support.
> >   Is it already in mainline ?
> >   From what kernel version ?
> 
> Ext4's defragmentation code is still under development.  There are
> multiple versions of e4defrag, paired with kernel patches that have
> been submitted to linux-ext4@vger.kernel.org.  The patches are not yet
> in in mainline, and the kernel/userspace interface is still subject to
> change (which is why it's not in mainline yet).
> 
> > - I have seen that some distros refer to a package called 'e4fsprogs'.
> >   Does that thing exist ? Where ?
> 
> e4fsprogs was a snapshot of something between e2fprogs 1.41.1 and
> e2fsprogs 1.41.2, if I remember correctly, and it was created for
> RHEL5's technology preview of ext4; the concern was that Red Hat
> didn't want to risk making a major change to e2fsprogs in an
> enterprise distribution; so e4fsprogs is the e2fsprogs's various
> binaries with ext4 support, statically linked.
> 

So, in short, forget online defrag for the moment. No problem.

> > - Booting:
> >   - There seem to be patches for grub
> >     http://lists.openwall.net/linux-ext4/2008/11/19/8
> >     to boot ext4
> >   - and a Google SoC called 'grub4ext4'.
> >    -and there is also grub2 (ext4 support in svn).
> >   What would you recommend ?
> 
> I can't speak to that; I'm still using an ext3 boot partition since
> until very recently I was still using an somewhat oldish (Ubuntu
> Hardy) userspace.  It's exciting enough to be on the bleeding edge of
> the kernel, I don't need to be on the bleeding edge of userspace too.  :-)
> 
> (For example, after my laptop got stolen and I decided to finally move
> to Ubuntu Intrepid, with pulseaudio sound is only sporadically
> working.  And I don't have time to debug how and why the new
> complexities in the !@#!@ sound system is failing.  I'll probably have
> to carve out a few hours to figure out how to rip out pulseaudio by
> its roots, since it clearly isn't working for me.  Lovely...)

So for the full move to ext4 I need a patched grub. I'm not sure I
trust grub2 for the moment...

> 
> > Apart from that, it works pretty fine.
> > The thing that most surprised me was that after the needed fsck, it had
> > converted files with contiguous blocks to big extents ;). Nice.
> 
> Um, e2fsck won't convert files to use extents.  Once you set the
> extents flag, new files that are created will be created with extents,
> but existing files are left untouched.  

Mmmm. I had some big files on my drive, and some of them now use
only 1 extent and some 2. I supposed that fsck.ext4 recognized a
run of contiguous blocks and turned the list of inodes onto one
and only extent:

werewolf:/mnt/dsk0/music/zzz# ll
total 121760
-rw-r--r-- 1 magallon home 21892759 2008.03.02 23:30 01 zzz.mp3
-rw-r--r-- 1 magallon home  7192808 2008.03.02 23:30 02 zzz.mp3
-rw-r--r-- 1 magallon home  6709136 2008.03.02 23:30 03 zzz.mp3
-rw-r--r-- 1 magallon home  7155107 2008.03.02 23:30 04 zzz.mp3
-rw-r--r-- 1 magallon home 10457465 2008.03.02 23:30 05 zzz.mp3
-rw-r--r-- 1 magallon home  6082273 2008.03.02 23:30 06 zzz.mp3
-rw-r--r-- 1 magallon home 10052444 2008.03.02 23:30 07 zzz.mp3

werewolf:/mnt/dsk0/music/zzz# filefrag *
01 zzz.mp3: 1 extent found
02 zzz.mp3: 1 extent found
03 zzz.mp3: 1 extent found
04 zzz.mp3: 1 extent found
05 zzz.mp3: 2 extents found, perfection would be 1 extent
06 zzz.mp3: 1 extent found
07 zzz.mp3: 2 extents found, perfection would be 1 extent

But I have just discovered that filefrag also works on ext3, so
it has not so much meaning...

> Also, some of ext4's
> performance improvements will only come after doing a
> dump/mke2fs/restore pass.  People who are thinking about doing that
> would be advised to wait until after 2.6.30 or 2.6.31, though, since
> we have a patch that changes how the block/inode allocator works
> currently pending in development.
> 
> Eventually, I hope the online defragger will be able to fully convert
> an ext3 filesystem to use extents, with files defragged and placed in
> locations that is as close as possible to a dump/mke2fs/restore pass
> --- but we're not there yet.  Obviously you'll get a some of the
> performance improvements simply going to ext4, and more by enabling
> various ext4 features, such as extents et. al.  But for example, the
> defragger still tries to keep blocks in the same block group, and it's
> not aware of the new allocation policies that tries to keep inodes and
> blocks within the same flex_bg (which is a collection of block
> groups).  So that's an example of how the defragger needs to be made
> more intelligent, and that work hasn't happened yet.
> 

Many thanks forr the info !!

-- 
J.A. Magallon <jamagallon()ono!com>     \               Software is like sex:
                                         \         It's better when it's free
Mandriva Linux release 2009.1 (Cooker) for x86_64
Linux 2.6.28.2-desktop-1mnb (gcc 4.3.2 (GCC) #1 Wed Jan