Yea old defrag

Yea old defrag

[Phillip Susi]

I'm writing this to Theodore Ts'o because he has worked on the package 
in the past, and currently maintains e2fsprogs.   I am CCing lkml since 
there are likely to be other interested parties, and there does not seem 
to be a separate list for e2fsprogs.

The old defrag package for doing offline defragmentation of ext2 has not 
been maintained in many years.  About two years ago I fixed a few bugs 
that were causing it to not work on ext3 and placed the changes in 
Ubuntu.  I would like to see this program continue to live since it 
still serves a purpose, but because it has had no upstream maintainer in 
so long, Debian and thus Ubuntu have dropped the package.  I am trying 
to resurrect it.

I was wondering why it was never integrated into e2fsprogs, and if this 
could now be done.  It seems to me that e2fsprogs would be the proper 
home for it, and a neutral upstream for everyone.  If not, then I will 
try to maintain it as an Ubuntu native package in a bazaar branch on 
launchpad.

Re: Yea old defrag

[Theodore Tso]

On Wed, Oct 22, 2008 at 12:10:23PM -0400, Phillip Susi wrote:
> I'm writing this to Theodore Ts'o because he has worked on the package  
> in the past, and currently maintains e2fsprogs.   I am CCing lkml since  
> there are likely to be other interested parties, and there does not seem  
> to be a separate list for e2fsprogs.
>
> The old defrag package for doing offline defragmentation of ext2 has not  
> been maintained in many years.  About two years ago I fixed a few bugs  
> that were causing it to not work on ext3 and placed the changes in  
> Ubuntu.  I would like to see this program continue to live since it  
> still serves a purpose, but because it has had no upstream maintainer in  
> so long, Debian and thus Ubuntu have dropped the package.  I am trying  
> to resurrect it.
>
> I was wondering why it was never integrated into e2fsprogs, and if this  
> could now be done.  It seems to me that e2fsprogs would be the proper  
> home for it, and a neutral upstream for everyone.  If not, then I will  
> try to maintain it as an Ubuntu native package in a bazaar branch on  
> launchpad.

The problem with the defrag code has always been that I haven't had
time to give it proper love and attention, and it needs a *lot* of
work before I would be willing to stand behind it as a useful and
"safe" tool to use.  Basically, I don't want to get the hate mail from
people who lose their data due to bugs or other potential failures
from e2defrag.

I have a vague memory it doesn't even work if the filesystem blocksize
is greater than 1k, but I'm not 100% certain that is still true.

There is also an on-line defragmetter that requires kernel support
that some folks from NEC are working on for ext4 (which can also
support ext3 filesystems).  My thinking is that it's more worthwhile
to focus my attentions on helping Akira Fujita and Takashi Sato finish
theiron-line ext4 defragmentation patches than to worry about
e2defrag.  Of course, if you'd like to give some love and attention to
e2defrag, that's great.  But e2defrag doesn't use the ext2fs
libraries, and it would be a huge amount of work to make sure it
supports things like extended attributes and other newer format
enhancements that have been made to the ext3 filesystem.

Regards,

						- Ted

Re: Yea old defrag

[Phillip Susi]

Theodore Tso wrote:
> The problem with the defrag code has always been that I haven't had
> time to give it proper love and attention, and it needs a *lot* of
> work before I would be willing to stand behind it as a useful and
> "safe" tool to use.  Basically, I don't want to get the hate mail from
> people who lose their data due to bugs or other potential failures
> from e2defrag.

Indeed, it rightly should be covered with warnings about it possibly 
munching your data and should only be used on fully backed up 
filesystems.  As far as I know currently though, it works properly and 
I'm willing to use what free time I have to fix any bugs that are found, 
but I need to decide where to house the source in a proper VCS instead 
of just continuing to add dpatches to the ubuntu source package.

> I have a vague memory it doesn't even work if the filesystem blocksize
> is greater than 1k, but I'm not 100% certain that is still true.

I have an active 110 gig ext3 root fs ( that is only about 8% used ) 
that I tested on with no errors ( after removing one disk from the 
mirror and using that to test with ).  It uses 4k blocks, has_journal, 
resize_inode, dir_index, filetype, sparse_super, large_file.  I think it 
won't support blocks larger than 4k, but I don't think the kernel will 
either.  I also verified the md5sums of all files on the disk after the 
defrag.

> There is also an on-line defragmetter that requires kernel support
> that some folks from NEC are working on for ext4 (which can also
> support ext3 filesystems).  My thinking is that it's more worthwhile
> to focus my attentions on helping Akira Fujita and Takashi Sato finish
> theiron-line ext4 defragmentation patches than to worry about
> e2defrag.  Of course, if you'd like to give some love and attention to
> e2defrag, that's great.  But e2defrag doesn't use the ext2fs
> libraries, and it would be a huge amount of work to make sure it
> supports things like extended attributes and other newer format
> enhancements that have been made to the ext3 filesystem.

I've been reading a lot in the last few hours about the proposed online 
defragmenter, and while it seems quite interesting, it appears to not be 
ready yet, whereas e2defrag is readily available.  It will be 
interesting to compare the two approaches.

I have been trying to identify any new features that cause problems and 
fix any that arise.  Aren't extended attributes stored in the extra 
space of large inodes?  If so, I would need to format a new fs using 
large inodes to test.  I don't think e2defrag currently supports large 
inodes, but it shouldn't be too hard to add support.  Other than being 
aware of the correct size of the inodes, I don't see anything special it 
would need to do to support extended attributes if they are just extra 
data in the inode, or can they be stored in data blocks and so it would 
need to understand the block pointers in the inode?

Are there any other ext3 features that might cause trouble?  I know 
there are plenty in ext4, but I want to make sure everything works well 
in ext3 first.

Re: Yea old defrag

[Theodore Tso]

On Wed, Oct 22, 2008 at 04:01:48PM -0400, Phillip Susi wrote:
>
> I have been trying to identify any new features that cause problems and  
> fix any that arise.  Aren't extended attributes stored in the extra  
> space of large inodes?  If so, I would need to format a new fs using  
> large inodes to test.

Extended attributes *can* be stored in the extra space of large
inodes, but they can also be stored in separate, stand-alone data
blocks --- and just to make life entertaining, if multiple files have
exactly the same set of extended attributes (for example, if all of
the files inone directory have the same ACL and/or SELinux property),
an extended attribute block can be shared across multiple inodes for
space efficiency.

> Are there any other ext3 features that might cause trouble?  I know  
> there are plenty in ext4, but I want to make sure everything works well  
> in ext3 first.

That's probably the main one; and as you say, there are plenty of new
features that will require special handling for e2defrag, especially
since it doesn't use the standard libext2fs library.

						- Ted