[RFC][PATCH 0/8]ext4: online defrag (ver 0.9)

[RFC][PATCH 0/8]ext4: online defrag (ver 0.9)

[Akira Fujita]

Hello,

I've updated the ext4 online defragmentation patches.
I've addressed the review comment from Mingming, and there are some cleanups
but no functional changes since the previous version.

Changelog:
- 0.9 (May 30, 2008)
  - Create some new functions (ext4_defrag_fill_ar(),
    ext4_defrag_check_phase() ...) to separate block allocation function,
    since the phase mode plug into the allocation function isn't good.
  - Add the description of ext4_defrag() which is main function.
  - Add the capability check.
  - Some cleanups.
- 0.8 (April 8, 2008)
    Fix sparse warnings and change the construction of patches.
- 0.7 (January 10, 2008)
    Interchange the data blocks of the target and temporary files
    in an atomic manner.

Outline for ext4 online defragmentation:
Ext4 online defrag has the following three functions.

 no option  Solving a single file fragmentation.
            Single file fragmentation is solved by moving file
            data to contiguous free blocks.

    -r      Solving a relevant file fragmentation.
            Relevant file fragmentation could be solved by moving
            the files under the specified directory close together with
            the block containing the directory data.

    -f      Solving a free space fragmentation.
            If there is no contiguous free blocks in the filesystem,
            the other files are moved to make sufficient space to allocate
            contiguous blocks for the target file.

Notes:
- Ext4 online defarg needs "mballoc" and "extents" mount options,
  and it only supports 4KB block size (It will go away soon).
- Current mballoc in the ext4 patch queue (linux-2.6.26-rc4) doesn't
  work fine, ext4_mb_init_caches() which is changed by
  ext4-add-error-handling-in-ext4-mb_laod_buddy.patch returns -EIO.
  You can reproduce this kind of issue easily with following procedure.
  1. mke2fs -j -E test_fs /devname
  2. mount -t ext4dev -o mballoc,extents /devname /mountpoint
  3. mkdir DIR /mountpint
     mkdir: cannot create directory `/mnt/mp2/DIR1': No space left on device

Next steps:
1. Implement FS_IOC_FIEMAP instead of EXT4_EXTENTS_INFO to reduce ioctl count.
2. Remove the restriction of block size (Enable defrag with 1KB and 2KB).
3. Remove the limit of target file size (now 128MB) in -f mode.
   * The post about -f mode can be found at:
   http://marc.info/?l=linux-ext4&m=118239067704899&w=4

Summary of patches:
* The following patches are new ext4 online defrag patches.
  Mingming, please replace these patches with the ext4 patch queue's.

[PATCH 1/8] Main function of ext4 online defrag and ioctl implementation
- Create the temporary inode and do defrag per
  defrag_size (defalut 64MB).

[PATCH 2/8] Allocate new contiguous blocks with mballoc
- Search contiguous free blocks with mutil-block allocation
  and allocate them for the temporary inode.

[PATCH 3/8] Read and write file data with memory page
- Read the file data from the old blocks to the page cache and
  write the file data on the page into the new blocks.

[PATCH 4/8]  Exchange the blocks between two inodes
- Exchange the data blocks between the temporary inode and
  the original inode.

[PATCH 5/8] Defragmentation for the relevant files (-r mode)
- Relevant file fragmentation could be solved by moving
  the files under the specified directory close together with
  the block containing the directory data.

[PATCH 6/8] Check the free space fragmentation (-f mode)
- Check the free space fragmentation in the block group
  where target file is located.

[PATCH 7/8] Move victim files for the target file (-f mode)
- Move victim files to make sufficient space and reallocate
  the contiguous blocks for the target file.

[PATCH 8/8] Online defrag command
- The defrag command. Usage is as follows:
  - Defrag for a single file.
    # e4defrag file-name
  - Defrag for all files on ext4.
    # e4defrag device-name
  - Put the multiple files closer together.
    # e4defrag -r directory-name
  - Defrag for free space fragmentation.
    # e4defrag -f file-name

Performance test:
I created 1GB files with parallel writing then ran the e4defrag to it.
As a result, I got the following improvement.

                        <Before>           <After>
--------------------------------------------------------------------
Fragments                  127      ->       20
I/O performance(sec)       40.3     ->       36 (10% improved)

		Kernel: 2.6.26-rc4 + ext4 patch queue Arch: x86_64
		Mount options: extents, mballoc, delalloc, data=writeback

* "Fragments" is the number of fragments in 1GB file shown by filefrag.
  "I/O performance" is the execution time for reading 1GB file
   with "cat" command (cat file* > /dev/null).

Any comments are welcome.

Regards,
Akira Fujita

Re: [RFC][PATCH 0/8]ext4: online defrag (ver 0.9)

[Mingming]

Thanks for the update.
On Fri, 2008-05-30 at 20:17 +0900, Akira Fujita wrote:
> - Current mballoc in the ext4 patch queue (linux-2.6.26-rc4) doesn't
>   work fine, ext4_mb_init_caches() which is changed by
>   ext4-add-error-handling-in-ext4-mb_laod_buddy.patch returns -EIO.
>   You can reproduce this kind of issue easily with following
> procedure.
>   1. mke2fs -j -E test_fs /devname
>   2. mount -t ext4dev -o mballoc,extents /devname /mountpoint
>   3. mkdir DIR /mountpint
>      mkdir: cannot create directory `/mnt/mp2/DIR1': No space left on
> device
> 
> 
Ccing Shen Feng for this bug...

> Next steps:
> 1. Implement FS_IOC_FIEMAP instead of EXT4_EXTENTS_INFO to reduce ioctl count.
> 2. Remove the restriction of block size (Enable defrag with 1KB and 2KB).
> 3. Remove the limit of target file size (now 128MB) in -f mode.
>    * The post about -f mode can be found at:
>    http://marc.info/?l=linux-ext4&m=118239067704899&w=4
> 
> Summary of patches:
> * The following patches are new ext4 online defrag patches.
>   Mingming, please replace these patches with the ext4 patch queue's.
> 
Done.

Regards,
Mingming

[PATCH]ext4: Fix ext4_mb_init_cache return error

[Mingming Cao]

ext4: Fix ext4_mb_init_cache return error

From: Mingming Cao <cmm@us.ibm.com>

ext4_mb_init_cache() incorrectly always return EIO on success. This
causes the caller of ext4_mb_init_cache() fail when it checks
 the return value.

Signed-off-by: Mingming Cao <cmm@us.ibm.com>
---
 fs/ext4/mballoc.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: linux-2.6.26-rc4/fs/ext4/mballoc.c
===================================================================
--- linux-2.6.26-rc4.orig/fs/ext4/mballoc.c	2008-06-01 12:55:19.000000000 -0700
+++ linux-2.6.26-rc4/fs/ext4/mballoc.c	2008-06-01 12:55:23.000000000 -0700
@@ -798,7 +798,7 @@ static int ext4_mb_init_cache(struct pag
 	for (i = 0; i < groups_per_page && bh[i]; i++)
 		wait_on_buffer(bh[i]);
 
-	err = -EIO;
+	err = 0;
 	for (i = 0; i < groups_per_page && bh[i]; i++)
 		if (!buffer_uptodate(bh[i]))
 			goto out;

Re: [PATCH]ext4: Fix ext4_mb_init_cache return error

[Shen Feng]

Mingming Cao Wrote:
> ext4: Fix ext4_mb_init_cache return error
> 
> From: Mingming Cao <cmm@us.ibm.com>
> 
> ext4_mb_init_cache() incorrectly always return EIO on success. This
> causes the caller of ext4_mb_init_cache() fail when it checks
>  the return value.
> 
> Signed-off-by: Mingming Cao <cmm@us.ibm.com>
> ---
>  fs/ext4/mballoc.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> Index: linux-2.6.26-rc4/fs/ext4/mballoc.c
> ===================================================================
> --- linux-2.6.26-rc4.orig/fs/ext4/mballoc.c	2008-06-01 12:55:19.000000000 -0700
> +++ linux-2.6.26-rc4/fs/ext4/mballoc.c	2008-06-01 12:55:23.000000000 -0700
> @@ -798,7 +798,7 @@ static int ext4_mb_init_cache(struct pag
>  	for (i = 0; i < groups_per_page && bh[i]; i++)
>  		wait_on_buffer(bh[i]);
>  
> -	err = -EIO;
> +	err = 0;
>  	for (i = 0; i < groups_per_page && bh[i]; i++)
>  		if (!buffer_uptodate(bh[i]))
>  			goto out;
> 
> 

I think the fix should be

   	err = -EIO;
  	for (i = 0; i < groups_per_page && bh[i]; i++)
  		if (!buffer_uptodate(bh[i]))
  			goto out;
 +	err = 0;

Is that right?

Shen Feng

Re: [PATCH]ext4: Fix ext4_mb_init_cache return error

[Mingming Cao]

On Mon, 2008-06-02 at 09:16 +0800, Shen Feng wrote:
> 
> Mingming Cao Wrote:
> > ext4: Fix ext4_mb_init_cache return error
> > 
> > From: Mingming Cao <cmm@us.ibm.com>
> > 
> > ext4_mb_init_cache() incorrectly always return EIO on success. This
> > causes the caller of ext4_mb_init_cache() fail when it checks
> >  the return value.
> > 
> > Signed-off-by: Mingming Cao <cmm@us.ibm.com>
> > ---
> >  fs/ext4/mballoc.c |    2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > Index: linux-2.6.26-rc4/fs/ext4/mballoc.c
> > ===================================================================
> > --- linux-2.6.26-rc4.orig/fs/ext4/mballoc.c	2008-06-01 12:55:19.000000000 -0700
> > +++ linux-2.6.26-rc4/fs/ext4/mballoc.c	2008-06-01 12:55:23.000000000 -0700
> > @@ -798,7 +798,7 @@ static int ext4_mb_init_cache(struct pag
> >  	for (i = 0; i < groups_per_page && bh[i]; i++)
> >  		wait_on_buffer(bh[i]);
> >  
> > -	err = -EIO;
> > +	err = 0;
> >  	for (i = 0; i < groups_per_page && bh[i]; i++)
> >  		if (!buffer_uptodate(bh[i]))
> >  			goto out;
> > 
> > 
> 
> I think the fix should be
> 
>    	err = -EIO;
>   	for (i = 0; i < groups_per_page && bh[i]; i++)
>   		if (!buffer_uptodate(bh[i]))
>   			goto out;
>  +	err = 0;
> 
> Is that right?
> 

you are right, thanks for point this out.

Mingming
> Shen Feng
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html