Options for migrating existing files on ext2 partitions to ext4

Options for migrating existing files on ext2 partitions to ext4

[Xiang Wang]

Hey,

Since we are thinking of switching into using the ext4 file systems on
some of our machines with purely ext2 partitions, we face the problem
of how to migrate all those existing files on ext2 partitions. Seems
like there are two major directions to go:

1) do a backup, umount, mke2fs into ext4, and restore
In this approach, we may actually take advantage of the layout changes
so as to benefit from some features of ext4.

2) umount, tune2fs to add the 'extent' feature, mount back using type
ext4, migrate the existing files(directories) one by one through the
ioctl EXT4_IOC_MIGRATE
Motivated by the ext4 ioctl interface for migration, we are also
looking into this in-place approach. But we are mostly concerned with
the fact that if the existing files are highly fragmented, will we
still get benefit from the extents? Or will we even get hurt by
turning the files into being extents-based?

Before we do more detailed experiments on both approaches, we are here
to ask for advice, suggestion and comments.

So If you have ever run into either direction or even both, do you
have any experience you can share with us? Or simply express your
feeling:

Does it work? What could go wrong?
Does it take very long to complete the migration(of course, it could
depends on the number of files and their data block locations)?
Are you happy with the resulting files being migrated?
Has anyone every tried approach 2)? How did that work for you?

Thanks,
Xiang

Re: Options for migrating existing files on ext2 partitions to ext4

[Andreas Dilger]

On Feb 11, 2009  12:18 -0800, Xiang Wang wrote:
> Since we are thinking of switching into using the ext4 file systems on
> some of our machines with purely ext2 partitions, we face the problem
> of how to migrate all those existing files on ext2 partitions. Seems
> like there are two major directions to go:
> 
> 1) do a backup, umount, mke2fs into ext4, and restore
> In this approach, we may actually take advantage of the layout changes
> so as to benefit from some features of ext4.
> 
> 2) umount, tune2fs to add the 'extent' feature, mount back using type
> ext4,

Note that you can really stop at this point - new files will be created
with extents, old ones will be left alone.

> migrate the existing files(directories) one by one through the
> ioctl EXT4_IOC_MIGRATE

Note that you don't really need to do this.  If the filesystem is
offline to users you can just do "cp file file.tmp; mv file.tmp file"
and it will do the same thing, with exception of getting a new inode
number and requiring that files are not being modified at the time.

> Motivated by the ext4 ioctl interface for migration, we are also
> looking into this in-place approach. But we are mostly concerned with
> the fact that if the existing files are highly fragmented, will we
> still get benefit from the extents? Or will we even get hurt by
> turning the files into being extents-based?

The mballoc allocator is pretty good in conjunction with delalloc, so
I expect you would get mostly defragmented files as you migrate.  This
would depend on having some decent amount of free space in the filesystem
so that it is likely that there is enough contiguous free space for some
number of the >= average size files.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Re: Options for migrating existing files on ext2 partitions to ext4

[Aneesh Kumar K.V]

On Thu, Feb 12, 2009 at 02:23:34AM -0500, Andreas Dilger wrote:
> On Feb 11, 2009  12:18 -0800, Xiang Wang wrote:
> 
> > Motivated by the ext4 ioctl interface for migration, we are also
> > looking into this in-place approach. But we are mostly concerned with
> > the fact that if the existing files are highly fragmented, will we
> > still get benefit from the extents? Or will we even get hurt by
> > turning the files into being extents-based?
> 
> The mballoc allocator is pretty good in conjunction with delalloc, so
> I expect you would get mostly defragmented files as you migrate.  This
> would depend on having some decent amount of free space in the filesystem
> so that it is likely that there is enough contiguous free space for some
> number of the >= average size files.
> 

We don't touch the data block during migrate. So we are not actually
doing block allocation. So if the file is fragmented it would remain so
after migrate also. But if it is not we would end up having lesser
number of meta-data information with extent format.

The plan was to hook the migrate interface with defrag so that after
migrate we run a defrag on that that would move data blocks around.

-aneesh

Re: Options for migrating existing files on ext2 partitions to ext4

[Michael Rubin]

On Wed, Feb 11, 2009 at 11:57 PM, Aneesh Kumar K.V
<aneesh.kumar@linux.vnet.ibm.com> wrote:
> The plan was to hook the migrate interface with defrag so that after
> migrate we run a defrag on that that would move data blocks around.

Any data showing amount of time it takes to perform either the migrate
only or defrag-then-migrate operations on a disk both offline and
online?

mrubin