merging of two file system

merging of two file system

[Vineet Agarwal]

can you please help about porting the feature of file system
encapsulation on ext2

-- 
>From :
Vineet Agarwal

Re: merging of two file system

[Theodore Tso]

On Sun, Jul 27, 2008 at 04:48:47PM +0530, Vineet Agarwal wrote:
> can you please help about porting the feature of file system
> encapsulation on ext2

I'm not sure what you mean by that.

Regards,

					- Ted

merging of two file system

[Vineet Agarwal]

can we merge two file system into one in ext2/3 ?

-- 
>From :
Vineet Agarwal

Re: merging of two file system

[Theodore Tso]

On Sun, Jul 27, 2008 at 09:35:30PM +0530, Vineet Agarwal wrote:
> can we merge two file system into one in ext2/3 ?

What exactly do you mean by that?

					- Ted

Re: merging of two file system

[SandeepKsinha]

Theodore Tso wrote:
> 
> On Sun, Jul 27, 2008 at 09:35:30PM +0530, Vineet Agarwal wrote:
>> can we merge two file system into one in ext2/3 ?
> 
>>What exactly do you mean by that?
> 
> 

What I understand is that, he means to say that is there any existing
mechanism or support by which we can merger two ext2/ext3 file systems on
the fly.
i.e. One of them would be the primary and the other would be a secondary.
The secondary after merging will reside in the directory structure of the
primary file system.

HTH,

Thanks,
SandeepKsinha



> 					- Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

-- 
View this message in context: http://www.nabble.com/merging-of-two-file-system-tp18677733p18697116.html
Sent from the linux-ext4 mailing list archive at Nabble.com.

Re: merging of two file system

[Vineet Agarwal]

yeah i meant that we have to separate filesystems,
stored for example on /dev/hda1 and /dev/hda2, and we want to combine
the data in the two filesystems into a single filesystem /dev/hda3, or
combining the contents of /dev/hda2 so that it is also in /dev/hda1

so is there any mechanism exist to do so either online or offline . please help
thank you

Re: merging of two file system

[Theodore Tso]

On Tue, Jul 29, 2008 at 12:41:25AM +0530, Vineet Agarwal wrote:
> yeah i meant that we have to separate filesystems,
> stored for example on /dev/hda1 and /dev/hda2, and we want to combine
> the data in the two filesystems into a single filesystem /dev/hda3, or
> combining the contents of /dev/hda2 so that it is also in /dev/hda1
> 
> so is there any mechanism exist to do so either online or offline . please help

There is no direct way to do this.

How big are the filesystems involved?  The standard and easist way to
do this would be to copy /dev/hda2 to another device, then if
/dev/hda1 was created with on-line resizing inode, expanding /dev/hda1
to fill the space taken by /dev/hda2, and then simply use "cp" to copy
the contents that had previously been in /dev/hda2 into the filesystem
on /dev/hda1.

In theory it would be possible to write a program would take to
adjacent filesystems, and map out where the blocks would be once the
two partitions were combined, and then relocate blocks to make a
single filesystem.  No one has done such a thing, however, for any
filesytem.  It is definitely not a trivial thing to do, but it is not
impossible; it's not that different from some of what an off-line
resize2fs operations does, albeit maybe 3-4 times more complex.

The main thing is that no one has ever taken the time to do such a
thing, because except for truly large filesystems, it's cheaper just
to get an extra disk drive, and just copy the contents off, and then
recreate the filesystem.

							- Ted

Re: merging of two file system

[Goswin von Brederlow]

Theodore Tso <tytso@mit.edu> writes:

> On Tue, Jul 29, 2008 at 12:41:25AM +0530, Vineet Agarwal wrote:
>> yeah i meant that we have to separate filesystems,
>> stored for example on /dev/hda1 and /dev/hda2, and we want to combine
>> the data in the two filesystems into a single filesystem /dev/hda3, or
>> combining the contents of /dev/hda2 so that it is also in /dev/hda1
>> 
>> so is there any mechanism exist to do so either online or offline . please help
>
> There is no direct way to do this.
>
> How big are the filesystems involved?  The standard and easist way to
> do this would be to copy /dev/hda2 to another device, then if
> /dev/hda1 was created with on-line resizing inode, expanding /dev/hda1
> to fill the space taken by /dev/hda2, and then simply use "cp" to copy
> the contents that had previously been in /dev/hda2 into the filesystem
> on /dev/hda1.

Or if you have no extra disk but some free space you can copy as much
as possible to hda1, shrink hda2, move hda2, grow hda1, repeat.

At which point you really start to wish you had used LVM in the first
place.

> In theory it would be possible to write a program would take to
> adjacent filesystems, and map out where the blocks would be once the
> two partitions were combined, and then relocate blocks to make a
> single filesystem.  No one has done such a thing, however, for any
> filesytem.  It is definitely not a trivial thing to do, but it is not
> impossible; it's not that different from some of what an off-line
> resize2fs operations does, albeit maybe 3-4 times more complex.
>
> The main thing is that no one has ever taken the time to do such a
> thing, because except for truly large filesystems, it's cheaper just
> to get an extra disk drive, and just copy the contents off, and then
> recreate the filesystem.

I've started such a thing. Not specifically for merging 2 filesystem
but for changing from one filesystem to another. Actually I did it
even more generic by attacking the problem at the block devie layer.
The method was this:

1) Convert the physical block device into a sparse device
   - Copy the first few blocks to a safe place
   - Map the first few blocks to the safe place and the rest to the
     physical device
   - Fill the old filesystem with zero (this frees the blocks from the
     mapping and creates empty space)
2) Create a 2nd sparse device (fully empty)
3) successivley move files from old to new and write more zeroes to
   old (this allocates free blocks to the 2nd device as data gets
   written and frees blocks from the 1st devcie as zeroes are written)
4) Destroy empty 1st device
5) Convert sparse 2nd device to physical
   - Move first few virtual blocks to safe place
   - Defragment all other blocks so virtual == physical
   - move blocks from safe palce to physical overwriting the sparse
     device metadata

The safe palce is used as extra storage when the physical device has
not enough blocks and to store recovery state infos. With enough free
space on the FS only a few MB are needed.


I used this to convert from xfs to ext3 but then run into a bug during
defragmenting and had to restore from backup. As I used ext3 to
restore to I had no need for this anymore so I never finished it.

MfG
        Goswin

Re: merging of two file system

[Vineet Agarwal]

On Thu, Aug 7, 2008 at 6:03 PM, Goswin von Brederlow <goswin-v-b@web.de> wrote:
> Theodore Tso <tytso@mit.edu> writes:
>
>> On Tue, Jul 29, 2008 at 12:41:25AM +0530, Vineet Agarwal wrote:
>>> yeah i meant that we have to separate filesystems,
>>> stored for example on /dev/hda1 and /dev/hda2, and we want to combine
>>> the data in the two filesystems into a single filesystem /dev/hda3, or
>>> combining the contents of /dev/hda2 so that it is also in /dev/hda1
>>>
>>> so is there any mechanism exist to do so either online or offline . please help
>>
>> There is no direct way to do this.
>>
>> How big are the filesystems involved?  The standard and easist way to
>> do this would be to copy /dev/hda2 to another device, then if
>> /dev/hda1 was created with on-line resizing inode, expanding /dev/hda1
>> to fill the space taken by /dev/hda2, and then simply use "cp" to copy
>> the contents that had previously been in /dev/hda2 into the filesystem
>> on /dev/hda1.
>
> Or if you have no extra disk but some free space you can copy as much
> as possible to hda1, shrink hda2, move hda2, grow hda1, repeat.
>
> At which point you really start to wish you had used LVM in the first
> place.
>
>> In theory it would be possible to write a program would take to
>> adjacent filesystems, and map out where the blocks would be once the
>> two partitions were combined, and then relocate blocks to make a
>> single filesystem.  No one has done such a thing, however, for any
>> filesytem.  It is definitely not a trivial thing to do, but it is not
>> impossible; it's not that different from some of what an off-line
>> resize2fs operations does, albeit maybe 3-4 times more complex.
>>
>> The main thing is that no one has ever taken the time to do such a
>> thing, because except for truly large filesystems, it's cheaper just
>> to get an extra disk drive, and just copy the contents off, and then
>> recreate the filesystem.
>
> I've started such a thing. Not specifically for merging 2 filesystem
> but for changing from one filesystem to another. Actually I did it
> even more generic by attacking the problem at the block device layer.
> The method was this:
>
> 1) Convert the physical block device into a sparse device
>   - Copy the first few blocks to a safe place
>   - Map the first few blocks to the safe place and the rest to the
>     physical device
>   - Fill the old filesystem with zero (this frees the blocks from the
>     mapping and creates empty space)
> 2) Create a 2nd sparse device (fully empty)
> 3) successivley move files from old to new and write more zeroes to
>   old (this allocates free blocks to the 2nd device as data gets
>   written and frees blocks from the 1st devcie as zeroes are written)
> 4) Destroy empty 1st device
> 5) Convert sparse 2nd device to physical
>   - Move first few virtual blocks to safe place
>   - Defragment all other blocks so virtual == physical
>   - move blocks from safe palce to physical overwriting the sparse
>     device metadata
>
> The safe palce is used as extra storage when the physical device has
> not enough blocks and to store recovery state infos. With enough free
> space on the FS only a few MB are needed.
>
>
> I used this to convert from xfs to ext3 but then run into a bug during
> defragmenting and had to restore from backup. As I used ext3 to
> restore to I had no need for this anymore so I never finished it.
>
> MfG
>        Goswin
>

the concept  thought is a bit similar but have not yet implemented
i thought to append or copy the the superblock information from second
hdd to first and then map then all or
i also thought to use lvm over two fs but i don't have much knowledge
about it, still working on it and can u help me to figure it out

i am still working for the design if you have any idea please suggest.
Thankyou

-- 
>From :
Vineet Agarwal

Re: merging of two file system

[Shehjar Tikoo]

Vineet Agarwal wrote:
> can we merge two file system into one in ext2/3 ?
> 

UnionFS might be what you're looking for.

-Shehjar

Re: merging of two file system

[Goswin von Brederlow]

Shehjar Tikoo <shehjart@cse.unsw.edu.au> writes:

> Vineet Agarwal wrote:
>> can we merge two file system into one in ext2/3 ?
>>
>
> UnionFS might be what you're looking for.
>
> -Shehjar

I prefer unionfs fuse. When merging directories on NFS the kernel
unionFS would oops when someone changed files on the server. Verry
nasty.

MfG
        Goswin