Online migration of arbitrary filesystems, possible?

Online migration of arbitrary filesystems, possible?

[Daniel Hilst]

Suppose that I have two big filesystems, dest fs and source fs.. I want 
to copy whole source fs to dest fs, but I need to keep
source filesystem online I can't mount source fs read only and copy..

The idea is, mount both filesystems "together", and make write/read 
operations go on this way
Read operations:
     1. See if data is already on dest fs,
     2. If is then read data and bright back to caller (lets call this 
cold read)
     3. If is not, then read file from source fs, put it on page cache, 
and change the backstorage of that page..
     3.1 So when this page get dirty or too old, it will be writed to 
dest fs

     The problem here is, I need to remap every data and metadata on 
step 3 (inodes and stuff), I think that read only data, isn't writed to 
disk,
     so in case of old page, this page would be freed from memory and 
not write to dest fs.. so I need read only pages to be forced dirty..

Write operations:
    1. Write data to dest fs
    2. Mark data as present on dest fs


Also I need to know when copy is finished, so I need a tool to 
crosscheck both filesystems and say what was copied and what wasn't, and 
give me some percent of remaining unsynced data. When I get 100% of data 
in sync I can umount source filesystem and use its storage for other 
propose.

The go here is move data online, where I can't bring data offline..

I'm a kernel newbie, I have just read a lot about page cache and VSF, 
but this seems possible, right?


-- 

Follow the white rabbit!

Online migration of arbitrary filesystems, possible?

[Greg Freemyer]

On Fri, Mar 29, 2013 at 4:09 PM, Daniel Hilst <danielhilst@gmail.com> wrote:
> Suppose that I have two big filesystems, dest fs and source fs.. I want
> to copy whole source fs to dest fs, but I need to keep
> source filesystem online I can't mount source fs read only and copy..
>
> The idea is, mount both filesystems "together", and make write/read
> operations go on this way
> Read operations:
>      1. See if data is already on dest fs,
>      2. If is then read data and bright back to caller (lets call this
> cold read)
>      3. If is not, then read file from source fs, put it on page cache,
> and change the backstorage of that page..
>      3.1 So when this page get dirty or too old, it will be writed to
> dest fs
>
>      The problem here is, I need to remap every data and metadata on
> step 3 (inodes and stuff), I think that read only data, isn't writed to
> disk,
>      so in case of old page, this page would be freed from memory and
> not write to dest fs.. so I need read only pages to be forced dirty..
>
> Write operations:
>     1. Write data to dest fs
>     2. Mark data as present on dest fs
>
>
> Also I need to know when copy is finished, so I need a tool to
> crosscheck both filesystems and say what was copied and what wasn't, and
> give me some percent of remaining unsynced data. When I get 100% of data
> in sync I can umount source filesystem and use its storage for other
> propose.
>
> The go here is move data online, where I can't bring data offline..
>
> I'm a kernel newbie, I have just read a lot about page cache and VSF,
> but this seems possible, right?

Are you looking for a pre-existing solution?  Or you want to write something?

Greg

Online migration of arbitrary filesystems, possible?

[Daniel Hilst]

On 29-03-2013 17:28, Greg Freemyer wrote:
> On Fri, Mar 29, 2013 at 4:09 PM, Daniel Hilst <danielhilst@gmail.com> wrote:
>> Suppose that I have two big filesystems, dest fs and source fs.. I want
>> to copy whole source fs to dest fs, but I need to keep
>> source filesystem online I can't mount source fs read only and copy..
>>
>> The idea is, mount both filesystems "together", and make write/read
>> operations go on this way
>> Read operations:
>>       1. See if data is already on dest fs,
>>       2. If is then read data and bright back to caller (lets call this
>> cold read)
>>       3. If is not, then read file from source fs, put it on page cache,
>> and change the backstorage of that page..
>>       3.1 So when this page get dirty or too old, it will be writed to
>> dest fs
>>
>>       The problem here is, I need to remap every data and metadata on
>> step 3 (inodes and stuff), I think that read only data, isn't writed to
>> disk,
>>       so in case of old page, this page would be freed from memory and
>> not write to dest fs.. so I need read only pages to be forced dirty..
>>
>> Write operations:
>>      1. Write data to dest fs
>>      2. Mark data as present on dest fs
>>
>>
>> Also I need to know when copy is finished, so I need a tool to
>> crosscheck both filesystems and say what was copied and what wasn't, and
>> give me some percent of remaining unsynced data. When I get 100% of data
>> in sync I can umount source filesystem and use its storage for other
>> propose.
>>
>> The go here is move data online, where I can't bring data offline..
>>
>> I'm a kernel newbie, I have just read a lot about page cache and VSF,
>> but this seems possible, right?
>
> Are you looking for a pre-existing solution?  Or you want to write something?
>
> Greg
>
I can do this with btrfs snapshots, I think LVM2 can do this with 
snapshots too... I was searching for something like that, but using a 
kind of VFS hack

So I want to know if is possible, in theory, then I can start write 
something like that

-- 
Follow the white rabbit!

Online migration of arbitrary filesystems, possible?

[Jason Ball]

>
>
> Ive had to do something similar before.   It is painful, but doable.

* create the directory structure on the target. (find -exec...)

* create a symlink for every file on the target device, obviously pointing
to the origin device. (Find)

File system should be useable.

*  progressively replace the symlinks with the actual file (find or script)

* watch out for actual symlinks...

Optionally replace the origin file with a symlink to the new file.


-- 
--
Teach your kids Science, or somebody else will :/

jason at ball.net
vk2vjb at google.com <vk2flnx@google.com>
callsign: vk2vjb
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130402/c8603a58/attachment.html 

Online migration of arbitrary filesystems, possible?

[Valdis.Kletnieks at vt.edu]

On Fri, 29 Mar 2013 17:09:14 -0300, Daniel Hilst said:

> The idea is, mount both filesystems "together", and make write/read
> operations go on this way
> Read operations:
>      1. See if data is already on dest fs,
>      2. If is then read data and bright back to caller (lets call this
> cold read)
>      3. If is not, then read file from source fs, put it on page cache,
> and change the backstorage of that page..
>      3.1 So when this page get dirty or too old, it will be writed to

Any reason you can't just 'rsync /source-fs /dest-fs'?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 865 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130329/56ab327c/attachment-0001.bin 

Online migration of arbitrary filesystems, possible?

[Daniel Hilst]

On 29-03-2013 18:06, Valdis.Kletnieks at vt.edu wrote:
> On Fri, 29 Mar 2013 17:09:14 -0300, Daniel Hilst said:
>
>> The idea is, mount both filesystems "together", and make write/read
>> operations go on this way
>> Read operations:
>>       1. See if data is already on dest fs,
>>       2. If is then read data and bright back to caller (lets call this
>> cold read)
>>       3. If is not, then read file from source fs, put it on page cache,
>> and change the backstorage of that page..
>>       3.1 So when this page get dirty or too old, it will be writed to
>
> Any reason you can't just 'rsync /source-fs /dest-fs'?
>
because I can't use dest-fs while rsynching

-- 
Follow the white rabbit!

Online migration of arbitrary filesystems, possible?

[Valdis.Kletnieks at vt.edu]

On Mon, 01 Apr 2013 17:50:43 -0300, Daniel Hilst said:

> > Any reason you can't just 'rsync /source-fs /dest-fs'?
> >
> because I can't use dest-fs while rsynching

Sure you can. You just have to remember to pay attention to race
conditions - if you create foo/bar.dat on the dest and then rsync
wants to copy over a foo/bar.tar from the source, things will go
poorly.

However, if you wanted to write to the dest while doing your sync,
you'll have that issue no matter *what* method you use to do it.

>> Read operations:
>>       1. See if data is already on dest fs,
>>       2. If is then read data and bright back to caller (lets call this
>> cold read)
>>       3. If is not, then read file from source fs, put it on page cache,
>> and change the backstorage of that page..
>>       3.1 So when this page get dirty or too old, it will be writed to

You may want to look for 'overlayfs' and 'unionfs', which may provide
you the function you need. (Note there's several different patchsets
calling themselves 'unionfs').

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 865 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20130402/778b299c/attachment.bin