Re: [RFC] dm snapshot based asynchronous replication

Re: [RFC] dm snapshot based asynchronous replication

[christophe.varoqui]

> Should I understand there's no way for userspace to extract from the device mapper the cow'ed chunk list of a vanilla snapshot ?

Replying to myself, I found an instructive post from Lars of the drdb folks : http://markmail.org/message/7qdbp36tohasojuy (thanks for sharing Lars)

This script is easy enough to morph from its original snapshot reverting purpose to the replication model using the cow device as a changelog I described.

Thanks for the inputs. I'll keep posting if I something of general interest comes out of this stuff.


PS:
example per-chunk dump/load dd generated by the script :

root@tstparunx1:/$ /tmp/exceptions.pl  /dev/mapper/tstservcva-snap0-cow
# found snapshot header for chunk_size=8 sectors
# dump: dd if=$lastsnap of=$replayfile seek=125 iflag=direct count=1 bs=8b
# load: dd if=$replayfile of=$replica skip=0 seek=125 iflag=direct count=1 bs=8b
# dump: dd if=$lastsnap of=$replayfile seek=1250 iflag=direct count=1 bs=8b
# load: dd if=$replayfile of=$replica skip=1 seek=1250 iflag=direct count=1 bs=8b
# dump: dd if=$lastsnap of=$replayfile seek=50 iflag=direct count=2 bs=8b
# load: dd if=$replayfile of=$replica skip=2 seek=50 iflag=direct count=2 bs=8b
# dump: dd if=$lastsnap of=$replayfile seek=151 iflag=direct count=1 bs=8b
# load: dd if=$replayfile of=$replica skip=4 seek=151 iflag=direct count=1 bs=8b
# dump: dd if=$lastsnap of=$replayfile seek=126 iflag=direct count=1 bs=8b
# load: dd if=$replayfile of=$replica skip=5 seek=126 iflag=direct count=1 bs=8b
# found 6 exceptions (24 kB)
## use these dd commands AT YOUR OWN RISK

Re: [RFC] dm snapshot based asynchronous replication

[Lars Ellenberg]

On Wed, Apr 15, 2009 at 08:31:44PM +0200, christophe.varoqui@free.fr wrote:
> > Should I understand there's no way for userspace to extract from the device mapper the cow'ed chunk list of a vanilla snapshot ?
> 
> Replying to myself, I found an instructive post from Lars of the drdb folks : http://markmail.org/message/7qdbp36tohasojuy (thanks for sharing Lars)
> 
> This script is easy enough to morph from its original snapshot reverting purpose to the replication model using the cow device as a changelog I described.
> 
> Thanks for the inputs. I'll keep posting if I something of general interest comes out of this stuff.

that perl hack was meanwhile ported to C by someone.
posted earlier this year on dm-devel.
also please do not ignore the Zumastor work.

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.

Re: [RFC] dm snapshot based asynchronous replication

[Jonathan Brassow]

On Apr 16, 2009, at 2:02 AM, Lars Ellenberg wrote:

> On Wed, Apr 15, 2009 at 08:31:44PM +0200, christophe.varoqui@free.fr  
> wrote:
>>> Should I understand there's no way for userspace to extract from  
>>> the device mapper the cow'ed chunk list of a vanilla snapshot ?
>>
>> Replying to myself, I found an instructive post from Lars of the  
>> drdb folks : http://markmail.org/message/7qdbp36tohasojuy (thanks  
>> for sharing Lars)
>>
>> This script is easy enough to morph from its original snapshot  
>> reverting purpose to the replication model using the cow device as  
>> a changelog I described.
>>
>> Thanks for the inputs. I'll keep posting if I something of general  
>> interest comes out of this stuff.
>
> that perl hack was meanwhile ported to C by someone.
> posted earlier this year on dm-devel.
> also please do not ignore the Zumastor work.

It also seems that the script is not generic enough to handle the  
variety of exception stores that are coming.  I just wrote a 'snapshot- 
merge' target that will do what you ask.  I will post the patch  
shortly.  However, this won't help you in RHEL5.

  brassow

Re: [RFC] dm snapshot based asynchronous replication

[Christophe Varoqui]

> It also seems that the script is not generic enough to handle the  
> variety of exception stores that are coming.  I just wrote a 'snapshot- 
> merge' target that will do what you ask.  I will post the patch  
> shortly.  However, this won't help you in RHEL5.
> 
Jonathan,

thank you for this new snapshot-merge target. This plus the shared
exception patches will certainly bring the definitive answer for
LVM2-based replication/backup/time-navigation needs for RHEL6. I hope
btrfs will even bring to the table an interesting alternative sometime
in the el6 lifecycle.

But for those stuck with (entitlements on thousands of) el4 and el5
deployments, here is my contribution to the poor-man's async
replication :

http://christophe.varoqui.free.fr/rcow/

* rcow.pl
parse a cow device to generate a delta-file of 2 snapshots

* rcowd
handles full sync of remote device and schedule (one-shot or loop) :
- delta-file creation
- snapshot rotation
- delta-file sending to remote
- delta-file load on remote


Todo:
- more error checking
- sequencing cookies :
  o store on remote the last delta-file cookie succesfully loaded
  o store {prev_cookie, this_cookie} in delta_files
  o verify on remote that delta-file(prev_cookie)==last applied cookie
- replace the dd plumbing, though the readability of delta-file +
command-file is quite nice for understanding what happen (and may go
wrong). May be port to C.
- delta-file historization and rotation


Comments and contributions welcome.

Regards,
cvaroqui

[RFC] dm snapshot based asynchronous replication

[christophe.varoqui]

dm developers,

I'm in need of an asynchronous replication method for active/passive clusters using direct-attached disks. Please advise if you know tools (other than drdb an rsync) to do the job. (How goes dm-replicator btw ?).

Pending an existing project, I toyed with a replication scheme I'd like to sketch here for comments and fool-proofing.

Source and destination are logical volumes : lv_src => lv_dst
Upon cold start lv_src and lv_dst are unsynchronized.

==== pseudo-code start
snap lv_src (lv_src_snap0)
full copy lv_src_snap0 on lv_dst
while true:
  wait n seconds
  snap lv_src (lv_src_snap1)
  for each changed chunk in lv_src_snap0 cow table:
    copy chunk from lv_src_snap1 to lv_dst
  drop lv_src_snap0, and rename lv_src_snap1 to lv_src_snap0
==== pseudo-code end

Snapshoting the lv_dst would make sense as a rollback safety net ... even if the rollback is a bit hard pending the snapshot merging  patchset inclusion.

If this sketch looks sane to you, I'm interested in how to list the changed chunks for a snap. A pointer to existing code would do great.

Regards,
cvaroqui

Re: [RFC] dm snapshot based asynchronous replication

[FUJITA Tomonori]

On Mon, 13 Apr 2009 23:08:20 +0200 (CEST)
christophe.varoqui@free.fr wrote:

> dm developers,
> 
> I'm in need of an asynchronous replication method for active/passive clusters using direct-attached disks. Please advise if you know tools (other than drdb an rsync) to do the job. (How goes dm-replicator btw ?).

Zumastor?

http://zumastor.org/

An asynchronous remote replication based on snapshots is one of
reasons why I worked a new dm snapshot implementation:

https://www.redhat.com/archives/dm-devel/2008-August/msg00003.html


Jonathan Brassow has been working on it (Jonathan, Thanks!):

http://marc.info/?l=dm-devel&m=123905441719141&w=2

If you add the interface to get the delta between two snapshots, then
you can implement a daemon in user space that does an asynchronous
remote replication.

Re: [RFC] dm snapshot based asynchronous replication

[Jonathan Brassow]

On Apr 13, 2009, at 10:29 PM, FUJITA Tomonori wrote:

> On Mon, 13 Apr 2009 23:08:20 +0200 (CEST)
> christophe.varoqui@free.fr wrote:
>
>> dm developers,
>>
>> I'm in need of an asynchronous replication method for active/ 
>> passive clusters using direct-attached disks. Please advise if you  
>> know tools (other than drdb an rsync) to do the job. (How goes dm- 
>> replicator btw ?).
>
> Zumastor?
>
> http://zumastor.org/
>
> An asynchronous remote replication based on snapshots is one of
> reasons why I worked a new dm snapshot implementation:
>
> https://www.redhat.com/archives/dm-devel/2008-August/msg00003.html
>
>
> Jonathan Brassow has been working on it (Jonathan, Thanks!):
>
> http://marc.info/?l=dm-devel&m=123905441719141&w=2
>
> If you add the interface to get the delta between two snapshots, then
> you can implement a daemon in user space that does an asynchronous
> remote replication.

The exception store API is waiting for agk to push upstream.  The  
'shared' (aka tomonori/phillips) exception store module is working and  
ready for testing.  I am working on the LVM patches to allow people to  
use the shared exception store, but this will take me a while.  (The  
cluster-aware exception store is ready for testing as well.)

Regarding the solution you are talking about... I'm not sure how you  
will get the information out about the deltas between the snapshots.   
One method I see working would be to use device-mapper's 'message' and  
'wait/status' interface.  The API already has the ability to pass on  
messages to the exception stores below, which are free to implement  
their own features.  Still, it would be nice if there was a cleaner  
way to do this....

Of course, this dialog assumes you've hit upon the right solution to  
your problem.  There may be another way...  Direct-attached disks is  
what is complicating the problem.  You could use 'rgmanager' if the  
devices where shared.  You could also use a network block device  
driver (gnbd, etc) and then use mirroring, but this isn't much  
different than DRBD.

  brassow

  brassow

Re: [RFC] dm snapshot based asynchronous replication

[christophe.varoqui]

> Zumastor?
>
> http://zumastor.org/

thanks for your prompt response. zumastor is indeed exactly what I need feature-wize. Thanks for pointing it.

Though I'm eager to play with shared exception snapshots, my more pressing need will need to run on unmodified redhat el5. Thus I'm tied to vanilla snapshots, but that could be sufficient for my simplistic replication algorithm if I find a way to list cow'ed chunks : no need for snap-deltas, I'm ok with replicating more data than necessary (that can't be worse a ressource drain than rsync on big vm files).

Should I understand there's no way for userspace to extract from the device mapper the cow'ed chunk list of a vanilla snapshot ?