[ANNOUNCE] dds snapshot based remote async replication

[ANNOUNCE] dds snapshot based remote async replication

[Christophe Varoqui]

Hi,

dds is a small GPLv2-licensed C program I developed to replicate LVM2
logical volumes to bandwidth-constrained targets (primarily off-site
nodes for disaster recovery). It's main use-case is to replicate opaque
logical volume, like those given to virtual machine as virtual disks.

On source, dds parses a snapshot COW metadata and extracts changed data
from a tiers block device. Data and metadata can be dumped to a file or
piped (the format is the same as the LVM2 dedicated COW devices).

On target, dds parses a COW-file (generated by dds on source) or stdin,
and applies deltas to a specified file or block device.


Example command:

dds --extract --cow /dev/mapper/unxtstsvc02-data_osvc_snap1-cow
--source /dev/unxtstsvc02/data_osvc_snap2 | /usr/bin/ssh vm5 dds --merge
--dest /tmp/dds.img -v

Output of the remote dds execution:

Snapshot header:
  magic             0x70416e53
  version           1
  valid             yes
  chunk_size        4 KB
Report:
  Exception chunks  11
  Exceptions        2728
  Output size meta  48 KB
  Output size data  10912 KB
  Output size total 10960 KB


dds pros:
o low cpu resource overhead
o low bandwidth usage: only delta are synced
o cleanly wrapped into OpenSVC as a replication strategy

dds cons:
o no easy way to verify src/dst integrity (checksumming the whole src/dst is not fair)
o async only: hourly to daily is the reasonable range
o need free extents in the VG to create 1 long-lived snap to keep track of changed extents and 1 short-lived snap to extract fresh data from
o need to run a full sync in case of a corruption
o the target device is corrupted during the sync: the snap merge-back feature integration would provide a clean way to address this problem

OpenSVC is a clustered service manager embedding data replication drivers.
More details on OpenSVC integration at http://www.opensvc.com/

dds source file is tracked at http://git.opensvc.com/
Comments and reviews are welcome.

Regards,

-- 
Christophe Varoqui <christophe.varoqui@opensvc.com>
OpenSVC