From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christophe Varoqui Subject: [ANNOUNCE] dds snapshot based remote async replication Date: Sun, 27 Jun 2010 20:38:15 +0200 Message-ID: <1277663895.20849.906.camel@localhost> Reply-To: christophe.varoqui@opensvc.com, device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: dm-devel List-Id: dm-devel.ids Hi, dds is a small GPLv2-licensed C program I developed to replicate LVM2 logical volumes to bandwidth-constrained targets (primarily off-site nodes for disaster recovery). It's main use-case is to replicate opaque logical volume, like those given to virtual machine as virtual disks. On source, dds parses a snapshot COW metadata and extracts changed data from a tiers block device. Data and metadata can be dumped to a file or piped (the format is the same as the LVM2 dedicated COW devices). On target, dds parses a COW-file (generated by dds on source) or stdin, and applies deltas to a specified file or block device. Example command: dds --extract --cow /dev/mapper/unxtstsvc02-data_osvc_snap1-cow --source /dev/unxtstsvc02/data_osvc_snap2 | /usr/bin/ssh vm5 dds --merge --dest /tmp/dds.img -v Output of the remote dds execution: Snapshot header: magic 0x70416e53 version 1 valid yes chunk_size 4 KB Report: Exception chunks 11 Exceptions 2728 Output size meta 48 KB Output size data 10912 KB Output size total 10960 KB dds pros: o low cpu resource overhead o low bandwidth usage: only delta are synced o cleanly wrapped into OpenSVC as a replication strategy dds cons: o no easy way to verify src/dst integrity (checksumming the whole src/dst is not fair) o async only: hourly to daily is the reasonable range o need free extents in the VG to create 1 long-lived snap to keep track of changed extents and 1 short-lived snap to extract fresh data from o need to run a full sync in case of a corruption o the target device is corrupted during the sync: the snap merge-back feature integration would provide a clean way to address this problem OpenSVC is a clustered service manager embedding data replication drivers. More details on OpenSVC integration at http://www.opensvc.com/ dds source file is tracked at http://git.opensvc.com/ Comments and reviews are welcome. Regards, -- Christophe Varoqui OpenSVC