From mboxrd@z Thu Jan 1 00:00:00 1970 From: Lukas Laukamp Subject: Re: [User Question] How to create a backup of an LVM based maschine without wasting space Date: Fri, 12 Oct 2012 15:02:37 +0200 Message-ID: <507814ED.5060001@laukamp.me> References: <5077BE30.5010209@laukamp.me> <20121012084221.GC14822@stefanha-thinkpad.redhat.com> <5077DB9F.6050809@laukamp.me> <5077E03E.2040507@laukamp.me> <5077EDEB.6050703@laukamp.me> <5077F600.7080809@laukamp.me> <5078044D.1080904@laukamp.me> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org To: Stefan Hajnoczi Return-path: Received: from mailer0.lippux.de ([5.9.218.245]:48642 "EHLO mailer0.lippux.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933615Ab2JLNCp (ORCPT ); Fri, 12 Oct 2012 09:02:45 -0400 In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: Am 12.10.2012 14:59, schrieb Stefan Hajnoczi: > On Fri, Oct 12, 2012 at 1:51 PM, Lukas Laukamp wrote: >> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: >> >>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp wrote: >>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: >>>> >>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp >>>>> wrote: >>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>>>>> >>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp >>>>>>> wrote: >>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>>>>> >>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests >>>>>>>>>>> and >>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be >>>>>>>>>>> to >>>>>>>>>>> create a complete output file with dd, which wastes very much >>>>>>>>>>> space. >>>>>>>>>>> So I would like to create a backup of the LVM to a file which only >>>>>>>>>>> locates the space which is used on the LVM. Would be create when >>>>>>>>>>> the >>>>>>>>>>> output file would be something like a qcow2 file which could be >>>>>>>>>>> also >>>>>>>>>>> simply startet with KVM. >>>>>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>>>>> >>>>>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>>>>> vm001-backup.qcow2 >>>>>>>>>> >>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see the >>>>>>>>>> --sparse option in the man page). So you don't need to use qcow2, >>>>>>>>>> you >>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not >>>>>>>>>> use >>>>>>>>>> disk space for zero regions. >>>>>>>>>> >>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>>>>> >>>>>>>>>> Stefan >>>>>>>>>> -- >>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>>> >>>>>>>>> Hello Stefano, >>>>>>>>> >>>>>>>>> thanks for the fast reply. I will test this later. In my case now it >>>>>>>>> would >>>>>>>>> be a offline backup. For the online backup I think about a seperated >>>>>>>>> system >>>>>>>>> which every day makes incremental backups and once a week a full >>>>>>>>> backup. >>>>>>>>> The >>>>>>>>> main problem is, that the systems are in a WAN network and I need >>>>>>>>> encryption >>>>>>>>> between the systems. Would it be possible to do something like this: >>>>>>>>> create >>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>>>>> remote >>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>>>>> files >>>>>>>>> on >>>>>>>>> the backup system? And in which format could be the incremental >>>>>>>>> backups >>>>>>>>> be >>>>>>>>> stored? >>>>>>> Since there is a WAN link it's important to use a compact image >>>>>>> representation before hitting the network. I would use qemu-img >>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output. The >>>>>>> qcow2 file does not contain zero regions and will therefore save a lot >>>>>>> of network bandwidth compared to accessing the LVM volume over the >>>>>>> WAN. >>>>>>> >>>>>>> If you are using rsync or another tool it's a different story. You >>>>>>> could rsync the current LVM volume on the host over the last full >>>>>>> backup, it should avoid transferring image data which is already >>>>>>> present in the last full backup - the result is that you only transfer >>>>>>> changed data plus the rsync metadata. >>>>>>> >>>>>>> Stefan >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>>> Hello Stefan, >>>>>> >>>>>> the rsync part I don't have understood fully. So to create a qcow2 on >>>>>> the >>>>>> host, transfer this to the backup server will result in the weekly full >>>>>> backup. So do you mean I could use rsync to read the LVM from the host, >>>>>> compare the LVM data with the data in the qcow2 on the backup server >>>>>> and >>>>>> simply transfer the differences to the file? Or does it work on another >>>>>> way? >>>>> When using rsync you can skip qcow2. Only two objects are needed: >>>>> 1. The LVM volume on the host. >>>>> 2. The last full backup on the backup client. >>>>> >>>>> rsync compares #1 and #2 efficiently over the network and only >>>>> transfers data from #1 which has changed. >>>>> >>>>> After rsync completes your full backup image is identical to the LVM >>>>> volume. Next week you can use it as the "last" image to rsync >>>>> against. >>>>> >>>>> Stefan >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> So I simply update the full backup, which is simply a raw file which get >>>> mounted while the backup? >>> The image file does not need to be mounted. Just rsync the raw image >>> file. >>> >>> Stefan >> >> Ah, thats great so to have a complete task in mind: >> >> 1. Create a qcow2 of a snapshot of an running VM >> 2. Transfer the qcow2 to the backup node >> 3. For incremental backup sync a daily LVM snapshot with the image on the >> backup node via rsync > Yes. Only make sure not to rsync the raw LVM image onto the qcow2 > image - the backup client needs to have a raw image if you're syncing > directly against the raw LVM volume. > > Stefan So I should use a raw file for the syncing and convert it on the backup node to a qcow2 to have a good file format? Best Regards