From mboxrd@z Thu Jan 1 00:00:00 1970 From: Lukas Laukamp Subject: Re: [User Question] How to create a backup of an LVM based maschine without wasting space Date: Fri, 12 Oct 2012 20:14:06 +0200 Message-ID: <50785DEE.6060204@laukamp.me> References: <5077BE30.5010209@laukamp.me> <20121012084221.GC14822@stefanha-thinkpad.redhat.com> <5077DB9F.6050809@laukamp.me> <5077E03E.2040507@laukamp.me> <5077EDEB.6050703@laukamp.me> <5077F600.7080809@laukamp.me> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org To: Stefan Hajnoczi Return-path: Received: from mailer0.lippux.de ([5.9.218.245]:50384 "EHLO mailer0.lippux.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1945954Ab2JLSOM (ORCPT ); Fri, 12 Oct 2012 14:14:12 -0400 In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: > On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp wrote: >> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: >> >>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp wrote: >>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>>> >>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp >>>>> wrote: >>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>>> >>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>>> I have a simple user question. I have a few LVM based KVM guests and >>>>>>>>> wan't to backup them to files. The simple and nasty way would be to >>>>>>>>> create a complete output file with dd, which wastes very much space. >>>>>>>>> So I would like to create a backup of the LVM to a file which only >>>>>>>>> locates the space which is used on the LVM. Would be create when the >>>>>>>>> output file would be something like a qcow2 file which could be also >>>>>>>>> simply startet with KVM. >>>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>>> >>>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>>> vm001-backup.qcow2 >>>>>>>> >>>>>>>> Note that cp(1) tries to make the destination file sparse (see the >>>>>>>> --sparse option in the man page). So you don't need to use qcow2, >>>>>>>> you >>>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not use >>>>>>>> disk space for zero regions. >>>>>>>> >>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>>> >>>>>>>> Stefan >>>>>>>> -- >>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>> >>>>>>> Hello Stefano, >>>>>>> >>>>>>> thanks for the fast reply. I will test this later. In my case now it >>>>>>> would >>>>>>> be a offline backup. For the online backup I think about a seperated >>>>>>> system >>>>>>> which every day makes incremental backups and once a week a full >>>>>>> backup. >>>>>>> The >>>>>>> main problem is, that the systems are in a WAN network and I need >>>>>>> encryption >>>>>>> between the systems. Would it be possible to do something like this: >>>>>>> create >>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>>> remote >>>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>>> files >>>>>>> on >>>>>>> the backup system? And in which format could be the incremental >>>>>>> backups >>>>>>> be >>>>>>> stored? >>>>> Since there is a WAN link it's important to use a compact image >>>>> representation before hitting the network. I would use qemu-img >>>>> convert -O qcow2 on the host and only transfer the qcow2 output. The >>>>> qcow2 file does not contain zero regions and will therefore save a lot >>>>> of network bandwidth compared to accessing the LVM volume over the >>>>> WAN. >>>>> >>>>> If you are using rsync or another tool it's a different story. You >>>>> could rsync the current LVM volume on the host over the last full >>>>> backup, it should avoid transferring image data which is already >>>>> present in the last full backup - the result is that you only transfer >>>>> changed data plus the rsync metadata. >>>>> >>>>> Stefan >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> Hello Stefan, >>>> >>>> the rsync part I don't have understood fully. So to create a qcow2 on the >>>> host, transfer this to the backup server will result in the weekly full >>>> backup. So do you mean I could use rsync to read the LVM from the host, >>>> compare the LVM data with the data in the qcow2 on the backup server and >>>> simply transfer the differences to the file? Or does it work on another >>>> way? >>> When using rsync you can skip qcow2. Only two objects are needed: >>> 1. The LVM volume on the host. >>> 2. The last full backup on the backup client. >>> >>> rsync compares #1 and #2 efficiently over the network and only >>> transfers data from #1 which has changed. >>> >>> After rsync completes your full backup image is identical to the LVM >>> volume. Next week you can use it as the "last" image to rsync >>> against. >>> >>> Stefan >>> -- >>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> So I simply update the full backup, which is simply a raw file which get >> mounted while the backup? > The image file does not need to be mounted. Just rsync the raw image file. > > Stefan I tested the qemu-img command now, but it does not do that what I want. I have a VM with a 5GB disk, this disk is not allocated with 1GB of data. When I do the convert command the output is a 5GB qcow2 disk. What do I have to do to get a qcow2 file with only the allocated space/data from the LVM? I also tried the -c option of qemu-img convert but the result was nearly the same. Best Regards