From mboxrd@z Thu Jan 1 00:00:00 1970 From: Lukas Laukamp Subject: Re: [User Question] How to create a backup of an LVM based maschine without wasting space Date: Fri, 12 Oct 2012 22:56:58 +0200 Message-ID: <5078841A.6040706@laukamp.me> References: <5077BE30.5010209@laukamp.me> <20121012084221.GC14822@stefanha-thinkpad.redhat.com> <5077DB9F.6050809@laukamp.me> <5077E03E.2040507@laukamp.me> <5077EDEB.6050703@laukamp.me> <5077F600.7080809@laukamp.me> <50785DEE.6060204@laukamp.me> <50786EEC.2030507@laukamp.me> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org To: Stefan Hajnoczi Return-path: Received: from mailer0.lippux.de ([5.9.218.245]:51197 "EHLO mailer0.lippux.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751957Ab2JLU5F (ORCPT ); Fri, 12 Oct 2012 16:57:05 -0400 In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: Am 12.10.2012 22:43, schrieb Stefan Hajnoczi: > On Fri, Oct 12, 2012 at 9:26 PM, Lukas Laukamp wrote: >> Am 12.10.2012 21:13, schrieb Stefan Hajnoczi: >> >>> On Fri, Oct 12, 2012 at 8:14 PM, Lukas Laukamp wrote: >>>> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: >>>> >>>>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp >>>>> wrote: >>>>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: >>>>>> >>>>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp >>>>>>> wrote: >>>>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>>>>>>> >>>>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp >>>>>>>>> wrote: >>>>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>>>>>>> >>>>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests >>>>>>>>>>>>> and >>>>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be >>>>>>>>>>>>> to >>>>>>>>>>>>> create a complete output file with dd, which wastes very much >>>>>>>>>>>>> space. >>>>>>>>>>>>> So I would like to create a backup of the LVM to a file which >>>>>>>>>>>>> only >>>>>>>>>>>>> locates the space which is used on the LVM. Would be create when >>>>>>>>>>>>> the >>>>>>>>>>>>> output file would be something like a qcow2 file which could be >>>>>>>>>>>>> also >>>>>>>>>>>>> simply startet with KVM. >>>>>>>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>>>>>>> >>>>>>>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>>>>>>> vm001-backup.qcow2 >>>>>>>>>>>> >>>>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see >>>>>>>>>>>> the >>>>>>>>>>>> --sparse option in the man page). So you don't need to use >>>>>>>>>>>> qcow2, >>>>>>>>>>>> you >>>>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not >>>>>>>>>>>> use >>>>>>>>>>>> disk space for zero regions. >>>>>>>>>>>> >>>>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>>>>>>> >>>>>>>>>>>> Stefan >>>>>>>>>>>> -- >>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>>>>> >>>>>>>>>>> Hello Stefano, >>>>>>>>>>> >>>>>>>>>>> thanks for the fast reply. I will test this later. In my case now >>>>>>>>>>> it >>>>>>>>>>> would >>>>>>>>>>> be a offline backup. For the online backup I think about a >>>>>>>>>>> seperated >>>>>>>>>>> system >>>>>>>>>>> which every day makes incremental backups and once a week a full >>>>>>>>>>> backup. >>>>>>>>>>> The >>>>>>>>>>> main problem is, that the systems are in a WAN network and I need >>>>>>>>>>> encryption >>>>>>>>>>> between the systems. Would it be possible to do something like >>>>>>>>>>> this: >>>>>>>>>>> create >>>>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>>>>>>> remote >>>>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>>>>>>> files >>>>>>>>>>> on >>>>>>>>>>> the backup system? And in which format could be the incremental >>>>>>>>>>> backups >>>>>>>>>>> be >>>>>>>>>>> stored? >>>>>>>>> Since there is a WAN link it's important to use a compact image >>>>>>>>> representation before hitting the network. I would use qemu-img >>>>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output. >>>>>>>>> The >>>>>>>>> qcow2 file does not contain zero regions and will therefore save a >>>>>>>>> lot >>>>>>>>> of network bandwidth compared to accessing the LVM volume over the >>>>>>>>> WAN. >>>>>>>>> >>>>>>>>> If you are using rsync or another tool it's a different story. You >>>>>>>>> could rsync the current LVM volume on the host over the last full >>>>>>>>> backup, it should avoid transferring image data which is already >>>>>>>>> present in the last full backup - the result is that you only >>>>>>>>> transfer >>>>>>>>> changed data plus the rsync metadata. >>>>>>>>> >>>>>>>>> Stefan >>>>>>>>> -- >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>> >>>>>>>> Hello Stefan, >>>>>>>> >>>>>>>> the rsync part I don't have understood fully. So to create a qcow2 on >>>>>>>> the >>>>>>>> host, transfer this to the backup server will result in the weekly >>>>>>>> full >>>>>>>> backup. So do you mean I could use rsync to read the LVM from the >>>>>>>> host, >>>>>>>> compare the LVM data with the data in the qcow2 on the backup server >>>>>>>> and >>>>>>>> simply transfer the differences to the file? Or does it work on >>>>>>>> another >>>>>>>> way? >>>>>>> When using rsync you can skip qcow2. Only two objects are needed: >>>>>>> 1. The LVM volume on the host. >>>>>>> 2. The last full backup on the backup client. >>>>>>> >>>>>>> rsync compares #1 and #2 efficiently over the network and only >>>>>>> transfers data from #1 which has changed. >>>>>>> >>>>>>> After rsync completes your full backup image is identical to the LVM >>>>>>> volume. Next week you can use it as the "last" image to rsync >>>>>>> against. >>>>>>> >>>>>>> Stefan >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>>> So I simply update the full backup, which is simply a raw file which >>>>>> get >>>>>> mounted while the backup? >>>>> The image file does not need to be mounted. Just rsync the raw image >>>>> file. >>>>> >>>>> Stefan >>>> >>>> I tested the qemu-img command now, but it does not do that what I want. I >>>> have a VM with a 5GB disk, this disk is not allocated with 1GB of data. >>>> When >>>> I do the convert command the output is a 5GB qcow2 disk. What do I have >>>> to >>>> do to get a qcow2 file with only the allocated space/data from the LVM? I >>>> also tried the -c option of qemu-img convert but the result was nearly >>>> the >>>> same. >>> Please show the exact command-lines you are using and the qemu-img >>> info output afterwards. >>> >>> Stefan >>> -- >>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> Hello, >> >> the VM is called mailer and I used this command: qemu-img convert -f raw -c >> -O qcow2 /dev/vmdisks/mailer ./vmachines/mailer.qcow2 >> >> Im on the /mnt dir because it's a seperate HDD with the folder vmachines. >> >> Thats the output of qemu-img info: >> >> image: ./vmachines/mailer.qcow2 >> file format: qcow2 >> virtual size: 5.0G (5368709120 bytes) >> disk size: 4.1G >> cluster_size: 65536 >> >> I checked the info of the fs inside the guest. The / partition is 4,7GB big >> and 1,2GB are used. The fs is ext3. > I wonder if you see better results without the -c option. > > Stefan > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html So I would say that the results are very similar. So with -c option the file is 650-800MB smaller but thats not very much. I have a few bigger VM disks with sizes like 100GB when we calculat that in % so we see that the compression is very ecactly 20% so I would have an 80GB image of a disk where only 3GBs are used inside the VM. I think that it must be possible to create an image with a size like the used space + a few hundret MB with metadata or something like that. Best Regards