Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp <lukas@laukamp.me>]

Am 12.10.2012 14:59, schrieb Stefan Hajnoczi:
> On Fri, Oct 12, 2012 at 1:51 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi:
>>
>>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi:
>>>>
>>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me>
>>>>> wrote:
>>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>>>>>>
>>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me>
>>>>>>> wrote:
>>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>>>>>>
>>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests
>>>>>>>>>>> and
>>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be
>>>>>>>>>>> to
>>>>>>>>>>> create a complete output file with dd, which wastes very much
>>>>>>>>>>> space.
>>>>>>>>>>> So I would like to create a backup of the LVM to a file which only
>>>>>>>>>>> locates the space which is used on the LVM. Would be create when
>>>>>>>>>>> the
>>>>>>>>>>> output file would be something like a qcow2 file which could be
>>>>>>>>>>> also
>>>>>>>>>>> simply startet with KVM.
>>>>>>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>>>>>>
>>>>>>>>>>        qemu-img convert -f raw -O qcow2 /dev/vg/vm001
>>>>>>>>>> vm001-backup.qcow2
>>>>>>>>>>
>>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see the
>>>>>>>>>> --sparse option in the man page).  So you don't need to use qcow2,
>>>>>>>>>> you
>>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not
>>>>>>>>>> use
>>>>>>>>>> disk space for zero regions.
>>>>>>>>>>
>>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>>>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>> --
>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>>
>>>>>>>>> Hello Stefano,
>>>>>>>>>
>>>>>>>>> thanks for the fast reply. I will test this later. In my case now it
>>>>>>>>> would
>>>>>>>>> be a offline backup. For the online backup I think about a seperated
>>>>>>>>> system
>>>>>>>>> which every day makes incremental backups and once a week a full
>>>>>>>>> backup.
>>>>>>>>> The
>>>>>>>>> main problem is, that the systems are in a WAN network and I need
>>>>>>>>> encryption
>>>>>>>>> between the systems. Would it be possible to do something like this:
>>>>>>>>> create
>>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the
>>>>>>>>> remote
>>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2
>>>>>>>>> files
>>>>>>>>> on
>>>>>>>>> the backup system? And in which format could be the incremental
>>>>>>>>> backups
>>>>>>>>> be
>>>>>>>>> stored?
>>>>>>> Since there is a WAN link it's important to use a compact image
>>>>>>> representation before hitting the network. I would use qemu-img
>>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output.  The
>>>>>>> qcow2 file does not contain zero regions and will therefore save a lot
>>>>>>> of network bandwidth compared to accessing the LVM volume over the
>>>>>>> WAN.
>>>>>>>
>>>>>>> If you are using rsync or another tool it's a different story.  You
>>>>>>> could rsync the current LVM volume on the host over the last full
>>>>>>> backup, it should avoid transferring image data which is already
>>>>>>> present in the last full backup - the result is that you only transfer
>>>>>>> changed data plus the rsync metadata.
>>>>>>>
>>>>>>> Stefan
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>> Hello Stefan,
>>>>>>
>>>>>> the rsync part I don't have understood fully. So to create a qcow2 on
>>>>>> the
>>>>>> host, transfer this to the backup server will result in the weekly full
>>>>>> backup. So do you mean I could use rsync to read the LVM from the host,
>>>>>> compare the LVM data with the data in the qcow2 on the backup server
>>>>>> and
>>>>>> simply transfer the differences to the file? Or does it work on another
>>>>>> way?
>>>>> When using rsync you can skip qcow2.  Only two objects are needed:
>>>>> 1. The LVM volume on the host.
>>>>> 2. The last full backup on the backup client.
>>>>>
>>>>> rsync compares #1 and #2 efficiently over the network and only
>>>>> transfers data from #1 which has changed.
>>>>>
>>>>> After rsync completes your full backup image is identical to the LVM
>>>>> volume.  Next week you can use it as the "last" image to rsync
>>>>> against.
>>>>>
>>>>> Stefan
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>> So I simply update the full backup, which is simply a raw file which get
>>>> mounted while the backup?
>>> The image file does not need to be mounted.  Just rsync the raw image
>>> file.
>>>
>>> Stefan
>>
>> Ah, thats great so to have a complete task in mind:
>>
>> 1. Create a qcow2 of a snapshot of an running VM
>> 2. Transfer the qcow2 to the backup node
>> 3. For incremental backup sync a daily LVM snapshot with the image on the
>> backup node via rsync
> Yes.  Only make sure not to rsync the raw LVM image onto the qcow2
> image - the backup client needs to have a raw image if you're syncing
> directly against the raw LVM volume.
>
> Stefan

So I should use a raw file for the syncing and convert it on the backup 
node to a qcow2 to have a good file format?

Best Regards