[User Question] How to create a backup of an LVM based maschine without wasting space

[User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Hey all,

I have a simple user question. I have a few LVM based KVM guests and 
wan't to backup them to files. The simple and nasty way would be to 
create a complete output file with dd, which wastes very much space. So 
I would like to create a backup of the LVM to a file which only locates 
the space which is used on the LVM. Would be create when the output file 
would be something like a qcow2 file which could be also simply startet 
with KVM.

Is this supported by qemu_backup or what would be the best way to do this?

PS: Sorry for my bed english

Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Stefan Hajnoczi]

On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
> I have a simple user question. I have a few LVM based KVM guests and
> wan't to backup them to files. The simple and nasty way would be to
> create a complete output file with dd, which wastes very much space.
> So I would like to create a backup of the LVM to a file which only
> locates the space which is used on the LVM. Would be create when the
> output file would be something like a qcow2 file which could be also
> simply startet with KVM.

If the VM is not running you can use "qemu-img convert":

  qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2

Note that cp(1) tries to make the destination file sparse (see the
--sparse option in the man page).  So you don't need to use qcow2, you
can use cp(1) to copy the LVM volume to a raw file.  It will not use
disk space for zero regions.

If the VM is running you need to use LVM snapshots or stop the VM
temporarily so a crash-consistent backup can be taken.

Stefan

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Am 12.10.2012 10:58, schrieb Lukas Laukamp:
> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>> I have a simple user question. I have a few LVM based KVM guests and
>>> wan't to backup them to files. The simple and nasty way would be to
>>> create a complete output file with dd, which wastes very much space.
>>> So I would like to create a backup of the LVM to a file which only
>>> locates the space which is used on the LVM. Would be create when the
>>> output file would be something like a qcow2 file which could be also
>>> simply startet with KVM.
>> If the VM is not running you can use "qemu-img convert":
>>
>>    qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2
>>
>> Note that cp(1) tries to make the destination file sparse (see the
>> --sparse option in the man page).  So you don't need to use qcow2, you
>> can use cp(1) to copy the LVM volume to a raw file.  It will not use
>> disk space for zero regions.
>>
>> If the VM is running you need to use LVM snapshots or stop the VM
>> temporarily so a crash-consistent backup can be taken.
>>
>> Stefan
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> Hello Stefano,
>
> thanks for the fast reply. I will test this later. In my case now it 
> would be a offline backup. For the online backup I think about a 
> seperated system which every day makes incremental backups and once a 
> week a full backup. The main problem is, that the systems are in a WAN 
> network and I need encryption between the systems. Would it be 
> possible to do something like this: create the LVM snapshot for the 
> backup, read this LVM snapshot with the remote backup system via ssh 
> tunnel and save the output of this to qcow2 files on the backup 
> system? And in which format could be the incremental backups be stored?
>
> Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Stefan Hajnoczi]

On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> wrote:
> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>
>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>
>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>
>>>> I have a simple user question. I have a few LVM based KVM guests and
>>>> wan't to backup them to files. The simple and nasty way would be to
>>>> create a complete output file with dd, which wastes very much space.
>>>> So I would like to create a backup of the LVM to a file which only
>>>> locates the space which is used on the LVM. Would be create when the
>>>> output file would be something like a qcow2 file which could be also
>>>> simply startet with KVM.
>>>
>>> If the VM is not running you can use "qemu-img convert":
>>>
>>>    qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2
>>>
>>> Note that cp(1) tries to make the destination file sparse (see the
>>> --sparse option in the man page).  So you don't need to use qcow2, you
>>> can use cp(1) to copy the LVM volume to a raw file.  It will not use
>>> disk space for zero regions.
>>>
>>> If the VM is running you need to use LVM snapshots or stop the VM
>>> temporarily so a crash-consistent backup can be taken.
>>>
>>> Stefan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
>> Hello Stefano,
>>
>> thanks for the fast reply. I will test this later. In my case now it would
>> be a offline backup. For the online backup I think about a seperated system
>> which every day makes incremental backups and once a week a full backup. The
>> main problem is, that the systems are in a WAN network and I need encryption
>> between the systems. Would it be possible to do something like this: create
>> the LVM snapshot for the backup, read this LVM snapshot with the remote
>> backup system via ssh tunnel and save the output of this to qcow2 files on
>> the backup system? And in which format could be the incremental backups be
>> stored?

Since there is a WAN link it's important to use a compact image
representation before hitting the network. I would use qemu-img
convert -O qcow2 on the host and only transfer the qcow2 output.  The
qcow2 file does not contain zero regions and will therefore save a lot
of network bandwidth compared to accessing the LVM volume over the
WAN.

If you are using rsync or another tool it's a different story.  You
could rsync the current LVM volume on the host over the last full
backup, it should avoid transferring image data which is already
present in the last full backup - the result is that you only transfer
changed data plus the rsync metadata.

Stefan

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> wrote:
>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>
>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>> I have a simple user question. I have a few LVM based KVM guests and
>>>>> wan't to backup them to files. The simple and nasty way would be to
>>>>> create a complete output file with dd, which wastes very much space.
>>>>> So I would like to create a backup of the LVM to a file which only
>>>>> locates the space which is used on the LVM. Would be create when the
>>>>> output file would be something like a qcow2 file which could be also
>>>>> simply startet with KVM.
>>>> If the VM is not running you can use "qemu-img convert":
>>>>
>>>>     qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2
>>>>
>>>> Note that cp(1) tries to make the destination file sparse (see the
>>>> --sparse option in the man page).  So you don't need to use qcow2, you
>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not use
>>>> disk space for zero regions.
>>>>
>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>> temporarily so a crash-consistent backup can be taken.
>>>>
>>>> Stefan
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>> Hello Stefano,
>>>
>>> thanks for the fast reply. I will test this later. In my case now it would
>>> be a offline backup. For the online backup I think about a seperated system
>>> which every day makes incremental backups and once a week a full backup. The
>>> main problem is, that the systems are in a WAN network and I need encryption
>>> between the systems. Would it be possible to do something like this: create
>>> the LVM snapshot for the backup, read this LVM snapshot with the remote
>>> backup system via ssh tunnel and save the output of this to qcow2 files on
>>> the backup system? And in which format could be the incremental backups be
>>> stored?
> Since there is a WAN link it's important to use a compact image
> representation before hitting the network. I would use qemu-img
> convert -O qcow2 on the host and only transfer the qcow2 output.  The
> qcow2 file does not contain zero regions and will therefore save a lot
> of network bandwidth compared to accessing the LVM volume over the
> WAN.
>
> If you are using rsync or another tool it's a different story.  You
> could rsync the current LVM volume on the host over the last full
> backup, it should avoid transferring image data which is already
> present in the last full backup - the result is that you only transfer
> changed data plus the rsync metadata.
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hello Stefan,

the rsync part I don't have understood fully. So to create a qcow2 on 
the host, transfer this to the backup server will result in the weekly 
full backup. So do you mean I could use rsync to read the LVM from the 
host, compare the LVM data with the data in the qcow2 on the backup 
server and simply transfer the differences to the file? Or does it work 
on another way?

Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Stefan Hajnoczi]

On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>
>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>
>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>
>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>
>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>
>>>>>> I have a simple user question. I have a few LVM based KVM guests and
>>>>>> wan't to backup them to files. The simple and nasty way would be to
>>>>>> create a complete output file with dd, which wastes very much space.
>>>>>> So I would like to create a backup of the LVM to a file which only
>>>>>> locates the space which is used on the LVM. Would be create when the
>>>>>> output file would be something like a qcow2 file which could be also
>>>>>> simply startet with KVM.
>>>>>
>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>
>>>>>     qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2
>>>>>
>>>>> Note that cp(1) tries to make the destination file sparse (see the
>>>>> --sparse option in the man page).  So you don't need to use qcow2, you
>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not use
>>>>> disk space for zero regions.
>>>>>
>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>
>>>>> Stefan
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>> Hello Stefano,
>>>>
>>>> thanks for the fast reply. I will test this later. In my case now it
>>>> would
>>>> be a offline backup. For the online backup I think about a seperated
>>>> system
>>>> which every day makes incremental backups and once a week a full backup.
>>>> The
>>>> main problem is, that the systems are in a WAN network and I need
>>>> encryption
>>>> between the systems. Would it be possible to do something like this:
>>>> create
>>>> the LVM snapshot for the backup, read this LVM snapshot with the remote
>>>> backup system via ssh tunnel and save the output of this to qcow2 files
>>>> on
>>>> the backup system? And in which format could be the incremental backups
>>>> be
>>>> stored?
>>
>> Since there is a WAN link it's important to use a compact image
>> representation before hitting the network. I would use qemu-img
>> convert -O qcow2 on the host and only transfer the qcow2 output.  The
>> qcow2 file does not contain zero regions and will therefore save a lot
>> of network bandwidth compared to accessing the LVM volume over the
>> WAN.
>>
>> If you are using rsync or another tool it's a different story.  You
>> could rsync the current LVM volume on the host over the last full
>> backup, it should avoid transferring image data which is already
>> present in the last full backup - the result is that you only transfer
>> changed data plus the rsync metadata.
>>
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> Hello Stefan,
>
> the rsync part I don't have understood fully. So to create a qcow2 on the
> host, transfer this to the backup server will result in the weekly full
> backup. So do you mean I could use rsync to read the LVM from the host,
> compare the LVM data with the data in the qcow2 on the backup server and
> simply transfer the differences to the file? Or does it work on another way?

When using rsync you can skip qcow2.  Only two objects are needed:
1. The LVM volume on the host.
2. The last full backup on the backup client.

rsync compares #1 and #2 efficiently over the network and only
transfers data from #1 which has changed.

After rsync completes your full backup image is identical to the LVM
volume.  Next week you can use it as the "last" image to rsync
against.

Stefan

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Am 12.10.2012 12:47, schrieb Stefan Hajnoczi:
> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>>
>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>>
>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>> I have a simple user question. I have a few LVM based KVM guests and
>>>>>>> wan't to backup them to files. The simple and nasty way would be to
>>>>>>> create a complete output file with dd, which wastes very much space.
>>>>>>> So I would like to create a backup of the LVM to a file which only
>>>>>>> locates the space which is used on the LVM. Would be create when the
>>>>>>> output file would be something like a qcow2 file which could be also
>>>>>>> simply startet with KVM.
>>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>>
>>>>>>      qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2
>>>>>>
>>>>>> Note that cp(1) tries to make the destination file sparse (see the
>>>>>> --sparse option in the man page).  So you don't need to use qcow2, you
>>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not use
>>>>>> disk space for zero regions.
>>>>>>
>>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>>
>>>>>> Stefan
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>> Hello Stefano,
>>>>>
>>>>> thanks for the fast reply. I will test this later. In my case now it
>>>>> would
>>>>> be a offline backup. For the online backup I think about a seperated
>>>>> system
>>>>> which every day makes incremental backups and once a week a full backup.
>>>>> The
>>>>> main problem is, that the systems are in a WAN network and I need
>>>>> encryption
>>>>> between the systems. Would it be possible to do something like this:
>>>>> create
>>>>> the LVM snapshot for the backup, read this LVM snapshot with the remote
>>>>> backup system via ssh tunnel and save the output of this to qcow2 files
>>>>> on
>>>>> the backup system? And in which format could be the incremental backups
>>>>> be
>>>>> stored?
>>> Since there is a WAN link it's important to use a compact image
>>> representation before hitting the network. I would use qemu-img
>>> convert -O qcow2 on the host and only transfer the qcow2 output.  The
>>> qcow2 file does not contain zero regions and will therefore save a lot
>>> of network bandwidth compared to accessing the LVM volume over the
>>> WAN.
>>>
>>> If you are using rsync or another tool it's a different story.  You
>>> could rsync the current LVM volume on the host over the last full
>>> backup, it should avoid transferring image data which is already
>>> present in the last full backup - the result is that you only transfer
>>> changed data plus the rsync metadata.
>>>
>>> Stefan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> Hello Stefan,
>>
>> the rsync part I don't have understood fully. So to create a qcow2 on the
>> host, transfer this to the backup server will result in the weekly full
>> backup. So do you mean I could use rsync to read the LVM from the host,
>> compare the LVM data with the data in the qcow2 on the backup server and
>> simply transfer the differences to the file? Or does it work on another way?
> When using rsync you can skip qcow2.  Only two objects are needed:
> 1. The LVM volume on the host.
> 2. The last full backup on the backup client.
>
> rsync compares #1 and #2 efficiently over the network and only
> transfers data from #1 which has changed.
>
> After rsync completes your full backup image is identical to the LVM
> volume.  Next week you can use it as the "last" image to rsync
> against.
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

So I simply update the full backup, which is simply a raw file which get 
mounted while the backup?

Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Stefan Hajnoczi]

On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi:
>
>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>
>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>>>
>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me>
>>>> wrote:
>>>>>
>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>>>
>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>>>
>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>>>
>>>>>>>> I have a simple user question. I have a few LVM based KVM guests and
>>>>>>>> wan't to backup them to files. The simple and nasty way would be to
>>>>>>>> create a complete output file with dd, which wastes very much space.
>>>>>>>> So I would like to create a backup of the LVM to a file which only
>>>>>>>> locates the space which is used on the LVM. Would be create when the
>>>>>>>> output file would be something like a qcow2 file which could be also
>>>>>>>> simply startet with KVM.
>>>>>>>
>>>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>>>
>>>>>>>      qemu-img convert -f raw -O qcow2 /dev/vg/vm001
>>>>>>> vm001-backup.qcow2
>>>>>>>
>>>>>>> Note that cp(1) tries to make the destination file sparse (see the
>>>>>>> --sparse option in the man page).  So you don't need to use qcow2,
>>>>>>> you
>>>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not use
>>>>>>> disk space for zero regions.
>>>>>>>
>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>>>
>>>>>>> Stefan
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>
>>>>>> Hello Stefano,
>>>>>>
>>>>>> thanks for the fast reply. I will test this later. In my case now it
>>>>>> would
>>>>>> be a offline backup. For the online backup I think about a seperated
>>>>>> system
>>>>>> which every day makes incremental backups and once a week a full
>>>>>> backup.
>>>>>> The
>>>>>> main problem is, that the systems are in a WAN network and I need
>>>>>> encryption
>>>>>> between the systems. Would it be possible to do something like this:
>>>>>> create
>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the
>>>>>> remote
>>>>>> backup system via ssh tunnel and save the output of this to qcow2
>>>>>> files
>>>>>> on
>>>>>> the backup system? And in which format could be the incremental
>>>>>> backups
>>>>>> be
>>>>>> stored?
>>>>
>>>> Since there is a WAN link it's important to use a compact image
>>>> representation before hitting the network. I would use qemu-img
>>>> convert -O qcow2 on the host and only transfer the qcow2 output.  The
>>>> qcow2 file does not contain zero regions and will therefore save a lot
>>>> of network bandwidth compared to accessing the LVM volume over the
>>>> WAN.
>>>>
>>>> If you are using rsync or another tool it's a different story.  You
>>>> could rsync the current LVM volume on the host over the last full
>>>> backup, it should avoid transferring image data which is already
>>>> present in the last full backup - the result is that you only transfer
>>>> changed data plus the rsync metadata.
>>>>
>>>> Stefan
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>> Hello Stefan,
>>>
>>> the rsync part I don't have understood fully. So to create a qcow2 on the
>>> host, transfer this to the backup server will result in the weekly full
>>> backup. So do you mean I could use rsync to read the LVM from the host,
>>> compare the LVM data with the data in the qcow2 on the backup server and
>>> simply transfer the differences to the file? Or does it work on another
>>> way?
>>
>> When using rsync you can skip qcow2.  Only two objects are needed:
>> 1. The LVM volume on the host.
>> 2. The last full backup on the backup client.
>>
>> rsync compares #1 and #2 efficiently over the network and only
>> transfers data from #1 which has changed.
>>
>> After rsync completes your full backup image is identical to the LVM
>> volume.  Next week you can use it as the "last" image to rsync
>> against.
>>
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> So I simply update the full backup, which is simply a raw file which get
> mounted while the backup?

The image file does not need to be mounted.  Just rsync the raw image file.

Stefan

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Am 12.10.2012 13:36, schrieb Stefan Hajnoczi:
> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi:
>>
>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>>>>
>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me>
>>>>> wrote:
>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>>>>
>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests and
>>>>>>>>> wan't to backup them to files. The simple and nasty way would be to
>>>>>>>>> create a complete output file with dd, which wastes very much space.
>>>>>>>>> So I would like to create a backup of the LVM to a file which only
>>>>>>>>> locates the space which is used on the LVM. Would be create when the
>>>>>>>>> output file would be something like a qcow2 file which could be also
>>>>>>>>> simply startet with KVM.
>>>>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>>>>
>>>>>>>>       qemu-img convert -f raw -O qcow2 /dev/vg/vm001
>>>>>>>> vm001-backup.qcow2
>>>>>>>>
>>>>>>>> Note that cp(1) tries to make the destination file sparse (see the
>>>>>>>> --sparse option in the man page).  So you don't need to use qcow2,
>>>>>>>> you
>>>>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not use
>>>>>>>> disk space for zero regions.
>>>>>>>>
>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>> Hello Stefano,
>>>>>>>
>>>>>>> thanks for the fast reply. I will test this later. In my case now it
>>>>>>> would
>>>>>>> be a offline backup. For the online backup I think about a seperated
>>>>>>> system
>>>>>>> which every day makes incremental backups and once a week a full
>>>>>>> backup.
>>>>>>> The
>>>>>>> main problem is, that the systems are in a WAN network and I need
>>>>>>> encryption
>>>>>>> between the systems. Would it be possible to do something like this:
>>>>>>> create
>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the
>>>>>>> remote
>>>>>>> backup system via ssh tunnel and save the output of this to qcow2
>>>>>>> files
>>>>>>> on
>>>>>>> the backup system? And in which format could be the incremental
>>>>>>> backups
>>>>>>> be
>>>>>>> stored?
>>>>> Since there is a WAN link it's important to use a compact image
>>>>> representation before hitting the network. I would use qemu-img
>>>>> convert -O qcow2 on the host and only transfer the qcow2 output.  The
>>>>> qcow2 file does not contain zero regions and will therefore save a lot
>>>>> of network bandwidth compared to accessing the LVM volume over the
>>>>> WAN.
>>>>>
>>>>> If you are using rsync or another tool it's a different story.  You
>>>>> could rsync the current LVM volume on the host over the last full
>>>>> backup, it should avoid transferring image data which is already
>>>>> present in the last full backup - the result is that you only transfer
>>>>> changed data plus the rsync metadata.
>>>>>
>>>>> Stefan
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>> Hello Stefan,
>>>>
>>>> the rsync part I don't have understood fully. So to create a qcow2 on the
>>>> host, transfer this to the backup server will result in the weekly full
>>>> backup. So do you mean I could use rsync to read the LVM from the host,
>>>> compare the LVM data with the data in the qcow2 on the backup server and
>>>> simply transfer the differences to the file? Or does it work on another
>>>> way?
>>> When using rsync you can skip qcow2.  Only two objects are needed:
>>> 1. The LVM volume on the host.
>>> 2. The last full backup on the backup client.
>>>
>>> rsync compares #1 and #2 efficiently over the network and only
>>> transfers data from #1 which has changed.
>>>
>>> After rsync completes your full backup image is identical to the LVM
>>> volume.  Next week you can use it as the "last" image to rsync
>>> against.
>>>
>>> Stefan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> So I simply update the full backup, which is simply a raw file which get
>> mounted while the backup?
> The image file does not need to be mounted.  Just rsync the raw image file.
>
> Stefan

Ah, thats great so to have a complete task in mind:

1. Create a qcow2 of a snapshot of an running VM
2. Transfer the qcow2 to the backup node
3. For incremental backup sync a daily LVM snapshot with the image on 
the backup node via rsync

Is that right?

Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Stefan Hajnoczi]

On Fri, Oct 12, 2012 at 1:51 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi:
>
>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>
>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi:
>>>
>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me>
>>>> wrote:
>>>>>
>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>>>>>
>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me>
>>>>>> wrote:
>>>>>>>
>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>>>>>
>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>>>>>
>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>>>>>
>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests
>>>>>>>>>> and
>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be
>>>>>>>>>> to
>>>>>>>>>> create a complete output file with dd, which wastes very much
>>>>>>>>>> space.
>>>>>>>>>> So I would like to create a backup of the LVM to a file which only
>>>>>>>>>> locates the space which is used on the LVM. Would be create when
>>>>>>>>>> the
>>>>>>>>>> output file would be something like a qcow2 file which could be
>>>>>>>>>> also
>>>>>>>>>> simply startet with KVM.
>>>>>>>>>
>>>>>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>>>>>
>>>>>>>>>       qemu-img convert -f raw -O qcow2 /dev/vg/vm001
>>>>>>>>> vm001-backup.qcow2
>>>>>>>>>
>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see the
>>>>>>>>> --sparse option in the man page).  So you don't need to use qcow2,
>>>>>>>>> you
>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not
>>>>>>>>> use
>>>>>>>>> disk space for zero regions.
>>>>>>>>>
>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>>>>>
>>>>>>>> Hello Stefano,
>>>>>>>>
>>>>>>>> thanks for the fast reply. I will test this later. In my case now it
>>>>>>>> would
>>>>>>>> be a offline backup. For the online backup I think about a seperated
>>>>>>>> system
>>>>>>>> which every day makes incremental backups and once a week a full
>>>>>>>> backup.
>>>>>>>> The
>>>>>>>> main problem is, that the systems are in a WAN network and I need
>>>>>>>> encryption
>>>>>>>> between the systems. Would it be possible to do something like this:
>>>>>>>> create
>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the
>>>>>>>> remote
>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2
>>>>>>>> files
>>>>>>>> on
>>>>>>>> the backup system? And in which format could be the incremental
>>>>>>>> backups
>>>>>>>> be
>>>>>>>> stored?
>>>>>>
>>>>>> Since there is a WAN link it's important to use a compact image
>>>>>> representation before hitting the network. I would use qemu-img
>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output.  The
>>>>>> qcow2 file does not contain zero regions and will therefore save a lot
>>>>>> of network bandwidth compared to accessing the LVM volume over the
>>>>>> WAN.
>>>>>>
>>>>>> If you are using rsync or another tool it's a different story.  You
>>>>>> could rsync the current LVM volume on the host over the last full
>>>>>> backup, it should avoid transferring image data which is already
>>>>>> present in the last full backup - the result is that you only transfer
>>>>>> changed data plus the rsync metadata.
>>>>>>
>>>>>> Stefan
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>>> Hello Stefan,
>>>>>
>>>>> the rsync part I don't have understood fully. So to create a qcow2 on
>>>>> the
>>>>> host, transfer this to the backup server will result in the weekly full
>>>>> backup. So do you mean I could use rsync to read the LVM from the host,
>>>>> compare the LVM data with the data in the qcow2 on the backup server
>>>>> and
>>>>> simply transfer the differences to the file? Or does it work on another
>>>>> way?
>>>>
>>>> When using rsync you can skip qcow2.  Only two objects are needed:
>>>> 1. The LVM volume on the host.
>>>> 2. The last full backup on the backup client.
>>>>
>>>> rsync compares #1 and #2 efficiently over the network and only
>>>> transfers data from #1 which has changed.
>>>>
>>>> After rsync completes your full backup image is identical to the LVM
>>>> volume.  Next week you can use it as the "last" image to rsync
>>>> against.
>>>>
>>>> Stefan
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>> So I simply update the full backup, which is simply a raw file which get
>>> mounted while the backup?
>>
>> The image file does not need to be mounted.  Just rsync the raw image
>> file.
>>
>> Stefan
>
>
> Ah, thats great so to have a complete task in mind:
>
> 1. Create a qcow2 of a snapshot of an running VM
> 2. Transfer the qcow2 to the backup node
> 3. For incremental backup sync a daily LVM snapshot with the image on the
> backup node via rsync

Yes.  Only make sure not to rsync the raw LVM image onto the qcow2
image - the backup client needs to have a raw image if you're syncing
directly against the raw LVM volume.

Stefan

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Am 12.10.2012 14:59, schrieb Stefan Hajnoczi:
> On Fri, Oct 12, 2012 at 1:51 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi:
>>
>>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi:
>>>>
>>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me>
>>>>> wrote:
>>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>>>>>>
>>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me>
>>>>>>> wrote:
>>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>>>>>>
>>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests
>>>>>>>>>>> and
>>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be
>>>>>>>>>>> to
>>>>>>>>>>> create a complete output file with dd, which wastes very much
>>>>>>>>>>> space.
>>>>>>>>>>> So I would like to create a backup of the LVM to a file which only
>>>>>>>>>>> locates the space which is used on the LVM. Would be create when
>>>>>>>>>>> the
>>>>>>>>>>> output file would be something like a qcow2 file which could be
>>>>>>>>>>> also
>>>>>>>>>>> simply startet with KVM.
>>>>>>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>>>>>>
>>>>>>>>>>        qemu-img convert -f raw -O qcow2 /dev/vg/vm001
>>>>>>>>>> vm001-backup.qcow2
>>>>>>>>>>
>>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see the
>>>>>>>>>> --sparse option in the man page).  So you don't need to use qcow2,
>>>>>>>>>> you
>>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not
>>>>>>>>>> use
>>>>>>>>>> disk space for zero regions.
>>>>>>>>>>
>>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>>>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>> --
>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>>
>>>>>>>>> Hello Stefano,
>>>>>>>>>
>>>>>>>>> thanks for the fast reply. I will test this later. In my case now it
>>>>>>>>> would
>>>>>>>>> be a offline backup. For the online backup I think about a seperated
>>>>>>>>> system
>>>>>>>>> which every day makes incremental backups and once a week a full
>>>>>>>>> backup.
>>>>>>>>> The
>>>>>>>>> main problem is, that the systems are in a WAN network and I need
>>>>>>>>> encryption
>>>>>>>>> between the systems. Would it be possible to do something like this:
>>>>>>>>> create
>>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the
>>>>>>>>> remote
>>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2
>>>>>>>>> files
>>>>>>>>> on
>>>>>>>>> the backup system? And in which format could be the incremental
>>>>>>>>> backups
>>>>>>>>> be
>>>>>>>>> stored?
>>>>>>> Since there is a WAN link it's important to use a compact image
>>>>>>> representation before hitting the network. I would use qemu-img
>>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output.  The
>>>>>>> qcow2 file does not contain zero regions and will therefore save a lot
>>>>>>> of network bandwidth compared to accessing the LVM volume over the
>>>>>>> WAN.
>>>>>>>
>>>>>>> If you are using rsync or another tool it's a different story.  You
>>>>>>> could rsync the current LVM volume on the host over the last full
>>>>>>> backup, it should avoid transferring image data which is already
>>>>>>> present in the last full backup - the result is that you only transfer
>>>>>>> changed data plus the rsync metadata.
>>>>>>>
>>>>>>> Stefan
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>> Hello Stefan,
>>>>>>
>>>>>> the rsync part I don't have understood fully. So to create a qcow2 on
>>>>>> the
>>>>>> host, transfer this to the backup server will result in the weekly full
>>>>>> backup. So do you mean I could use rsync to read the LVM from the host,
>>>>>> compare the LVM data with the data in the qcow2 on the backup server
>>>>>> and
>>>>>> simply transfer the differences to the file? Or does it work on another
>>>>>> way?
>>>>> When using rsync you can skip qcow2.  Only two objects are needed:
>>>>> 1. The LVM volume on the host.
>>>>> 2. The last full backup on the backup client.
>>>>>
>>>>> rsync compares #1 and #2 efficiently over the network and only
>>>>> transfers data from #1 which has changed.
>>>>>
>>>>> After rsync completes your full backup image is identical to the LVM
>>>>> volume.  Next week you can use it as the "last" image to rsync
>>>>> against.
>>>>>
>>>>> Stefan
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>> So I simply update the full backup, which is simply a raw file which get
>>>> mounted while the backup?
>>> The image file does not need to be mounted.  Just rsync the raw image
>>> file.
>>>
>>> Stefan
>>
>> Ah, thats great so to have a complete task in mind:
>>
>> 1. Create a qcow2 of a snapshot of an running VM
>> 2. Transfer the qcow2 to the backup node
>> 3. For incremental backup sync a daily LVM snapshot with the image on the
>> backup node via rsync
> Yes.  Only make sure not to rsync the raw LVM image onto the qcow2
> image - the backup client needs to have a raw image if you're syncing
> directly against the raw LVM volume.
>
> Stefan

So I should use a raw file for the syncing and convert it on the backup 
node to a qcow2 to have a good file format?

Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Stefan Hajnoczi]

On Fri, Oct 12, 2012 at 3:02 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
> Am 12.10.2012 14:59, schrieb Stefan Hajnoczi:
>
>> On Fri, Oct 12, 2012 at 1:51 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>
>>> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi:
>>>
>>>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me>
>>>> wrote:
>>>>>
>>>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi:
>>>>>
>>>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me>
>>>>>> wrote:
>>>>>>>
>>>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>>>>>>>
>>>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>>>>>>>
>>>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests
>>>>>>>>>>>> and
>>>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be
>>>>>>>>>>>> to
>>>>>>>>>>>> create a complete output file with dd, which wastes very much
>>>>>>>>>>>> space.
>>>>>>>>>>>> So I would like to create a backup of the LVM to a file which
>>>>>>>>>>>> only
>>>>>>>>>>>> locates the space which is used on the LVM. Would be create when
>>>>>>>>>>>> the
>>>>>>>>>>>> output file would be something like a qcow2 file which could be
>>>>>>>>>>>> also
>>>>>>>>>>>> simply startet with KVM.
>>>>>>>>>>>
>>>>>>>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>>>>>>>
>>>>>>>>>>>        qemu-img convert -f raw -O qcow2 /dev/vg/vm001
>>>>>>>>>>> vm001-backup.qcow2
>>>>>>>>>>>
>>>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see
>>>>>>>>>>> the
>>>>>>>>>>> --sparse option in the man page).  So you don't need to use
>>>>>>>>>>> qcow2,
>>>>>>>>>>> you
>>>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not
>>>>>>>>>>> use
>>>>>>>>>>> disk space for zero regions.
>>>>>>>>>>>
>>>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>>>>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>>>>>>>
>>>>>>>>>>> Stefan
>>>>>>>>>>> --
>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hello Stefano,
>>>>>>>>>>
>>>>>>>>>> thanks for the fast reply. I will test this later. In my case now
>>>>>>>>>> it
>>>>>>>>>> would
>>>>>>>>>> be a offline backup. For the online backup I think about a
>>>>>>>>>> seperated
>>>>>>>>>> system
>>>>>>>>>> which every day makes incremental backups and once a week a full
>>>>>>>>>> backup.
>>>>>>>>>> The
>>>>>>>>>> main problem is, that the systems are in a WAN network and I need
>>>>>>>>>> encryption
>>>>>>>>>> between the systems. Would it be possible to do something like
>>>>>>>>>> this:
>>>>>>>>>> create
>>>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the
>>>>>>>>>> remote
>>>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2
>>>>>>>>>> files
>>>>>>>>>> on
>>>>>>>>>> the backup system? And in which format could be the incremental
>>>>>>>>>> backups
>>>>>>>>>> be
>>>>>>>>>> stored?
>>>>>>>>
>>>>>>>> Since there is a WAN link it's important to use a compact image
>>>>>>>> representation before hitting the network. I would use qemu-img
>>>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output.
>>>>>>>> The
>>>>>>>> qcow2 file does not contain zero regions and will therefore save a
>>>>>>>> lot
>>>>>>>> of network bandwidth compared to accessing the LVM volume over the
>>>>>>>> WAN.
>>>>>>>>
>>>>>>>> If you are using rsync or another tool it's a different story.  You
>>>>>>>> could rsync the current LVM volume on the host over the last full
>>>>>>>> backup, it should avoid transferring image data which is already
>>>>>>>> present in the last full backup - the result is that you only
>>>>>>>> transfer
>>>>>>>> changed data plus the rsync metadata.
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>>
>>>>>>> Hello Stefan,
>>>>>>>
>>>>>>> the rsync part I don't have understood fully. So to create a qcow2 on
>>>>>>> the
>>>>>>> host, transfer this to the backup server will result in the weekly
>>>>>>> full
>>>>>>> backup. So do you mean I could use rsync to read the LVM from the
>>>>>>> host,
>>>>>>> compare the LVM data with the data in the qcow2 on the backup server
>>>>>>> and
>>>>>>> simply transfer the differences to the file? Or does it work on
>>>>>>> another
>>>>>>> way?
>>>>>>
>>>>>> When using rsync you can skip qcow2.  Only two objects are needed:
>>>>>> 1. The LVM volume on the host.
>>>>>> 2. The last full backup on the backup client.
>>>>>>
>>>>>> rsync compares #1 and #2 efficiently over the network and only
>>>>>> transfers data from #1 which has changed.
>>>>>>
>>>>>> After rsync completes your full backup image is identical to the LVM
>>>>>> volume.  Next week you can use it as the "last" image to rsync
>>>>>> against.
>>>>>>
>>>>>> Stefan
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>>> So I simply update the full backup, which is simply a raw file which
>>>>> get
>>>>> mounted while the backup?
>>>>
>>>> The image file does not need to be mounted.  Just rsync the raw image
>>>> file.
>>>>
>>>> Stefan
>>>
>>>
>>> Ah, thats great so to have a complete task in mind:
>>>
>>> 1. Create a qcow2 of a snapshot of an running VM
>>> 2. Transfer the qcow2 to the backup node
>>> 3. For incremental backup sync a daily LVM snapshot with the image on the
>>> backup node via rsync
>>
>> Yes.  Only make sure not to rsync the raw LVM image onto the qcow2
>> image - the backup client needs to have a raw image if you're syncing
>> directly against the raw LVM volume.
>>
>> Stefan
>
>
> So I should use a raw file for the syncing and convert it on the backup node
> to a qcow2 to have a good file format?

I would leave them raw as long as they are sparse (zero regions do not
take up space).  If you need to copy them you can either convert to
qcow2 or use tools that preserve sparseness (BTW compression tools are
good at this).

But it's up to you.

Stefan

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Javier Guerra Giraldez]

On Fri, Oct 12, 2012 at 9:25 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> I would leave them raw as long as they are sparse (zero regions do not
> take up space).  If you need to copy them you can either convert to
> qcow2 or use tools that preserve sparseness (BTW compression tools are
> good at this).

note that free blocks previously used by deleted files won't be
sparse, won't be zero and won't be much reduced by compression.

i'd say the usual advice stays:

A: if you run any non-trivial application on that VM, then use a real
network backup tool 'from the inside' of the VM
B: if real point-in-time application-cache-storage consistency is not
important, then you can:

- make a read-write LVM snapshot
- mount that and fsck.  (it will appear as not-cleanly unmounted)
- backup the files. (i like rsync, especially if you have an
uncompressed previous backup)
- umount and destroy the snapshot
- optionally compress the backup


but seriously consider option A before.  especially important if you
run any DB on that VM

-- 
Javier

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Am 12.10.2012 16:36, schrieb Javier Guerra Giraldez:
> On Fri, Oct 12, 2012 at 9:25 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> I would leave them raw as long as they are sparse (zero regions do not
>> take up space).  If you need to copy them you can either convert to
>> qcow2 or use tools that preserve sparseness (BTW compression tools are
>> good at this).
> note that free blocks previously used by deleted files won't be
> sparse, won't be zero and won't be much reduced by compression.
>
> i'd say the usual advice stays:
>
> A: if you run any non-trivial application on that VM, then use a real
> network backup tool 'from the inside' of the VM
> B: if real point-in-time application-cache-storage consistency is not
> important, then you can:
>
> - make a read-write LVM snapshot
> - mount that and fsck.  (it will appear as not-cleanly unmounted)
> - backup the files. (i like rsync, especially if you have an
> uncompressed previous backup)
> - umount and destroy the snapshot
> - optionally compress the backup
>
>
> but seriously consider option A before.  especially important if you
> run any DB on that VM
>

I already thought about such situations. So DB systems are a real 
problem. I know that some data can be better backup directly but for 
example a VM for a Web- or Mailserver I want to backup completely to get 
this services get to work again very fast if there are problems. So it's 
a complex problem I think and not every machine can be backuped as another.

I think that I must start by the hardware and software configuration of 
the backup node.

Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Am 12.10.2012 13:36, schrieb Stefan Hajnoczi:
> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi:
>>
>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>>>>
>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me>
>>>>> wrote:
>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>>>>
>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests and
>>>>>>>>> wan't to backup them to files. The simple and nasty way would be to
>>>>>>>>> create a complete output file with dd, which wastes very much space.
>>>>>>>>> So I would like to create a backup of the LVM to a file which only
>>>>>>>>> locates the space which is used on the LVM. Would be create when the
>>>>>>>>> output file would be something like a qcow2 file which could be also
>>>>>>>>> simply startet with KVM.
>>>>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>>>>
>>>>>>>>       qemu-img convert -f raw -O qcow2 /dev/vg/vm001
>>>>>>>> vm001-backup.qcow2
>>>>>>>>
>>>>>>>> Note that cp(1) tries to make the destination file sparse (see the
>>>>>>>> --sparse option in the man page).  So you don't need to use qcow2,
>>>>>>>> you
>>>>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not use
>>>>>>>> disk space for zero regions.
>>>>>>>>
>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>> Hello Stefano,
>>>>>>>
>>>>>>> thanks for the fast reply. I will test this later. In my case now it
>>>>>>> would
>>>>>>> be a offline backup. For the online backup I think about a seperated
>>>>>>> system
>>>>>>> which every day makes incremental backups and once a week a full
>>>>>>> backup.
>>>>>>> The
>>>>>>> main problem is, that the systems are in a WAN network and I need
>>>>>>> encryption
>>>>>>> between the systems. Would it be possible to do something like this:
>>>>>>> create
>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the
>>>>>>> remote
>>>>>>> backup system via ssh tunnel and save the output of this to qcow2
>>>>>>> files
>>>>>>> on
>>>>>>> the backup system? And in which format could be the incremental
>>>>>>> backups
>>>>>>> be
>>>>>>> stored?
>>>>> Since there is a WAN link it's important to use a compact image
>>>>> representation before hitting the network. I would use qemu-img
>>>>> convert -O qcow2 on the host and only transfer the qcow2 output.  The
>>>>> qcow2 file does not contain zero regions and will therefore save a lot
>>>>> of network bandwidth compared to accessing the LVM volume over the
>>>>> WAN.
>>>>>
>>>>> If you are using rsync or another tool it's a different story.  You
>>>>> could rsync the current LVM volume on the host over the last full
>>>>> backup, it should avoid transferring image data which is already
>>>>> present in the last full backup - the result is that you only transfer
>>>>> changed data plus the rsync metadata.
>>>>>
>>>>> Stefan
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>> Hello Stefan,
>>>>
>>>> the rsync part I don't have understood fully. So to create a qcow2 on the
>>>> host, transfer this to the backup server will result in the weekly full
>>>> backup. So do you mean I could use rsync to read the LVM from the host,
>>>> compare the LVM data with the data in the qcow2 on the backup server and
>>>> simply transfer the differences to the file? Or does it work on another
>>>> way?
>>> When using rsync you can skip qcow2.  Only two objects are needed:
>>> 1. The LVM volume on the host.
>>> 2. The last full backup on the backup client.
>>>
>>> rsync compares #1 and #2 efficiently over the network and only
>>> transfers data from #1 which has changed.
>>>
>>> After rsync completes your full backup image is identical to the LVM
>>> volume.  Next week you can use it as the "last" image to rsync
>>> against.
>>>
>>> Stefan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> So I simply update the full backup, which is simply a raw file which get
>> mounted while the backup?
> The image file does not need to be mounted.  Just rsync the raw image file.
>
> Stefan

I tested the qemu-img command now, but it does not do that what I want. 
I have a VM with a 5GB disk, this disk is not allocated with 1GB of 
data. When I do the convert command the output is a 5GB qcow2 disk. What 
do I have to do to get a qcow2 file with only the allocated space/data 
from the LVM? I also tried the -c option of qemu-img convert but the 
result was nearly the same.

Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Stefan Hajnoczi]

On Fri, Oct 12, 2012 at 8:14 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi:
>
>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>
>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi:
>>>
>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me>
>>>> wrote:
>>>>>
>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>>>>>
>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me>
>>>>>> wrote:
>>>>>>>
>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>>>>>
>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>>>>>
>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>>>>>
>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests
>>>>>>>>>> and
>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be
>>>>>>>>>> to
>>>>>>>>>> create a complete output file with dd, which wastes very much
>>>>>>>>>> space.
>>>>>>>>>> So I would like to create a backup of the LVM to a file which only
>>>>>>>>>> locates the space which is used on the LVM. Would be create when
>>>>>>>>>> the
>>>>>>>>>> output file would be something like a qcow2 file which could be
>>>>>>>>>> also
>>>>>>>>>> simply startet with KVM.
>>>>>>>>>
>>>>>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>>>>>
>>>>>>>>>       qemu-img convert -f raw -O qcow2 /dev/vg/vm001
>>>>>>>>> vm001-backup.qcow2
>>>>>>>>>
>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see the
>>>>>>>>> --sparse option in the man page).  So you don't need to use qcow2,
>>>>>>>>> you
>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not
>>>>>>>>> use
>>>>>>>>> disk space for zero regions.
>>>>>>>>>
>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>>>>>
>>>>>>>> Hello Stefano,
>>>>>>>>
>>>>>>>> thanks for the fast reply. I will test this later. In my case now it
>>>>>>>> would
>>>>>>>> be a offline backup. For the online backup I think about a seperated
>>>>>>>> system
>>>>>>>> which every day makes incremental backups and once a week a full
>>>>>>>> backup.
>>>>>>>> The
>>>>>>>> main problem is, that the systems are in a WAN network and I need
>>>>>>>> encryption
>>>>>>>> between the systems. Would it be possible to do something like this:
>>>>>>>> create
>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the
>>>>>>>> remote
>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2
>>>>>>>> files
>>>>>>>> on
>>>>>>>> the backup system? And in which format could be the incremental
>>>>>>>> backups
>>>>>>>> be
>>>>>>>> stored?
>>>>>>
>>>>>> Since there is a WAN link it's important to use a compact image
>>>>>> representation before hitting the network. I would use qemu-img
>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output.  The
>>>>>> qcow2 file does not contain zero regions and will therefore save a lot
>>>>>> of network bandwidth compared to accessing the LVM volume over the
>>>>>> WAN.
>>>>>>
>>>>>> If you are using rsync or another tool it's a different story.  You
>>>>>> could rsync the current LVM volume on the host over the last full
>>>>>> backup, it should avoid transferring image data which is already
>>>>>> present in the last full backup - the result is that you only transfer
>>>>>> changed data plus the rsync metadata.
>>>>>>
>>>>>> Stefan
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>>> Hello Stefan,
>>>>>
>>>>> the rsync part I don't have understood fully. So to create a qcow2 on
>>>>> the
>>>>> host, transfer this to the backup server will result in the weekly full
>>>>> backup. So do you mean I could use rsync to read the LVM from the host,
>>>>> compare the LVM data with the data in the qcow2 on the backup server
>>>>> and
>>>>> simply transfer the differences to the file? Or does it work on another
>>>>> way?
>>>>
>>>> When using rsync you can skip qcow2.  Only two objects are needed:
>>>> 1. The LVM volume on the host.
>>>> 2. The last full backup on the backup client.
>>>>
>>>> rsync compares #1 and #2 efficiently over the network and only
>>>> transfers data from #1 which has changed.
>>>>
>>>> After rsync completes your full backup image is identical to the LVM
>>>> volume.  Next week you can use it as the "last" image to rsync
>>>> against.
>>>>
>>>> Stefan
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>> So I simply update the full backup, which is simply a raw file which get
>>> mounted while the backup?
>>
>> The image file does not need to be mounted.  Just rsync the raw image
>> file.
>>
>> Stefan
>
>
> I tested the qemu-img command now, but it does not do that what I want. I
> have a VM with a 5GB disk, this disk is not allocated with 1GB of data. When
> I do the convert command the output is a 5GB qcow2 disk. What do I have to
> do to get a qcow2 file with only the allocated space/data from the LVM? I
> also tried the -c option of qemu-img convert but the result was nearly the
> same.

Please show the exact command-lines you are using and the qemu-img
info <filename> output afterwards.

Stefan

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Stefan Hajnoczi]

On Fri, Oct 12, 2012 at 4:36 PM, Javier Guerra Giraldez
<javier@guerrag.com> wrote:
> On Fri, Oct 12, 2012 at 9:25 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> I would leave them raw as long as they are sparse (zero regions do not
>> take up space).  If you need to copy them you can either convert to
>> qcow2 or use tools that preserve sparseness (BTW compression tools are
>> good at this).
>
> note that free blocks previously used by deleted files won't be
> sparse, won't be zero and won't be much reduced by compression.
>
> i'd say the usual advice stays:
>
> A: if you run any non-trivial application on that VM, then use a real
> network backup tool 'from the inside' of the VM
> B: if real point-in-time application-cache-storage consistency is not
> important, then you can:
>
> - make a read-write LVM snapshot
> - mount that and fsck.  (it will appear as not-cleanly unmounted)
> - backup the files. (i like rsync, especially if you have an
> uncompressed previous backup)
> - umount and destroy the snapshot
> - optionally compress the backup
>
>
> but seriously consider option A before.  especially important if you
> run any DB on that VM

Valid points.  People seem to like crash-consistent image file backup
though because it's convenient.  It may not be backing up all
application state though :(.

Regarding option B, your idea is better than what I've been
suggesting.  By mounting the image on the host and rsyncing the
mounted files you avoid syncing dirty blocks in the image file (e.g.
deleted data in the guest file system).

Stefan

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Am 12.10.2012 21:13, schrieb Stefan Hajnoczi:
> On Fri, Oct 12, 2012 at 8:14 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi:
>>
>>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi:
>>>>
>>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me>
>>>>> wrote:
>>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>>>>>>
>>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me>
>>>>>>> wrote:
>>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>>>>>>
>>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests
>>>>>>>>>>> and
>>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be
>>>>>>>>>>> to
>>>>>>>>>>> create a complete output file with dd, which wastes very much
>>>>>>>>>>> space.
>>>>>>>>>>> So I would like to create a backup of the LVM to a file which only
>>>>>>>>>>> locates the space which is used on the LVM. Would be create when
>>>>>>>>>>> the
>>>>>>>>>>> output file would be something like a qcow2 file which could be
>>>>>>>>>>> also
>>>>>>>>>>> simply startet with KVM.
>>>>>>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>>>>>>
>>>>>>>>>>        qemu-img convert -f raw -O qcow2 /dev/vg/vm001
>>>>>>>>>> vm001-backup.qcow2
>>>>>>>>>>
>>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see the
>>>>>>>>>> --sparse option in the man page).  So you don't need to use qcow2,
>>>>>>>>>> you
>>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not
>>>>>>>>>> use
>>>>>>>>>> disk space for zero regions.
>>>>>>>>>>
>>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>>>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>> --
>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>>
>>>>>>>>> Hello Stefano,
>>>>>>>>>
>>>>>>>>> thanks for the fast reply. I will test this later. In my case now it
>>>>>>>>> would
>>>>>>>>> be a offline backup. For the online backup I think about a seperated
>>>>>>>>> system
>>>>>>>>> which every day makes incremental backups and once a week a full
>>>>>>>>> backup.
>>>>>>>>> The
>>>>>>>>> main problem is, that the systems are in a WAN network and I need
>>>>>>>>> encryption
>>>>>>>>> between the systems. Would it be possible to do something like this:
>>>>>>>>> create
>>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the
>>>>>>>>> remote
>>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2
>>>>>>>>> files
>>>>>>>>> on
>>>>>>>>> the backup system? And in which format could be the incremental
>>>>>>>>> backups
>>>>>>>>> be
>>>>>>>>> stored?
>>>>>>> Since there is a WAN link it's important to use a compact image
>>>>>>> representation before hitting the network. I would use qemu-img
>>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output.  The
>>>>>>> qcow2 file does not contain zero regions and will therefore save a lot
>>>>>>> of network bandwidth compared to accessing the LVM volume over the
>>>>>>> WAN.
>>>>>>>
>>>>>>> If you are using rsync or another tool it's a different story.  You
>>>>>>> could rsync the current LVM volume on the host over the last full
>>>>>>> backup, it should avoid transferring image data which is already
>>>>>>> present in the last full backup - the result is that you only transfer
>>>>>>> changed data plus the rsync metadata.
>>>>>>>
>>>>>>> Stefan
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>> Hello Stefan,
>>>>>>
>>>>>> the rsync part I don't have understood fully. So to create a qcow2 on
>>>>>> the
>>>>>> host, transfer this to the backup server will result in the weekly full
>>>>>> backup. So do you mean I could use rsync to read the LVM from the host,
>>>>>> compare the LVM data with the data in the qcow2 on the backup server
>>>>>> and
>>>>>> simply transfer the differences to the file? Or does it work on another
>>>>>> way?
>>>>> When using rsync you can skip qcow2.  Only two objects are needed:
>>>>> 1. The LVM volume on the host.
>>>>> 2. The last full backup on the backup client.
>>>>>
>>>>> rsync compares #1 and #2 efficiently over the network and only
>>>>> transfers data from #1 which has changed.
>>>>>
>>>>> After rsync completes your full backup image is identical to the LVM
>>>>> volume.  Next week you can use it as the "last" image to rsync
>>>>> against.
>>>>>
>>>>> Stefan
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>> So I simply update the full backup, which is simply a raw file which get
>>>> mounted while the backup?
>>> The image file does not need to be mounted.  Just rsync the raw image
>>> file.
>>>
>>> Stefan
>>
>> I tested the qemu-img command now, but it does not do that what I want. I
>> have a VM with a 5GB disk, this disk is not allocated with 1GB of data. When
>> I do the convert command the output is a 5GB qcow2 disk. What do I have to
>> do to get a qcow2 file with only the allocated space/data from the LVM? I
>> also tried the -c option of qemu-img convert but the result was nearly the
>> same.
> Please show the exact command-lines you are using and the qemu-img
> info <filename> output afterwards.
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hello,

the VM is called mailer and I used this command: qemu-img convert -f raw 
-c -O qcow2 /dev/vmdisks/mailer ./vmachines/mailer.qcow2

Im on the /mnt dir because it's a seperate HDD with the folder vmachines.

Thats the output of qemu-img info:

image: ./vmachines/mailer.qcow2
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 4.1G
cluster_size: 65536

I checked the info of the fs inside the guest. The / partition is 4,7GB 
big and 1,2GB are used. The fs is ext3.

Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Stefan Hajnoczi]

On Fri, Oct 12, 2012 at 9:26 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
> Am 12.10.2012 21:13, schrieb Stefan Hajnoczi:
>
>> On Fri, Oct 12, 2012 at 8:14 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>
>>> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi:
>>>
>>>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me>
>>>> wrote:
>>>>>
>>>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi:
>>>>>
>>>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me>
>>>>>> wrote:
>>>>>>>
>>>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>>>>>>>
>>>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>>>>>>>
>>>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests
>>>>>>>>>>>> and
>>>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be
>>>>>>>>>>>> to
>>>>>>>>>>>> create a complete output file with dd, which wastes very much
>>>>>>>>>>>> space.
>>>>>>>>>>>> So I would like to create a backup of the LVM to a file which
>>>>>>>>>>>> only
>>>>>>>>>>>> locates the space which is used on the LVM. Would be create when
>>>>>>>>>>>> the
>>>>>>>>>>>> output file would be something like a qcow2 file which could be
>>>>>>>>>>>> also
>>>>>>>>>>>> simply startet with KVM.
>>>>>>>>>>>
>>>>>>>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>>>>>>>
>>>>>>>>>>>        qemu-img convert -f raw -O qcow2 /dev/vg/vm001
>>>>>>>>>>> vm001-backup.qcow2
>>>>>>>>>>>
>>>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see
>>>>>>>>>>> the
>>>>>>>>>>> --sparse option in the man page).  So you don't need to use
>>>>>>>>>>> qcow2,
>>>>>>>>>>> you
>>>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not
>>>>>>>>>>> use
>>>>>>>>>>> disk space for zero regions.
>>>>>>>>>>>
>>>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>>>>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>>>>>>>
>>>>>>>>>>> Stefan
>>>>>>>>>>> --
>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hello Stefano,
>>>>>>>>>>
>>>>>>>>>> thanks for the fast reply. I will test this later. In my case now
>>>>>>>>>> it
>>>>>>>>>> would
>>>>>>>>>> be a offline backup. For the online backup I think about a
>>>>>>>>>> seperated
>>>>>>>>>> system
>>>>>>>>>> which every day makes incremental backups and once a week a full
>>>>>>>>>> backup.
>>>>>>>>>> The
>>>>>>>>>> main problem is, that the systems are in a WAN network and I need
>>>>>>>>>> encryption
>>>>>>>>>> between the systems. Would it be possible to do something like
>>>>>>>>>> this:
>>>>>>>>>> create
>>>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the
>>>>>>>>>> remote
>>>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2
>>>>>>>>>> files
>>>>>>>>>> on
>>>>>>>>>> the backup system? And in which format could be the incremental
>>>>>>>>>> backups
>>>>>>>>>> be
>>>>>>>>>> stored?
>>>>>>>>
>>>>>>>> Since there is a WAN link it's important to use a compact image
>>>>>>>> representation before hitting the network. I would use qemu-img
>>>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output.
>>>>>>>> The
>>>>>>>> qcow2 file does not contain zero regions and will therefore save a
>>>>>>>> lot
>>>>>>>> of network bandwidth compared to accessing the LVM volume over the
>>>>>>>> WAN.
>>>>>>>>
>>>>>>>> If you are using rsync or another tool it's a different story.  You
>>>>>>>> could rsync the current LVM volume on the host over the last full
>>>>>>>> backup, it should avoid transferring image data which is already
>>>>>>>> present in the last full backup - the result is that you only
>>>>>>>> transfer
>>>>>>>> changed data plus the rsync metadata.
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>>
>>>>>>> Hello Stefan,
>>>>>>>
>>>>>>> the rsync part I don't have understood fully. So to create a qcow2 on
>>>>>>> the
>>>>>>> host, transfer this to the backup server will result in the weekly
>>>>>>> full
>>>>>>> backup. So do you mean I could use rsync to read the LVM from the
>>>>>>> host,
>>>>>>> compare the LVM data with the data in the qcow2 on the backup server
>>>>>>> and
>>>>>>> simply transfer the differences to the file? Or does it work on
>>>>>>> another
>>>>>>> way?
>>>>>>
>>>>>> When using rsync you can skip qcow2.  Only two objects are needed:
>>>>>> 1. The LVM volume on the host.
>>>>>> 2. The last full backup on the backup client.
>>>>>>
>>>>>> rsync compares #1 and #2 efficiently over the network and only
>>>>>> transfers data from #1 which has changed.
>>>>>>
>>>>>> After rsync completes your full backup image is identical to the LVM
>>>>>> volume.  Next week you can use it as the "last" image to rsync
>>>>>> against.
>>>>>>
>>>>>> Stefan
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>>> So I simply update the full backup, which is simply a raw file which
>>>>> get
>>>>> mounted while the backup?
>>>>
>>>> The image file does not need to be mounted.  Just rsync the raw image
>>>> file.
>>>>
>>>> Stefan
>>>
>>>
>>> I tested the qemu-img command now, but it does not do that what I want. I
>>> have a VM with a 5GB disk, this disk is not allocated with 1GB of data.
>>> When
>>> I do the convert command the output is a 5GB qcow2 disk. What do I have
>>> to
>>> do to get a qcow2 file with only the allocated space/data from the LVM? I
>>> also tried the -c option of qemu-img convert but the result was nearly
>>> the
>>> same.
>>
>> Please show the exact command-lines you are using and the qemu-img
>> info <filename> output afterwards.
>>
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
> Hello,
>
> the VM is called mailer and I used this command: qemu-img convert -f raw -c
> -O qcow2 /dev/vmdisks/mailer ./vmachines/mailer.qcow2
>
> Im on the /mnt dir because it's a seperate HDD with the folder vmachines.
>
> Thats the output of qemu-img info:
>
> image: ./vmachines/mailer.qcow2
> file format: qcow2
> virtual size: 5.0G (5368709120 bytes)
> disk size: 4.1G
> cluster_size: 65536
>
> I checked the info of the fs inside the guest. The / partition is 4,7GB big
> and 1,2GB are used. The fs is ext3.

I wonder if you see better results without the -c option.

Stefan

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Am 12.10.2012 22:43, schrieb Stefan Hajnoczi:
> On Fri, Oct 12, 2012 at 9:26 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>> Am 12.10.2012 21:13, schrieb Stefan Hajnoczi:
>>
>>> On Fri, Oct 12, 2012 at 8:14 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>>> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi:
>>>>
>>>>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me>
>>>>> wrote:
>>>>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi:
>>>>>>
>>>>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me>
>>>>>>> wrote:
>>>>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi:
>>>>>>>>
>>>>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me>
>>>>>>>>> wrote:
>>>>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp:
>>>>>>>>>>
>>>>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi:
>>>>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote:
>>>>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests
>>>>>>>>>>>>> and
>>>>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be
>>>>>>>>>>>>> to
>>>>>>>>>>>>> create a complete output file with dd, which wastes very much
>>>>>>>>>>>>> space.
>>>>>>>>>>>>> So I would like to create a backup of the LVM to a file which
>>>>>>>>>>>>> only
>>>>>>>>>>>>> locates the space which is used on the LVM. Would be create when
>>>>>>>>>>>>> the
>>>>>>>>>>>>> output file would be something like a qcow2 file which could be
>>>>>>>>>>>>> also
>>>>>>>>>>>>> simply startet with KVM.
>>>>>>>>>>>> If the VM is not running you can use "qemu-img convert":
>>>>>>>>>>>>
>>>>>>>>>>>>         qemu-img convert -f raw -O qcow2 /dev/vg/vm001
>>>>>>>>>>>> vm001-backup.qcow2
>>>>>>>>>>>>
>>>>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see
>>>>>>>>>>>> the
>>>>>>>>>>>> --sparse option in the man page).  So you don't need to use
>>>>>>>>>>>> qcow2,
>>>>>>>>>>>> you
>>>>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file.  It will not
>>>>>>>>>>>> use
>>>>>>>>>>>> disk space for zero regions.
>>>>>>>>>>>>
>>>>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM
>>>>>>>>>>>> temporarily so a crash-consistent backup can be taken.
>>>>>>>>>>>>
>>>>>>>>>>>> Stefan
>>>>>>>>>>>> --
>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>>>>
>>>>>>>>>>> Hello Stefano,
>>>>>>>>>>>
>>>>>>>>>>> thanks for the fast reply. I will test this later. In my case now
>>>>>>>>>>> it
>>>>>>>>>>> would
>>>>>>>>>>> be a offline backup. For the online backup I think about a
>>>>>>>>>>> seperated
>>>>>>>>>>> system
>>>>>>>>>>> which every day makes incremental backups and once a week a full
>>>>>>>>>>> backup.
>>>>>>>>>>> The
>>>>>>>>>>> main problem is, that the systems are in a WAN network and I need
>>>>>>>>>>> encryption
>>>>>>>>>>> between the systems. Would it be possible to do something like
>>>>>>>>>>> this:
>>>>>>>>>>> create
>>>>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the
>>>>>>>>>>> remote
>>>>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2
>>>>>>>>>>> files
>>>>>>>>>>> on
>>>>>>>>>>> the backup system? And in which format could be the incremental
>>>>>>>>>>> backups
>>>>>>>>>>> be
>>>>>>>>>>> stored?
>>>>>>>>> Since there is a WAN link it's important to use a compact image
>>>>>>>>> representation before hitting the network. I would use qemu-img
>>>>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output.
>>>>>>>>> The
>>>>>>>>> qcow2 file does not contain zero regions and will therefore save a
>>>>>>>>> lot
>>>>>>>>> of network bandwidth compared to accessing the LVM volume over the
>>>>>>>>> WAN.
>>>>>>>>>
>>>>>>>>> If you are using rsync or another tool it's a different story.  You
>>>>>>>>> could rsync the current LVM volume on the host over the last full
>>>>>>>>> backup, it should avoid transferring image data which is already
>>>>>>>>> present in the last full backup - the result is that you only
>>>>>>>>> transfer
>>>>>>>>> changed data plus the rsync metadata.
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>>>>> Hello Stefan,
>>>>>>>>
>>>>>>>> the rsync part I don't have understood fully. So to create a qcow2 on
>>>>>>>> the
>>>>>>>> host, transfer this to the backup server will result in the weekly
>>>>>>>> full
>>>>>>>> backup. So do you mean I could use rsync to read the LVM from the
>>>>>>>> host,
>>>>>>>> compare the LVM data with the data in the qcow2 on the backup server
>>>>>>>> and
>>>>>>>> simply transfer the differences to the file? Or does it work on
>>>>>>>> another
>>>>>>>> way?
>>>>>>> When using rsync you can skip qcow2.  Only two objects are needed:
>>>>>>> 1. The LVM volume on the host.
>>>>>>> 2. The last full backup on the backup client.
>>>>>>>
>>>>>>> rsync compares #1 and #2 efficiently over the network and only
>>>>>>> transfers data from #1 which has changed.
>>>>>>>
>>>>>>> After rsync completes your full backup image is identical to the LVM
>>>>>>> volume.  Next week you can use it as the "last" image to rsync
>>>>>>> against.
>>>>>>>
>>>>>>> Stefan
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>> So I simply update the full backup, which is simply a raw file which
>>>>>> get
>>>>>> mounted while the backup?
>>>>> The image file does not need to be mounted.  Just rsync the raw image
>>>>> file.
>>>>>
>>>>> Stefan
>>>>
>>>> I tested the qemu-img command now, but it does not do that what I want. I
>>>> have a VM with a 5GB disk, this disk is not allocated with 1GB of data.
>>>> When
>>>> I do the convert command the output is a 5GB qcow2 disk. What do I have
>>>> to
>>>> do to get a qcow2 file with only the allocated space/data from the LVM? I
>>>> also tried the -c option of qemu-img convert but the result was nearly
>>>> the
>>>> same.
>>> Please show the exact command-lines you are using and the qemu-img
>>> info <filename> output afterwards.
>>>
>>> Stefan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> Hello,
>>
>> the VM is called mailer and I used this command: qemu-img convert -f raw -c
>> -O qcow2 /dev/vmdisks/mailer ./vmachines/mailer.qcow2
>>
>> Im on the /mnt dir because it's a seperate HDD with the folder vmachines.
>>
>> Thats the output of qemu-img info:
>>
>> image: ./vmachines/mailer.qcow2
>> file format: qcow2
>> virtual size: 5.0G (5368709120 bytes)
>> disk size: 4.1G
>> cluster_size: 65536
>>
>> I checked the info of the fs inside the guest. The / partition is 4,7GB big
>> and 1,2GB are used. The fs is ext3.
> I wonder if you see better results without the -c option.
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

So I would say that the results are very similar. So with -c option the 
file is 650-800MB smaller but thats not very much. I have a few bigger 
VM disks with sizes like 100GB when we calculat that in % so we see that 
the compression is very ecactly 20% so I would have an 80GB image of a 
disk where only 3GBs are used inside the VM.

I think that it must be possible to create an image with a size like the 
used space + a few hundret MB with metadata or something like that.

Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Javier Guerra Giraldez]

On Fri, Oct 12, 2012 at 3:56 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
> I think that it must be possible to create an image with a size like the
> used space + a few hundret MB with metadata or something like that.

the 'best' way to do it is 'from within' the VM

the typical workaround is to mount/fsck a LVM snapshot (don't skip the
fsck, or at least a journal replay)

beyond that, there are a few utilities for specific filesystems:
PartImage [1], dump/restore [2], and i'm sure some others.  I don't
know how would these behave with unclean images, which is what you get
if you pull the image under the VM's feet.


[1] http://www.partimage.org/Main_Page
[2] http://dump.sourceforge.net/


-- 
Javier

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Am 12.10.2012 23:53, schrieb Javier Guerra Giraldez:
> On Fri, Oct 12, 2012 at 3:56 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>> I think that it must be possible to create an image with a size like the
>> used space + a few hundret MB with metadata or something like that.
> the 'best' way to do it is 'from within' the VM
>
> the typical workaround is to mount/fsck a LVM snapshot (don't skip the
> fsck, or at least a journal replay)
>
> beyond that, there are a few utilities for specific filesystems:
> PartImage [1], dump/restore [2], and i'm sure some others.  I don't
> know how would these behave with unclean images, which is what you get
> if you pull the image under the VM's feet.
>
>
> [1] http://www.partimage.org/Main_Page
> [2] http://dump.sourceforge.net/
>
>

So when I boot the VM with for example a live debian system, what would 
be the best method to backup the complete disk /dev/vda with only the 
data which is on it over a secure network connection to a destination in 
the network.

Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Am 13.10.2012 12:28, schrieb Lukas Laukamp:
> Am 12.10.2012 23:53, schrieb Javier Guerra Giraldez:
>> On Fri, Oct 12, 2012 at 3:56 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>>> I think that it must be possible to create an image with a size like 
>>> the
>>> used space + a few hundret MB with metadata or something like that.
>> the 'best' way to do it is 'from within' the VM
>>
>> the typical workaround is to mount/fsck a LVM snapshot (don't skip the
>> fsck, or at least a journal replay)
>>
>> beyond that, there are a few utilities for specific filesystems:
>> PartImage [1], dump/restore [2], and i'm sure some others.  I don't
>> know how would these behave with unclean images, which is what you get
>> if you pull the image under the VM's feet.
>>
>>
>> [1] http://www.partimage.org/Main_Page
>> [2] http://dump.sourceforge.net/
>>
>>
>
> So when I boot the VM with for example a live debian system, what 
> would be the best method to backup the complete disk /dev/vda with 
> only the data which is on it over a secure network connection to a 
> destination in the network.
>
> Best Regards
> -- 
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

I have backed up the data within the machine with partimage and 
fsarchiver. But it would be greate to have a better way than doing this 
over a live system.

Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Javier Guerra Giraldez]

On Sat, Oct 13, 2012 at 5:25 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
> I have backed up the data within the machine with partimage and fsarchiver.
> But it would be greate to have a better way than doing this over a live
> system.

make no mistake, the absolutely best way is from within the VM.  It's
the most consistent, safe and efficient method.

Doing it "from the outside" is attractive, but it's a hack, and in
some cases you have to jump through several hoops to make it safe.


-- 
Javier

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Lukas Laukamp]

Am 15.10.2012 05:06, schrieb Javier Guerra Giraldez:
> On Sat, Oct 13, 2012 at 5:25 PM, Lukas Laukamp <lukas@laukamp.me> wrote:
>> I have backed up the data within the machine with partimage and fsarchiver.
>> But it would be greate to have a better way than doing this over a live
>> system.
> make no mistake, the absolutely best way is from within the VM.  It's
> the most consistent, safe and efficient method.
>
> Doing it "from the outside" is attractive, but it's a hack, and in
> some cases you have to jump through several hoops to make it safe.
>
>

Because of the fact that this is a "problem" I bootet the VM with a live 
cd and made a backup of the important filesystems with the live system 
which runs inside the VM.

Best Regards

Re: [User Question] How to create a backup of an LVM based maschine without wasting space

[Avi Kivity]

On 10/12/2012 10:43 PM, Stefan Hajnoczi wrote:
>>
>> the VM is called mailer and I used this command: qemu-img convert -f raw -c
>> -O qcow2 /dev/vmdisks/mailer ./vmachines/mailer.qcow2
>>
>> Im on the /mnt dir because it's a seperate HDD with the folder vmachines.
>>
>> Thats the output of qemu-img info:
>>
>> image: ./vmachines/mailer.qcow2
>> file format: qcow2
>> virtual size: 5.0G (5368709120 bytes)
>> disk size: 4.1G
>> cluster_size: 65536
>>
>> I checked the info of the fs inside the guest. The / partition is 4,7GB big
>> and 1,2GB are used. The fs is ext3.
> 
> I wonder if you see better results without the -c option.

Since the image is on a logical volume, it may not have been cleared
prior to use by the guest; or the guest is busy enough that it has
overwritten all the blocks in the image.

You can try a manual trim by creating a large file in the guest
consisting of all zeros, then deleting it:

  dd < /dev/zero < /zeroes
  rm zeroes

Afterwards you'll get a much smaller qcow image.  Of course this
requires working within the guest, and there is the risk of disrupting
the guest by reaching the ENOSPC condition.

-- 
error compiling committee.c: too many arguments to function