* [User Question] How to create a backup of an LVM based maschine without wasting space @ 2012-10-12 6:52 Lukas Laukamp 2012-10-12 8:42 ` Stefan Hajnoczi 0 siblings, 1 reply; 26+ messages in thread From: Lukas Laukamp @ 2012-10-12 6:52 UTC (permalink / raw) To: kvm Hey all, I have a simple user question. I have a few LVM based KVM guests and wan't to backup them to files. The simple and nasty way would be to create a complete output file with dd, which wastes very much space. So I would like to create a backup of the LVM to a file which only locates the space which is used on the LVM. Would be create when the output file would be something like a qcow2 file which could be also simply startet with KVM. Is this supported by qemu_backup or what would be the best way to do this? PS: Sorry for my bed english Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 6:52 [User Question] How to create a backup of an LVM based maschine without wasting space Lukas Laukamp @ 2012-10-12 8:42 ` Stefan Hajnoczi [not found] ` <5077DB9F.6050809@laukamp.me> 0 siblings, 1 reply; 26+ messages in thread From: Stefan Hajnoczi @ 2012-10-12 8:42 UTC (permalink / raw) To: Lukas Laukamp; +Cc: kvm On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: > I have a simple user question. I have a few LVM based KVM guests and > wan't to backup them to files. The simple and nasty way would be to > create a complete output file with dd, which wastes very much space. > So I would like to create a backup of the LVM to a file which only > locates the space which is used on the LVM. Would be create when the > output file would be something like a qcow2 file which could be also > simply startet with KVM. If the VM is not running you can use "qemu-img convert": qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 Note that cp(1) tries to make the destination file sparse (see the --sparse option in the man page). So you don't need to use qcow2, you can use cp(1) to copy the LVM volume to a raw file. It will not use disk space for zero regions. If the VM is running you need to use LVM snapshots or stop the VM temporarily so a crash-consistent backup can be taken. Stefan ^ permalink raw reply [flat|nested] 26+ messages in thread
[parent not found: <5077DB9F.6050809@laukamp.me>]
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space [not found] ` <5077DB9F.6050809@laukamp.me> @ 2012-10-12 9:17 ` Lukas Laukamp 2012-10-12 10:11 ` Stefan Hajnoczi 0 siblings, 1 reply; 26+ messages in thread From: Lukas Laukamp @ 2012-10-12 9:17 UTC (permalink / raw) To: kvm Am 12.10.2012 10:58, schrieb Lukas Laukamp: > Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>> I have a simple user question. I have a few LVM based KVM guests and >>> wan't to backup them to files. The simple and nasty way would be to >>> create a complete output file with dd, which wastes very much space. >>> So I would like to create a backup of the LVM to a file which only >>> locates the space which is used on the LVM. Would be create when the >>> output file would be something like a qcow2 file which could be also >>> simply startet with KVM. >> If the VM is not running you can use "qemu-img convert": >> >> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 >> >> Note that cp(1) tries to make the destination file sparse (see the >> --sparse option in the man page). So you don't need to use qcow2, you >> can use cp(1) to copy the LVM volume to a raw file. It will not use >> disk space for zero regions. >> >> If the VM is running you need to use LVM snapshots or stop the VM >> temporarily so a crash-consistent backup can be taken. >> >> Stefan >> -- >> To unsubscribe from this list: send the line "unsubscribe kvm" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > Hello Stefano, > > thanks for the fast reply. I will test this later. In my case now it > would be a offline backup. For the online backup I think about a > seperated system which every day makes incremental backups and once a > week a full backup. The main problem is, that the systems are in a WAN > network and I need encryption between the systems. Would it be > possible to do something like this: create the LVM snapshot for the > backup, read this LVM snapshot with the remote backup system via ssh > tunnel and save the output of this to qcow2 files on the backup > system? And in which format could be the incremental backups be stored? > > Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 9:17 ` Lukas Laukamp @ 2012-10-12 10:11 ` Stefan Hajnoczi 2012-10-12 10:16 ` Lukas Laukamp 0 siblings, 1 reply; 26+ messages in thread From: Stefan Hajnoczi @ 2012-10-12 10:11 UTC (permalink / raw) To: Lukas Laukamp; +Cc: kvm On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> wrote: > Am 12.10.2012 10:58, schrieb Lukas Laukamp: > >> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>> >>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>> >>>> I have a simple user question. I have a few LVM based KVM guests and >>>> wan't to backup them to files. The simple and nasty way would be to >>>> create a complete output file with dd, which wastes very much space. >>>> So I would like to create a backup of the LVM to a file which only >>>> locates the space which is used on the LVM. Would be create when the >>>> output file would be something like a qcow2 file which could be also >>>> simply startet with KVM. >>> >>> If the VM is not running you can use "qemu-img convert": >>> >>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 >>> >>> Note that cp(1) tries to make the destination file sparse (see the >>> --sparse option in the man page). So you don't need to use qcow2, you >>> can use cp(1) to copy the LVM volume to a raw file. It will not use >>> disk space for zero regions. >>> >>> If the VM is running you need to use LVM snapshots or stop the VM >>> temporarily so a crash-consistent backup can be taken. >>> >>> Stefan >>> -- >>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> Hello Stefano, >> >> thanks for the fast reply. I will test this later. In my case now it would >> be a offline backup. For the online backup I think about a seperated system >> which every day makes incremental backups and once a week a full backup. The >> main problem is, that the systems are in a WAN network and I need encryption >> between the systems. Would it be possible to do something like this: create >> the LVM snapshot for the backup, read this LVM snapshot with the remote >> backup system via ssh tunnel and save the output of this to qcow2 files on >> the backup system? And in which format could be the incremental backups be >> stored? Since there is a WAN link it's important to use a compact image representation before hitting the network. I would use qemu-img convert -O qcow2 on the host and only transfer the qcow2 output. The qcow2 file does not contain zero regions and will therefore save a lot of network bandwidth compared to accessing the LVM volume over the WAN. If you are using rsync or another tool it's a different story. You could rsync the current LVM volume on the host over the last full backup, it should avoid transferring image data which is already present in the last full backup - the result is that you only transfer changed data plus the rsync metadata. Stefan ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 10:11 ` Stefan Hajnoczi @ 2012-10-12 10:16 ` Lukas Laukamp 2012-10-12 10:47 ` Stefan Hajnoczi 0 siblings, 1 reply; 26+ messages in thread From: Lukas Laukamp @ 2012-10-12 10:16 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: kvm Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: > On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> wrote: >> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >> >>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>> I have a simple user question. I have a few LVM based KVM guests and >>>>> wan't to backup them to files. The simple and nasty way would be to >>>>> create a complete output file with dd, which wastes very much space. >>>>> So I would like to create a backup of the LVM to a file which only >>>>> locates the space which is used on the LVM. Would be create when the >>>>> output file would be something like a qcow2 file which could be also >>>>> simply startet with KVM. >>>> If the VM is not running you can use "qemu-img convert": >>>> >>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 >>>> >>>> Note that cp(1) tries to make the destination file sparse (see the >>>> --sparse option in the man page). So you don't need to use qcow2, you >>>> can use cp(1) to copy the LVM volume to a raw file. It will not use >>>> disk space for zero regions. >>>> >>>> If the VM is running you need to use LVM snapshots or stop the VM >>>> temporarily so a crash-consistent backup can be taken. >>>> >>>> Stefan >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> Hello Stefano, >>> >>> thanks for the fast reply. I will test this later. In my case now it would >>> be a offline backup. For the online backup I think about a seperated system >>> which every day makes incremental backups and once a week a full backup. The >>> main problem is, that the systems are in a WAN network and I need encryption >>> between the systems. Would it be possible to do something like this: create >>> the LVM snapshot for the backup, read this LVM snapshot with the remote >>> backup system via ssh tunnel and save the output of this to qcow2 files on >>> the backup system? And in which format could be the incremental backups be >>> stored? > Since there is a WAN link it's important to use a compact image > representation before hitting the network. I would use qemu-img > convert -O qcow2 on the host and only transfer the qcow2 output. The > qcow2 file does not contain zero regions and will therefore save a lot > of network bandwidth compared to accessing the LVM volume over the > WAN. > > If you are using rsync or another tool it's a different story. You > could rsync the current LVM volume on the host over the last full > backup, it should avoid transferring image data which is already > present in the last full backup - the result is that you only transfer > changed data plus the rsync metadata. > > Stefan > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html Hello Stefan, the rsync part I don't have understood fully. So to create a qcow2 on the host, transfer this to the backup server will result in the weekly full backup. So do you mean I could use rsync to read the LVM from the host, compare the LVM data with the data in the qcow2 on the backup server and simply transfer the differences to the file? Or does it work on another way? Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 10:16 ` Lukas Laukamp @ 2012-10-12 10:47 ` Stefan Hajnoczi 2012-10-12 10:50 ` Lukas Laukamp 0 siblings, 1 reply; 26+ messages in thread From: Stefan Hajnoczi @ 2012-10-12 10:47 UTC (permalink / raw) To: Lukas Laukamp; +Cc: kvm On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> wrote: > Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: > >> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> wrote: >>> >>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>> >>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>> >>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>> >>>>>> I have a simple user question. I have a few LVM based KVM guests and >>>>>> wan't to backup them to files. The simple and nasty way would be to >>>>>> create a complete output file with dd, which wastes very much space. >>>>>> So I would like to create a backup of the LVM to a file which only >>>>>> locates the space which is used on the LVM. Would be create when the >>>>>> output file would be something like a qcow2 file which could be also >>>>>> simply startet with KVM. >>>>> >>>>> If the VM is not running you can use "qemu-img convert": >>>>> >>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 >>>>> >>>>> Note that cp(1) tries to make the destination file sparse (see the >>>>> --sparse option in the man page). So you don't need to use qcow2, you >>>>> can use cp(1) to copy the LVM volume to a raw file. It will not use >>>>> disk space for zero regions. >>>>> >>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>> temporarily so a crash-consistent backup can be taken. >>>>> >>>>> Stefan >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> >>>> Hello Stefano, >>>> >>>> thanks for the fast reply. I will test this later. In my case now it >>>> would >>>> be a offline backup. For the online backup I think about a seperated >>>> system >>>> which every day makes incremental backups and once a week a full backup. >>>> The >>>> main problem is, that the systems are in a WAN network and I need >>>> encryption >>>> between the systems. Would it be possible to do something like this: >>>> create >>>> the LVM snapshot for the backup, read this LVM snapshot with the remote >>>> backup system via ssh tunnel and save the output of this to qcow2 files >>>> on >>>> the backup system? And in which format could be the incremental backups >>>> be >>>> stored? >> >> Since there is a WAN link it's important to use a compact image >> representation before hitting the network. I would use qemu-img >> convert -O qcow2 on the host and only transfer the qcow2 output. The >> qcow2 file does not contain zero regions and will therefore save a lot >> of network bandwidth compared to accessing the LVM volume over the >> WAN. >> >> If you are using rsync or another tool it's a different story. You >> could rsync the current LVM volume on the host over the last full >> backup, it should avoid transferring image data which is already >> present in the last full backup - the result is that you only transfer >> changed data plus the rsync metadata. >> >> Stefan >> -- >> To unsubscribe from this list: send the line "unsubscribe kvm" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > Hello Stefan, > > the rsync part I don't have understood fully. So to create a qcow2 on the > host, transfer this to the backup server will result in the weekly full > backup. So do you mean I could use rsync to read the LVM from the host, > compare the LVM data with the data in the qcow2 on the backup server and > simply transfer the differences to the file? Or does it work on another way? When using rsync you can skip qcow2. Only two objects are needed: 1. The LVM volume on the host. 2. The last full backup on the backup client. rsync compares #1 and #2 efficiently over the network and only transfers data from #1 which has changed. After rsync completes your full backup image is identical to the LVM volume. Next week you can use it as the "last" image to rsync against. Stefan ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 10:47 ` Stefan Hajnoczi @ 2012-10-12 10:50 ` Lukas Laukamp 2012-10-12 11:36 ` Stefan Hajnoczi 0 siblings, 1 reply; 26+ messages in thread From: Lukas Laukamp @ 2012-10-12 10:50 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: kvm Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: > On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >> >>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> wrote: >>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>> >>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>> I have a simple user question. I have a few LVM based KVM guests and >>>>>>> wan't to backup them to files. The simple and nasty way would be to >>>>>>> create a complete output file with dd, which wastes very much space. >>>>>>> So I would like to create a backup of the LVM to a file which only >>>>>>> locates the space which is used on the LVM. Would be create when the >>>>>>> output file would be something like a qcow2 file which could be also >>>>>>> simply startet with KVM. >>>>>> If the VM is not running you can use "qemu-img convert": >>>>>> >>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 vm001-backup.qcow2 >>>>>> >>>>>> Note that cp(1) tries to make the destination file sparse (see the >>>>>> --sparse option in the man page). So you don't need to use qcow2, you >>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not use >>>>>> disk space for zero regions. >>>>>> >>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>> temporarily so a crash-consistent backup can be taken. >>>>>> >>>>>> Stefan >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>> the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> Hello Stefano, >>>>> >>>>> thanks for the fast reply. I will test this later. In my case now it >>>>> would >>>>> be a offline backup. For the online backup I think about a seperated >>>>> system >>>>> which every day makes incremental backups and once a week a full backup. >>>>> The >>>>> main problem is, that the systems are in a WAN network and I need >>>>> encryption >>>>> between the systems. Would it be possible to do something like this: >>>>> create >>>>> the LVM snapshot for the backup, read this LVM snapshot with the remote >>>>> backup system via ssh tunnel and save the output of this to qcow2 files >>>>> on >>>>> the backup system? And in which format could be the incremental backups >>>>> be >>>>> stored? >>> Since there is a WAN link it's important to use a compact image >>> representation before hitting the network. I would use qemu-img >>> convert -O qcow2 on the host and only transfer the qcow2 output. The >>> qcow2 file does not contain zero regions and will therefore save a lot >>> of network bandwidth compared to accessing the LVM volume over the >>> WAN. >>> >>> If you are using rsync or another tool it's a different story. You >>> could rsync the current LVM volume on the host over the last full >>> backup, it should avoid transferring image data which is already >>> present in the last full backup - the result is that you only transfer >>> changed data plus the rsync metadata. >>> >>> Stefan >>> -- >>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> Hello Stefan, >> >> the rsync part I don't have understood fully. So to create a qcow2 on the >> host, transfer this to the backup server will result in the weekly full >> backup. So do you mean I could use rsync to read the LVM from the host, >> compare the LVM data with the data in the qcow2 on the backup server and >> simply transfer the differences to the file? Or does it work on another way? > When using rsync you can skip qcow2. Only two objects are needed: > 1. The LVM volume on the host. > 2. The last full backup on the backup client. > > rsync compares #1 and #2 efficiently over the network and only > transfers data from #1 which has changed. > > After rsync completes your full backup image is identical to the LVM > volume. Next week you can use it as the "last" image to rsync > against. > > Stefan > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html So I simply update the full backup, which is simply a raw file which get mounted while the backup? Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 10:50 ` Lukas Laukamp @ 2012-10-12 11:36 ` Stefan Hajnoczi 2012-10-12 11:51 ` Lukas Laukamp 2012-10-12 18:14 ` Lukas Laukamp 0 siblings, 2 replies; 26+ messages in thread From: Stefan Hajnoczi @ 2012-10-12 11:36 UTC (permalink / raw) To: Lukas Laukamp; +Cc: kvm On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote: > Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: > >> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >>> >>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>> >>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> >>>> wrote: >>>>> >>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>> >>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>> >>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>> >>>>>>>> I have a simple user question. I have a few LVM based KVM guests and >>>>>>>> wan't to backup them to files. The simple and nasty way would be to >>>>>>>> create a complete output file with dd, which wastes very much space. >>>>>>>> So I would like to create a backup of the LVM to a file which only >>>>>>>> locates the space which is used on the LVM. Would be create when the >>>>>>>> output file would be something like a qcow2 file which could be also >>>>>>>> simply startet with KVM. >>>>>>> >>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>> >>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>> vm001-backup.qcow2 >>>>>>> >>>>>>> Note that cp(1) tries to make the destination file sparse (see the >>>>>>> --sparse option in the man page). So you don't need to use qcow2, >>>>>>> you >>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not use >>>>>>> disk space for zero regions. >>>>>>> >>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>> >>>>>>> Stefan >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>>> >>>>>> Hello Stefano, >>>>>> >>>>>> thanks for the fast reply. I will test this later. In my case now it >>>>>> would >>>>>> be a offline backup. For the online backup I think about a seperated >>>>>> system >>>>>> which every day makes incremental backups and once a week a full >>>>>> backup. >>>>>> The >>>>>> main problem is, that the systems are in a WAN network and I need >>>>>> encryption >>>>>> between the systems. Would it be possible to do something like this: >>>>>> create >>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>> remote >>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>> files >>>>>> on >>>>>> the backup system? And in which format could be the incremental >>>>>> backups >>>>>> be >>>>>> stored? >>>> >>>> Since there is a WAN link it's important to use a compact image >>>> representation before hitting the network. I would use qemu-img >>>> convert -O qcow2 on the host and only transfer the qcow2 output. The >>>> qcow2 file does not contain zero regions and will therefore save a lot >>>> of network bandwidth compared to accessing the LVM volume over the >>>> WAN. >>>> >>>> If you are using rsync or another tool it's a different story. You >>>> could rsync the current LVM volume on the host over the last full >>>> backup, it should avoid transferring image data which is already >>>> present in the last full backup - the result is that you only transfer >>>> changed data plus the rsync metadata. >>>> >>>> Stefan >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> >>> Hello Stefan, >>> >>> the rsync part I don't have understood fully. So to create a qcow2 on the >>> host, transfer this to the backup server will result in the weekly full >>> backup. So do you mean I could use rsync to read the LVM from the host, >>> compare the LVM data with the data in the qcow2 on the backup server and >>> simply transfer the differences to the file? Or does it work on another >>> way? >> >> When using rsync you can skip qcow2. Only two objects are needed: >> 1. The LVM volume on the host. >> 2. The last full backup on the backup client. >> >> rsync compares #1 and #2 efficiently over the network and only >> transfers data from #1 which has changed. >> >> After rsync completes your full backup image is identical to the LVM >> volume. Next week you can use it as the "last" image to rsync >> against. >> >> Stefan >> -- >> To unsubscribe from this list: send the line "unsubscribe kvm" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > So I simply update the full backup, which is simply a raw file which get > mounted while the backup? The image file does not need to be mounted. Just rsync the raw image file. Stefan ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 11:36 ` Stefan Hajnoczi @ 2012-10-12 11:51 ` Lukas Laukamp 2012-10-12 12:59 ` Stefan Hajnoczi 2012-10-12 18:14 ` Lukas Laukamp 1 sibling, 1 reply; 26+ messages in thread From: Lukas Laukamp @ 2012-10-12 11:51 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: kvm Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: > On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: >> >>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>>> >>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> >>>>> wrote: >>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>>> >>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>>> I have a simple user question. I have a few LVM based KVM guests and >>>>>>>>> wan't to backup them to files. The simple and nasty way would be to >>>>>>>>> create a complete output file with dd, which wastes very much space. >>>>>>>>> So I would like to create a backup of the LVM to a file which only >>>>>>>>> locates the space which is used on the LVM. Would be create when the >>>>>>>>> output file would be something like a qcow2 file which could be also >>>>>>>>> simply startet with KVM. >>>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>>> >>>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>>> vm001-backup.qcow2 >>>>>>>> >>>>>>>> Note that cp(1) tries to make the destination file sparse (see the >>>>>>>> --sparse option in the man page). So you don't need to use qcow2, >>>>>>>> you >>>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not use >>>>>>>> disk space for zero regions. >>>>>>>> >>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>>> >>>>>>>> Stefan >>>>>>>> -- >>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>> >>>>>>> Hello Stefano, >>>>>>> >>>>>>> thanks for the fast reply. I will test this later. In my case now it >>>>>>> would >>>>>>> be a offline backup. For the online backup I think about a seperated >>>>>>> system >>>>>>> which every day makes incremental backups and once a week a full >>>>>>> backup. >>>>>>> The >>>>>>> main problem is, that the systems are in a WAN network and I need >>>>>>> encryption >>>>>>> between the systems. Would it be possible to do something like this: >>>>>>> create >>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>>> remote >>>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>>> files >>>>>>> on >>>>>>> the backup system? And in which format could be the incremental >>>>>>> backups >>>>>>> be >>>>>>> stored? >>>>> Since there is a WAN link it's important to use a compact image >>>>> representation before hitting the network. I would use qemu-img >>>>> convert -O qcow2 on the host and only transfer the qcow2 output. The >>>>> qcow2 file does not contain zero regions and will therefore save a lot >>>>> of network bandwidth compared to accessing the LVM volume over the >>>>> WAN. >>>>> >>>>> If you are using rsync or another tool it's a different story. You >>>>> could rsync the current LVM volume on the host over the last full >>>>> backup, it should avoid transferring image data which is already >>>>> present in the last full backup - the result is that you only transfer >>>>> changed data plus the rsync metadata. >>>>> >>>>> Stefan >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> Hello Stefan, >>>> >>>> the rsync part I don't have understood fully. So to create a qcow2 on the >>>> host, transfer this to the backup server will result in the weekly full >>>> backup. So do you mean I could use rsync to read the LVM from the host, >>>> compare the LVM data with the data in the qcow2 on the backup server and >>>> simply transfer the differences to the file? Or does it work on another >>>> way? >>> When using rsync you can skip qcow2. Only two objects are needed: >>> 1. The LVM volume on the host. >>> 2. The last full backup on the backup client. >>> >>> rsync compares #1 and #2 efficiently over the network and only >>> transfers data from #1 which has changed. >>> >>> After rsync completes your full backup image is identical to the LVM >>> volume. Next week you can use it as the "last" image to rsync >>> against. >>> >>> Stefan >>> -- >>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> So I simply update the full backup, which is simply a raw file which get >> mounted while the backup? > The image file does not need to be mounted. Just rsync the raw image file. > > Stefan Ah, thats great so to have a complete task in mind: 1. Create a qcow2 of a snapshot of an running VM 2. Transfer the qcow2 to the backup node 3. For incremental backup sync a daily LVM snapshot with the image on the backup node via rsync Is that right? Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 11:51 ` Lukas Laukamp @ 2012-10-12 12:59 ` Stefan Hajnoczi 2012-10-12 13:02 ` Lukas Laukamp 0 siblings, 1 reply; 26+ messages in thread From: Stefan Hajnoczi @ 2012-10-12 12:59 UTC (permalink / raw) To: Lukas Laukamp; +Cc: kvm On Fri, Oct 12, 2012 at 1:51 PM, Lukas Laukamp <lukas@laukamp.me> wrote: > Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: > >> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >>> >>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: >>> >>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> >>>> wrote: >>>>> >>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>>>> >>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> >>>>>> wrote: >>>>>>> >>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>>>> >>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>>>> >>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>>>> >>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests >>>>>>>>>> and >>>>>>>>>> wan't to backup them to files. The simple and nasty way would be >>>>>>>>>> to >>>>>>>>>> create a complete output file with dd, which wastes very much >>>>>>>>>> space. >>>>>>>>>> So I would like to create a backup of the LVM to a file which only >>>>>>>>>> locates the space which is used on the LVM. Would be create when >>>>>>>>>> the >>>>>>>>>> output file would be something like a qcow2 file which could be >>>>>>>>>> also >>>>>>>>>> simply startet with KVM. >>>>>>>>> >>>>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>>>> >>>>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>>>> vm001-backup.qcow2 >>>>>>>>> >>>>>>>>> Note that cp(1) tries to make the destination file sparse (see the >>>>>>>>> --sparse option in the man page). So you don't need to use qcow2, >>>>>>>>> you >>>>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not >>>>>>>>> use >>>>>>>>> disk space for zero regions. >>>>>>>>> >>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>>>> >>>>>>>>> Stefan >>>>>>>>> -- >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>> >>>>>>>> >>>>>>>> Hello Stefano, >>>>>>>> >>>>>>>> thanks for the fast reply. I will test this later. In my case now it >>>>>>>> would >>>>>>>> be a offline backup. For the online backup I think about a seperated >>>>>>>> system >>>>>>>> which every day makes incremental backups and once a week a full >>>>>>>> backup. >>>>>>>> The >>>>>>>> main problem is, that the systems are in a WAN network and I need >>>>>>>> encryption >>>>>>>> between the systems. Would it be possible to do something like this: >>>>>>>> create >>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>>>> remote >>>>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>>>> files >>>>>>>> on >>>>>>>> the backup system? And in which format could be the incremental >>>>>>>> backups >>>>>>>> be >>>>>>>> stored? >>>>>> >>>>>> Since there is a WAN link it's important to use a compact image >>>>>> representation before hitting the network. I would use qemu-img >>>>>> convert -O qcow2 on the host and only transfer the qcow2 output. The >>>>>> qcow2 file does not contain zero regions and will therefore save a lot >>>>>> of network bandwidth compared to accessing the LVM volume over the >>>>>> WAN. >>>>>> >>>>>> If you are using rsync or another tool it's a different story. You >>>>>> could rsync the current LVM volume on the host over the last full >>>>>> backup, it should avoid transferring image data which is already >>>>>> present in the last full backup - the result is that you only transfer >>>>>> changed data plus the rsync metadata. >>>>>> >>>>>> Stefan >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>> the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> >>>>> Hello Stefan, >>>>> >>>>> the rsync part I don't have understood fully. So to create a qcow2 on >>>>> the >>>>> host, transfer this to the backup server will result in the weekly full >>>>> backup. So do you mean I could use rsync to read the LVM from the host, >>>>> compare the LVM data with the data in the qcow2 on the backup server >>>>> and >>>>> simply transfer the differences to the file? Or does it work on another >>>>> way? >>>> >>>> When using rsync you can skip qcow2. Only two objects are needed: >>>> 1. The LVM volume on the host. >>>> 2. The last full backup on the backup client. >>>> >>>> rsync compares #1 and #2 efficiently over the network and only >>>> transfers data from #1 which has changed. >>>> >>>> After rsync completes your full backup image is identical to the LVM >>>> volume. Next week you can use it as the "last" image to rsync >>>> against. >>>> >>>> Stefan >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> >>> So I simply update the full backup, which is simply a raw file which get >>> mounted while the backup? >> >> The image file does not need to be mounted. Just rsync the raw image >> file. >> >> Stefan > > > Ah, thats great so to have a complete task in mind: > > 1. Create a qcow2 of a snapshot of an running VM > 2. Transfer the qcow2 to the backup node > 3. For incremental backup sync a daily LVM snapshot with the image on the > backup node via rsync Yes. Only make sure not to rsync the raw LVM image onto the qcow2 image - the backup client needs to have a raw image if you're syncing directly against the raw LVM volume. Stefan ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 12:59 ` Stefan Hajnoczi @ 2012-10-12 13:02 ` Lukas Laukamp 2012-10-12 14:25 ` Stefan Hajnoczi 0 siblings, 1 reply; 26+ messages in thread From: Lukas Laukamp @ 2012-10-12 13:02 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: kvm Am 12.10.2012 14:59, schrieb Stefan Hajnoczi: > On Fri, Oct 12, 2012 at 1:51 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: >> >>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: >>>> >>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> >>>>> wrote: >>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>>>>> >>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> >>>>>>> wrote: >>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>>>>> >>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests >>>>>>>>>>> and >>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be >>>>>>>>>>> to >>>>>>>>>>> create a complete output file with dd, which wastes very much >>>>>>>>>>> space. >>>>>>>>>>> So I would like to create a backup of the LVM to a file which only >>>>>>>>>>> locates the space which is used on the LVM. Would be create when >>>>>>>>>>> the >>>>>>>>>>> output file would be something like a qcow2 file which could be >>>>>>>>>>> also >>>>>>>>>>> simply startet with KVM. >>>>>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>>>>> >>>>>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>>>>> vm001-backup.qcow2 >>>>>>>>>> >>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see the >>>>>>>>>> --sparse option in the man page). So you don't need to use qcow2, >>>>>>>>>> you >>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not >>>>>>>>>> use >>>>>>>>>> disk space for zero regions. >>>>>>>>>> >>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>>>>> >>>>>>>>>> Stefan >>>>>>>>>> -- >>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>>> >>>>>>>>> Hello Stefano, >>>>>>>>> >>>>>>>>> thanks for the fast reply. I will test this later. In my case now it >>>>>>>>> would >>>>>>>>> be a offline backup. For the online backup I think about a seperated >>>>>>>>> system >>>>>>>>> which every day makes incremental backups and once a week a full >>>>>>>>> backup. >>>>>>>>> The >>>>>>>>> main problem is, that the systems are in a WAN network and I need >>>>>>>>> encryption >>>>>>>>> between the systems. Would it be possible to do something like this: >>>>>>>>> create >>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>>>>> remote >>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>>>>> files >>>>>>>>> on >>>>>>>>> the backup system? And in which format could be the incremental >>>>>>>>> backups >>>>>>>>> be >>>>>>>>> stored? >>>>>>> Since there is a WAN link it's important to use a compact image >>>>>>> representation before hitting the network. I would use qemu-img >>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output. The >>>>>>> qcow2 file does not contain zero regions and will therefore save a lot >>>>>>> of network bandwidth compared to accessing the LVM volume over the >>>>>>> WAN. >>>>>>> >>>>>>> If you are using rsync or another tool it's a different story. You >>>>>>> could rsync the current LVM volume on the host over the last full >>>>>>> backup, it should avoid transferring image data which is already >>>>>>> present in the last full backup - the result is that you only transfer >>>>>>> changed data plus the rsync metadata. >>>>>>> >>>>>>> Stefan >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>>> Hello Stefan, >>>>>> >>>>>> the rsync part I don't have understood fully. So to create a qcow2 on >>>>>> the >>>>>> host, transfer this to the backup server will result in the weekly full >>>>>> backup. So do you mean I could use rsync to read the LVM from the host, >>>>>> compare the LVM data with the data in the qcow2 on the backup server >>>>>> and >>>>>> simply transfer the differences to the file? Or does it work on another >>>>>> way? >>>>> When using rsync you can skip qcow2. Only two objects are needed: >>>>> 1. The LVM volume on the host. >>>>> 2. The last full backup on the backup client. >>>>> >>>>> rsync compares #1 and #2 efficiently over the network and only >>>>> transfers data from #1 which has changed. >>>>> >>>>> After rsync completes your full backup image is identical to the LVM >>>>> volume. Next week you can use it as the "last" image to rsync >>>>> against. >>>>> >>>>> Stefan >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> So I simply update the full backup, which is simply a raw file which get >>>> mounted while the backup? >>> The image file does not need to be mounted. Just rsync the raw image >>> file. >>> >>> Stefan >> >> Ah, thats great so to have a complete task in mind: >> >> 1. Create a qcow2 of a snapshot of an running VM >> 2. Transfer the qcow2 to the backup node >> 3. For incremental backup sync a daily LVM snapshot with the image on the >> backup node via rsync > Yes. Only make sure not to rsync the raw LVM image onto the qcow2 > image - the backup client needs to have a raw image if you're syncing > directly against the raw LVM volume. > > Stefan So I should use a raw file for the syncing and convert it on the backup node to a qcow2 to have a good file format? Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 13:02 ` Lukas Laukamp @ 2012-10-12 14:25 ` Stefan Hajnoczi 2012-10-12 14:36 ` Javier Guerra Giraldez 0 siblings, 1 reply; 26+ messages in thread From: Stefan Hajnoczi @ 2012-10-12 14:25 UTC (permalink / raw) To: Lukas Laukamp; +Cc: kvm On Fri, Oct 12, 2012 at 3:02 PM, Lukas Laukamp <lukas@laukamp.me> wrote: > Am 12.10.2012 14:59, schrieb Stefan Hajnoczi: > >> On Fri, Oct 12, 2012 at 1:51 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >>> >>> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: >>> >>>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> >>>> wrote: >>>>> >>>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: >>>>> >>>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> >>>>>> wrote: >>>>>>> >>>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>>>>>> >>>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>>>>>> >>>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>>>>>> >>>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>>>>>> >>>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests >>>>>>>>>>>> and >>>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be >>>>>>>>>>>> to >>>>>>>>>>>> create a complete output file with dd, which wastes very much >>>>>>>>>>>> space. >>>>>>>>>>>> So I would like to create a backup of the LVM to a file which >>>>>>>>>>>> only >>>>>>>>>>>> locates the space which is used on the LVM. Would be create when >>>>>>>>>>>> the >>>>>>>>>>>> output file would be something like a qcow2 file which could be >>>>>>>>>>>> also >>>>>>>>>>>> simply startet with KVM. >>>>>>>>>>> >>>>>>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>>>>>> >>>>>>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>>>>>> vm001-backup.qcow2 >>>>>>>>>>> >>>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see >>>>>>>>>>> the >>>>>>>>>>> --sparse option in the man page). So you don't need to use >>>>>>>>>>> qcow2, >>>>>>>>>>> you >>>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not >>>>>>>>>>> use >>>>>>>>>>> disk space for zero regions. >>>>>>>>>>> >>>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>>>>>> >>>>>>>>>>> Stefan >>>>>>>>>>> -- >>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hello Stefano, >>>>>>>>>> >>>>>>>>>> thanks for the fast reply. I will test this later. In my case now >>>>>>>>>> it >>>>>>>>>> would >>>>>>>>>> be a offline backup. For the online backup I think about a >>>>>>>>>> seperated >>>>>>>>>> system >>>>>>>>>> which every day makes incremental backups and once a week a full >>>>>>>>>> backup. >>>>>>>>>> The >>>>>>>>>> main problem is, that the systems are in a WAN network and I need >>>>>>>>>> encryption >>>>>>>>>> between the systems. Would it be possible to do something like >>>>>>>>>> this: >>>>>>>>>> create >>>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>>>>>> remote >>>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>>>>>> files >>>>>>>>>> on >>>>>>>>>> the backup system? And in which format could be the incremental >>>>>>>>>> backups >>>>>>>>>> be >>>>>>>>>> stored? >>>>>>>> >>>>>>>> Since there is a WAN link it's important to use a compact image >>>>>>>> representation before hitting the network. I would use qemu-img >>>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output. >>>>>>>> The >>>>>>>> qcow2 file does not contain zero regions and will therefore save a >>>>>>>> lot >>>>>>>> of network bandwidth compared to accessing the LVM volume over the >>>>>>>> WAN. >>>>>>>> >>>>>>>> If you are using rsync or another tool it's a different story. You >>>>>>>> could rsync the current LVM volume on the host over the last full >>>>>>>> backup, it should avoid transferring image data which is already >>>>>>>> present in the last full backup - the result is that you only >>>>>>>> transfer >>>>>>>> changed data plus the rsync metadata. >>>>>>>> >>>>>>>> Stefan >>>>>>>> -- >>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>> >>>>>>> >>>>>>> Hello Stefan, >>>>>>> >>>>>>> the rsync part I don't have understood fully. So to create a qcow2 on >>>>>>> the >>>>>>> host, transfer this to the backup server will result in the weekly >>>>>>> full >>>>>>> backup. So do you mean I could use rsync to read the LVM from the >>>>>>> host, >>>>>>> compare the LVM data with the data in the qcow2 on the backup server >>>>>>> and >>>>>>> simply transfer the differences to the file? Or does it work on >>>>>>> another >>>>>>> way? >>>>>> >>>>>> When using rsync you can skip qcow2. Only two objects are needed: >>>>>> 1. The LVM volume on the host. >>>>>> 2. The last full backup on the backup client. >>>>>> >>>>>> rsync compares #1 and #2 efficiently over the network and only >>>>>> transfers data from #1 which has changed. >>>>>> >>>>>> After rsync completes your full backup image is identical to the LVM >>>>>> volume. Next week you can use it as the "last" image to rsync >>>>>> against. >>>>>> >>>>>> Stefan >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>> the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> >>>>> So I simply update the full backup, which is simply a raw file which >>>>> get >>>>> mounted while the backup? >>>> >>>> The image file does not need to be mounted. Just rsync the raw image >>>> file. >>>> >>>> Stefan >>> >>> >>> Ah, thats great so to have a complete task in mind: >>> >>> 1. Create a qcow2 of a snapshot of an running VM >>> 2. Transfer the qcow2 to the backup node >>> 3. For incremental backup sync a daily LVM snapshot with the image on the >>> backup node via rsync >> >> Yes. Only make sure not to rsync the raw LVM image onto the qcow2 >> image - the backup client needs to have a raw image if you're syncing >> directly against the raw LVM volume. >> >> Stefan > > > So I should use a raw file for the syncing and convert it on the backup node > to a qcow2 to have a good file format? I would leave them raw as long as they are sparse (zero regions do not take up space). If you need to copy them you can either convert to qcow2 or use tools that preserve sparseness (BTW compression tools are good at this). But it's up to you. Stefan ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 14:25 ` Stefan Hajnoczi @ 2012-10-12 14:36 ` Javier Guerra Giraldez 2012-10-12 15:51 ` Lukas Laukamp 2012-10-12 19:18 ` Stefan Hajnoczi 0 siblings, 2 replies; 26+ messages in thread From: Javier Guerra Giraldez @ 2012-10-12 14:36 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: Lukas Laukamp, kvm On Fri, Oct 12, 2012 at 9:25 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote: > I would leave them raw as long as they are sparse (zero regions do not > take up space). If you need to copy them you can either convert to > qcow2 or use tools that preserve sparseness (BTW compression tools are > good at this). note that free blocks previously used by deleted files won't be sparse, won't be zero and won't be much reduced by compression. i'd say the usual advice stays: A: if you run any non-trivial application on that VM, then use a real network backup tool 'from the inside' of the VM B: if real point-in-time application-cache-storage consistency is not important, then you can: - make a read-write LVM snapshot - mount that and fsck. (it will appear as not-cleanly unmounted) - backup the files. (i like rsync, especially if you have an uncompressed previous backup) - umount and destroy the snapshot - optionally compress the backup but seriously consider option A before. especially important if you run any DB on that VM -- Javier ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 14:36 ` Javier Guerra Giraldez @ 2012-10-12 15:51 ` Lukas Laukamp 2012-10-12 19:18 ` Stefan Hajnoczi 1 sibling, 0 replies; 26+ messages in thread From: Lukas Laukamp @ 2012-10-12 15:51 UTC (permalink / raw) To: Javier Guerra Giraldez; +Cc: Stefan Hajnoczi, kvm Am 12.10.2012 16:36, schrieb Javier Guerra Giraldez: > On Fri, Oct 12, 2012 at 9:25 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote: >> I would leave them raw as long as they are sparse (zero regions do not >> take up space). If you need to copy them you can either convert to >> qcow2 or use tools that preserve sparseness (BTW compression tools are >> good at this). > note that free blocks previously used by deleted files won't be > sparse, won't be zero and won't be much reduced by compression. > > i'd say the usual advice stays: > > A: if you run any non-trivial application on that VM, then use a real > network backup tool 'from the inside' of the VM > B: if real point-in-time application-cache-storage consistency is not > important, then you can: > > - make a read-write LVM snapshot > - mount that and fsck. (it will appear as not-cleanly unmounted) > - backup the files. (i like rsync, especially if you have an > uncompressed previous backup) > - umount and destroy the snapshot > - optionally compress the backup > > > but seriously consider option A before. especially important if you > run any DB on that VM > I already thought about such situations. So DB systems are a real problem. I know that some data can be better backup directly but for example a VM for a Web- or Mailserver I want to backup completely to get this services get to work again very fast if there are problems. So it's a complex problem I think and not every machine can be backuped as another. I think that I must start by the hardware and software configuration of the backup node. Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 14:36 ` Javier Guerra Giraldez 2012-10-12 15:51 ` Lukas Laukamp @ 2012-10-12 19:18 ` Stefan Hajnoczi 1 sibling, 0 replies; 26+ messages in thread From: Stefan Hajnoczi @ 2012-10-12 19:18 UTC (permalink / raw) To: Javier Guerra Giraldez; +Cc: Lukas Laukamp, kvm On Fri, Oct 12, 2012 at 4:36 PM, Javier Guerra Giraldez <javier@guerrag.com> wrote: > On Fri, Oct 12, 2012 at 9:25 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote: >> I would leave them raw as long as they are sparse (zero regions do not >> take up space). If you need to copy them you can either convert to >> qcow2 or use tools that preserve sparseness (BTW compression tools are >> good at this). > > note that free blocks previously used by deleted files won't be > sparse, won't be zero and won't be much reduced by compression. > > i'd say the usual advice stays: > > A: if you run any non-trivial application on that VM, then use a real > network backup tool 'from the inside' of the VM > B: if real point-in-time application-cache-storage consistency is not > important, then you can: > > - make a read-write LVM snapshot > - mount that and fsck. (it will appear as not-cleanly unmounted) > - backup the files. (i like rsync, especially if you have an > uncompressed previous backup) > - umount and destroy the snapshot > - optionally compress the backup > > > but seriously consider option A before. especially important if you > run any DB on that VM Valid points. People seem to like crash-consistent image file backup though because it's convenient. It may not be backing up all application state though :(. Regarding option B, your idea is better than what I've been suggesting. By mounting the image on the host and rsyncing the mounted files you avoid syncing dirty blocks in the image file (e.g. deleted data in the guest file system). Stefan ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 11:36 ` Stefan Hajnoczi 2012-10-12 11:51 ` Lukas Laukamp @ 2012-10-12 18:14 ` Lukas Laukamp 2012-10-12 19:13 ` Stefan Hajnoczi 1 sibling, 1 reply; 26+ messages in thread From: Lukas Laukamp @ 2012-10-12 18:14 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: kvm Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: > On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: >> >>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>>> >>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> >>>>> wrote: >>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>>> >>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>>> I have a simple user question. I have a few LVM based KVM guests and >>>>>>>>> wan't to backup them to files. The simple and nasty way would be to >>>>>>>>> create a complete output file with dd, which wastes very much space. >>>>>>>>> So I would like to create a backup of the LVM to a file which only >>>>>>>>> locates the space which is used on the LVM. Would be create when the >>>>>>>>> output file would be something like a qcow2 file which could be also >>>>>>>>> simply startet with KVM. >>>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>>> >>>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>>> vm001-backup.qcow2 >>>>>>>> >>>>>>>> Note that cp(1) tries to make the destination file sparse (see the >>>>>>>> --sparse option in the man page). So you don't need to use qcow2, >>>>>>>> you >>>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not use >>>>>>>> disk space for zero regions. >>>>>>>> >>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>>> >>>>>>>> Stefan >>>>>>>> -- >>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>> >>>>>>> Hello Stefano, >>>>>>> >>>>>>> thanks for the fast reply. I will test this later. In my case now it >>>>>>> would >>>>>>> be a offline backup. For the online backup I think about a seperated >>>>>>> system >>>>>>> which every day makes incremental backups and once a week a full >>>>>>> backup. >>>>>>> The >>>>>>> main problem is, that the systems are in a WAN network and I need >>>>>>> encryption >>>>>>> between the systems. Would it be possible to do something like this: >>>>>>> create >>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>>> remote >>>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>>> files >>>>>>> on >>>>>>> the backup system? And in which format could be the incremental >>>>>>> backups >>>>>>> be >>>>>>> stored? >>>>> Since there is a WAN link it's important to use a compact image >>>>> representation before hitting the network. I would use qemu-img >>>>> convert -O qcow2 on the host and only transfer the qcow2 output. The >>>>> qcow2 file does not contain zero regions and will therefore save a lot >>>>> of network bandwidth compared to accessing the LVM volume over the >>>>> WAN. >>>>> >>>>> If you are using rsync or another tool it's a different story. You >>>>> could rsync the current LVM volume on the host over the last full >>>>> backup, it should avoid transferring image data which is already >>>>> present in the last full backup - the result is that you only transfer >>>>> changed data plus the rsync metadata. >>>>> >>>>> Stefan >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> Hello Stefan, >>>> >>>> the rsync part I don't have understood fully. So to create a qcow2 on the >>>> host, transfer this to the backup server will result in the weekly full >>>> backup. So do you mean I could use rsync to read the LVM from the host, >>>> compare the LVM data with the data in the qcow2 on the backup server and >>>> simply transfer the differences to the file? Or does it work on another >>>> way? >>> When using rsync you can skip qcow2. Only two objects are needed: >>> 1. The LVM volume on the host. >>> 2. The last full backup on the backup client. >>> >>> rsync compares #1 and #2 efficiently over the network and only >>> transfers data from #1 which has changed. >>> >>> After rsync completes your full backup image is identical to the LVM >>> volume. Next week you can use it as the "last" image to rsync >>> against. >>> >>> Stefan >>> -- >>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> So I simply update the full backup, which is simply a raw file which get >> mounted while the backup? > The image file does not need to be mounted. Just rsync the raw image file. > > Stefan I tested the qemu-img command now, but it does not do that what I want. I have a VM with a 5GB disk, this disk is not allocated with 1GB of data. When I do the convert command the output is a 5GB qcow2 disk. What do I have to do to get a qcow2 file with only the allocated space/data from the LVM? I also tried the -c option of qemu-img convert but the result was nearly the same. Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 18:14 ` Lukas Laukamp @ 2012-10-12 19:13 ` Stefan Hajnoczi 2012-10-12 19:26 ` Lukas Laukamp 0 siblings, 1 reply; 26+ messages in thread From: Stefan Hajnoczi @ 2012-10-12 19:13 UTC (permalink / raw) To: Lukas Laukamp; +Cc: kvm On Fri, Oct 12, 2012 at 8:14 PM, Lukas Laukamp <lukas@laukamp.me> wrote: > Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: > >> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >>> >>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: >>> >>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> >>>> wrote: >>>>> >>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>>>> >>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> >>>>>> wrote: >>>>>>> >>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>>>> >>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>>>> >>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>>>> >>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests >>>>>>>>>> and >>>>>>>>>> wan't to backup them to files. The simple and nasty way would be >>>>>>>>>> to >>>>>>>>>> create a complete output file with dd, which wastes very much >>>>>>>>>> space. >>>>>>>>>> So I would like to create a backup of the LVM to a file which only >>>>>>>>>> locates the space which is used on the LVM. Would be create when >>>>>>>>>> the >>>>>>>>>> output file would be something like a qcow2 file which could be >>>>>>>>>> also >>>>>>>>>> simply startet with KVM. >>>>>>>>> >>>>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>>>> >>>>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>>>> vm001-backup.qcow2 >>>>>>>>> >>>>>>>>> Note that cp(1) tries to make the destination file sparse (see the >>>>>>>>> --sparse option in the man page). So you don't need to use qcow2, >>>>>>>>> you >>>>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not >>>>>>>>> use >>>>>>>>> disk space for zero regions. >>>>>>>>> >>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>>>> >>>>>>>>> Stefan >>>>>>>>> -- >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>> >>>>>>>> >>>>>>>> Hello Stefano, >>>>>>>> >>>>>>>> thanks for the fast reply. I will test this later. In my case now it >>>>>>>> would >>>>>>>> be a offline backup. For the online backup I think about a seperated >>>>>>>> system >>>>>>>> which every day makes incremental backups and once a week a full >>>>>>>> backup. >>>>>>>> The >>>>>>>> main problem is, that the systems are in a WAN network and I need >>>>>>>> encryption >>>>>>>> between the systems. Would it be possible to do something like this: >>>>>>>> create >>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>>>> remote >>>>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>>>> files >>>>>>>> on >>>>>>>> the backup system? And in which format could be the incremental >>>>>>>> backups >>>>>>>> be >>>>>>>> stored? >>>>>> >>>>>> Since there is a WAN link it's important to use a compact image >>>>>> representation before hitting the network. I would use qemu-img >>>>>> convert -O qcow2 on the host and only transfer the qcow2 output. The >>>>>> qcow2 file does not contain zero regions and will therefore save a lot >>>>>> of network bandwidth compared to accessing the LVM volume over the >>>>>> WAN. >>>>>> >>>>>> If you are using rsync or another tool it's a different story. You >>>>>> could rsync the current LVM volume on the host over the last full >>>>>> backup, it should avoid transferring image data which is already >>>>>> present in the last full backup - the result is that you only transfer >>>>>> changed data plus the rsync metadata. >>>>>> >>>>>> Stefan >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>> the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> >>>>> Hello Stefan, >>>>> >>>>> the rsync part I don't have understood fully. So to create a qcow2 on >>>>> the >>>>> host, transfer this to the backup server will result in the weekly full >>>>> backup. So do you mean I could use rsync to read the LVM from the host, >>>>> compare the LVM data with the data in the qcow2 on the backup server >>>>> and >>>>> simply transfer the differences to the file? Or does it work on another >>>>> way? >>>> >>>> When using rsync you can skip qcow2. Only two objects are needed: >>>> 1. The LVM volume on the host. >>>> 2. The last full backup on the backup client. >>>> >>>> rsync compares #1 and #2 efficiently over the network and only >>>> transfers data from #1 which has changed. >>>> >>>> After rsync completes your full backup image is identical to the LVM >>>> volume. Next week you can use it as the "last" image to rsync >>>> against. >>>> >>>> Stefan >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> >>> So I simply update the full backup, which is simply a raw file which get >>> mounted while the backup? >> >> The image file does not need to be mounted. Just rsync the raw image >> file. >> >> Stefan > > > I tested the qemu-img command now, but it does not do that what I want. I > have a VM with a 5GB disk, this disk is not allocated with 1GB of data. When > I do the convert command the output is a 5GB qcow2 disk. What do I have to > do to get a qcow2 file with only the allocated space/data from the LVM? I > also tried the -c option of qemu-img convert but the result was nearly the > same. Please show the exact command-lines you are using and the qemu-img info <filename> output afterwards. Stefan ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 19:13 ` Stefan Hajnoczi @ 2012-10-12 19:26 ` Lukas Laukamp 2012-10-12 20:43 ` Stefan Hajnoczi 0 siblings, 1 reply; 26+ messages in thread From: Lukas Laukamp @ 2012-10-12 19:26 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: kvm Am 12.10.2012 21:13, schrieb Stefan Hajnoczi: > On Fri, Oct 12, 2012 at 8:14 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: >> >>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: >>>> >>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> >>>>> wrote: >>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>>>>> >>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> >>>>>>> wrote: >>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>>>>> >>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests >>>>>>>>>>> and >>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be >>>>>>>>>>> to >>>>>>>>>>> create a complete output file with dd, which wastes very much >>>>>>>>>>> space. >>>>>>>>>>> So I would like to create a backup of the LVM to a file which only >>>>>>>>>>> locates the space which is used on the LVM. Would be create when >>>>>>>>>>> the >>>>>>>>>>> output file would be something like a qcow2 file which could be >>>>>>>>>>> also >>>>>>>>>>> simply startet with KVM. >>>>>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>>>>> >>>>>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>>>>> vm001-backup.qcow2 >>>>>>>>>> >>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see the >>>>>>>>>> --sparse option in the man page). So you don't need to use qcow2, >>>>>>>>>> you >>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not >>>>>>>>>> use >>>>>>>>>> disk space for zero regions. >>>>>>>>>> >>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>>>>> >>>>>>>>>> Stefan >>>>>>>>>> -- >>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>>> >>>>>>>>> Hello Stefano, >>>>>>>>> >>>>>>>>> thanks for the fast reply. I will test this later. In my case now it >>>>>>>>> would >>>>>>>>> be a offline backup. For the online backup I think about a seperated >>>>>>>>> system >>>>>>>>> which every day makes incremental backups and once a week a full >>>>>>>>> backup. >>>>>>>>> The >>>>>>>>> main problem is, that the systems are in a WAN network and I need >>>>>>>>> encryption >>>>>>>>> between the systems. Would it be possible to do something like this: >>>>>>>>> create >>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>>>>> remote >>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>>>>> files >>>>>>>>> on >>>>>>>>> the backup system? And in which format could be the incremental >>>>>>>>> backups >>>>>>>>> be >>>>>>>>> stored? >>>>>>> Since there is a WAN link it's important to use a compact image >>>>>>> representation before hitting the network. I would use qemu-img >>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output. The >>>>>>> qcow2 file does not contain zero regions and will therefore save a lot >>>>>>> of network bandwidth compared to accessing the LVM volume over the >>>>>>> WAN. >>>>>>> >>>>>>> If you are using rsync or another tool it's a different story. You >>>>>>> could rsync the current LVM volume on the host over the last full >>>>>>> backup, it should avoid transferring image data which is already >>>>>>> present in the last full backup - the result is that you only transfer >>>>>>> changed data plus the rsync metadata. >>>>>>> >>>>>>> Stefan >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>>> Hello Stefan, >>>>>> >>>>>> the rsync part I don't have understood fully. So to create a qcow2 on >>>>>> the >>>>>> host, transfer this to the backup server will result in the weekly full >>>>>> backup. So do you mean I could use rsync to read the LVM from the host, >>>>>> compare the LVM data with the data in the qcow2 on the backup server >>>>>> and >>>>>> simply transfer the differences to the file? Or does it work on another >>>>>> way? >>>>> When using rsync you can skip qcow2. Only two objects are needed: >>>>> 1. The LVM volume on the host. >>>>> 2. The last full backup on the backup client. >>>>> >>>>> rsync compares #1 and #2 efficiently over the network and only >>>>> transfers data from #1 which has changed. >>>>> >>>>> After rsync completes your full backup image is identical to the LVM >>>>> volume. Next week you can use it as the "last" image to rsync >>>>> against. >>>>> >>>>> Stefan >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>>> So I simply update the full backup, which is simply a raw file which get >>>> mounted while the backup? >>> The image file does not need to be mounted. Just rsync the raw image >>> file. >>> >>> Stefan >> >> I tested the qemu-img command now, but it does not do that what I want. I >> have a VM with a 5GB disk, this disk is not allocated with 1GB of data. When >> I do the convert command the output is a 5GB qcow2 disk. What do I have to >> do to get a qcow2 file with only the allocated space/data from the LVM? I >> also tried the -c option of qemu-img convert but the result was nearly the >> same. > Please show the exact command-lines you are using and the qemu-img > info <filename> output afterwards. > > Stefan > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html Hello, the VM is called mailer and I used this command: qemu-img convert -f raw -c -O qcow2 /dev/vmdisks/mailer ./vmachines/mailer.qcow2 Im on the /mnt dir because it's a seperate HDD with the folder vmachines. Thats the output of qemu-img info: image: ./vmachines/mailer.qcow2 file format: qcow2 virtual size: 5.0G (5368709120 bytes) disk size: 4.1G cluster_size: 65536 I checked the info of the fs inside the guest. The / partition is 4,7GB big and 1,2GB are used. The fs is ext3. Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 19:26 ` Lukas Laukamp @ 2012-10-12 20:43 ` Stefan Hajnoczi 2012-10-12 20:56 ` Lukas Laukamp 2012-10-18 12:57 ` Avi Kivity 0 siblings, 2 replies; 26+ messages in thread From: Stefan Hajnoczi @ 2012-10-12 20:43 UTC (permalink / raw) To: Lukas Laukamp; +Cc: kvm On Fri, Oct 12, 2012 at 9:26 PM, Lukas Laukamp <lukas@laukamp.me> wrote: > Am 12.10.2012 21:13, schrieb Stefan Hajnoczi: > >> On Fri, Oct 12, 2012 at 8:14 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >>> >>> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: >>> >>>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> >>>> wrote: >>>>> >>>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: >>>>> >>>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> >>>>>> wrote: >>>>>>> >>>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>>>>>> >>>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>>>>>> >>>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>>>>>> >>>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>>>>>> >>>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests >>>>>>>>>>>> and >>>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be >>>>>>>>>>>> to >>>>>>>>>>>> create a complete output file with dd, which wastes very much >>>>>>>>>>>> space. >>>>>>>>>>>> So I would like to create a backup of the LVM to a file which >>>>>>>>>>>> only >>>>>>>>>>>> locates the space which is used on the LVM. Would be create when >>>>>>>>>>>> the >>>>>>>>>>>> output file would be something like a qcow2 file which could be >>>>>>>>>>>> also >>>>>>>>>>>> simply startet with KVM. >>>>>>>>>>> >>>>>>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>>>>>> >>>>>>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>>>>>> vm001-backup.qcow2 >>>>>>>>>>> >>>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see >>>>>>>>>>> the >>>>>>>>>>> --sparse option in the man page). So you don't need to use >>>>>>>>>>> qcow2, >>>>>>>>>>> you >>>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not >>>>>>>>>>> use >>>>>>>>>>> disk space for zero regions. >>>>>>>>>>> >>>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>>>>>> >>>>>>>>>>> Stefan >>>>>>>>>>> -- >>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hello Stefano, >>>>>>>>>> >>>>>>>>>> thanks for the fast reply. I will test this later. In my case now >>>>>>>>>> it >>>>>>>>>> would >>>>>>>>>> be a offline backup. For the online backup I think about a >>>>>>>>>> seperated >>>>>>>>>> system >>>>>>>>>> which every day makes incremental backups and once a week a full >>>>>>>>>> backup. >>>>>>>>>> The >>>>>>>>>> main problem is, that the systems are in a WAN network and I need >>>>>>>>>> encryption >>>>>>>>>> between the systems. Would it be possible to do something like >>>>>>>>>> this: >>>>>>>>>> create >>>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>>>>>> remote >>>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>>>>>> files >>>>>>>>>> on >>>>>>>>>> the backup system? And in which format could be the incremental >>>>>>>>>> backups >>>>>>>>>> be >>>>>>>>>> stored? >>>>>>>> >>>>>>>> Since there is a WAN link it's important to use a compact image >>>>>>>> representation before hitting the network. I would use qemu-img >>>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output. >>>>>>>> The >>>>>>>> qcow2 file does not contain zero regions and will therefore save a >>>>>>>> lot >>>>>>>> of network bandwidth compared to accessing the LVM volume over the >>>>>>>> WAN. >>>>>>>> >>>>>>>> If you are using rsync or another tool it's a different story. You >>>>>>>> could rsync the current LVM volume on the host over the last full >>>>>>>> backup, it should avoid transferring image data which is already >>>>>>>> present in the last full backup - the result is that you only >>>>>>>> transfer >>>>>>>> changed data plus the rsync metadata. >>>>>>>> >>>>>>>> Stefan >>>>>>>> -- >>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>> >>>>>>> >>>>>>> Hello Stefan, >>>>>>> >>>>>>> the rsync part I don't have understood fully. So to create a qcow2 on >>>>>>> the >>>>>>> host, transfer this to the backup server will result in the weekly >>>>>>> full >>>>>>> backup. So do you mean I could use rsync to read the LVM from the >>>>>>> host, >>>>>>> compare the LVM data with the data in the qcow2 on the backup server >>>>>>> and >>>>>>> simply transfer the differences to the file? Or does it work on >>>>>>> another >>>>>>> way? >>>>>> >>>>>> When using rsync you can skip qcow2. Only two objects are needed: >>>>>> 1. The LVM volume on the host. >>>>>> 2. The last full backup on the backup client. >>>>>> >>>>>> rsync compares #1 and #2 efficiently over the network and only >>>>>> transfers data from #1 which has changed. >>>>>> >>>>>> After rsync completes your full backup image is identical to the LVM >>>>>> volume. Next week you can use it as the "last" image to rsync >>>>>> against. >>>>>> >>>>>> Stefan >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>> the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>>> >>>>> So I simply update the full backup, which is simply a raw file which >>>>> get >>>>> mounted while the backup? >>>> >>>> The image file does not need to be mounted. Just rsync the raw image >>>> file. >>>> >>>> Stefan >>> >>> >>> I tested the qemu-img command now, but it does not do that what I want. I >>> have a VM with a 5GB disk, this disk is not allocated with 1GB of data. >>> When >>> I do the convert command the output is a 5GB qcow2 disk. What do I have >>> to >>> do to get a qcow2 file with only the allocated space/data from the LVM? I >>> also tried the -c option of qemu-img convert but the result was nearly >>> the >>> same. >> >> Please show the exact command-lines you are using and the qemu-img >> info <filename> output afterwards. >> >> Stefan >> -- >> To unsubscribe from this list: send the line "unsubscribe kvm" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > Hello, > > the VM is called mailer and I used this command: qemu-img convert -f raw -c > -O qcow2 /dev/vmdisks/mailer ./vmachines/mailer.qcow2 > > Im on the /mnt dir because it's a seperate HDD with the folder vmachines. > > Thats the output of qemu-img info: > > image: ./vmachines/mailer.qcow2 > file format: qcow2 > virtual size: 5.0G (5368709120 bytes) > disk size: 4.1G > cluster_size: 65536 > > I checked the info of the fs inside the guest. The / partition is 4,7GB big > and 1,2GB are used. The fs is ext3. I wonder if you see better results without the -c option. Stefan ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 20:43 ` Stefan Hajnoczi @ 2012-10-12 20:56 ` Lukas Laukamp 2012-10-12 21:53 ` Javier Guerra Giraldez 2012-10-18 12:57 ` Avi Kivity 1 sibling, 1 reply; 26+ messages in thread From: Lukas Laukamp @ 2012-10-12 20:56 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: kvm Am 12.10.2012 22:43, schrieb Stefan Hajnoczi: > On Fri, Oct 12, 2012 at 9:26 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >> Am 12.10.2012 21:13, schrieb Stefan Hajnoczi: >> >>> On Fri, Oct 12, 2012 at 8:14 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >>>> Am 12.10.2012 13:36, schrieb Stefan Hajnoczi: >>>> >>>>> On Fri, Oct 12, 2012 at 12:50 PM, Lukas Laukamp <lukas@laukamp.me> >>>>> wrote: >>>>>> Am 12.10.2012 12:47, schrieb Stefan Hajnoczi: >>>>>> >>>>>>> On Fri, Oct 12, 2012 at 12:16 PM, Lukas Laukamp <lukas@laukamp.me> >>>>>>> wrote: >>>>>>>> Am 12.10.2012 12:11, schrieb Stefan Hajnoczi: >>>>>>>> >>>>>>>>> On Fri, Oct 12, 2012 at 11:17 AM, Lukas Laukamp <lukas@laukamp.me> >>>>>>>>> wrote: >>>>>>>>>> Am 12.10.2012 10:58, schrieb Lukas Laukamp: >>>>>>>>>> >>>>>>>>>>> Am 12.10.2012 10:42, schrieb Stefan Hajnoczi: >>>>>>>>>>>> On Fri, Oct 12, 2012 at 08:52:32AM +0200, Lukas Laukamp wrote: >>>>>>>>>>>>> I have a simple user question. I have a few LVM based KVM guests >>>>>>>>>>>>> and >>>>>>>>>>>>> wan't to backup them to files. The simple and nasty way would be >>>>>>>>>>>>> to >>>>>>>>>>>>> create a complete output file with dd, which wastes very much >>>>>>>>>>>>> space. >>>>>>>>>>>>> So I would like to create a backup of the LVM to a file which >>>>>>>>>>>>> only >>>>>>>>>>>>> locates the space which is used on the LVM. Would be create when >>>>>>>>>>>>> the >>>>>>>>>>>>> output file would be something like a qcow2 file which could be >>>>>>>>>>>>> also >>>>>>>>>>>>> simply startet with KVM. >>>>>>>>>>>> If the VM is not running you can use "qemu-img convert": >>>>>>>>>>>> >>>>>>>>>>>> qemu-img convert -f raw -O qcow2 /dev/vg/vm001 >>>>>>>>>>>> vm001-backup.qcow2 >>>>>>>>>>>> >>>>>>>>>>>> Note that cp(1) tries to make the destination file sparse (see >>>>>>>>>>>> the >>>>>>>>>>>> --sparse option in the man page). So you don't need to use >>>>>>>>>>>> qcow2, >>>>>>>>>>>> you >>>>>>>>>>>> can use cp(1) to copy the LVM volume to a raw file. It will not >>>>>>>>>>>> use >>>>>>>>>>>> disk space for zero regions. >>>>>>>>>>>> >>>>>>>>>>>> If the VM is running you need to use LVM snapshots or stop the VM >>>>>>>>>>>> temporarily so a crash-consistent backup can be taken. >>>>>>>>>>>> >>>>>>>>>>>> Stefan >>>>>>>>>>>> -- >>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>>>>> >>>>>>>>>>> Hello Stefano, >>>>>>>>>>> >>>>>>>>>>> thanks for the fast reply. I will test this later. In my case now >>>>>>>>>>> it >>>>>>>>>>> would >>>>>>>>>>> be a offline backup. For the online backup I think about a >>>>>>>>>>> seperated >>>>>>>>>>> system >>>>>>>>>>> which every day makes incremental backups and once a week a full >>>>>>>>>>> backup. >>>>>>>>>>> The >>>>>>>>>>> main problem is, that the systems are in a WAN network and I need >>>>>>>>>>> encryption >>>>>>>>>>> between the systems. Would it be possible to do something like >>>>>>>>>>> this: >>>>>>>>>>> create >>>>>>>>>>> the LVM snapshot for the backup, read this LVM snapshot with the >>>>>>>>>>> remote >>>>>>>>>>> backup system via ssh tunnel and save the output of this to qcow2 >>>>>>>>>>> files >>>>>>>>>>> on >>>>>>>>>>> the backup system? And in which format could be the incremental >>>>>>>>>>> backups >>>>>>>>>>> be >>>>>>>>>>> stored? >>>>>>>>> Since there is a WAN link it's important to use a compact image >>>>>>>>> representation before hitting the network. I would use qemu-img >>>>>>>>> convert -O qcow2 on the host and only transfer the qcow2 output. >>>>>>>>> The >>>>>>>>> qcow2 file does not contain zero regions and will therefore save a >>>>>>>>> lot >>>>>>>>> of network bandwidth compared to accessing the LVM volume over the >>>>>>>>> WAN. >>>>>>>>> >>>>>>>>> If you are using rsync or another tool it's a different story. You >>>>>>>>> could rsync the current LVM volume on the host over the last full >>>>>>>>> backup, it should avoid transferring image data which is already >>>>>>>>> present in the last full backup - the result is that you only >>>>>>>>> transfer >>>>>>>>> changed data plus the rsync metadata. >>>>>>>>> >>>>>>>>> Stefan >>>>>>>>> -- >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>> >>>>>>>> Hello Stefan, >>>>>>>> >>>>>>>> the rsync part I don't have understood fully. So to create a qcow2 on >>>>>>>> the >>>>>>>> host, transfer this to the backup server will result in the weekly >>>>>>>> full >>>>>>>> backup. So do you mean I could use rsync to read the LVM from the >>>>>>>> host, >>>>>>>> compare the LVM data with the data in the qcow2 on the backup server >>>>>>>> and >>>>>>>> simply transfer the differences to the file? Or does it work on >>>>>>>> another >>>>>>>> way? >>>>>>> When using rsync you can skip qcow2. Only two objects are needed: >>>>>>> 1. The LVM volume on the host. >>>>>>> 2. The last full backup on the backup client. >>>>>>> >>>>>>> rsync compares #1 and #2 efficiently over the network and only >>>>>>> transfers data from #1 which has changed. >>>>>>> >>>>>>> After rsync completes your full backup image is identical to the LVM >>>>>>> volume. Next week you can use it as the "last" image to rsync >>>>>>> against. >>>>>>> >>>>>>> Stefan >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>>>>>> the body of a message to majordomo@vger.kernel.org >>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>>> So I simply update the full backup, which is simply a raw file which >>>>>> get >>>>>> mounted while the backup? >>>>> The image file does not need to be mounted. Just rsync the raw image >>>>> file. >>>>> >>>>> Stefan >>>> >>>> I tested the qemu-img command now, but it does not do that what I want. I >>>> have a VM with a 5GB disk, this disk is not allocated with 1GB of data. >>>> When >>>> I do the convert command the output is a 5GB qcow2 disk. What do I have >>>> to >>>> do to get a qcow2 file with only the allocated space/data from the LVM? I >>>> also tried the -c option of qemu-img convert but the result was nearly >>>> the >>>> same. >>> Please show the exact command-lines you are using and the qemu-img >>> info <filename> output afterwards. >>> >>> Stefan >>> -- >>> To unsubscribe from this list: send the line "unsubscribe kvm" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> Hello, >> >> the VM is called mailer and I used this command: qemu-img convert -f raw -c >> -O qcow2 /dev/vmdisks/mailer ./vmachines/mailer.qcow2 >> >> Im on the /mnt dir because it's a seperate HDD with the folder vmachines. >> >> Thats the output of qemu-img info: >> >> image: ./vmachines/mailer.qcow2 >> file format: qcow2 >> virtual size: 5.0G (5368709120 bytes) >> disk size: 4.1G >> cluster_size: 65536 >> >> I checked the info of the fs inside the guest. The / partition is 4,7GB big >> and 1,2GB are used. The fs is ext3. > I wonder if you see better results without the -c option. > > Stefan > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html So I would say that the results are very similar. So with -c option the file is 650-800MB smaller but thats not very much. I have a few bigger VM disks with sizes like 100GB when we calculat that in % so we see that the compression is very ecactly 20% so I would have an 80GB image of a disk where only 3GBs are used inside the VM. I think that it must be possible to create an image with a size like the used space + a few hundret MB with metadata or something like that. Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 20:56 ` Lukas Laukamp @ 2012-10-12 21:53 ` Javier Guerra Giraldez 2012-10-13 10:28 ` Lukas Laukamp 0 siblings, 1 reply; 26+ messages in thread From: Javier Guerra Giraldez @ 2012-10-12 21:53 UTC (permalink / raw) To: Lukas Laukamp; +Cc: Stefan Hajnoczi, kvm On Fri, Oct 12, 2012 at 3:56 PM, Lukas Laukamp <lukas@laukamp.me> wrote: > I think that it must be possible to create an image with a size like the > used space + a few hundret MB with metadata or something like that. the 'best' way to do it is 'from within' the VM the typical workaround is to mount/fsck a LVM snapshot (don't skip the fsck, or at least a journal replay) beyond that, there are a few utilities for specific filesystems: PartImage [1], dump/restore [2], and i'm sure some others. I don't know how would these behave with unclean images, which is what you get if you pull the image under the VM's feet. [1] http://www.partimage.org/Main_Page [2] http://dump.sourceforge.net/ -- Javier ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 21:53 ` Javier Guerra Giraldez @ 2012-10-13 10:28 ` Lukas Laukamp 2012-10-13 22:25 ` Lukas Laukamp 0 siblings, 1 reply; 26+ messages in thread From: Lukas Laukamp @ 2012-10-13 10:28 UTC (permalink / raw) To: Javier Guerra Giraldez; +Cc: Stefan Hajnoczi, kvm Am 12.10.2012 23:53, schrieb Javier Guerra Giraldez: > On Fri, Oct 12, 2012 at 3:56 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >> I think that it must be possible to create an image with a size like the >> used space + a few hundret MB with metadata or something like that. > the 'best' way to do it is 'from within' the VM > > the typical workaround is to mount/fsck a LVM snapshot (don't skip the > fsck, or at least a journal replay) > > beyond that, there are a few utilities for specific filesystems: > PartImage [1], dump/restore [2], and i'm sure some others. I don't > know how would these behave with unclean images, which is what you get > if you pull the image under the VM's feet. > > > [1] http://www.partimage.org/Main_Page > [2] http://dump.sourceforge.net/ > > So when I boot the VM with for example a live debian system, what would be the best method to backup the complete disk /dev/vda with only the data which is on it over a secure network connection to a destination in the network. Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-13 10:28 ` Lukas Laukamp @ 2012-10-13 22:25 ` Lukas Laukamp 2012-10-15 3:06 ` Javier Guerra Giraldez 0 siblings, 1 reply; 26+ messages in thread From: Lukas Laukamp @ 2012-10-13 22:25 UTC (permalink / raw) To: Javier Guerra Giraldez; +Cc: Stefan Hajnoczi, kvm Am 13.10.2012 12:28, schrieb Lukas Laukamp: > Am 12.10.2012 23:53, schrieb Javier Guerra Giraldez: >> On Fri, Oct 12, 2012 at 3:56 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >>> I think that it must be possible to create an image with a size like >>> the >>> used space + a few hundret MB with metadata or something like that. >> the 'best' way to do it is 'from within' the VM >> >> the typical workaround is to mount/fsck a LVM snapshot (don't skip the >> fsck, or at least a journal replay) >> >> beyond that, there are a few utilities for specific filesystems: >> PartImage [1], dump/restore [2], and i'm sure some others. I don't >> know how would these behave with unclean images, which is what you get >> if you pull the image under the VM's feet. >> >> >> [1] http://www.partimage.org/Main_Page >> [2] http://dump.sourceforge.net/ >> >> > > So when I boot the VM with for example a live debian system, what > would be the best method to backup the complete disk /dev/vda with > only the data which is on it over a secure network connection to a > destination in the network. > > Best Regards > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html I have backed up the data within the machine with partimage and fsarchiver. But it would be greate to have a better way than doing this over a live system. Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-13 22:25 ` Lukas Laukamp @ 2012-10-15 3:06 ` Javier Guerra Giraldez 2012-10-15 6:17 ` Lukas Laukamp 0 siblings, 1 reply; 26+ messages in thread From: Javier Guerra Giraldez @ 2012-10-15 3:06 UTC (permalink / raw) To: Lukas Laukamp; +Cc: Stefan Hajnoczi, kvm On Sat, Oct 13, 2012 at 5:25 PM, Lukas Laukamp <lukas@laukamp.me> wrote: > I have backed up the data within the machine with partimage and fsarchiver. > But it would be greate to have a better way than doing this over a live > system. make no mistake, the absolutely best way is from within the VM. It's the most consistent, safe and efficient method. Doing it "from the outside" is attractive, but it's a hack, and in some cases you have to jump through several hoops to make it safe. -- Javier ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-15 3:06 ` Javier Guerra Giraldez @ 2012-10-15 6:17 ` Lukas Laukamp 0 siblings, 0 replies; 26+ messages in thread From: Lukas Laukamp @ 2012-10-15 6:17 UTC (permalink / raw) To: Javier Guerra Giraldez; +Cc: Stefan Hajnoczi, kvm Am 15.10.2012 05:06, schrieb Javier Guerra Giraldez: > On Sat, Oct 13, 2012 at 5:25 PM, Lukas Laukamp <lukas@laukamp.me> wrote: >> I have backed up the data within the machine with partimage and fsarchiver. >> But it would be greate to have a better way than doing this over a live >> system. > make no mistake, the absolutely best way is from within the VM. It's > the most consistent, safe and efficient method. > > Doing it "from the outside" is attractive, but it's a hack, and in > some cases you have to jump through several hoops to make it safe. > > Because of the fact that this is a "problem" I bootet the VM with a live cd and made a backup of the important filesystems with the live system which runs inside the VM. Best Regards ^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [User Question] How to create a backup of an LVM based maschine without wasting space 2012-10-12 20:43 ` Stefan Hajnoczi 2012-10-12 20:56 ` Lukas Laukamp @ 2012-10-18 12:57 ` Avi Kivity 1 sibling, 0 replies; 26+ messages in thread From: Avi Kivity @ 2012-10-18 12:57 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: Lukas Laukamp, kvm On 10/12/2012 10:43 PM, Stefan Hajnoczi wrote: >> >> the VM is called mailer and I used this command: qemu-img convert -f raw -c >> -O qcow2 /dev/vmdisks/mailer ./vmachines/mailer.qcow2 >> >> Im on the /mnt dir because it's a seperate HDD with the folder vmachines. >> >> Thats the output of qemu-img info: >> >> image: ./vmachines/mailer.qcow2 >> file format: qcow2 >> virtual size: 5.0G (5368709120 bytes) >> disk size: 4.1G >> cluster_size: 65536 >> >> I checked the info of the fs inside the guest. The / partition is 4,7GB big >> and 1,2GB are used. The fs is ext3. > > I wonder if you see better results without the -c option. Since the image is on a logical volume, it may not have been cleared prior to use by the guest; or the guest is busy enough that it has overwritten all the blocks in the image. You can try a manual trim by creating a large file in the guest consisting of all zeros, then deleting it: dd < /dev/zero < /zeroes rm zeroes Afterwards you'll get a much smaller qcow image. Of course this requires working within the guest, and there is the risk of disrupting the guest by reaching the ENOSPC condition. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2012-10-18 12:57 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-10-12 6:52 [User Question] How to create a backup of an LVM based maschine without wasting space Lukas Laukamp
2012-10-12 8:42 ` Stefan Hajnoczi
[not found] ` <5077DB9F.6050809@laukamp.me>
2012-10-12 9:17 ` Lukas Laukamp
2012-10-12 10:11 ` Stefan Hajnoczi
2012-10-12 10:16 ` Lukas Laukamp
2012-10-12 10:47 ` Stefan Hajnoczi
2012-10-12 10:50 ` Lukas Laukamp
2012-10-12 11:36 ` Stefan Hajnoczi
2012-10-12 11:51 ` Lukas Laukamp
2012-10-12 12:59 ` Stefan Hajnoczi
2012-10-12 13:02 ` Lukas Laukamp
2012-10-12 14:25 ` Stefan Hajnoczi
2012-10-12 14:36 ` Javier Guerra Giraldez
2012-10-12 15:51 ` Lukas Laukamp
2012-10-12 19:18 ` Stefan Hajnoczi
2012-10-12 18:14 ` Lukas Laukamp
2012-10-12 19:13 ` Stefan Hajnoczi
2012-10-12 19:26 ` Lukas Laukamp
2012-10-12 20:43 ` Stefan Hajnoczi
2012-10-12 20:56 ` Lukas Laukamp
2012-10-12 21:53 ` Javier Guerra Giraldez
2012-10-13 10:28 ` Lukas Laukamp
2012-10-13 22:25 ` Lukas Laukamp
2012-10-15 3:06 ` Javier Guerra Giraldez
2012-10-15 6:17 ` Lukas Laukamp
2012-10-18 12:57 ` Avi Kivity
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.