* [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image [not found] <54c55fc6-b12a-42e5-b99d-9752c8a9b19b@mailpro> @ 2014-10-08 11:15 ` Alexandre DERUMIER 2014-10-11 7:01 ` Fam Zheng 2014-10-12 13:02 ` Paolo Bonzini 0 siblings, 2 replies; 11+ messages in thread From: Alexandre DERUMIER @ 2014-10-08 11:15 UTC (permalink / raw) To: qemu-devel; +Cc: Ceph Devel Hi, I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd). Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion. Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ? (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers). Regards, Alexandre Derumier ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 2014-10-08 11:15 ` [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image Alexandre DERUMIER @ 2014-10-11 7:01 ` Fam Zheng 2014-10-11 8:00 ` Alexandre DERUMIER 2014-10-12 13:02 ` Paolo Bonzini 1 sibling, 1 reply; 11+ messages in thread From: Fam Zheng @ 2014-10-11 7:01 UTC (permalink / raw) To: Alexandre DERUMIER; +Cc: Ceph Devel, qemu-devel On Wed, 10/08 13:15, Alexandre DERUMIER wrote: > Hi, > > I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror > > and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd). > > Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion. What is the source format? If the zero clusters are actually unallocated in the source image, drive-mirror will not write those clusters either. I.e. with "drive-mirror sync=top", both source and target should have the same "qemu-img map" output. Fam > > > Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ? > > (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers). > > > > Regards, > > Alexandre Derumier > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 2014-10-11 7:01 ` Fam Zheng @ 2014-10-11 8:00 ` Alexandre DERUMIER 2014-10-11 8:25 ` Fam Zheng 0 siblings, 1 reply; 11+ messages in thread From: Alexandre DERUMIER @ 2014-10-11 8:00 UTC (permalink / raw) To: Fam Zheng; +Cc: Ceph Devel, qemu-devel >>What is the source format? If the zero clusters are actually unallocated in the >>source image, drive-mirror will not write those clusters either. I.e. with >>"drive-mirror sync=top", both source and target should have the same "qemu-img >>map" output. Thanks for your reply, I had tried drive mirror (sync=full) with raw file (sparse) -> rbd (no sparse) rbd (sparse) -> rbd (no sparse) raw file (sparse) -> qcow2 on ext4 (sparse) rbd (sparse) -> raw on ext4 (sparse) Also I see that I have the same problem with target file format on xfs. raw file (sparse) -> qcow2 on xfs (no sparse) rbd (sparse) -> raw on xfs (no sparse) I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks. Or maybe this is because I'm using sync=full ? What is the difference between full and top ? ""sync": what parts of the disk image should be copied to the destination; possibilities include "full" for all the disk, "top" for only the sectors allocated in the topmost image". (what is topmost image ?) ----- Mail original ----- De: "Fam Zheng" <famz@redhat.com> À: "Alexandre DERUMIER" <aderumier@odiso.com> Cc: "qemu-devel" <qemu-devel@nongnu.org>, "Ceph Devel" <ceph-devel@vger.kernel.org> Envoyé: Samedi 11 Octobre 2014 09:01:18 Objet: Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image On Wed, 10/08 13:15, Alexandre DERUMIER wrote: > Hi, > > I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror > > and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd). > > Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion. What is the source format? If the zero clusters are actually unallocated in the source image, drive-mirror will not write those clusters either. I.e. with "drive-mirror sync=top", both source and target should have the same "qemu-img map" output. Fam > > > Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ? > > (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers). > > > > Regards, > > Alexandre Derumier > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 2014-10-11 8:00 ` Alexandre DERUMIER @ 2014-10-11 8:25 ` Fam Zheng 2014-10-11 8:30 ` Andrey Korolyov 2014-10-12 10:33 ` Alexandre DERUMIER 0 siblings, 2 replies; 11+ messages in thread From: Fam Zheng @ 2014-10-11 8:25 UTC (permalink / raw) To: Alexandre DERUMIER; +Cc: Ceph Devel, qemu-devel On Sat, 10/11 10:00, Alexandre DERUMIER wrote: > >>What is the source format? If the zero clusters are actually unallocated in the > >>source image, drive-mirror will not write those clusters either. I.e. with > >>"drive-mirror sync=top", both source and target should have the same "qemu-img > >>map" output. > > Thanks for your reply, > > I had tried drive mirror (sync=full) with > > raw file (sparse) -> rbd (no sparse) > rbd (sparse) -> rbd (no sparse) > raw file (sparse) -> qcow2 on ext4 (sparse) > rbd (sparse) -> raw on ext4 (sparse) > > Also I see that I have the same problem with target file format on xfs. > > raw file (sparse) -> qcow2 on xfs (no sparse) > rbd (sparse) -> raw on xfs (no sparse) > These don't tell me much. Maybe it's better to show the actual commands and how you tell sparse from no sparse? Does "qcow2 -> qcow2" work for you on xfs? > > I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks. > > > Or maybe this is because I'm using sync=full ? > > What is the difference between full and top ? > > ""sync": what parts of the disk image should be copied to the destination; > possibilities include "full" for all the disk, "top" for only the sectors > allocated in the topmost image". > > (what is topmost image ?) For "sync=top", only the clusters allocated in the image itself is copied; for "full", all those clusters allocated in the image itself, and its backing image, and it's backing's backing image, ..., are copied. The image itself, having a backing image or not, is called the topmost image. Fam ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 2014-10-11 8:25 ` Fam Zheng @ 2014-10-11 8:30 ` Andrey Korolyov 2014-10-12 10:02 ` Alexandre DERUMIER 2014-10-12 10:33 ` Alexandre DERUMIER 1 sibling, 1 reply; 11+ messages in thread From: Andrey Korolyov @ 2014-10-11 8:30 UTC (permalink / raw) To: Fam Zheng; +Cc: Ceph Devel, qemu-devel, Alexandre DERUMIER On Sat, Oct 11, 2014 at 12:25 PM, Fam Zheng <famz@redhat.com> wrote: > On Sat, 10/11 10:00, Alexandre DERUMIER wrote: >> >>What is the source format? If the zero clusters are actually unallocated in the >> >>source image, drive-mirror will not write those clusters either. I.e. with >> >>"drive-mirror sync=top", both source and target should have the same "qemu-img >> >>map" output. >> >> Thanks for your reply, >> >> I had tried drive mirror (sync=full) with >> >> raw file (sparse) -> rbd (no sparse) >> rbd (sparse) -> rbd (no sparse) >> raw file (sparse) -> qcow2 on ext4 (sparse) >> rbd (sparse) -> raw on ext4 (sparse) >> >> Also I see that I have the same problem with target file format on xfs. >> >> raw file (sparse) -> qcow2 on xfs (no sparse) >> rbd (sparse) -> raw on xfs (no sparse) >> > > These don't tell me much. Maybe it's better to show the actual commands and how > you tell sparse from no sparse? > > Does "qcow2 -> qcow2" work for you on xfs? > >> >> I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks. >> >> >> Or maybe this is because I'm using sync=full ? >> >> What is the difference between full and top ? >> >> ""sync": what parts of the disk image should be copied to the destination; >> possibilities include "full" for all the disk, "top" for only the sectors >> allocated in the topmost image". >> >> (what is topmost image ?) > > For "sync=top", only the clusters allocated in the image itself is copied; for > "full", all those clusters allocated in the image itself, and its backing > image, and it's backing's backing image, ..., are copied. > > The image itself, having a backing image or not, is called the topmost image. > > Fam > -- Just a wild guess - Alexandre, did you tried detect-zeroes blk option for mirroring targets? ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 2014-10-11 8:30 ` Andrey Korolyov @ 2014-10-12 10:02 ` Alexandre DERUMIER 0 siblings, 0 replies; 11+ messages in thread From: Alexandre DERUMIER @ 2014-10-12 10:02 UTC (permalink / raw) To: Andrey Korolyov; +Cc: Ceph Devel, Fam Zheng, qemu-devel >>Just a wild guess - Alexandre, did you tried detect-zeroes blk option >>for mirroring targets? Hi, yes, I have also tried with detect-zeroes (on or discard) with virtio and virtio-scsi, doesn't help. (I'm not sure that is implemtend in drive-mirror). As workaround currently, after drive-mirror, I doing fstrim inside the guest (with virtio-scsi + discard), and like this I can free space on rbd storage. ----- Mail original ----- De: "Andrey Korolyov" <andrey@xdel.ru> À: "Fam Zheng" <famz@redhat.com> Cc: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org>, "Ceph Devel" <ceph-devel@vger.kernel.org> Envoyé: Samedi 11 Octobre 2014 10:30:40 Objet: Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image On Sat, Oct 11, 2014 at 12:25 PM, Fam Zheng <famz@redhat.com> wrote: > On Sat, 10/11 10:00, Alexandre DERUMIER wrote: >> >>What is the source format? If the zero clusters are actually unallocated in the >> >>source image, drive-mirror will not write those clusters either. I.e. with >> >>"drive-mirror sync=top", both source and target should have the same "qemu-img >> >>map" output. >> >> Thanks for your reply, >> >> I had tried drive mirror (sync=full) with >> >> raw file (sparse) -> rbd (no sparse) >> rbd (sparse) -> rbd (no sparse) >> raw file (sparse) -> qcow2 on ext4 (sparse) >> rbd (sparse) -> raw on ext4 (sparse) >> >> Also I see that I have the same problem with target file format on xfs. >> >> raw file (sparse) -> qcow2 on xfs (no sparse) >> rbd (sparse) -> raw on xfs (no sparse) >> > > These don't tell me much. Maybe it's better to show the actual commands and how > you tell sparse from no sparse? > > Does "qcow2 -> qcow2" work for you on xfs? > >> >> I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks. >> >> >> Or maybe this is because I'm using sync=full ? >> >> What is the difference between full and top ? >> >> ""sync": what parts of the disk image should be copied to the destination; >> possibilities include "full" for all the disk, "top" for only the sectors >> allocated in the topmost image". >> >> (what is topmost image ?) > > For "sync=top", only the clusters allocated in the image itself is copied; for > "full", all those clusters allocated in the image itself, and its backing > image, and it's backing's backing image, ..., are copied. > > The image itself, having a backing image or not, is called the topmost image. > > Fam > -- Just a wild guess - Alexandre, did you tried detect-zeroes blk option for mirroring targets? ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 2014-10-11 8:25 ` Fam Zheng 2014-10-11 8:30 ` Andrey Korolyov @ 2014-10-12 10:33 ` Alexandre DERUMIER 1 sibling, 0 replies; 11+ messages in thread From: Alexandre DERUMIER @ 2014-10-12 10:33 UTC (permalink / raw) To: Fam Zheng; +Cc: Ceph Devel, qemu-devel >>These don't tell me much. Maybe it's better to show the actual commands and how >>you tell sparse from no sparse? Well, I create 2 empty source images files of 10G. (source.qcow2 and source.raw) then: du -sh source.qcow2 : 2M du -sh source.raw : 0M then I convert them with qemu-img convert : (source.qcow2 -> target.qcow2 , source.raw -> target.raw) du -sh target.qcow2 : 2M du -sh target.raw : 0M (So it's ok here) But If I convert them with drive-mirror: du -sh target.qcow2 : 11G du -sh target.raw : 11G I have also double check with #df, and I see space allocated on filesystem when using drive-mirror. I have the same behavior if target is a rbd storage. Also I have done test again ext3,ext4, and I see the same problem than with xfs. I'm pretty sure that drive-mirror copy zero block, qemu-img convert take around 2s to convert the empty file (because it's skipping zero block), and drive mirror take around 5min. ----- Mail original ----- De: "Fam Zheng" <famz@redhat.com> À: "Alexandre DERUMIER" <aderumier@odiso.com> Cc: "qemu-devel" <qemu-devel@nongnu.org>, "Ceph Devel" <ceph-devel@vger.kernel.org> Envoyé: Samedi 11 Octobre 2014 10:25:35 Objet: Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image On Sat, 10/11 10:00, Alexandre DERUMIER wrote: > >>What is the source format? If the zero clusters are actually unallocated in the > >>source image, drive-mirror will not write those clusters either. I.e. with > >>"drive-mirror sync=top", both source and target should have the same "qemu-img > >>map" output. > > Thanks for your reply, > > I had tried drive mirror (sync=full) with > > raw file (sparse) -> rbd (no sparse) > rbd (sparse) -> rbd (no sparse) > raw file (sparse) -> qcow2 on ext4 (sparse) > rbd (sparse) -> raw on ext4 (sparse) > > Also I see that I have the same problem with target file format on xfs. > > raw file (sparse) -> qcow2 on xfs (no sparse) > rbd (sparse) -> raw on xfs (no sparse) > These don't tell me much. Maybe it's better to show the actual commands and how you tell sparse from no sparse? Does "qcow2 -> qcow2" work for you on xfs? > > I only have this problem with drive-mirror, qemu-img convert seem to simply skip zero blocks. > > > Or maybe this is because I'm using sync=full ? > > What is the difference between full and top ? > > ""sync": what parts of the disk image should be copied to the destination; > possibilities include "full" for all the disk, "top" for only the sectors > allocated in the topmost image". > > (what is topmost image ?) For "sync=top", only the clusters allocated in the image itself is copied; for "full", all those clusters allocated in the image itself, and its backing image, and it's backing's backing image, ..., are copied. The image itself, having a backing image or not, is called the topmost image. Fam ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 2014-10-08 11:15 ` [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image Alexandre DERUMIER 2014-10-11 7:01 ` Fam Zheng @ 2014-10-12 13:02 ` Paolo Bonzini 2014-10-13 6:06 ` Alexandre DERUMIER 1 sibling, 1 reply; 11+ messages in thread From: Paolo Bonzini @ 2014-10-12 13:02 UTC (permalink / raw) To: Alexandre DERUMIER, qemu-devel; +Cc: Ceph Devel Il 08/10/2014 13:15, Alexandre DERUMIER ha scritto: > Hi, > > I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror > > and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd). > > Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion. > > > Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ? > > (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers). Lack of bdrv_co_write_zeroes is why detect-zeroes does not work. Lack of bdrv_get_block_status is why sparse->sparse does not work without detect-zeroes. Paolo ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 2014-10-12 13:02 ` Paolo Bonzini @ 2014-10-13 6:06 ` Alexandre DERUMIER 2014-10-13 7:06 ` Paolo Bonzini 0 siblings, 1 reply; 11+ messages in thread From: Alexandre DERUMIER @ 2014-10-13 6:06 UTC (permalink / raw) To: Paolo Bonzini; +Cc: Ceph Devel, qemu-devel >>Lack of bdrv_co_write_zeroes is why detect-zeroes does not work. >> >>Lack of bdrv_get_block_status is why sparse->sparse does not work >>without detect-zeroes. Ok, thanks Paolo ! Both are missing in rbd block driver. @ceph-devel . Could it be possible to implement them ? Also, about drive-mirror, I had tried with detect-zeroes with simple qcow2 file, and It don't seem to help. I'm not sure that detect-zeroes is implement in drive-mirror. also, the target mirrored volume don't seem to have the detect-zeroes option # info block drive-virtio1: /source.qcow2 (qcow2) Detect zeroes: on #du -sh source.qcow2 : 2M drive-mirror source.qcow2 -> target.qcow2 # info block drive-virtio1: /target.qcow2 (qcow2) #du -sh target.qcow2 : 11G ----- Mail original ----- De: "Paolo Bonzini" <pbonzini@redhat.com> À: "Alexandre DERUMIER" <aderumier@odiso.com>, "qemu-devel" <qemu-devel@nongnu.org> Cc: "Ceph Devel" <ceph-devel@vger.kernel.org> Envoyé: Dimanche 12 Octobre 2014 15:02:12 Objet: Re: qemu drive-mirror to rbd storage : no sparse rbd image Il 08/10/2014 13:15, Alexandre DERUMIER ha scritto: > Hi, > > I'm currently planning to migrate our storage to ceph/rbd through qemu drive-mirror > > and It seem that drive-mirror with rbd block driver, don't create a sparse image. (all zeros are copied to the target rbd). > > Also note, that it's working fine with "qemu-img convert" , the rbd volume is sparse after conversion. > > > Could it be related to the "bdrv_co_write_zeroes" missing features in block/rbd.c ? > > (It's available in other block drivers (scsi,gluster,raw-aio) , and I don't have this problem with theses block drivers). Lack of bdrv_co_write_zeroes is why detect-zeroes does not work. Lack of bdrv_get_block_status is why sparse->sparse does not work without detect-zeroes. Paolo ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 2014-10-13 6:06 ` Alexandre DERUMIER @ 2014-10-13 7:06 ` Paolo Bonzini 2014-10-13 8:08 ` Alexandre DERUMIER 0 siblings, 1 reply; 11+ messages in thread From: Paolo Bonzini @ 2014-10-13 7:06 UTC (permalink / raw) To: Alexandre DERUMIER; +Cc: Ceph Devel, qemu-devel Il 13/10/2014 08:06, Alexandre DERUMIER ha scritto: > > Also, about drive-mirror, I had tried with detect-zeroes with simple qcow2 file, > and It don't seem to help. > I'm not sure that detect-zeroes is implement in drive-mirror. > > also, the target mirrored volume don't seem to have the detect-zeroes option > > > # info block > drive-virtio1: /source.qcow2 (qcow2) > Detect zeroes: on > > #du -sh source.qcow2 : 2M > > drive-mirror source.qcow2 -> target.qcow2 > > # info block > drive-virtio1: /target.qcow2 (qcow2) > > #du -sh target.qcow2 : 11G > Ah, you're right. We need to add an options field, or use a new blockdev-mirror command. Paolo ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image 2014-10-13 7:06 ` Paolo Bonzini @ 2014-10-13 8:08 ` Alexandre DERUMIER 0 siblings, 0 replies; 11+ messages in thread From: Alexandre DERUMIER @ 2014-10-13 8:08 UTC (permalink / raw) To: Paolo Bonzini; +Cc: Ceph Devel, qemu-devel >>Ah, you're right. We need to add an options field, or use a new >>blockdev-mirror command. Ok, thanks. Can't help to implement this, but I'll glad to help for testing. ----- Mail original ----- De: "Paolo Bonzini" <pbonzini@redhat.com> À: "Alexandre DERUMIER" <aderumier@odiso.com> Cc: "Ceph Devel" <ceph-devel@vger.kernel.org>, "qemu-devel" <qemu-devel@nongnu.org> Envoyé: Lundi 13 Octobre 2014 09:06:01 Objet: Re: qemu drive-mirror to rbd storage : no sparse rbd image Il 13/10/2014 08:06, Alexandre DERUMIER ha scritto: > > Also, about drive-mirror, I had tried with detect-zeroes with simple qcow2 file, > and It don't seem to help. > I'm not sure that detect-zeroes is implement in drive-mirror. > > also, the target mirrored volume don't seem to have the detect-zeroes option > > > # info block > drive-virtio1: /source.qcow2 (qcow2) > Detect zeroes: on > > #du -sh source.qcow2 : 2M > > drive-mirror source.qcow2 -> target.qcow2 > > # info block > drive-virtio1: /target.qcow2 (qcow2) > > #du -sh target.qcow2 : 11G > Ah, you're right. We need to add an options field, or use a new blockdev-mirror command. Paolo ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2014-10-13 8:09 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <54c55fc6-b12a-42e5-b99d-9752c8a9b19b@mailpro>
2014-10-08 11:15 ` [Qemu-devel] qemu drive-mirror to rbd storage : no sparse rbd image Alexandre DERUMIER
2014-10-11 7:01 ` Fam Zheng
2014-10-11 8:00 ` Alexandre DERUMIER
2014-10-11 8:25 ` Fam Zheng
2014-10-11 8:30 ` Andrey Korolyov
2014-10-12 10:02 ` Alexandre DERUMIER
2014-10-12 10:33 ` Alexandre DERUMIER
2014-10-12 13:02 ` Paolo Bonzini
2014-10-13 6:06 ` Alexandre DERUMIER
2014-10-13 7:06 ` Paolo Bonzini
2014-10-13 8:08 ` Alexandre DERUMIER
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).