Re: QEMU RBD is slow with QCOW2 images

[Stefano Garzarella <sgarzare@redhat.com>]

On Wed, Mar 03, 2021 at 10:26:12PM +0100, Peter Lieven wrote:
>Am 03.03.21 um 19:47 schrieb Jason Dillaman:
>> On Wed, Mar 3, 2021 at 12:41 PM Stefano Garzarella <sgarzare@redhat.com> wrote:
>>> Hi Jason,
>>> as reported in this BZ [1], when qemu-img creates a QCOW2 image on RBD
>>> writing data is very slow compared to a raw file.
>>>
>>> Comparing raw vs QCOW2 image creation with RBD I found that we use a
>>> different object size, for the raw file I see '4 MiB objects', for QCOW2
>>> I see '64 KiB objects' as reported on comment 14 [2].
>>> This should be the main issue of slowness, indeed forcing in the code 4
>>> MiB object size also for QCOW2 increased the speed a lot.
>>>
>>> Looking better I discovered that for raw files, we call rbd_create()
>>> with obj_order = 0 (if 'cluster_size' options is not defined), so the
>>> default object size is used.
>>> Instead for QCOW2, we use obj_order = 16, since the default
>>> 'cluster_size' defined for QCOW2, is 64 KiB.
>>>
>>> Using '-o cluster_size=2M' with qemu-img changed only the qcow2 cluster
>>> size, since in qcow2_co_create_opts() we remove the 'cluster_size' from
>>> QemuOpts calling qemu_opts_to_qdict_filtered().
>>> For some reason that I have yet to understand, after this deletion,
>>> however remains in QemuOpts the default value of 'cluster_size' for
>>> qcow2 (64 KiB), that it's used in qemu_rbd_co_create_opts()
>>>
>>> At this point my doubts are:
>>> Does it make sense to use the same cluster_size as qcow2 as object_size
>>> in RBD?
>> No, not really. But it also doesn't really make any sense to put a
>> QCOW2 image within an RBD image. To clarify from the BZ, OpenStack
>> does not put QCOW2 images on RBD, it converts QCOW2 images into raw
>> images to store in RBD.
>
>
>As discussed earlier the only reasonable format for rbd image is raw.
>What is the idea behind putting a qcow2 on an rbd pool?
>Jason and I even discussed shortly durign the review of the rbd driver 
>rewrite I posted
>earlier if it was ok to drop support for writing past the end of file.
>
>Anyway the reason why it is so slow is that write requests serialize if the
>qcow2 file grows. If there is a sane reason why we need qcow2 on rbd
>we need to implement at least preallocation mode = full to overcome
>the serialization.

Agree, at most we could deprecate it by printing a message and then 
remove the support in future releases.

Thanks,
Stefano