From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([209.51.188.92]:36937) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hIqMz-000860-De for qemu-devel@nongnu.org; Tue, 23 Apr 2019 04:01:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hIqMx-0001qk-Fj for qemu-devel@nongnu.org; Tue, 23 Apr 2019 04:01:37 -0400 Date: Tue, 23 Apr 2019 09:56:19 +0200 From: Kevin Wolf Message-ID: <20190423075619.GB9041@localhost.localdomain> References: <20190411105025.97397-2-sgarzare@redhat.com> <20190411130245.ltfyp7d2dfe2cr3h@steredhat> <20190414132008.uxoia6avdpp4jov6@steredhat.homenet.telecomitalia.it> <20190415080452.GA6031@localhost.localdomain> <20190417073438.r57lemi6emu4x3ld@steredhat> <20190417080443.GA8330@localhost.localdomain> <20190419122356.7dmwwdbtnzgimydw@steredhat> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190419122356.7dmwwdbtnzgimydw@steredhat> Subject: Re: [Qemu-devel] [PATCH RFC 1/1] block/rbd: increase dynamically the image size List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefano Garzarella Cc: dillaman@redhat.com, qemu-devel , Josh Durgin , qemu-block , Max Reitz Am 19.04.2019 um 14:23 hat Stefano Garzarella geschrieben: > Hi Kevin, > > On Wed, Apr 17, 2019 at 10:04:43AM +0200, Kevin Wolf wrote: > > Am 17.04.2019 um 09:34 hat Stefano Garzarella geschrieben: > > > On Mon, Apr 15, 2019 at 10:04:52AM +0200, Kevin Wolf wrote: > > > > > > > > I think a potential actual use case could be persistent dirty bitmaps > > > > for incremental backup. Though maybe that would be better served by > > > > using the rbd image just as a raw external data file and keeping the > > > > qcow2 metadata on a filesystem. > > > > > > Thanks to point it out! I'll take a look to understand how to keep > > > metadata separated from the data. > > > > I'd consider the feature still experimental, but for local files, it > > works like this: > > > > qemu-img create -f qcow2 -o data_file=test.raw test.qcow2 4G > > > > And then just use test.qcow2. As long as you can put everything you need > > into an rbd URL, the same approach should work. Otherwise, you may need > > to use QMP blockdev-create on creation and possibly the data-file option > > of the qcow2 driver for opening. > > Very interesting, I'll try to add this support also in the rbd driver. I don't understand - what is the thing you want to add to the rbd driver? qcow2 doesn't need special protocol driver support for doing this, and I don't think the QEMU rbd driver has any metadata that could be split off. > > > > How fast is rbd_resize()? Does automatically resizing for every write > > > > request actually work reasonably well in practice? If it does, there is > > > > probably little reason not to allow it, even if the use cases are rather > > > > obscure. > > > > > > I'll try to measure the percentage of the time spent in the rbd_resize. > > > > > > Another solution could be to pass to the rbd driver the virtual size of > > > the image and resize it only one time also if the preallocation is > > > disabled, because RBD will not allocate blocks but IIUC it only set the max > > > size. > > > > > > Do you think make sense? Is it feasible? > > > > In theory yes, though it requires modification of every driver that > > should be usable together with rbd (i.e. ideally all of the drivers). If > > automatic resize works good enough, I'd prefer that > > I did some tests and it seems that the cost of rbd_resize() is > negligible. IIUC it only updates the metadata without allocating any > blocks (if we are growing, like that case). > > Anyway the automatic resize will not affect the current use-case (raw > images on rbd), where the file size is set during the creation, so I > think there should not be side effects with this patch. Okay, sounds good. > I'm also adding the support for preallocation (i.e. full) in the rbd > driver that can be useful for qcow2 images. > > If you prefer I can resend this patch with the preallocation series. Let's keep seperate things separate. Huge patch series are always harder to handle. Kevin From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F16EDC10F14 for ; Tue, 23 Apr 2019 08:04:08 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C6D5C2077C for ; Tue, 23 Apr 2019 08:04:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C6D5C2077C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([127.0.0.1]:49738 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hIqPQ-0000nP-4P for qemu-devel@archiver.kernel.org; Tue, 23 Apr 2019 04:04:08 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36937) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hIqMz-000860-De for qemu-devel@nongnu.org; Tue, 23 Apr 2019 04:01:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hIqMx-0001qk-Fj for qemu-devel@nongnu.org; Tue, 23 Apr 2019 04:01:37 -0400 Received: from mx1.redhat.com ([209.132.183.28]:56468) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hIqI2-0007uY-Nb; Tue, 23 Apr 2019 03:56:30 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 87AD666968; Tue, 23 Apr 2019 07:56:26 +0000 (UTC) Received: from localhost.localdomain (ovpn-116-143.ams2.redhat.com [10.36.116.143]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D6CC91001DE6; Tue, 23 Apr 2019 07:56:20 +0000 (UTC) Date: Tue, 23 Apr 2019 09:56:19 +0200 From: Kevin Wolf To: Stefano Garzarella Message-ID: <20190423075619.GB9041@localhost.localdomain> References: <20190411105025.97397-2-sgarzare@redhat.com> <20190411130245.ltfyp7d2dfe2cr3h@steredhat> <20190414132008.uxoia6avdpp4jov6@steredhat.homenet.telecomitalia.it> <20190415080452.GA6031@localhost.localdomain> <20190417073438.r57lemi6emu4x3ld@steredhat> <20190417080443.GA8330@localhost.localdomain> <20190419122356.7dmwwdbtnzgimydw@steredhat> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Disposition: inline In-Reply-To: <20190419122356.7dmwwdbtnzgimydw@steredhat> User-Agent: Mutt/1.11.3 (2019-02-01) X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Tue, 23 Apr 2019 07:56:26 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: Re: [Qemu-devel] [PATCH RFC 1/1] block/rbd: increase dynamically the image size X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Josh Durgin , dillaman@redhat.com, qemu-devel , qemu-block , Max Reitz Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Message-ID: <20190423075619.VXHvG9Liht_Jku8hLj2dNlcLkuXQG1l2TlzHhgJIrJI@z> Am 19.04.2019 um 14:23 hat Stefano Garzarella geschrieben: > Hi Kevin, > > On Wed, Apr 17, 2019 at 10:04:43AM +0200, Kevin Wolf wrote: > > Am 17.04.2019 um 09:34 hat Stefano Garzarella geschrieben: > > > On Mon, Apr 15, 2019 at 10:04:52AM +0200, Kevin Wolf wrote: > > > > > > > > I think a potential actual use case could be persistent dirty bitmaps > > > > for incremental backup. Though maybe that would be better served by > > > > using the rbd image just as a raw external data file and keeping the > > > > qcow2 metadata on a filesystem. > > > > > > Thanks to point it out! I'll take a look to understand how to keep > > > metadata separated from the data. > > > > I'd consider the feature still experimental, but for local files, it > > works like this: > > > > qemu-img create -f qcow2 -o data_file=test.raw test.qcow2 4G > > > > And then just use test.qcow2. As long as you can put everything you need > > into an rbd URL, the same approach should work. Otherwise, you may need > > to use QMP blockdev-create on creation and possibly the data-file option > > of the qcow2 driver for opening. > > Very interesting, I'll try to add this support also in the rbd driver. I don't understand - what is the thing you want to add to the rbd driver? qcow2 doesn't need special protocol driver support for doing this, and I don't think the QEMU rbd driver has any metadata that could be split off. > > > > How fast is rbd_resize()? Does automatically resizing for every write > > > > request actually work reasonably well in practice? If it does, there is > > > > probably little reason not to allow it, even if the use cases are rather > > > > obscure. > > > > > > I'll try to measure the percentage of the time spent in the rbd_resize. > > > > > > Another solution could be to pass to the rbd driver the virtual size of > > > the image and resize it only one time also if the preallocation is > > > disabled, because RBD will not allocate blocks but IIUC it only set the max > > > size. > > > > > > Do you think make sense? Is it feasible? > > > > In theory yes, though it requires modification of every driver that > > should be usable together with rbd (i.e. ideally all of the drivers). If > > automatic resize works good enough, I'd prefer that > > I did some tests and it seems that the cost of rbd_resize() is > negligible. IIUC it only updates the metadata without allocating any > blocks (if we are growing, like that case). > > Anyway the automatic resize will not affect the current use-case (raw > images on rbd), where the file size is set during the creation, so I > think there should not be side effects with this patch. Okay, sounds good. > I'm also adding the support for preallocation (i.e. full) in the rbd > driver that can be useful for qcow2 images. > > If you prefer I can resend this patch with the preallocation series. Let's keep seperate things separate. Huge patch series are always harder to handle. Kevin