Re: [Qemu-devel] Estimation of qcow2 image size converted from raw image

[Stefan Hajnoczi <stefanha@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3453 bytes --]

On Wed, Feb 22, 2017 at 06:15:47PM +0200, Maor Lipchuk wrote:
> Hi all,
> 
> Thank you very much for your help, it was much helpful
> We adopted John Snow advice and implemented our own calculation so we
> can resolve the issue now,
> We plan to drop this code once we can get this estimation from qemu-img.
> 
> This is the link to the patch introducing the calculation:
> https://gerrit.ovirt.org/#/c/65039/14/lib/vdsm/storage/qcow2.py
> 
> And here are link to the tests that we added:
> https://gerrit.ovirt.org/#/c/65039/14/tests/storage_qcow2_test.py
> 
> Here is how the calculation goes:
> 
> We first use qemuimg map to get the used clusters and count all the
> clusters for each run returned from qemuimg.map(filename):
> 
>  def count_clusters(runs):
>     count = 0
>      last = -1
>      for r in runs:
>          # Find the cluster when start and end are located.
>          start = r["start"] // CLUSTER_SIZE
>          end = (r["start"] + r["length"]) // CLUSTER_SIZE
>          if r["data"]:
>              if start == end:
>                  # This run is smaller then a cluster. If we have several runs
>                  # in the same cluster, we want to count the cluster only once.
>                  if start != last:
>                      count += 1
>              else:
>                  # This run span over multiple clusters - we want to count all
>                  # the clusters this run touch.
>                  count += end - start
>              last = end
>      return count
> 
> 
> The following calculation is based on Kevin's comments on the original
> bug, and qcow2 spec:
> https://github.com/qemu/qemu/blob/master/docs/specs/qcow2.txt:
> 
>      header_size = 3 * CLUSTER_SIZE

Are you including the L1 table in these 3 clusters?

>      virtual_size = os.stat(filename).st_size

This assumes the input file is in raw format.  If the input file is in
vmdk, vhdx, qcow2, etc then the POSIX file size does not represent the
virtual disk size.

> 
>      # Each 512MiB has one l2 table (64K)
>      l2_tables = (virtual_size + (512 * 1024**2) - 1) // (512 * 1024**2)
>      l2_tables_size = l2_tables * CLUSTER_SIZE
> 
>      # Each cluster have a refcount entry (16 bits) in the refcount tables. It
>      # is not clear what is the size of the refcount table, lets assume it is
>      # the same size as the l2 tables.
>      refcounts_tables_size = l2_tables_size

There is a formula for calculating refcount blocks in qcow2_create2():

  /* total size of refcount blocks
   *
   * note: every host cluster is reference-counted, including metadata
   * (even refcount blocks are recursively included).
   * Let:
   *   a = total_size (this is the guest disk size)
   *   m = meta size not including refcount blocks and refcount tables
   *   c = cluster size
   *   y1 = number of refcount blocks entries
   *   y2 = meta size including everything
   *   rces = refcount entry size in bytes
   * then,
   *   y1 = (y2 + a)/c
   *   y2 = y1 * rces + y1 * rces * sizeof(u64) / c + m
   * we can get y1:
   *   y1 = (a + m) / (c - rces - rces * sizeof(u64) / c)
   */
  nrefblocke = (aligned_total_size + meta_size + cluster_size)
             / (cluster_size - rces - rces * sizeof(uint64_t)
                                           / cluster_size);
  meta_size += DIV_ROUND_UP(nrefblocke, refblock_size) * cluster_size;

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]