[Qemu-devel] [PATCH 0/4] Introduction of l2-cache-full option for qcow2 images

[Qemu-devel] [PATCH 0/4] Introduction of l2-cache-full option for qcow2 images

[Leonid Bloch]

This series introduces an option to calculate and allocate
automatically enough qcow2 L2 cache to cover the entire image.
Using cache that covers the entire image can benefit performance,
while having only a small memory overhead (just 1 MB for every 8 GB
of virtual image size with the default cluster size).

Leonid Bloch (4):
  qcow2: Introduce an option for sufficient L2 cache for the entire
    image
  docs: Fix an inaccuracy due to recent changes
  docs: Document the l2-cache-full option
  iotests: Add tests for the new l2-cache-full option

 block/qcow2.c              | 37 +++++++++++++++++++++++++++++--------
 block/qcow2.h              |  1 +
 docs/qcow2-cache.txt       | 18 +++++++++++-------
 qapi/block-core.json       |  4 ++++
 qemu-options.hx            |  4 ++++
 tests/qemu-iotests/103     |  6 ++++++
 tests/qemu-iotests/103.out |  4 +++-
 tests/qemu-iotests/137     |  2 ++
 tests/qemu-iotests/137.out |  4 +++-
 9 files changed, 63 insertions(+), 17 deletions(-)

-- 
2.14.1

[Qemu-devel] [PATCH 1/4] qcow2: Introduce an option for sufficient L2 cache for the entire image

[Leonid Bloch]

An option "l2-cache-full" is introduced to automatically set the qcow2
L2 cache to a sufficient value for covering the entire image. The memory
overhead when using this option is not big (1 MB for each 8 GB of
virtual image size with the default cluster size) and it can noticeably
improve performance when using large images with frequent I/O.
Previously, for this functionality the correct L2 cache size needed to
be calculated manually or with a script, and then this size needed to be
passed to the "l2-cache-size" option. Now it is sufficient to just pass
the boolean "l2-cache-full" option.

Signed-off-by: Leonid Bloch <lbloch@janustech.com>
---
 block/qcow2.c        | 37 +++++++++++++++++++++++++++++--------
 block/qcow2.h        |  1 +
 qapi/block-core.json |  4 ++++
 qemu-options.hx      |  4 ++++
 4 files changed, 38 insertions(+), 8 deletions(-)

diff --git a/block/qcow2.c b/block/qcow2.c
index 6162ed8be2..101b8b474b 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -695,6 +695,11 @@ static QemuOptsList qcow2_runtime_opts = {
             .type = QEMU_OPT_SIZE,
             .help = "Maximum L2 table cache size",
         },
+        {
+            .name = QCOW2_OPT_L2_CACHE_FULL,
+            .type = QEMU_OPT_BOOL,
+            .help = "Create full coverage of the image with the L2 cache",
+        },
         {
             .name = QCOW2_OPT_L2_CACHE_ENTRY_SIZE,
             .type = QEMU_OPT_SIZE,
@@ -779,10 +784,12 @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts,
     BDRVQcow2State *s = bs->opaque;
     uint64_t combined_cache_size;
     bool l2_cache_size_set, refcount_cache_size_set, combined_cache_size_set;
+    bool l2_cache_full_set;
     int min_refcount_cache = MIN_REFCOUNT_CACHE_SIZE * s->cluster_size;
 
     combined_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_CACHE_SIZE);
     l2_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_L2_CACHE_SIZE);
+    l2_cache_full_set = qemu_opt_get(opts, QCOW2_OPT_L2_CACHE_FULL);
     refcount_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_REFCOUNT_CACHE_SIZE);
 
     combined_cache_size = qemu_opt_get_size(opts, QCOW2_OPT_CACHE_SIZE, 0);
@@ -793,15 +800,32 @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts,
     *l2_cache_entry_size = qemu_opt_get_size(
         opts, QCOW2_OPT_L2_CACHE_ENTRY_SIZE, s->cluster_size);
 
+    uint64_t virtual_disk_size = bs->total_sectors * BDRV_SECTOR_SIZE;
+    uint64_t max_l2_cache = virtual_disk_size / (s->cluster_size / 8);
+
+    if (l2_cache_size_set && l2_cache_full_set) {
+        error_setg(errp, QCOW2_OPT_L2_CACHE_SIZE " and "
+                   QCOW2_OPT_L2_CACHE_FULL " may not be set at the same time");
+        return;
+    } else if (l2_cache_full_set) {
+        *l2_cache_size = max_l2_cache;
+    }
+
     if (combined_cache_size_set) {
         if (l2_cache_size_set && refcount_cache_size_set) {
             error_setg(errp, QCOW2_OPT_CACHE_SIZE ", " QCOW2_OPT_L2_CACHE_SIZE
                        " and " QCOW2_OPT_REFCOUNT_CACHE_SIZE " may not be set "
-                       "the same time");
+                       "at the same time");
             return;
         } else if (*l2_cache_size > combined_cache_size) {
-            error_setg(errp, QCOW2_OPT_L2_CACHE_SIZE " may not exceed "
-                       QCOW2_OPT_CACHE_SIZE);
+            if (l2_cache_full_set) {
+                error_setg(errp, QCOW2_OPT_CACHE_SIZE " must be greater than "
+                           "the full L2 cache if " QCOW2_OPT_L2_CACHE_FULL
+                           " is used");
+            } else {
+                error_setg(errp, QCOW2_OPT_L2_CACHE_SIZE " may not exceed "
+                           QCOW2_OPT_CACHE_SIZE);
+            }
             return;
         } else if (*refcount_cache_size > combined_cache_size) {
             error_setg(errp, QCOW2_OPT_REFCOUNT_CACHE_SIZE " may not exceed "
@@ -809,14 +833,11 @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts,
             return;
         }
 
-        if (l2_cache_size_set) {
+        if (l2_cache_size_set || l2_cache_full_set) {
             *refcount_cache_size = combined_cache_size - *l2_cache_size;
         } else if (refcount_cache_size_set) {
             *l2_cache_size = combined_cache_size - *refcount_cache_size;
         } else {
-            uint64_t virtual_disk_size = bs->total_sectors * BDRV_SECTOR_SIZE;
-            uint64_t max_l2_cache = virtual_disk_size / (s->cluster_size / 8);
-
             /* Assign as much memory as possible to the L2 cache, and
              * use the remainder for the refcount cache */
             if (combined_cache_size >= max_l2_cache + min_refcount_cache) {
@@ -829,7 +850,7 @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts,
             }
         }
     } else {
-        if (!l2_cache_size_set) {
+        if (!l2_cache_size_set && !l2_cache_full_set) {
             *l2_cache_size = MAX(DEFAULT_L2_CACHE_BYTE_SIZE,
                                  (uint64_t)DEFAULT_L2_CACHE_CLUSTERS
                                  * s->cluster_size);
diff --git a/block/qcow2.h b/block/qcow2.h
index 81b844e936..151e014bd8 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -97,6 +97,7 @@
 #define QCOW2_OPT_OVERLAP_BITMAP_DIRECTORY "overlap-check.bitmap-directory"
 #define QCOW2_OPT_CACHE_SIZE "cache-size"
 #define QCOW2_OPT_L2_CACHE_SIZE "l2-cache-size"
+#define QCOW2_OPT_L2_CACHE_FULL "l2-cache-full"
 #define QCOW2_OPT_L2_CACHE_ENTRY_SIZE "l2-cache-entry-size"
 #define QCOW2_OPT_REFCOUNT_CACHE_SIZE "refcount-cache-size"
 #define QCOW2_OPT_CACHE_CLEAN_INTERVAL "cache-clean-interval"
diff --git a/qapi/block-core.json b/qapi/block-core.json
index d40d5ecc3b..53c7d2efd8 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -2814,6 +2814,9 @@
 # @l2-cache-size:         the maximum size of the L2 table cache in
 #                         bytes (since 2.2)
 #
+# @l2-cache-full:         make the L2 table cache large enough to cover the
+#                         entire image (since 3.1)
+#
 # @l2-cache-entry-size:   the size of each entry in the L2 cache in
 #                         bytes. It must be a power of two between 512
 #                         and the cluster size. The default value is
@@ -2840,6 +2843,7 @@
             '*overlap-check': 'Qcow2OverlapChecks',
             '*cache-size': 'int',
             '*l2-cache-size': 'int',
+            '*l2-cache-full': 'bool',
             '*l2-cache-entry-size': 'int',
             '*refcount-cache-size': 'int',
             '*cache-clean-interval': 'int',
diff --git a/qemu-options.hx b/qemu-options.hx
index b1bf0f485f..be4d862795 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -758,6 +758,10 @@ The maximum total size of the L2 table and refcount block caches in bytes
 The maximum size of the L2 table cache in bytes
 (default: 4/5 of the total cache size)
 
+@item l2-cache-full
+Make the L2 table cache large enough to cover the entire image
+(on/off; default: off)
+
 @item refcount-cache-size
 The maximum size of the refcount block cache in bytes
 (default: 1/5 of the total cache size)
-- 
2.14.1

Re: [Qemu-devel] [PATCH 1/4] qcow2: Introduce an option for sufficient L2 cache for the entire image

[Eric Blake]

On 07/24/2018 07:17 AM, Leonid Bloch wrote:
> An option "l2-cache-full" is introduced to automatically set the qcow2
> L2 cache to a sufficient value for covering the entire image. The memory
> overhead when using this option is not big (1 MB for each 8 GB of
> virtual image size with the default cluster size) and it can noticeably
> improve performance when using large images with frequent I/O.
> Previously, for this functionality the correct L2 cache size needed to
> be calculated manually or with a script, and then this size needed to be
> passed to the "l2-cache-size" option. Now it is sufficient to just pass
> the boolean "l2-cache-full" option.
> 
> Signed-off-by: Leonid Bloch <lbloch@janustech.com>
> ---

> @@ -793,15 +800,32 @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts,
>       *l2_cache_entry_size = qemu_opt_get_size(
>           opts, QCOW2_OPT_L2_CACHE_ENTRY_SIZE, s->cluster_size);
>   
> +    uint64_t virtual_disk_size = bs->total_sectors * BDRV_SECTOR_SIZE;
> +    uint64_t max_l2_cache = virtual_disk_size / (s->cluster_size / 8);
> +
> +    if (l2_cache_size_set && l2_cache_full_set) {
> +        error_setg(errp, QCOW2_OPT_L2_CACHE_SIZE " and "
> +                   QCOW2_OPT_L2_CACHE_FULL " may not be set at the same time");
> +        return;
> +    } else if (l2_cache_full_set) {
> +        *l2_cache_size = max_l2_cache;
> +    }

Since this makes the options mutually exclusive...

> +++ b/qapi/block-core.json
> @@ -2814,6 +2814,9 @@
>   # @l2-cache-size:         the maximum size of the L2 table cache in
>   #                         bytes (since 2.2)
>   #
> +# @l2-cache-full:         make the L2 table cache large enough to cover the
> +#                         entire image (since 3.1)
> +#

...you should probably document that fact here.

> +++ b/qemu-options.hx
> @@ -758,6 +758,10 @@ The maximum total size of the L2 table and refcount block caches in bytes
>   The maximum size of the L2 table cache in bytes
>   (default: 4/5 of the total cache size)
>   
> +@item l2-cache-full
> +Make the L2 table cache large enough to cover the entire image
> +(on/off; default: off)

Likewise.

> +
>   @item refcount-cache-size
>   The maximum size of the refcount block cache in bytes
>   (default: 1/5 of the total cache size)

Alberto, looks like we missed this documentation in your changes to the 
defaults.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH 1/4] qcow2: Introduce an option for sufficient L2 cache for the entire image

[Alberto Garcia]

On Tue 24 Jul 2018 04:52:17 PM CEST, Eric Blake wrote:
>>   @item refcount-cache-size
>>   The maximum size of the refcount block cache in bytes
>>   (default: 1/5 of the total cache size)
>
> Alberto, looks like we missed this documentation in your changes to
> the defaults.

Oh, indeed!

Berto

[Qemu-devel] [PATCH 2/4] docs: Fix an inaccuracy due to recent changes

[Leonid Bloch]

Signed-off-by: Leonid Bloch <lbloch@janustech.com>
---
 docs/qcow2-cache.txt | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt
index 8a09a5cc5f..a0a1267482 100644
--- a/docs/qcow2-cache.txt
+++ b/docs/qcow2-cache.txt
@@ -97,9 +97,9 @@ need:
    l2_cache_size = disk_size_GB * 131072
    refcount_cache_size = disk_size_GB * 32768
 
-QEMU has a default L2 cache of 1MB (1048576 bytes) and a refcount
-cache of 256KB (262144 bytes), so using the formulas we've just seen
-we have
+QEMU has a default L2 cache of 1MB (1048576 bytes) or 8 clusters (whichever
+is larger) and a refcount cache of 256KB (262144 bytes), so using the
+formulas we've just seen we have (assuming the L2 cache is 1MB):
 
    1048576 / 131072 = 8 GB of virtual disk covered by that cache
     262144 /  32768 = 8 GB
-- 
2.14.1

Re: [Qemu-devel] [PATCH 2/4] docs: Fix an inaccuracy due to recent changes

[Eric Blake]

On 07/24/2018 07:17 AM, Leonid Bloch wrote:
> Signed-off-by: Leonid Bloch <lbloch@janustech.com>
> ---
>   docs/qcow2-cache.txt | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt
> index 8a09a5cc5f..a0a1267482 100644
> --- a/docs/qcow2-cache.txt
> +++ b/docs/qcow2-cache.txt
> @@ -97,9 +97,9 @@ need:
>      l2_cache_size = disk_size_GB * 131072
>      refcount_cache_size = disk_size_GB * 32768
>   
> -QEMU has a default L2 cache of 1MB (1048576 bytes) and a refcount
> -cache of 256KB (262144 bytes), so using the formulas we've just seen
> -we have
> +QEMU has a default L2 cache of 1MB (1048576 bytes) or 8 clusters (whichever
> +is larger) and a refcount cache of 256KB (262144 bytes), so using the

Looks suspicious; isn't the refcount cache size also dependent on the 
cluster size?

> +formulas we've just seen we have (assuming the L2 cache is 1MB):
>   
>      1048576 / 131072 = 8 GB of virtual disk covered by that cache
>       262144 /  32768 = 8 GB
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH 2/4] docs: Fix an inaccuracy due to recent changes

[Leonid Bloch]

   On 07/24/2018 06:20 PM, Eric Blake wrote:

     On 07/24/2018 07:17 AM, Leonid Bloch wrote:

     Signed-off-by: Leonid Bloch [1]<lbloch@janustech.com>
     ---
       docs/qcow2-cache.txt | 6 +++---
       1 file changed, 3 insertions(+), 3 deletions(-)
     diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt
     index 8a09a5cc5f..a0a1267482 100644
     --- a/docs/qcow2-cache.txt
     +++ b/docs/qcow2-cache.txt
     @@ -97,9 +97,9 @@ need:
          l2_cache_size = disk_size_GB * 131072
          refcount_cache_size = disk_size_GB * 32768
       -QEMU has a default L2 cache of 1MB (1048576 bytes) and a refcount
     -cache of 256KB (262144 bytes), so using the formulas we've just
     seen
     -we have
     +QEMU has a default L2 cache of 1MB (1048576 bytes) or 8 clusters
     (whichever
     +is larger) and a refcount cache of 256KB (262144 bytes), so using
     the

     Looks suspicious; isn't the refcount cache size also dependent on
     the cluster size?

   Yes, it's 4*cluster_size. But I realize now that the text here speaks
   about the default sizes, which means with with 64KB clusters. So this
   fix is not necessary. Will drop this patch.

     +formulas we've just seen we have (assuming the L2 cache is 1MB):
            1048576 / 131072 = 8 GB of virtual disk covered by that cache
           262144 /  32768 = 8 GB

References

   1. mailto:lbloch@janustech.com

Re: [Qemu-devel] [PATCH 2/4 for-3.0] docs: Fix an inaccuracy due to recent changes

[Eric Blake]

On 07/24/2018 07:17 AM, Leonid Bloch wrote:
> Signed-off-by: Leonid Bloch <lbloch@janustech.com>
> ---
>   docs/qcow2-cache.txt | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)

Updating the subject line, since this doc fix is appropriate for 3.0, if 
it is correct.

> 
> diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt
> index 8a09a5cc5f..a0a1267482 100644
> --- a/docs/qcow2-cache.txt
> +++ b/docs/qcow2-cache.txt
> @@ -97,9 +97,9 @@ need:
>      l2_cache_size = disk_size_GB * 131072
>      refcount_cache_size = disk_size_GB * 32768
>   
> -QEMU has a default L2 cache of 1MB (1048576 bytes) and a refcount
> -cache of 256KB (262144 bytes), so using the formulas we've just seen
> -we have
> +QEMU has a default L2 cache of 1MB (1048576 bytes) or 8 clusters (whichever
> +is larger) and a refcount cache of 256KB (262144 bytes), so using the
> +formulas we've just seen we have (assuming the L2 cache is 1MB):
>   
>      1048576 / 131072 = 8 GB of virtual disk covered by that cache
>       262144 /  32768 = 8 GB
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[Qemu-devel] [PATCH 3/4] docs: Document the l2-cache-full option

[Leonid Bloch]

Signed-off-by: Leonid Bloch <lbloch@janustech.com>
---
 docs/qcow2-cache.txt | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt
index a0a1267482..43c0faaddb 100644
--- a/docs/qcow2-cache.txt
+++ b/docs/qcow2-cache.txt
@@ -110,11 +110,12 @@ How to configure the cache sizes
 Cache sizes can be configured using the -drive option in the
 command-line, or the 'blockdev-add' QMP command.
 
-There are three options available, and all of them take bytes:
+There are four options available:
 
-"l2-cache-size":         maximum size of the L2 table cache
-"refcount-cache-size":   maximum size of the refcount block cache
-"cache-size":            maximum size of both caches combined
+"l2-cache-size":         maximum size of the L2 table cache (bytes, K, M)
+"refcount-cache-size":   maximum size of the refcount block cache (bytes, K, M)
+"cache-size":            maximum size of both caches combined (bytes, K, M)
+"l2-cache-full":         make the L2 cache cover the full image (boolean)
 
 There are a few things that need to be taken into account:
 
@@ -130,6 +131,9 @@ There are a few things that need to be taken into account:
    memory as possible to the L2 cache before increasing the refcount
    cache size.
 
+- If "l2-cache-full" is specified, QEMU will assign enough memory
+  to the L2 cache to cover the entire size of the image.
+
 Unlike L2 tables, refcount blocks are not used during normal I/O but
 only during allocations and internal snapshots. In most cases they are
 accessed sequentially (even during random guest I/O) so increasing the
-- 
2.14.1

Re: [Qemu-devel] [PATCH 3/4] docs: Document the l2-cache-full option

[Eric Blake]

On 07/24/2018 07:17 AM, Leonid Bloch wrote:
> Signed-off-by: Leonid Bloch <lbloch@janustech.com>
> ---
>   docs/qcow2-cache.txt | 12 ++++++++----
>   1 file changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt
> index a0a1267482..43c0faaddb 100644
> --- a/docs/qcow2-cache.txt
> +++ b/docs/qcow2-cache.txt
> @@ -110,11 +110,12 @@ How to configure the cache sizes
>   Cache sizes can be configured using the -drive option in the
>   command-line, or the 'blockdev-add' QMP command.
>   
> -There are three options available, and all of them take bytes:
> +There are four options available:
>   
> -"l2-cache-size":         maximum size of the L2 table cache
> -"refcount-cache-size":   maximum size of the refcount block cache
> -"cache-size":            maximum size of both caches combined
> +"l2-cache-size":         maximum size of the L2 table cache (bytes, K, M)
> +"refcount-cache-size":   maximum size of the refcount block cache (bytes, K, M)
> +"cache-size":            maximum size of both caches combined (bytes, K, M)
> +"l2-cache-full":         make the L2 cache cover the full image (boolean)
>   
>   There are a few things that need to be taken into account:
>   

Worth mentioning that the first three can be combined (although usually 
you'll never specify more than 2; as specifying all 3 requires that you 
perform the correct addition), but that the third is mutually exclusive?

> @@ -130,6 +131,9 @@ There are a few things that need to be taken into account:
>      memory as possible to the L2 cache before increasing the refcount
>      cache size.
>   
> +- If "l2-cache-full" is specified, QEMU will assign enough memory
> +  to the L2 cache to cover the entire size of the image.
> +
>   Unlike L2 tables, refcount blocks are not used during normal I/O but
>   only during allocations and internal snapshots. In most cases they are
>   accessed sequentially (even during random guest I/O) so increasing the
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH 3/4] docs: Document the l2-cache-full option

[Leonid Bloch]

   On 07/24/2018 06:21 PM, Eric Blake wrote:

     On 07/24/2018 07:17 AM, Leonid Bloch wrote:

     Signed-off-by: Leonid Bloch [1]<lbloch@janustech.com>
     ---
       docs/qcow2-cache.txt | 12 ++++++++----
       1 file changed, 8 insertions(+), 4 deletions(-)
     diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt
     index a0a1267482..43c0faaddb 100644
     --- a/docs/qcow2-cache.txt
     +++ b/docs/qcow2-cache.txt
     @@ -110,11 +110,12 @@ How to configure the cache sizes
       Cache sizes can be configured using the -drive option in the
       command-line, or the 'blockdev-add' QMP command.
       -There are three options available, and all of them take bytes:
     +There are four options available:
       -"l2-cache-size":         maximum size of the L2 table cache
     -"refcount-cache-size":   maximum size of the refcount block cache
     -"cache-size":            maximum size of both caches combined
     +"l2-cache-size":         maximum size of the L2 table cache (bytes,
     K, M)
     +"refcount-cache-size":   maximum size of the refcount block cache
     (bytes, K, M)
     +"cache-size":            maximum size of both caches combined
     (bytes, K, M)
     +"l2-cache-full":         make the L2 cache cover the full image
     (boolean)
         There are a few things that need to be taken into account:


     Worth mentioning that the first three can be combined (although
     usually you'll never specify more than 2; as specifying all 3
     requires that you perform the correct addition), but that the third
     is mutually exclusive?

   Actually, only 2 of the first 3 can be specified. But you're right, I
   need to write about mutual exclusivity in the docs. Will fix in v2.
   Thanks!

     @@ -130,6 +131,9 @@ There are a few things that need to be taken
     into account:
          memory as possible to the L2 cache before increasing the
     refcount
          cache size.
       +- If "l2-cache-full" is specified, QEMU will assign enough memory
     +  to the L2 cache to cover the entire size of the image.
     +
       Unlike L2 tables, refcount blocks are not used during normal I/O
     but
       only during allocations and internal snapshots. In most cases they
     are
       accessed sequentially (even during random guest I/O) so increasing
     the

References

   1. mailto:lbloch@janustech.com

[Qemu-devel] [PATCH 4/4] iotests: Add tests for the new l2-cache-full option

[Leonid Bloch]

This patch adds tests for the l2-cache-full option, and also intcoduces
a small change due to a grammar fix in block/qcow2.c

Signed-off-by: Leonid Bloch <lbloch@janustech.com>
---
 tests/qemu-iotests/103     | 6 ++++++
 tests/qemu-iotests/103.out | 4 +++-
 tests/qemu-iotests/137     | 2 ++
 tests/qemu-iotests/137.out | 4 +++-
 4 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/103 b/tests/qemu-iotests/103
index 2841318492..a2886e8569 100755
--- a/tests/qemu-iotests/103
+++ b/tests/qemu-iotests/103
@@ -52,9 +52,15 @@ echo
 echo '=== Testing invalid option combinations ==='
 echo
 
+# l2-cache-size and l2-cache-full at the same time
+$QEMU_IO -c "open -o l2-cache-full,l2-cache-size=1M $TEST_IMG" 2>&1 |
+    _filter_testdir | _filter_imgfmt
 # all sizes set at the same time
 $QEMU_IO -c "open -o cache-size=1.25M,l2-cache-size=1M,refcount-cache-size=0.25M $TEST_IMG" \
     2>&1 | _filter_testdir | _filter_imgfmt
+# cache-size may not be smaller than the full L2 size if l2-cache-full is used
+$QEMU_IO -c "open -o l2-cache-full,cache-size=6K $TEST_IMG" 2>&1 |
+    _filter_testdir | _filter_imgfmt
 # l2-cache-size may not exceed cache-size
 $QEMU_IO -c "open -o cache-size=1M,l2-cache-size=2M $TEST_IMG" 2>&1 \
     | _filter_testdir | _filter_imgfmt
diff --git a/tests/qemu-iotests/103.out b/tests/qemu-iotests/103.out
index bd45d3875a..92afbff024 100644
--- a/tests/qemu-iotests/103.out
+++ b/tests/qemu-iotests/103.out
@@ -5,7 +5,9 @@ wrote 65536/65536 bytes at offset 0
 
 === Testing invalid option combinations ===
 
-can't open device TEST_DIR/t.IMGFMT: cache-size, l2-cache-size and refcount-cache-size may not be set the same time
+can't open device TEST_DIR/t.IMGFMT: l2-cache-full and l2-cache-size may not be set at the same time
+can't open device TEST_DIR/t.IMGFMT: cache-size, l2-cache-size and refcount-cache-size may not be set at the same time
+can't open device TEST_DIR/t.IMGFMT: cache-size must be greater than the full L2 cache if l2-cache-full is used
 can't open device TEST_DIR/t.IMGFMT: l2-cache-size may not exceed cache-size
 can't open device TEST_DIR/t.IMGFMT: refcount-cache-size may not exceed cache-size
 can't open device TEST_DIR/t.IMGFMT: cache-size, l2-cache-size and refcount-cache-size may not be set the same time
diff --git a/tests/qemu-iotests/137 b/tests/qemu-iotests/137
index 87965625d8..f460b5bfe1 100755
--- a/tests/qemu-iotests/137
+++ b/tests/qemu-iotests/137
@@ -106,7 +106,9 @@ echo
 
 $QEMU_IO \
     -c "reopen -o lazy-refcounts=42" \
+    -c "reopen -o l2-cache-full,l2-cache-size=64k" \
     -c "reopen -o cache-size=1M,l2-cache-size=64k,refcount-cache-size=64k" \
+    -c "reopen -o l2-cache-full,cache-size=6K" \
     -c "reopen -o cache-size=1M,l2-cache-size=2M" \
     -c "reopen -o cache-size=1M,refcount-cache-size=2M" \
     -c "reopen -o l2-cache-size=256T" \
diff --git a/tests/qemu-iotests/137.out b/tests/qemu-iotests/137.out
index 96724a6c33..b15dfc391a 100644
--- a/tests/qemu-iotests/137.out
+++ b/tests/qemu-iotests/137.out
@@ -16,7 +16,9 @@ read 33554432/33554432 bytes at offset 0
 === Try setting some invalid values ===
 
 Parameter 'lazy-refcounts' expects 'on' or 'off'
-cache-size, l2-cache-size and refcount-cache-size may not be set the same time
+l2-cache-full and l2-cache-size may not be set at the same time
+cache-size, l2-cache-size and refcount-cache-size may not be set at the same time
+cache-size must be greater than the full L2 cache if l2-cache-full is used
 l2-cache-size may not exceed cache-size
 refcount-cache-size may not exceed cache-size
 L2 cache size too big
-- 
2.14.1

Re: [Qemu-devel] [PATCH 4/4] iotests: Add tests for the new l2-cache-full option

[Eric Blake]

On 07/24/2018 07:17 AM, Leonid Bloch wrote:
> This patch adds tests for the l2-cache-full option, and also intcoduces

s/intcoduces/introduces/

> a small change due to a grammar fix in block/qcow2.c
> 
> Signed-off-by: Leonid Bloch <lbloch@janustech.com>
> ---
>   tests/qemu-iotests/103     | 6 ++++++
>   tests/qemu-iotests/103.out | 4 +++-
>   tests/qemu-iotests/137     | 2 ++
>   tests/qemu-iotests/137.out | 4 +++-
>   4 files changed, 14 insertions(+), 2 deletions(-)
> 

> +++ b/tests/qemu-iotests/103.out
> @@ -5,7 +5,9 @@ wrote 65536/65536 bytes at offset 0
>   
>   === Testing invalid option combinations ===
>   
> -can't open device TEST_DIR/t.IMGFMT: cache-size, l2-cache-size and refcount-cache-size may not be set the same time
> +can't open device TEST_DIR/t.IMGFMT: l2-cache-full and l2-cache-size may not be set at the same time
> +can't open device TEST_DIR/t.IMGFMT: cache-size, l2-cache-size and refcount-cache-size may not be set at the same time

This should be squashed into the patch that introduces the changed 
output (when bisecting, we don't want to land on a known-broken iotests 
state).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH 0/4] Introduction of l2-cache-full option for qcow2 images

[Eric Blake]

On 07/24/2018 07:17 AM, Leonid Bloch wrote:
> This series introduces an option to calculate and allocate
> automatically enough qcow2 L2 cache to cover the entire image.
> Using cache that covers the entire image can benefit performance,
> while having only a small memory overhead (just 1 MB for every 8 GB
> of virtual image size with the default cluster size).

Is this still needed after Berto's work on commit 603790e and related, 
that maximizes the l2 cache by minimizing the refcount cache?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH 0/4] Introduction of l2-cache-full option for qcow2 images

[Leonid Bloch]

   On 07/24/2018 05:49 PM, Eric Blake wrote:

     On 07/24/2018 07:17 AM, Leonid Bloch wrote:

     This series introduces an option to calculate and allocate
     automatically enough qcow2 L2 cache to cover the entire image.
     Using cache that covers the entire image can benefit performance,
     while having only a small memory overhead (just 1 MB for every 8 GB
     of virtual image size with the default cluster size).

     Is this still needed after Berto's work on commit 603790e and
     related, that maximizes the l2 cache by minimizing the refcount
     cache?

   Yes. In the commits you refer to, the L2 cache was set to cover all the
   image size only in the case that the combined cache size is set, and is
   larger than max_l2_cache+min_refcount_cache. This series adds an option
   to set the L2 cache to cover the entire image without the need to
   calculate the required size externally, and without dependency on the
   combined cache size setting.