[Qemu-devel] [PATCH 0/3] qemu-iotests: Variable cluster size

[Qemu-devel] [PATCH 0/3] qemu-iotests: Variable cluster size

[Kevin Wolf]

Currently, all qcow2 tests align their test requests to hit the critical spots
in images that use 4k clusters. By default we are using 64k clusters now for a
while, so we need to make the tests more dynamic.

This patch series makes the cluster size variable in some of the generic code
like common.pattern and adds a test case for the non-4k cluster case.

Kevin Wolf (3):
  qemu-iotests: Align test requests according to cluster size
  qemu-iotests: 019: Make cluster size dynamic
  qemu-iotests: Tests on non-4k clusters

 013            |    9 +-
 013.out        |  682 +-
 014            |    6 +-
 014.out        |  818 +-
 019            |   13 +-
 019.out        |    4 +-
 022            |    5 +-
 022.out        |  138 +-
 023            |  119 +
 023.out        |25706 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 common.pattern |   69 +-
 common.rc      |    4 +
 group          |    1 +
 13 files changed, 26709 insertions(+), 865 deletions(-)
 mode change 100644 => 100755 021
 create mode 100755 023
 create mode 100644 023.out

[Qemu-devel] [PATCH 1/3] qemu-iotests: Align test requests according to cluster size

[Kevin Wolf]

Change the io_test and io_test2 functions to take the cluster size of the image
and the number of test requests to issue. Tests are changed to specify a
cluster size (usually 4k), but expected test results stay the same for now
(apart from qemu-img printing the cluster size now).

Based on a patch written by Christoph Hellwig.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 013            |    9 ++++---
 013.out        |    2 +-
 014            |    6 +++-
 014.out        |    2 +-
 019            |    1 +
 019.out        |    4 +-
 022            |    5 ++-
 022.out        |    2 +-
 common.pattern |   68 +++++++++++++++++++++++++++++++------------------------
 common.rc      |    4 +++
 10 files changed, 60 insertions(+), 43 deletions(-)

diff --git a/013 b/013
index 565071f..bd9fdef 100755
--- a/013
+++ b/013
@@ -1,6 +1,6 @@
 #!/bin/sh
 #
-# qcow2 pattern test, empty and compressed image
+# qcow2 pattern test, empty and compressed image - 4k cluster patterns
 #
 # Copyright (C) 2009 Red Hat, Inc.
 #
@@ -45,6 +45,7 @@ _supported_os Linux
 
 TEST_OFFSETS="0 4294967296"
 TEST_OPS="writev read write readv"
+CLUSTER_SIZE=4096
 
 _make_test_img 6G
 
@@ -54,7 +55,7 @@ echo
 for offset in $TEST_OFFSETS; do
     echo "At offset $offset:"
     for op in $TEST_OPS; do
-        io_test $op $offset
+        io_test $op $offset $CLUSTER_SIZE 8
     done
     _check_test_img
 done
@@ -72,7 +73,7 @@ echo
 for offset in $TEST_OFFSETS; do
     echo "With offset $offset:"
     for op in read readv; do
-        io_test $op $offset
+        io_test $op $offset $CLUSTER_SIZE 8
     done
     _check_test_img
 done
@@ -85,7 +86,7 @@ for offset in $TEST_OFFSETS; do
     offset=$((offset + 512))
     echo "With offset $offset:"
     for op in $TEST_OPS; do
-        io_test $op $offset
+        io_test $op $offset $CLUSTER_SIZE 8
     done
     _check_test_img
 done
diff --git a/013.out b/013.out
index 01d306d..4072e8d 100644
--- a/013.out
+++ b/013.out
@@ -1,5 +1,5 @@
 QA output created by 013
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=6442450944 
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=6442450944 cluster_size=4096 
 Testing empty image
 
 At offset 0:
diff --git a/014 b/014
index d7f50af..ee7bf21 100755
--- a/014
+++ b/014
@@ -1,6 +1,7 @@
 #!/bin/sh
 #
 # qcow2 pattern test, complex patterns including compression and snapshots
+# Using patterns for 4k cluster size.
 #
 #
 # Copyright (C) 2009 Red Hat, Inc.
@@ -46,13 +47,14 @@ _supported_os Linux
 
 TEST_OFFSETS="0 4294967296"
 TEST_OPS="writev read write readv"
+CLUSTER_SIZE=4096
 
 _make_test_img 6G
 
 echo "Testing empty image:"
 for offset in $TEST_OFFSETS; do
     echo test2: With offset $offset
-    io_test2 $offset
+    io_test2 $offset $CLUSTER_SIZE 256
     _check_test_img
 done
 
@@ -62,7 +64,7 @@ for i in `seq 1 3`; do
     for offset in $TEST_OFFSETS; do
         echo With snapshot test$i, offset $offset
         for op in $TEST_OPS; do
-            io_test $op $offset
+            io_test $op $offset $CLUSTER_SIZE 8
         done
         _check_test_img
     done
diff --git a/014.out b/014.out
index b2c0176..3fbc3a5 100644
--- a/014.out
+++ b/014.out
@@ -1,5 +1,5 @@
 QA output created by 014
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=6442450944 
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=6442450944 cluster_size=4096 
 Testing empty image:
 test2: With offset 0
 === Clusters to be compressed [1]
diff --git a/019 b/019
index 0c84d2f..c3bf950 100755
--- a/019
+++ b/019
@@ -48,6 +48,7 @@ _supported_fmt qcow qcow2 vmdk
 _supported_os Linux
 
 TEST_OFFSETS="0 4294967296"
+CLUSTER_SIZE=65536
 
 _make_test_img 6G
 
diff --git a/019.out b/019.out
index 9ac1a21..ef82d02 100644
--- a/019.out
+++ b/019.out
@@ -1,5 +1,5 @@
 QA output created by 019
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=6442450944 
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=6442450944 cluster_size=65536 
 Filling base image
 
 === IO: pattern 0
@@ -269,7 +269,7 @@ qemu-io> wrote 65536/65536 bytes at offset 4296081408
 qemu-io> No errors were found on the image.
 Creating test image with backing file
 
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=6442450944 backing_file='TEST_DIR/t.IMGFMT.base' 
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=6442450944 backing_file='TEST_DIR/t.IMGFMT.base' cluster_size=65536 
 Filling test image
 
 === IO: pattern 1
diff --git a/022 b/022
index ca5df99..ccf2319 100755
--- a/022
+++ b/022
@@ -46,6 +46,7 @@ _supported_fmt qcow2
 _supported_os Linux
 
 TEST_OFFSETS="10485760 4294967296"
+CLUSTER_SIZE="4096"
 
 _make_test_img 6G
 
@@ -54,8 +55,8 @@ echo
 
 for offset in $TEST_OFFSETS; do
     echo "At offset $offset:"
-    io_test "write -b" $offset
-    io_test "read -b" $offset
+    io_test "write -b" $offset $CLUSTER_SIZE 8
+    io_test "read -b" $offset $CLUSTER_SIZE 8
     _check_test_img
 done
 
diff --git a/022.out b/022.out
index 0100f67..d1a610d 100644
--- a/022.out
+++ b/022.out
@@ -1,5 +1,5 @@
 QA output created by 022
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=6442450944 
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=6442450944 cluster_size=4096 
 Testing empty image
 
 At offset 10485760:
diff --git a/common.pattern b/common.pattern
index 08e1cda..ae28863 100644
--- a/common.pattern
+++ b/common.pattern
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
 #
 # Copyright (C) 2009 Red Hat, Inc.
 #
@@ -63,69 +63,77 @@ function io_zero() {
 function io_test() {
     local op=$1
     local offset=$2
+    local cluster_size=$3
 
-    # Complete clusters (size = 4k)
-    io "$op" $offset 4096 4096 256
-    offset=$((offset + 256 * 4096))
+    local num_large=$4
+    local num_medium=$((num_large * num_large))
+    local num_small=$((4 * num_medium))
+
+    local half_cluster=$((cluster_size / 2))
+    local quarter_cluster=$((cluster_size / 4))
+    local l2_size=$((cluster_size * cluster_size / 8))
+
+    # Complete clusters
+    io "$op" $offset $cluster_size $cluster_size $num_small
+    offset=$((offset + num_small * $cluster_size))
 
     # From somewhere in the middle to the end of a cluster
-    io "$op" $((offset + 2048)) 2048 4096 256
-    offset=$((offset + 256 * 4096))
+    io "$op" $((offset + $half_cluster)) $half_cluster $cluster_size $num_small
+    offset=$((offset + num_small * $cluster_size))
 
     # From the start to somewhere in the middle of a cluster
-    io "$op" $offset 2048 4096 256
-    offset=$((offset + 256 * 4096))
+    io "$op" $offset $half_cluster $cluster_size $num_small
+    offset=$((offset + num_small * $cluster_size))
 
     # Completely misaligned (and small)
-    io "$op" $((offset + 1024)) 2048 4096 256
-    offset=$((offset + 256 * 4096))
+    io "$op" $((offset + $quarter_cluster)) $half_cluster $cluster_size $num_small
+    offset=$((offset + num_small * $cluster_size))
 
     # Spanning multiple clusters
-    io "$op" $((offset + 2048)) 8192 12288 64
-    offset=$((offset + 64 * 12288))
+    io "$op" $((offset + $half_cluster)) $((cluster_size * 2)) $((cluster_size * 3)) $num_medium
+    offset=$((offset + num_medium * 3 * $cluster_size))
 
     # Spanning multiple L2 tables
     # L2 table size: 512 clusters of 4k = 2M
-    io "$op" $((offset + 2048)) 4194304 4999680 8
-    offset=$((offset + 8 * 4999680))
-
-    if false; then
-        true
-    fi
+    io "$op" $((offset + $half_cluster)) $((2 * l2_size)) 4999680 $num_large
+    offset=$((offset + num_large * (2 * l2_size + 512 * 1573)))
 }
 
 function io_test2() {
     local orig_offset=$1
+    local cluster_size=$2
+    local num=$3
 
     # Pattern (repeat after 9 clusters):
-    # used - used - free - used - compressed - compressed - free - free - compressed
+    #	used - used - free - used - compressed - compressed -
+    #	free - free - compressed
 
     # Write the clusters to be compressed
     echo === Clusters to be compressed [1]
-    io_pattern writev $((offset + 4 * 4096)) 4096 $((9 * 4096)) 256 165
+    io_pattern writev $((offset + 4 * $cluster_size)) $cluster_size $((9 * $cluster_size)) $num 165
     echo === Clusters to be compressed [2]
-    io_pattern writev $((offset + 5 * 4096)) 4096 $((9 * 4096)) 256 165
+    io_pattern writev $((offset + 5 * $cluster_size)) $cluster_size $((9 * $cluster_size)) $num 165
     echo === Clusters to be compressed [3]
-    io_pattern writev $((offset + 8 * 4096)) 4096 $((9 * 4096)) 256 165
+    io_pattern writev $((offset + 8 * $cluster_size)) $cluster_size $((9 * $cluster_size)) $num 165
 
     mv $TEST_IMG $TEST_IMG.orig
     $QEMU_IMG convert -f $IMGFMT -O $IMGFMT -c $TEST_IMG.orig $TEST_IMG
 
     # Write the used clusters
     echo === Used clusters [1]
-    io_pattern writev $((offset + 0 * 4096)) 4096 $((9 * 4096)) 256 165
+    io_pattern writev $((offset + 0 * $cluster_size)) $cluster_size $((9 * $cluster_size)) $num 165
     echo === Used clusters [2]
-    io_pattern writev $((offset + 1 * 4096)) 4096 $((9 * 4096)) 256 165
+    io_pattern writev $((offset + 1 * $cluster_size)) $cluster_size $((9 * $cluster_size)) $num 165
     echo === Used clusters [3]
-    io_pattern writev $((offset + 3 * 4096)) 4096 $((9 * 4096)) 256 165
+    io_pattern writev $((offset + 3 * $cluster_size)) $cluster_size $((9 * $cluster_size)) $num 165
 
     # Read them
     echo === Read used/compressed clusters
-    io_pattern readv $((offset + 0 * 4096)) $((2 * 4096)) $((9 * 4096)) 256 165
-    io_pattern readv $((offset + 3 * 4096)) $((3 * 4096)) $((9 * 4096)) 256 165
-    io_pattern readv $((offset + 8 * 4096)) $((1 * 4096)) $((9 * 4096)) 256 165
+    io_pattern readv $((offset + 0 * $cluster_size)) $((2 * $cluster_size)) $((9 * $cluster_size)) $num 165
+    io_pattern readv $((offset + 3 * $cluster_size)) $((3 * $cluster_size)) $((9 * $cluster_size)) $num 165
+    io_pattern readv $((offset + 8 * $cluster_size)) $((1 * $cluster_size)) $((9 * $cluster_size)) $num 165
 
     echo === Read zeros
-    io_zero readv $((offset + 2 * 4096)) $((1 * 4096)) $((9 * 4096)) 256
-    io_zero readv $((offset + 6 * 4096)) $((2 * 4096)) $((9 * 4096)) 256
+    io_zero readv $((offset + 2 * $cluster_size)) $((1 * $cluster_size)) $((9 * $cluster_size)) $num
+    io_zero readv $((offset + 6 * $cluster_size)) $((2 * $cluster_size)) $((9 * $cluster_size)) $num
 }
diff --git a/common.rc b/common.rc
index 6ef6b51..0a7ba00 100644
--- a/common.rc
+++ b/common.rc
@@ -55,6 +55,10 @@ _make_test_img()
     # at least one argument (the image size) needs to be added
     local extra_img_options=$*
 
+    if [ "$IMGFMT" = "qcow2" -a -n "$CLUSTER_SIZE" ]; then
+        extra_img_options="-o cluster_size=$CLUSTER_SIZE $extra_img_options"
+    fi
+
     # XXX(hch): have global image options?
     $QEMU_IMG create -f $IMGFMT $TEST_IMG $extra_img_options | \
     	sed -e "s#$TEST_DIR#TEST_DIR#g" | \
-- 
1.6.2.5

[Qemu-devel] [PATCH 2/3] qemu-iotests: 019: Make cluster size dynamic

[Kevin Wolf]

Change the offsets for test requests according to CLUSTER_SIZE.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 019 |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)
 mode change 100644 => 100755 021

diff --git a/019 b/019
index c3bf950..20893c5 100755
--- a/019
+++ b/019
@@ -60,7 +60,7 @@ for offset in $TEST_OFFSETS; do
     io writev $(( offset )) 512 1024 64
 
     # Complete backing clusters
-    io writev $(( offset  + 1024 * 1024))  65536 65536 1
+    io writev $(( offset  + 1024 * 1024))  $CLUSTER_SIZE $CLUSTER_SIZE 1
 done
 _check_test_img
 
@@ -78,7 +78,7 @@ for offset in $TEST_OFFSETS; do
     io writev $(( offset + 512 )) 512 1024 64
 
     # Complete test image clusters
-    io writev $(( offset + 1024 * 1024 + 65536))  65536 65536 1
+    io writev $(( offset + 1024 * 1024 + $CLUSTER_SIZE))  $CLUSTER_SIZE $CLUSTER_SIZE 1
 done
 _check_test_img
 
@@ -89,7 +89,7 @@ echo "Checking if backing clusters are allocated when they shouldn't"
 echo
 for offset in $TEST_OFFSETS; do
     # Complete backing clusters
-    is_allocated $(( offset  + 1024 * 1024))  65536 65536 1
+    is_allocated $(( offset  + 1024 * 1024))  $CLUSTER_SIZE $CLUSTER_SIZE 1
 done
 
 echo "Reading"
@@ -101,11 +101,11 @@ for offset in $TEST_OFFSETS; do
     io readv $(( offset + 512 )) 512 1024 64
 
     # Complete test image clusters
-    io readv $(( offset  + 1024 * 1024))  65536 65536 1
-    io readv $(( offset + 1024 * 1024 + 65536))  65536 65536 1
+    io readv $(( offset  + 1024 * 1024))  $CLUSTER_SIZE $CLUSTER_SIZE 1
+    io readv $(( offset + 1024 * 1024 + $CLUSTER_SIZE))  $CLUSTER_SIZE $CLUSTER_SIZE 1
 
     # Empty sectors
-    io_zero readv $(( offset + 1024 * 1024 + 65536 * 4 )) 65536 65536 1
+    io_zero readv $(( offset + 1024 * 1024 + $CLUSTER_SIZE * 4 )) $CLUSTER_SIZE $CLUSTER_SIZE 1
 done
 _check_test_img
 
diff --git a/021 b/021
old mode 100644
new mode 100755
-- 
1.6.2.5

[Qemu-devel] [PATCH 3/3] qemu-iotests: Tests on non-4k clusters

[Kevin Wolf]

[-- Attachment #1: Type: text/plain, Size: 1138 bytes --]

This one is a bit larger (1,5 MB) as it contains lots of expected test
output. So I decided it might be inappropriate to send an email of this
size to the list. Please find the compressed patch attached.


>From 0f6107fa4f4f58245b8fdd46f63dbba129bf9da3 Mon Sep 17 00:00:00 2001
From: Kevin Wolf <kwolf@redhat.com>
Date: Fri, 25 Sep 2009 19:19:44 +0200
Subject: [PATCH 3/3] qemu-iotests: Tests on non-4k clusters

This patch adds test case 023 which tests some more cluster sizes. For
anythinger larger than 4k clusters we can't use requests that are
l2_size or more (128k for 1k clusters, 2 MB for 4k clusters, 512 MB for
64k clusters). Therefore one of the common.pattern cases is changed and
needs new expected results for some old test cases.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 013.out        |  680 +-
 014.out        |  816 +-
 022.out        |  136 +-
 023            |  119 +
 023.out        |25706
++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 common.pattern |    5 +-
 group          |    1 +
 7 files changed, 26645 insertions(+), 818 deletions(-)
 create mode 100755 023
 create mode 100644 023.out

[-- Attachment #2: 0003-qemu-iotests-Tests-on-non-4k-clusters.patch.bz2 --]
[-- Type: application/x-bzip, Size: 30650 bytes --]

Re: [Qemu-devel] [PATCH 0/3] qemu-iotests: Variable cluster size

[Christoph Hellwig]

On Mon, Sep 28, 2009 at 02:06:30PM +0200, Kevin Wolf wrote:
> Currently, all qcow2 tests align their test requests to hit the critical spots
> in images that use 4k clusters. By default we are using 64k clusters now for a
> while, so we need to make the tests more dynamic.
> 
> This patch series makes the cluster size variable in some of the generic code
> like common.pattern and adds a test case for the non-4k cluster case.

Thanks, I've put all three patches in.