Re: [PATCH v18 4/4] iotests: 287: add qcow2 compression type test

[Eric Blake <eblake@redhat.com>]

On 4/2/20 1:36 AM, Denis Plotnikov wrote:
> The test checks fulfilling qcow2 requiriements for the compression

requirements

> type feature and zstd compression type operability.
> 
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
>   tests/qemu-iotests/287     | 167 +++++++++++++++++++++++++++++++++++++
>   tests/qemu-iotests/287.out |  70 ++++++++++++++++
>   tests/qemu-iotests/group   |   1 +
>   3 files changed, 238 insertions(+)
>   create mode 100755 tests/qemu-iotests/287
>   create mode 100644 tests/qemu-iotests/287.out
> 

> +# Check if we can run this test.
> +if IMGOPTS='compression_type=zstd' _make_test_img 64M |
> +    grep "Invalid parameter 'zstd'"; then
> +    _notrun "ZSTD is disabled"
> +fi

Side effect - this created an image (which gets cleaned up when 
skipping, so no problem there)...

> +
> +# Test: when compression is zlib the incompatible bit is unset
> +echo
> +echo "=== Testing compression type incompatible bit setting for zlib ==="
> +echo
> +
> +IMGOPTS='compression_type=zlib' _make_test_img 64M

...and this recreates the same image.  You could drop this line as 
redundant.

> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
> +
> +# Test: when compression differs from zlib the incompatible bit is set
> +echo
> +echo "=== Testing compression type incompatible bit setting for zstd ==="
> +echo

The duplication of '# Test xyz' and 'echo "=== Test xyz"' is awkward; 
you can safely delete the redundant # lines.

> +
> +IMGOPTS='compression_type=zstd' _make_test_img 64M
> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
> +
> +# Test: an image can't be opened if compression type is zlib and
> +#       incompatible feature compression type is set
> +echo
> +echo "=== Testing zlib with incompatible bit set ==="
> +echo
> +
> +IMGOPTS='compression_type=zlib' _make_test_img 64M
> +$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
> +# to make sure the bit was actually set
> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
> +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null

This creates a file named '1' populated with stderr from qemu-img.  I 
don't think that was your intent; you probably meant 2>&1 (if you wanted 
stderr to be logged with the rest of this script's output).

> +if (($?==0)); then
> +    echo "Error: The image opened successfully. The image must not be opened"
> +fi

Although this is valid bash, the use of (()) is documented as being 
something you should avoid in modern scripts (it can be confused for a 
nested subshell).  So, rewriting these last few lines:

if $QEMU_IMG info "$TEST_IMG" 2>&1 >/dev/null ; then
     echo "Error ..."
fi

> +
> +# Test: an image can't be opened if compression type is NOT zlib and
> +#       incompatible feature compression type is UNSET
> +echo
> +echo "=== Testing zstd with incompatible bit unset ==="
> +echo
> +
> +IMGOPTS='compression_type=zstd' _make_test_img 64M
> +$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
> +# to make sure the bit was actually unset
> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
> +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null

Another bad redirect,

> +if (($?==0)); then

and another awkward (()).

> +    echo "Error: The image opened successfully. The image must not be opened"
> +fi
> +# Test: check compression type values
> +echo
> +echo "=== Testing compression type values ==="
> +echo
> +# zlib=0
> +IMGOPTS='compression_type=zlib' _make_test_img 64M
> +od -j104 -N1 -An -vtu1 "$TEST_IMG"

We recently added peek_file_be in common.rc that would be a lot nicer 
than writing your own od command line.  Use it as:

peek_file_be "$TEST_IMG" 104 1

> +
> +# zstd=1
> +IMGOPTS='compression_type=zstd' _make_test_img 64M
> +od -j104 -N1 -An -vtu1 "$TEST_IMG"

and again

> +echo "=== Testing incompressible cluster processing with zstd ==="
> +echo
> +
> +dd if=/dev/urandom of="$RAND_FILE" bs=1M count=1
> +
> +_make_test_img 64M
> +
> +# fill the image with likely incompressible and compressible clusters
> +
> +# TODO: if RAND_FILE variable contain a whitespace, the following will fail.
> +# We need to support some kind of quotes to make possible file paths with
> +# white spaces for -s option

In the meantime, you can make this test robust, by adding up front 
(copying from test 197 for example):

# Sanity check: our use of $RAND_FILE fails if $TEST_DIR contains spaces
# or other problems
case "$TEST_DIR" in
     *[^-_a-zA-Z0-9/]*)
         _notrun "Suspicious TEST_DIR='$TEST_DIR', cowardly refusing to 
run" ;;
esac

> +$QEMU_IO -c "write -c -s $RAND_FILE 0 1M " "$TEST_IMG" | _filter_qemu_io
> +$QEMU_IO -c "write -c -P 0xFA 1M 1M " "$TEST_IMG" | _filter_qemu_io
> +$QEMU_IMG convert -O $IMGFMT -c -o compression_type=zstd \
> +                  "$TEST_IMG" "$COMPR_IMG"
> +$QEMU_IMG compare "$TEST_IMG" "$COMPR_IMG"

You test that data read in as compressed in zlib and written back out as 
zstd compares as equal, which is not quite as strong as whether what is 
read back out matches the original $RAND_FILE, but this is still a 
pretty good round-trip test (it did not prove whether zlib is 
corruption-free, but does show that zstd is coruption-free, and the 
point of this test is what zstd does).

If you want to avoid the issues with 'write -c -s $RAND_FILE' being 
risky, you could instead do:

$QEMU_IO -c "write -P 0xFA 1M 1M" "$RAND_FILE"
$QEMU_IMG convert -f raw -O $IMGFMT -c "$RAND_FILE" "$TEST_IMG"

for creating the zlib file.

Overall, I'm liking how this looks.  There are still a few shell bugs to 
clean up, and I'm not sure which maintainer will be incorporating this, 
but I'm hoping v19 goes into 5.1 rather early in the cycle.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org