[Qemu-devel] [PULL v2 0/3] NBD patches for 2019-02-25

[Qemu-devel] [PULL v2 0/3] NBD patches for 2019-02-25

[Eric Blake]

The following changes since commit d88d85f1f0625d57e9f354aa0874c4c8b5d1fb47:

  Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging (2019-02-25 17:28:04 +0000)

are available in the Git repository at:

  https://repo.or.cz/qemu/ericb.git tags/pull-nbd-2019-02-25-v2

for you to fetch changes up to 3e6f45446b11ccc20b4b751f70331f03d70369b8:

  iotests: avoid broken pipe with certtool (2019-02-26 10:45:37 -0600)

(v2: pull in Andrey's v3 python patch per last-minute list emails)

----------------------------------------------------------------
nbd patches for 2019-02-25

- iotest failure fixes for tests related to NBD

----------------------------------------------------------------
Andrey Shinkevich (1):
      iotests: handle TypeError for Python 3 in test 242

Daniel P. Berrangé (2):
      iotests: ensure we print nbd server log on error
      iotests: avoid broken pipe with certtool

 tests/qemu-iotests/common.tls | 48 ++++++++++++++++++++++++++++---------------
 tests/qemu-iotests/233        |  3 +++
 tests/qemu-iotests/242        |  8 +++++---
 3 files changed, 40 insertions(+), 19 deletions(-)

-- 
2.20.1

[Qemu-devel] [PULL v2 1/3] iotests: handle TypeError for Python 3 in test 242

[Eric Blake]

From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>

The data type for bytes in Python 3 differs from the one in Python 2.
The type cast that is compatible with both versions was applied.

Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Reported-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <1551197495-24425-1-git-send-email-andrey.shinkevich@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
 tests/qemu-iotests/242 | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/242 b/tests/qemu-iotests/242
index 16c65edcd7f..c176e92da6a 100755
--- a/tests/qemu-iotests/242
+++ b/tests/qemu-iotests/242
@@ -20,6 +20,7 @@

 import iotests
 import json
+import struct
 from iotests import qemu_img_create, qemu_io, qemu_img_pipe, \
     file_path, img_info_log, log, filter_qemu_io

@@ -64,10 +65,11 @@ def write_to_disk(offset, size):
 def toggle_flag(offset):
     with open(disk, "r+b") as f:
         f.seek(offset, 0)
-        c = f.read(1)
-        toggled = chr(ord(c) ^ bitmap_flag_unknown)
+        # Read one byte in a way compatible with Python 2
+        flags = struct.unpack("B", f.read(1))
+        toggled = flags[0] ^ bitmap_flag_unknown
         f.seek(-1, 1)
-        f.write(toggled)
+        f.write(struct.pack("B", toggled))


 qemu_img_create('-f', iotests.imgfmt, disk, '1M')
-- 
2.20.1

[Qemu-devel] [PULL v2 2/3] iotests: ensure we print nbd server log on error

[Eric Blake]

From: Daniel P. Berrangé <berrange@redhat.com>

If we abort the iotest early the server.log file might contain useful
information for diagnosing the problem. Ensure its contents are
displayed in this case.

Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20190220145819.30969-2-berrange@redhat.com>
[eblake: fix shell quoting]
Signed-off-by: Eric Blake <eblake@redhat.com>
---
 tests/qemu-iotests/233 | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/qemu-iotests/233 b/tests/qemu-iotests/233
index fc345a1a46e..adb742fafb0 100755
--- a/tests/qemu-iotests/233
+++ b/tests/qemu-iotests/233
@@ -30,6 +30,8 @@ _cleanup()
 {
     nbd_server_stop
     _cleanup_test_img
+    # If we aborted early we want to see this log for diagnosis
+    test -f "$TEST_DIR/server.log" && cat "$TEST_DIR/server.log"
     rm -f "$TEST_DIR/server.log"
     tls_x509_cleanup
 }
@@ -120,6 +122,7 @@ $QEMU_IO -f $IMGFMT -r -U -c 'r -P 0x22 1m 1m' "$TEST_IMG" | _filter_qemu_io
 echo
 echo "== final server log =="
 cat "$TEST_DIR/server.log"
+rm -f "$TEST_DIR/server.log"

 # success, all done
 echo "*** done"
-- 
2.20.1

[Qemu-devel] [PULL v2 3/3] iotests: avoid broken pipe with certtool

[Eric Blake]

From: Daniel P. Berrangé <berrange@redhat.com>

When we run "certtool 2>&1 | head -1" the latter command is likely to
complete and exit before certtool has written everything it wants to
stderr. In at least the RHEL-7 gnutls 3.3.29 this causes certtool to
quit with broken pipe before it has finished writing the desired
output file to disk. This causes non-deterministic failures of the
iotest 233 because the certs are sometimes zero length files.
If certtool fails the "head -1" means we also lose any useful error
message it would have printed.

Thus this patch gets rid of the pipe and post-processes the output in a
more flexible & reliable manner.

Reported-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20190220145819.30969-3-berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
 tests/qemu-iotests/common.tls | 48 +++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 16 deletions(-)

diff --git a/tests/qemu-iotests/common.tls b/tests/qemu-iotests/common.tls
index eae81789bbc..3caf989d28c 100644
--- a/tests/qemu-iotests/common.tls
+++ b/tests/qemu-iotests/common.tls
@@ -29,6 +29,17 @@ tls_x509_cleanup()
 }


+tls_certtool()
+{
+    certtool "$@" 1>"${tls_dir}"/certtool.log 2>&1
+    if test "$?" = 0; then
+      head -1 "${tls_dir}"/certtool.log
+    else
+      cat "${tls_dir}"/certtool.log
+    fi
+    rm -f "${tls_dir}"/certtool.log
+}
+
 tls_x509_init()
 {
     (certtool --help) >/dev/null 2>&1 || \
@@ -71,10 +82,11 @@ ca
 cert_signing_key
 EOF

-    certtool --generate-self-signed \
-             --load-privkey "${tls_dir}/key.pem" \
-             --template "${tls_dir}/ca.info" \
-             --outfile "${tls_dir}/$name-cert.pem" 2>&1 | head -1
+    tls_certtool \
+        --generate-self-signed \
+        --load-privkey "${tls_dir}/key.pem" \
+        --template "${tls_dir}/ca.info" \
+        --outfile "${tls_dir}/$name-cert.pem"

     rm -f "${tls_dir}/ca.info"
 }
@@ -98,12 +110,14 @@ encryption_key
 signing_key
 EOF

-    certtool --generate-certificate \
-             --load-ca-privkey "${tls_dir}/key.pem" \
-             --load-ca-certificate "${tls_dir}/$caname-cert.pem" \
-             --load-privkey "${tls_dir}/key.pem" \
-             --template "${tls_dir}/cert.info" \
-             --outfile "${tls_dir}/$name/server-cert.pem" 2>&1 | head -1
+    tls_certtool \
+        --generate-certificate \
+        --load-ca-privkey "${tls_dir}/key.pem" \
+        --load-ca-certificate "${tls_dir}/$caname-cert.pem" \
+        --load-privkey "${tls_dir}/key.pem" \
+        --template "${tls_dir}/cert.info" \
+        --outfile "${tls_dir}/$name/server-cert.pem"
+
     ln -s "${tls_dir}/$caname-cert.pem" "${tls_dir}/$name/ca-cert.pem"
     ln -s "${tls_dir}/key.pem" "${tls_dir}/$name/server-key.pem"

@@ -127,12 +141,14 @@ encryption_key
 signing_key
 EOF

-    certtool --generate-certificate \
-             --load-ca-privkey "${tls_dir}/key.pem" \
-             --load-ca-certificate "${tls_dir}/$caname-cert.pem" \
-             --load-privkey "${tls_dir}/key.pem" \
-             --template "${tls_dir}/cert.info" \
-             --outfile "${tls_dir}/$name/client-cert.pem" 2>&1 | head -1
+    tls_certtool \
+        --generate-certificate \
+        --load-ca-privkey "${tls_dir}/key.pem" \
+        --load-ca-certificate "${tls_dir}/$caname-cert.pem" \
+        --load-privkey "${tls_dir}/key.pem" \
+        --template "${tls_dir}/cert.info" \
+        --outfile "${tls_dir}/$name/client-cert.pem"
+
     ln -s "${tls_dir}/$caname-cert.pem" "${tls_dir}/$name/ca-cert.pem"
     ln -s "${tls_dir}/key.pem" "${tls_dir}/$name/client-key.pem"

-- 
2.20.1

Re: [Qemu-devel] [PULL v2 0/3] NBD patches for 2019-02-25

[Peter Maydell]

On Tue, 26 Feb 2019 at 16:56, Eric Blake <eblake@redhat.com> wrote:
>
> The following changes since commit d88d85f1f0625d57e9f354aa0874c4c8b5d1fb47:
>
>   Merge remote-tracking branch 'remotes/stefanha/tags/block-pull-request' into staging (2019-02-25 17:28:04 +0000)
>
> are available in the Git repository at:
>
>   https://repo.or.cz/qemu/ericb.git tags/pull-nbd-2019-02-25-v2
>
> for you to fetch changes up to 3e6f45446b11ccc20b4b751f70331f03d70369b8:
>
>   iotests: avoid broken pipe with certtool (2019-02-26 10:45:37 -0600)
>
> (v2: pull in Andrey's v3 python patch per last-minute list emails)
>
> ----------------------------------------------------------------
> nbd patches for 2019-02-25
>
> - iotest failure fixes for tests related to NBD
>
> ----------------------------------------------------------------

Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/4.0
for any user-visible changes.

-- PMM