[COMMITTED 0/3] build fixes

[COMMITTED 0/3] build fixes

[Richard Henderson]

This is a collection of fixes required to get CI back to green.
Given that 2 of 3 apply to cirrus, I had to push them to staging to
even try them.  Since it worked, I've now pushed them to master.


r~


Richard Henderson (1):
  gitlab-ci/cirrus: Update freebsd to python 3.9 packages

Vladimir Sementsov-Ogievskiy (2):
  iotests: fix copy-before-write for macOS and FreeBSD
  iotests/copy-before-write: specify required_fmts

 .gitlab-ci.d/cirrus/freebsd-12.vars        | 3 ++-
 .gitlab-ci.d/cirrus/freebsd-13.vars        | 3 ++-
 tests/qemu-iotests/tests/copy-before-write | 8 +++++++-
 3 files changed, 11 insertions(+), 3 deletions(-)

-- 
2.34.1

[COMMITTED 1/3] gitlab-ci/cirrus: Update freebsd to python 3.9 packages

[Richard Henderson]

FreeBSD has stopped shipping python 3.8, causing our cirrus
builds to fail immediately.  Upstream lcitool has an update
to address this, but has also reorganized its source tree so
additional changes are required for 'make lcitool-update'.

In the meantime, fix the build.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 .gitlab-ci.d/cirrus/freebsd-12.vars | 3 ++-
 .gitlab-ci.d/cirrus/freebsd-13.vars | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/cirrus/freebsd-12.vars b/.gitlab-ci.d/cirrus/freebsd-12.vars
index b4842271b2..f59263731f 100644
--- a/.gitlab-ci.d/cirrus/freebsd-12.vars
+++ b/.gitlab-ci.d/cirrus/freebsd-12.vars
@@ -1,4 +1,5 @@
 # THIS FILE WAS AUTO-GENERATED
+# ... and then edited to fix py39, pending proper lcitool update.
 #
 #  $ lcitool variables freebsd-12 qemu
 #
@@ -11,6 +12,6 @@ MAKE='/usr/local/bin/gmake'
 NINJA='/usr/local/bin/ninja'
 PACKAGING_COMMAND='pkg'
 PIP3='/usr/local/bin/pip-3.8'
-PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils dtc fusefs-libs3 gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 llvm lzo2 meson ncurses nettle ninja opencv perl5 pixman pkgconf png py38-numpy py38-pillow py38-pip py38-sphinx py38-sphinx_rtd_theme py38-virtualenv py38-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
+PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils dtc fusefs-libs3 gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 llvm lzo2 meson ncurses nettle ninja opencv perl5 pixman pkgconf png py39-numpy py39-pillow py39-pip py39-sphinx py39-sphinx_rtd_theme py39-virtualenv py39-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
 PYPI_PKGS=''
 PYTHON='/usr/local/bin/python3'
diff --git a/.gitlab-ci.d/cirrus/freebsd-13.vars b/.gitlab-ci.d/cirrus/freebsd-13.vars
index 546a82dd75..40fc961398 100644
--- a/.gitlab-ci.d/cirrus/freebsd-13.vars
+++ b/.gitlab-ci.d/cirrus/freebsd-13.vars
@@ -1,4 +1,5 @@
 # THIS FILE WAS AUTO-GENERATED
+# ... and then edited to fix py39, pending proper lcitool update.
 #
 #  $ lcitool variables freebsd-13 qemu
 #
@@ -11,6 +12,6 @@ MAKE='/usr/local/bin/gmake'
 NINJA='/usr/local/bin/ninja'
 PACKAGING_COMMAND='pkg'
 PIP3='/usr/local/bin/pip-3.8'
-PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils dtc fusefs-libs3 gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 llvm lzo2 meson ncurses nettle ninja opencv perl5 pixman pkgconf png py38-numpy py38-pillow py38-pip py38-sphinx py38-sphinx_rtd_theme py38-virtualenv py38-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
+PKGS='alsa-lib bash bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage ctags curl cyrus-sasl dbus diffutils dtc fusefs-libs3 gettext git glib gmake gnutls gsed gtk3 libepoxy libffi libgcrypt libjpeg-turbo libnfs libspice-server libssh libtasn1 llvm lzo2 meson ncurses nettle ninja opencv perl5 pixman pkgconf png py39-numpy py39-pillow py39-pip py39-sphinx py39-sphinx_rtd_theme py39-virtualenv py39-yaml python3 rpm2cpio sdl2 sdl2_image snappy spice-protocol tesseract texinfo usbredir virglrenderer vte3 zstd'
 PYPI_PKGS=''
 PYTHON='/usr/local/bin/python3'
-- 
2.34.1

[COMMITTED 2/3] iotests: fix copy-before-write for macOS and FreeBSD

[Richard Henderson]

From: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>

strerror() represents ETIMEDOUT a bit different in Linux and macOS /
FreeBSD. Let's support the latter too.

Fixes: 9d05a87b77 ("iotests: copy-before-write: add cases for cbw-timeout option")
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Tested-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Hanna Reitz <hreitz@redhat.com>
Message-Id: <20220705153708.186418-1-vsementsov@yandex-team.ru>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 tests/qemu-iotests/tests/copy-before-write | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/tests/qemu-iotests/tests/copy-before-write b/tests/qemu-iotests/tests/copy-before-write
index 16efebbf8f..56937b9dff 100755
--- a/tests/qemu-iotests/tests/copy-before-write
+++ b/tests/qemu-iotests/tests/copy-before-write
@@ -192,6 +192,11 @@ read 1048576/1048576 bytes at offset 0
 
     def test_timeout_break_guest(self):
         log = self.do_cbw_timeout('break-guest-write')
+        # macOS and FreeBSD tend to represent ETIMEDOUT as
+        # "Operation timed out", when Linux prefer
+        # "Connection timed out"
+        log = log.replace('Operation timed out',
+                          'Connection timed out')
         self.assertEqual(log, """\
 wrote 524288/524288 bytes at offset 0
 512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-- 
2.34.1

Re: [COMMITTED 2/3] iotests: fix copy-before-write for macOS and FreeBSD

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 2947 bytes --]

On Tue, Jul 12, 2022 at 4:02 AM Richard Henderson <
richard.henderson@linaro.org> wrote:

> From: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
>
> strerror() represents ETIMEDOUT a bit different in Linux and macOS /
> FreeBSD. Let's support the latter too.
>

tl;dr: This change goes back to 1990, but many forks of the BSD networking
code
and its APIs pre-date that and now FreeBSD and Linux are on different sides
of
that change...

It all starts with 4.2BSD's error string for ETIMEDOUT being:

./4.2/usr/src/sys/h/errno.h:#define ETIMEDOUT 60 /* Connection timed out */

but that changed in 1990 with 4.3reno to (though the kernel version of this
file
retained the old wording until it was changed in 4.4-LITE):

./4.3reno/usr/include/errno.h:#define ETIMEDOUT 60 /* Operation timed out */

and 4.4 picked it up from there. While MacOS is Mach based (which was 4.3BSD
based and the 86 and 92 versions both have the 4.3 wording), it had a large
infusion of code from NetBSD/FreeBSD which were derived by then from
4.4-LITE. NET.2 had the 4.3 wording, and early versions of both NetBSD
and FreeBSD (as well as 386BSD and the various patchkit versions) had the
4.3 wording, but that all went away with stipulations by the projects to use
4.4-LITE after BSDi related legal action...

As far as I can tell (and my historical archives are less complete for
Linux), Linux
always had the 4.3 wording because, I think, (a) that's what SunOS had and
(b)
System V had the 4.3 wording starting at System Vr3 when the TCP stack was
introduced which took large parts of its code from 4.3BSD.

Fixes: 9d05a87b77 ("iotests: copy-before-write: add cases for cbw-timeout
> option")
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
> Tested-by: Thomas Huth <thuth@redhat.com>
> Reviewed-by: Hanna Reitz <hreitz@redhat.com>
> Message-Id: <20220705153708.186418-1-vsementsov@yandex-team.ru>
> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
> ---
>  tests/qemu-iotests/tests/copy-before-write | 5 +++++
>  1 file changed, 5 insertions(+)
>

Reviewed-by: Warner Losh <imp@bsdimp.com>


> diff --git a/tests/qemu-iotests/tests/copy-before-write
> b/tests/qemu-iotests/tests/copy-before-write
> index 16efebbf8f..56937b9dff 100755
> --- a/tests/qemu-iotests/tests/copy-before-write
> +++ b/tests/qemu-iotests/tests/copy-before-write
> @@ -192,6 +192,11 @@ read 1048576/1048576 bytes at offset 0
>
>      def test_timeout_break_guest(self):
>          log = self.do_cbw_timeout('break-guest-write')
> +        # macOS and FreeBSD tend to represent ETIMEDOUT as
> +        # "Operation timed out", when Linux prefer
> +        # "Connection timed out"
> +        log = log.replace('Operation timed out',
> +                          'Connection timed out')
>          self.assertEqual(log, """\
>  wrote 524288/524288 bytes at offset 0
>  512 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> --
> 2.34.1
>
>
>

[-- Attachment #2: Type: text/html, Size: 4575 bytes --]

[COMMITTED 3/3] iotests/copy-before-write: specify required_fmts

[Richard Henderson]

From: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>

Declare that we need copy-before-write filter to avoid failure when
filter is not whitelisted.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Tested-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220706170834.242277-1-vsementsov@yandex-team.ru>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
---
 tests/qemu-iotests/tests/copy-before-write | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/tests/copy-before-write b/tests/qemu-iotests/tests/copy-before-write
index 56937b9dff..2ffe092b31 100755
--- a/tests/qemu-iotests/tests/copy-before-write
+++ b/tests/qemu-iotests/tests/copy-before-write
@@ -218,4 +218,5 @@ read failed: Permission denied
 
 if __name__ == '__main__':
     iotests.main(supported_fmts=['qcow2'],
-                 supported_protocols=['file'])
+                 supported_protocols=['file'],
+                 required_fmts=['copy-before-write'])
-- 
2.34.1

Re: [COMMITTED 0/3] build fixes

[Daniel P. Berrangé]

On Tue, Jul 12, 2022 at 03:28:58PM +0530, Richard Henderson wrote:
> This is a collection of fixes required to get CI back to green.
> Given that 2 of 3 apply to cirrus, I had to push them to staging to
> even try them.  Since it worked, I've now pushed them to master.

FWIW, it is possible to test Cirrus CI fixes in your own personal
fork too, but it needs a little extra setup first, detailed in

  .gitlab-ci.d/cirrus/README.rst

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [COMMITTED 0/3] build fixes

[Richard Henderson]

On 7/12/22 15:43, Daniel P. Berrangé wrote:
> On Tue, Jul 12, 2022 at 03:28:58PM +0530, Richard Henderson wrote:
>> This is a collection of fixes required to get CI back to green.
>> Given that 2 of 3 apply to cirrus, I had to push them to staging to
>> even try them.  Since it worked, I've now pushed them to master.
> 
> FWIW, it is possible to test Cirrus CI fixes in your own personal
> fork too, but it needs a little extra setup first, detailed in
> 
>    .gitlab-ci.d/cirrus/README.rst

How many of the 'github' strings therein should really be 'gitlab'?


r~

Re: [COMMITTED 0/3] build fixes

[Daniel P. Berrangé]

On Tue, Jul 12, 2022 at 03:56:54PM +0530, Richard Henderson wrote:
> On 7/12/22 15:43, Daniel P. Berrangé wrote:
> > On Tue, Jul 12, 2022 at 03:28:58PM +0530, Richard Henderson wrote:
> > > This is a collection of fixes required to get CI back to green.
> > > Given that 2 of 3 apply to cirrus, I had to push them to staging to
> > > even try them.  Since it worked, I've now pushed them to master.
> > 
> > FWIW, it is possible to test Cirrus CI fixes in your own personal
> > fork too, but it needs a little extra setup first, detailed in
> > 
> >    .gitlab-ci.d/cirrus/README.rst
> 
> How many of the 'github' strings therein should really be 'gitlab'?

None actually, this is all part of the (gross) hack. Cirrus CI officially
only integrates with GitHub, so we need a dummy GitHub repo there. Once
the job starts though, it actually pulls the real code from gitlab.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [COMMITTED 0/3] build fixes

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1052 bytes --]

On Tue, Jul 12, 2022 at 4:33 AM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Tue, Jul 12, 2022 at 03:56:54PM +0530, Richard Henderson wrote:
> > On 7/12/22 15:43, Daniel P. Berrangé wrote:
> > > On Tue, Jul 12, 2022 at 03:28:58PM +0530, Richard Henderson wrote:
> > > > This is a collection of fixes required to get CI back to green.
> > > > Given that 2 of 3 apply to cirrus, I had to push them to staging to
> > > > even try them.  Since it worked, I've now pushed them to master.
> > >
> > > FWIW, it is possible to test Cirrus CI fixes in your own personal
> > > fork too, but it needs a little extra setup first, detailed in
> > >
> > >    .gitlab-ci.d/cirrus/README.rst
> >
> > How many of the 'github' strings therein should really be 'gitlab'?
>
> None actually, this is all part of the (gross) hack. Cirrus CI officially
> only integrates with GitHub, so we need a dummy GitHub repo there. Once
> the job starts though, it actually pulls the real code from gitlab.
>

Yea, fun times that hack... :(

Warner

[-- Attachment #2: Type: text/html, Size: 1529 bytes --]