[PATCH 0/8] gitlab: speed up msys windows jobs with GCC

[PATCH 0/8] gitlab: speed up msys windows jobs with GCC

[Daniel P. Berrangé]

This is an alternative and/or complementary to Thomas' proposal
to use CLang with msys:

  https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html

First of all, the current msys installer we're using is over 12
months out of date. Thus after running the install, pacman then
replaces most of what we've just installed with new downloaded
content. Using the most update installer cuts 3+1/2 minutes off
the msys install time - 7 minutes becomes 3+1/2.

Secondly, QEMU defaults to compiling with -O2 and this is more
computationally expensive for GCC. Switching to -O0 drops the
build time from 60 minutes down to 45 minutes.

Thirdly, including debug symbols also has an overhead, and turning
that off reduces time still further down to 38 minutes.

IOW, between all three changes, we can cut approx 25-26 minutes
off the job execution time, bringing it nicely within the job
timeout.

The actually phase of installing the mingw deps still accounts
for about 10 minutes and has not been optimized.

Possibly the same trick of -O0 and skipping -g would also help
the clang alternative Thomas' proposed. If so, that could be
enough to let us enable more features / targets during the
msys build.

Daniel P. Berrangé (8):
  gitlab: remove duplication between msys jobs
  gitlab: print timestamps during windows msys jobs
  gitlab: always use updated msys installer
  gitlab: drop $CI_PROJECT_DIR from cache path
  gitlab: always populate cache for windows msys jobs
  configure: support passthrough of -Dxxx args to meson
  gitlab: disable optimization and debug symbols in msys build
  gitlab: disable FF_SCRIPT_SECTIONS on msys jobs

 .gitlab-ci.d/windows.yml | 174 +++++++++++++++++++--------------------
 configure                |   5 ++
 2 files changed, 92 insertions(+), 87 deletions(-)

-- 
2.41.0

[PATCH 1/8] gitlab: remove duplication between msys jobs

[Daniel P. Berrangé]

Although they share a common parent, the two msys jobs still have
massive duplication in their script definitions that can easily be
collapsed.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/windows.yml | 132 +++++++++++++++------------------------
 1 file changed, 49 insertions(+), 83 deletions(-)

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index f889a468b5..f086540e40 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -35,97 +35,63 @@
   - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu'  # Core update
   - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu'  # Normal update
   - taskkill /F /FI "MODULES eq msys-2.0.dll"
-
-msys2-64bit:
-  extends: .shared_msys2_builder
   script:
   - .\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
       bison diffutils flex
       git grep make sed
-      mingw-w64-x86_64-capstone
-      mingw-w64-x86_64-curl
-      mingw-w64-x86_64-cyrus-sasl
-      mingw-w64-x86_64-dtc
-      mingw-w64-x86_64-gcc
-      mingw-w64-x86_64-glib2
-      mingw-w64-x86_64-gnutls
-      mingw-w64-x86_64-gtk3
-      mingw-w64-x86_64-libgcrypt
-      mingw-w64-x86_64-libjpeg-turbo
-      mingw-w64-x86_64-libnfs
-      mingw-w64-x86_64-libpng
-      mingw-w64-x86_64-libssh
-      mingw-w64-x86_64-libtasn1
-      mingw-w64-x86_64-libusb
-      mingw-w64-x86_64-lzo2
-      mingw-w64-x86_64-nettle
-      mingw-w64-x86_64-ninja
-      mingw-w64-x86_64-pixman
-      mingw-w64-x86_64-pkgconf
-      mingw-w64-x86_64-python
-      mingw-w64-x86_64-SDL2
-      mingw-w64-x86_64-SDL2_image
-      mingw-w64-x86_64-snappy
-      mingw-w64-x86_64-spice
-      mingw-w64-x86_64-usbredir
-      mingw-w64-x86_64-zstd "
+      $MINGW_TARGET-capstone
+      $MINGW_TARGET-curl
+      $MINGW_TARGET-cyrus-sasl
+      $MINGW_TARGET-dtc
+      $MINGW_TARGET-gcc
+      $MINGW_TARGET-glib2
+      $MINGW_TARGET-gnutls
+      $MINGW_TARGET-gtk3
+      $MINGW_TARGET-libgcrypt
+      $MINGW_TARGET-libjpeg-turbo
+      $MINGW_TARGET-libnfs
+      $MINGW_TARGET-libpng
+      $MINGW_TARGET-libssh
+      $MINGW_TARGET-libtasn1
+      $MINGW_TARGET-libusb
+      $MINGW_TARGET-lzo2
+      $MINGW_TARGET-nettle
+      $MINGW_TARGET-ninja
+      $MINGW_TARGET-pixman
+      $MINGW_TARGET-pkgconf
+      $MINGW_TARGET-python
+      $MINGW_TARGET-SDL2
+      $MINGW_TARGET-SDL2_image
+      $MINGW_TARGET-snappy
+      $MINGW_TARGET-spice
+      $MINGW_TARGET-usbredir
+      $MINGW_TARGET-zstd "
   - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
-  - $env:MSYSTEM = 'MINGW64'     # Start a 64-bit MinGW environment
   - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
   - mkdir build
   - cd build
-  # Note: do not remove "--without-default-devices"!
-  # commit 9f8e6cad65a6 ("gitlab-ci: Speed up the msys2-64bit job by using --without-default-devices"
-  # changed to compile QEMU with the --without-default-devices switch
-  # for the msys2 64-bit job, due to the build could not complete within
-  # the project timeout.
-  - ..\msys64\usr\bin\bash -lc '../configure --target-list=x86_64-softmmu
-      --without-default-devices --enable-fdt=system'
-  - ..\msys64\usr\bin\bash -lc 'make'
-  # qTests don't run successfully with "--without-default-devices",
-  # so let's exclude the qtests from CI for now.
-  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS=\"--no-suite qtest\" || { cat meson-logs/testlog.txt; exit 1; } ;'
+  - ..\msys64\usr\bin\bash -lc "../configure --enable-fdt=system $CONFIGURE_ARGS"
+  - ..\msys64\usr\bin\bash -lc "make"
+  - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
+
+msys2-64bit:
+  extends: .shared_msys2_builder
+  variables:
+    MINGW_TARGET: mingw-w64-x86_64
+    MSYSTEM: MINGW64
+    # do not remove "--without-default-devices"!
+    # commit 9f8e6cad65a6 ("gitlab-ci: Speed up the msys2-64bit job by using --without-default-devices"
+    # changed to compile QEMU with the --without-default-devices switch
+    # for the msys2 64-bit job, due to the build could not complete within
+    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices
+    # qTests don't run successfully with "--without-default-devices",
+    # so let's exclude the qtests from CI for now.
+    TEST_ARGS: --no-suite qtest
 
 msys2-32bit:
   extends: .shared_msys2_builder
-  script:
-  - .\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
-      bison diffutils flex
-      git grep make sed
-      mingw-w64-i686-capstone
-      mingw-w64-i686-curl
-      mingw-w64-i686-cyrus-sasl
-      mingw-w64-i686-dtc
-      mingw-w64-i686-gcc
-      mingw-w64-i686-glib2
-      mingw-w64-i686-gnutls
-      mingw-w64-i686-gtk3
-      mingw-w64-i686-libgcrypt
-      mingw-w64-i686-libjpeg-turbo
-      mingw-w64-i686-libnfs
-      mingw-w64-i686-libpng
-      mingw-w64-i686-libssh
-      mingw-w64-i686-libtasn1
-      mingw-w64-i686-libusb
-      mingw-w64-i686-lzo2
-      mingw-w64-i686-nettle
-      mingw-w64-i686-ninja
-      mingw-w64-i686-pixman
-      mingw-w64-i686-pkgconf
-      mingw-w64-i686-python
-      mingw-w64-i686-SDL2
-      mingw-w64-i686-SDL2_image
-      mingw-w64-i686-snappy
-      mingw-w64-i686-spice
-      mingw-w64-i686-usbredir
-      mingw-w64-i686-zstd "
-  - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
-  - $env:MSYSTEM = 'MINGW32'     # Start a 32-bit MinGW environment
-  - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
-  - mkdir build
-  - cd build
-  - ..\msys64\usr\bin\bash -lc '../configure --target-list=ppc64-softmmu
-                                --enable-fdt=system'
-  - ..\msys64\usr\bin\bash -lc 'make'
-  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS=\"--no-suite qtest\" ||
-                                { cat meson-logs/testlog.txt; exit 1; }'
+  variables:
+    MINGW_TARGET: mingw-w64-i686
+    MSYSTEM: MINGW32
+    CONFIGURE_ARGS:  --target-list=ppc64-softmmu
+    TEST_ARGS: --no-suite qtest
-- 
2.41.0

Re: [PATCH 1/8] gitlab: remove duplication between msys jobs

[Thomas Huth]

On 01/08/2023 15.03, Daniel P. Berrangé wrote:
> Although they share a common parent, the two msys jobs still have
> massive duplication in their script definitions that can easily be
> collapsed.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/windows.yml | 132 +++++++++++++++------------------------
>   1 file changed, 49 insertions(+), 83 deletions(-)

As I discovered in the recent days, this is also quite helpful in case we 
ever want to switch to Clang, since we need to change the prefix of the 
packages there. And it's still easy to have distinct packages with some few 
lines of codes changes, as I tried it out in my Clang patch. So this patch 
now sounds fine to me:

Reviewed-by: Thomas Huth <thuth@redhat.com>

[PATCH 2/8] gitlab: print timestamps during windows msys jobs

[Daniel P. Berrangé]

It is hard to get visibility into where time is consumed in our Windows
msys jobs. Adding a few log console messages with the timestamp will
aid in our debugging.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/windows.yml | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index f086540e40..831b080d12 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -19,6 +19,7 @@
     reports:
       junit: "build/meson-logs/testlog.junit.xml"
   before_script:
+  - Write-Output "Acquiring msys2.exe installer at $(Get-Date -Format u)"
   - If ( !(Test-Path -Path msys64\var\cache ) ) {
       mkdir msys64\var\cache
     }
@@ -27,6 +28,7 @@
       "https://github.com/msys2/msys2-installer/releases/download/2022-06-03/msys2-base-x86_64-20220603.sfx.exe"
       -outfile "msys64\var\cache\msys2.exe"
     }
+  - Write-Output "Invoking msys2.exe installer at $(Get-Date -Format u)"
   - msys64\var\cache\msys2.exe -y
   - ((Get-Content -path .\msys64\etc\\post-install\\07-pacman-key.post -Raw)
       -replace '--refresh-keys', '--version') |
@@ -36,6 +38,7 @@
   - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu'  # Normal update
   - taskkill /F /FI "MODULES eq msys-2.0.dll"
   script:
+  - Write-Output "Installing mingw packages at $(Get-Date -Format u)"
   - .\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
       bison diffutils flex
       git grep make sed
@@ -66,6 +69,7 @@
       $MINGW_TARGET-spice
       $MINGW_TARGET-usbredir
       $MINGW_TARGET-zstd "
+  - Write-Output "Running build at $(Get-Date -Format u)"
   - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
   - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
   - mkdir build
@@ -73,6 +77,7 @@
   - ..\msys64\usr\bin\bash -lc "../configure --enable-fdt=system $CONFIGURE_ARGS"
   - ..\msys64\usr\bin\bash -lc "make"
   - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
+  - Write-Output "Finished build at $(Get-Date -Format u)"
 
 msys2-64bit:
   extends: .shared_msys2_builder
-- 
2.41.0

Re: [PATCH 2/8] gitlab: print timestamps during windows msys jobs

[Thomas Huth]

On 01/08/2023 15.03, Daniel P. Berrangé wrote:
> It is hard to get visibility into where time is consumed in our Windows
> msys jobs. Adding a few log console messages with the timestamp will
> aid in our debugging.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/windows.yml | 5 +++++
>   1 file changed, 5 insertions(+)
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index f086540e40..831b080d12 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -19,6 +19,7 @@
>       reports:
>         junit: "build/meson-logs/testlog.junit.xml"
>     before_script:
> +  - Write-Output "Acquiring msys2.exe installer at $(Get-Date -Format u)"
>     - If ( !(Test-Path -Path msys64\var\cache ) ) {
>         mkdir msys64\var\cache
>       }
> @@ -27,6 +28,7 @@
>         "https://github.com/msys2/msys2-installer/releases/download/2022-06-03/msys2-base-x86_64-20220603.sfx.exe"
>         -outfile "msys64\var\cache\msys2.exe"
>       }
> +  - Write-Output "Invoking msys2.exe installer at $(Get-Date -Format u)"
>     - msys64\var\cache\msys2.exe -y
>     - ((Get-Content -path .\msys64\etc\\post-install\\07-pacman-key.post -Raw)
>         -replace '--refresh-keys', '--version') |
> @@ -36,6 +38,7 @@
>     - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu'  # Normal update
>     - taskkill /F /FI "MODULES eq msys-2.0.dll"
>     script:
> +  - Write-Output "Installing mingw packages at $(Get-Date -Format u)"
>     - .\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
>         bison diffutils flex
>         git grep make sed
> @@ -66,6 +69,7 @@
>         $MINGW_TARGET-spice
>         $MINGW_TARGET-usbredir
>         $MINGW_TARGET-zstd "
> +  - Write-Output "Running build at $(Get-Date -Format u)"
>     - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
>     - $env:MSYS = 'winsymlinks:native' # Enable native Windows symlink
>     - mkdir build
> @@ -73,6 +77,7 @@
>     - ..\msys64\usr\bin\bash -lc "../configure --enable-fdt=system $CONFIGURE_ARGS"
>     - ..\msys64\usr\bin\bash -lc "make"
>     - ..\msys64\usr\bin\bash -lc "make check MTESTARGS='$TEST_ARGS' || { cat meson-logs/testlog.txt; exit 1; } ;"
> +  - Write-Output "Finished build at $(Get-Date -Format u)"
>   
>   msys2-64bit:
>     extends: .shared_msys2_builder

Reviewed-by: Thomas Huth <thuth@redhat.com>

[PATCH 3/8] gitlab: always use updated msys installer

[Daniel P. Berrangé]

We current reference an msys installer binary from mid-2022, which means
after installation, it immediately has to re-download a bunch of newer
content. This wastes precious CI time.

The msys project publishes an installer binary with a fixed URL that
always references the latest content. We cache the downloads in gitlab
though and so once downloaded we would never re-fetch the installer
leading back to the same problem.

To deal with this we also fetch the pgp signature for the installer
on every run, and compare that to the previously cached signature. If
the signature changes, we re-download the full installer.

This ensures we always have the latest installer for msys, while also
maximising use of the gitlab cache.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/windows.yml | 30 +++++++++++++++++++++++++++---
 1 file changed, 27 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index 831b080d12..0bc04ad068 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -23,10 +23,34 @@
   - If ( !(Test-Path -Path msys64\var\cache ) ) {
       mkdir msys64\var\cache
     }
-  - If ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
+  - Invoke-WebRequest
+    "https://repo.msys2.org/distrib/msys2-x86_64-latest.sfx.exe.sig"
+    -outfile "msys2.exe.sig"
+  - if ( Test-Path -Path msys64\var\cache\msys2.exe.sig ) {
+      Write-Output "Cached installer sig" ;
+      if ( ((Get-FileHash msys2.exe.sig).Hash -ne (Get-FileHash msys64\var\cache\msys2.exe.sig).Hash) ) {
+        Write-Output "Mis-matched installer sig, new installer download required" ;
+        Remove-Item -Path msys64\var\cache\msys2.exe.sig ;
+        if ( Test-Path -Path msys64\var\cache\msys2.exe ) {
+          Remove-Item -Path msys64\var\cache\msys2.exe
+        }
+      } else {
+        Write-Output "Matched installer sig, cached installer still valid"
+      }
+    } else {
+      Write-Output "No cached installer sig, new installer download required" ;
+      if ( Test-Path -Path msys64\var\cache\msys2.exe ) {
+        Remove-Item -Path msys64\var\cache\msys2.exe
+      }
+    }
+  - if ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
+      Write-Output "Fetching latest installer" ;
       Invoke-WebRequest
-      "https://github.com/msys2/msys2-installer/releases/download/2022-06-03/msys2-base-x86_64-20220603.sfx.exe"
-      -outfile "msys64\var\cache\msys2.exe"
+      "https://repo.msys2.org/distrib/msys2-x86_64-latest.sfx.exe"
+      -outfile "msys64\var\cache\msys2.exe" ;
+      Copy-Item -Path msys2.exe.sig -Destination msys64\var\cache\msys2.exe.sig
+    } else {
+      Write-Output "Using cached installer"
     }
   - Write-Output "Invoking msys2.exe installer at $(Get-Date -Format u)"
   - msys64\var\cache\msys2.exe -y
-- 
2.41.0

Re: [PATCH 3/8] gitlab: always use updated msys installer

[Thomas Huth]

On 01/08/2023 15.03, Daniel P. Berrangé wrote:
> We current reference an msys installer binary from mid-2022, which means
> after installation, it immediately has to re-download a bunch of newer
> content. This wastes precious CI time.
> 
> The msys project publishes an installer binary with a fixed URL that
> always references the latest content. We cache the downloads in gitlab
> though and so once downloaded we would never re-fetch the installer
> leading back to the same problem.
> 
> To deal with this we also fetch the pgp signature for the installer
> on every run, and compare that to the previously cached signature. If
> the signature changes, we re-download the full installer.
> 
> This ensures we always have the latest installer for msys, while also
> maximising use of the gitlab cache.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/windows.yml | 30 +++++++++++++++++++++++++++---
>   1 file changed, 27 insertions(+), 3 deletions(-)
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 831b080d12..0bc04ad068 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -23,10 +23,34 @@
>     - If ( !(Test-Path -Path msys64\var\cache ) ) {
>         mkdir msys64\var\cache
>       }
> -  - If ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
> +  - Invoke-WebRequest
> +    "https://repo.msys2.org/distrib/msys2-x86_64-latest.sfx.exe.sig"
> +    -outfile "msys2.exe.sig"
> +  - if ( Test-Path -Path msys64\var\cache\msys2.exe.sig ) {
> +      Write-Output "Cached installer sig" ;
> +      if ( ((Get-FileHash msys2.exe.sig).Hash -ne (Get-FileHash msys64\var\cache\msys2.exe.sig).Hash) ) {
> +        Write-Output "Mis-matched installer sig, new installer download required" ;
> +        Remove-Item -Path msys64\var\cache\msys2.exe.sig ;
> +        if ( Test-Path -Path msys64\var\cache\msys2.exe ) {
> +          Remove-Item -Path msys64\var\cache\msys2.exe
> +        }
> +      } else {
> +        Write-Output "Matched installer sig, cached installer still valid"
> +      }
> +    } else {
> +      Write-Output "No cached installer sig, new installer download required" ;
> +      if ( Test-Path -Path msys64\var\cache\msys2.exe ) {
> +        Remove-Item -Path msys64\var\cache\msys2.exe
> +      }
> +    }
> +  - if ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
> +      Write-Output "Fetching latest installer" ;
>         Invoke-WebRequest
> -      "https://github.com/msys2/msys2-installer/releases/download/2022-06-03/msys2-base-x86_64-20220603.sfx.exe"
> -      -outfile "msys64\var\cache\msys2.exe"
> +      "https://repo.msys2.org/distrib/msys2-x86_64-latest.sfx.exe"
> +      -outfile "msys64\var\cache\msys2.exe" ;
> +      Copy-Item -Path msys2.exe.sig -Destination msys64\var\cache\msys2.exe.sig
> +    } else {
> +      Write-Output "Using cached installer"
>       }
>     - Write-Output "Invoking msys2.exe installer at $(Get-Date -Format u)"
>     - msys64\var\cache\msys2.exe -y

Reviewed-by: Thomas Huth <thuth@redhat.com>

... I think the original idea was to use a "tagged" version to avoid that we 
have some kind of "rolling release" here, but since the latest content is 
fetched anyway during the following update, that idea was likely not working 
as expected...

Re: [PATCH 3/8] gitlab: always use updated msys installer

[Daniel P. Berrangé]

On Wed, Aug 02, 2023 at 05:49:42PM +0200, Thomas Huth wrote:
> On 01/08/2023 15.03, Daniel P. Berrangé wrote:
> > We current reference an msys installer binary from mid-2022, which means
> > after installation, it immediately has to re-download a bunch of newer
> > content. This wastes precious CI time.
> > 
> > The msys project publishes an installer binary with a fixed URL that
> > always references the latest content. We cache the downloads in gitlab
> > though and so once downloaded we would never re-fetch the installer
> > leading back to the same problem.
> > 
> > To deal with this we also fetch the pgp signature for the installer
> > on every run, and compare that to the previously cached signature. If
> > the signature changes, we re-download the full installer.
> > 
> > This ensures we always have the latest installer for msys, while also
> > maximising use of the gitlab cache.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>


> Reviewed-by: Thomas Huth <thuth@redhat.com>
> 
> ... I think the original idea was to use a "tagged" version to avoid that we
> have some kind of "rolling release" here, but since the latest content is
> fetched anyway during the following update, that idea was likely not working
> as expected...

I think using the "latest" installer is also more in keeping with what
we do for other distros, where we'll always pick up the latest content
when the containers get rebuilt, or macOS/FreeBSD where we pick the
latest from Ports/HomeBrew.  

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH 4/8] gitlab: drop $CI_PROJECT_DIR from cache path

[Daniel P. Berrangé]

The gitlab cache is limited to only handle content within the
$CI_PROJECT_DIR hierarchy, and as such relative paths are always
implicitly relative to $CI_PROJECT_DIR.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/windows.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index 0bc04ad068..6454880cb7 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -7,7 +7,7 @@
   cache:
     key: "${CI_JOB_NAME}-cache"
     paths:
-      - ${CI_PROJECT_DIR}/msys64/var/cache
+      - msys64/var/cache
   needs: []
   stage: build
   timeout: 80m
-- 
2.41.0

Re: [PATCH 4/8] gitlab: drop $CI_PROJECT_DIR from cache path

[Thomas Huth]

On 01/08/2023 15.03, Daniel P. Berrangé wrote:
> The gitlab cache is limited to only handle content within the
> $CI_PROJECT_DIR hierarchy, and as such relative paths are always
> implicitly relative to $CI_PROJECT_DIR.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/windows.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 0bc04ad068..6454880cb7 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -7,7 +7,7 @@
>     cache:
>       key: "${CI_JOB_NAME}-cache"
>       paths:
> -      - ${CI_PROJECT_DIR}/msys64/var/cache
> +      - msys64/var/cache
>     needs: []
>     stage: build
>     timeout: 80m

Reviewed-by: Thomas Huth <thuth@redhat.com>

[PATCH 5/8] gitlab: always populate cache for windows msys jobs

[Daniel P. Berrangé]

The cache is used to hold the msys installer. Even if the build phase
fails, we should still populate the cache as the installer will be
valid for next time.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/windows.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index 6454880cb7..34109a80f2 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -8,6 +8,7 @@
     key: "${CI_JOB_NAME}-cache"
     paths:
       - msys64/var/cache
+    when: always
   needs: []
   stage: build
   timeout: 80m
-- 
2.41.0

Re: [PATCH 5/8] gitlab: always populate cache for windows msys jobs

[Thomas Huth]

On 01/08/2023 15.04, Daniel P. Berrangé wrote:
> The cache is used to hold the msys installer. Even if the build phase
> fails, we should still populate the cache as the installer will be
> valid for next time.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/windows.yml | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 6454880cb7..34109a80f2 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -8,6 +8,7 @@
>       key: "${CI_JOB_NAME}-cache"
>       paths:
>         - msys64/var/cache
> +    when: always
>     needs: []
>     stage: build
>     timeout: 80m

Reviewed-by: Thomas Huth <thuth@redhat.com>

[PATCH 6/8] configure: support passthrough of -Dxxx args to meson

[Daniel P. Berrangé]

This can be useful for setting some meson global options, such as the
optimization level or debug state.xs

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 configure | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/configure b/configure
index 26ec5e4f54..9fe3718b77 100755
--- a/configure
+++ b/configure
@@ -757,6 +757,9 @@ for opt do
   # everything else has the same name in configure and meson
   --*) meson_option_parse "$opt" "$optarg"
   ;;
+  # Pass through -Dxxxx options to meson
+  -D*) meson_options="$meson_options $opt"
+  ;;
   esac
 done
 
@@ -887,6 +890,8 @@ cat << EOF
   pie             Position Independent Executables
   debug-tcg       TCG debugging (default is disabled)
 
+  -Dmesonoptname=val      passthrough option to meson unmodified
+
 NOTE: The object files are built at the place where configure is launched
 EOF
 exit 0
-- 
2.41.0

Re: [PATCH 6/8] configure: support passthrough of -Dxxx args to meson

[Thomas Huth]

On 01/08/2023 15.04, Daniel P. Berrangé wrote:
> This can be useful for setting some meson global options, such as the
> optimization level or debug state.xs
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   configure | 5 +++++
>   1 file changed, 5 insertions(+)
> 
> diff --git a/configure b/configure
> index 26ec5e4f54..9fe3718b77 100755
> --- a/configure
> +++ b/configure
> @@ -757,6 +757,9 @@ for opt do
>     # everything else has the same name in configure and meson
>     --*) meson_option_parse "$opt" "$optarg"
>     ;;
> +  # Pass through -Dxxxx options to meson
> +  -D*) meson_options="$meson_options $opt"
> +  ;;
>     esac
>   done
>   
> @@ -887,6 +890,8 @@ cat << EOF
>     pie             Position Independent Executables
>     debug-tcg       TCG debugging (default is disabled)
>   
> +  -Dmesonoptname=val      passthrough option to meson unmodified

I'd rather place that earlier in the help text, above the 
"meson_options_help" line in the configure script, next to the other 
--option=something lines.

  Thomas

>   NOTE: The object files are built at the place where configure is launched
>   EOF
>   exit 0

Re: [PATCH 6/8] configure: support passthrough of -Dxxx args to meson

[Daniel P. Berrangé]

On Tue, Aug 01, 2023 at 08:42:05PM +0200, Thomas Huth wrote:
> On 01/08/2023 15.04, Daniel P. Berrangé wrote:
> > This can be useful for setting some meson global options, such as the
> > optimization level or debug state.xs
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   configure | 5 +++++
> >   1 file changed, 5 insertions(+)
> > 
> > diff --git a/configure b/configure
> > index 26ec5e4f54..9fe3718b77 100755
> > --- a/configure
> > +++ b/configure
> > @@ -757,6 +757,9 @@ for opt do
> >     # everything else has the same name in configure and meson
> >     --*) meson_option_parse "$opt" "$optarg"
> >     ;;
> > +  # Pass through -Dxxxx options to meson
> > +  -D*) meson_options="$meson_options $opt"
> > +  ;;
> >     esac
> >   done
> > @@ -887,6 +890,8 @@ cat << EOF
> >     pie             Position Independent Executables
> >     debug-tcg       TCG debugging (default is disabled)
> > +  -Dmesonoptname=val      passthrough option to meson unmodified
> 
> I'd rather place that earlier in the help text, above the
> "meson_options_help" line in the configure script, next to the other
> --option=something lines.

Sure I've no preference. I was expecting Paolo to tell me to put it
somewhere else anyway :-)


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH 7/8] gitlab: disable optimization and debug symbols in msys build

[Daniel P. Berrangé]

Building at -O2, adds 33% to the build time, over -O2. IOW a build that
takes 45 minutes at -O0, takes 60 minutes at -O2. Turning off debug
symbols drops it further, down to 38 minutes.

IOW, a "-O2 -g" build is 58% slower than a "-O0" build on msys in the
gitlab CI windows shared runners.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/windows.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index 34109a80f2..552e3b751d 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -113,7 +113,7 @@ msys2-64bit:
     # commit 9f8e6cad65a6 ("gitlab-ci: Speed up the msys2-64bit job by using --without-default-devices"
     # changed to compile QEMU with the --without-default-devices switch
     # for the msys2 64-bit job, due to the build could not complete within
-    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices
+    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices -Ddebug=false -Doptimization=0
     # qTests don't run successfully with "--without-default-devices",
     # so let's exclude the qtests from CI for now.
     TEST_ARGS: --no-suite qtest
@@ -123,5 +123,5 @@ msys2-32bit:
   variables:
     MINGW_TARGET: mingw-w64-i686
     MSYSTEM: MINGW32
-    CONFIGURE_ARGS:  --target-list=ppc64-softmmu
+    CONFIGURE_ARGS:  --target-list=ppc64-softmmu -Ddebug=false -Doptimization=0
     TEST_ARGS: --no-suite qtest
-- 
2.41.0

Re: [PATCH 7/8] gitlab: disable optimization and debug symbols in msys build

[Thomas Huth]

On 01/08/2023 15.04, Daniel P. Berrangé wrote:
> Building at -O2, adds 33% to the build time, over -O2. IOW a build that
> takes 45 minutes at -O0, takes 60 minutes at -O2. Turning off debug
> symbols drops it further, down to 38 minutes.
> 
> IOW, a "-O2 -g" build is 58% slower than a "-O0" build on msys in the
> gitlab CI windows shared runners.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/windows.yml | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 34109a80f2..552e3b751d 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -113,7 +113,7 @@ msys2-64bit:
>       # commit 9f8e6cad65a6 ("gitlab-ci: Speed up the msys2-64bit job by using --without-default-devices"
>       # changed to compile QEMU with the --without-default-devices switch
>       # for the msys2 64-bit job, due to the build could not complete within
> -    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices
> +    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices -Ddebug=false -Doptimization=0
>       # qTests don't run successfully with "--without-default-devices",
>       # so let's exclude the qtests from CI for now.
>       TEST_ARGS: --no-suite qtest
> @@ -123,5 +123,5 @@ msys2-32bit:
>     variables:
>       MINGW_TARGET: mingw-w64-i686
>       MSYSTEM: MINGW32
> -    CONFIGURE_ARGS:  --target-list=ppc64-softmmu
> +    CONFIGURE_ARGS:  --target-list=ppc64-softmmu -Ddebug=false -Doptimization=0
>       TEST_ARGS: --no-suite qtest

This is IMHO a very good idea! But I think for now it's enough if you only 
change the 64-bit, isn't it?

  Thomas

Re: [PATCH 7/8] gitlab: disable optimization and debug symbols in msys build

[Daniel P. Berrangé]

On Tue, Aug 01, 2023 at 08:44:02PM +0200, Thomas Huth wrote:
> On 01/08/2023 15.04, Daniel P. Berrangé wrote:
> > Building at -O2, adds 33% to the build time, over -O2. IOW a build that
> > takes 45 minutes at -O0, takes 60 minutes at -O2. Turning off debug
> > symbols drops it further, down to 38 minutes.
> > 
> > IOW, a "-O2 -g" build is 58% slower than a "-O0" build on msys in the
> > gitlab CI windows shared runners.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   .gitlab-ci.d/windows.yml | 4 ++--
> >   1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> > index 34109a80f2..552e3b751d 100644
> > --- a/.gitlab-ci.d/windows.yml
> > +++ b/.gitlab-ci.d/windows.yml
> > @@ -113,7 +113,7 @@ msys2-64bit:
> >       # commit 9f8e6cad65a6 ("gitlab-ci: Speed up the msys2-64bit job by using --without-default-devices"
> >       # changed to compile QEMU with the --without-default-devices switch
> >       # for the msys2 64-bit job, due to the build could not complete within
> > -    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices
> > +    CONFIGURE_ARGS:  --target-list=x86_64-softmmu --without-default-devices -Ddebug=false -Doptimization=0
> >       # qTests don't run successfully with "--without-default-devices",
> >       # so let's exclude the qtests from CI for now.
> >       TEST_ARGS: --no-suite qtest
> > @@ -123,5 +123,5 @@ msys2-32bit:
> >     variables:
> >       MINGW_TARGET: mingw-w64-i686
> >       MSYSTEM: MINGW32
> > -    CONFIGURE_ARGS:  --target-list=ppc64-softmmu
> > +    CONFIGURE_ARGS:  --target-list=ppc64-softmmu -Ddebug=false -Doptimization=0
> >       TEST_ARGS: --no-suite qtest
> 
> This is IMHO a very good idea! But I think for now it's enough if you only
> change the 64-bit, isn't it?

My thought was if we do it for 32-bit too, we can enable some more targets
to get a more comprehensive build.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH 8/8] gitlab: disable FF_SCRIPT_SECTIONS on msys jobs

[Daniel P. Berrangé]

The FF_SCRIPT_SECTIONS=1 variable should ordinarily cause output from
each line of the job script to be presented in a collapsible section
with execution time listed.

While it works on Linux shared runners, when used with Windows runners
with PowerShell, this option does not create any sections, and actually
causes echo'ing of commands to be disabled, making it even worse to
debug the jobs.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/windows.yml | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index 552e3b751d..cd7622a761 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -12,6 +12,10 @@
   needs: []
   stage: build
   timeout: 80m
+  variables:
+    # This feature doesn't (currently) work with PowerShell, it stops
+    # the echo'ing of commands being run and doesn't show any timing
+    FF_SCRIPT_SECTIONS: 0
   artifacts:
     name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
     expire_in: 7 days
-- 
2.41.0

Re: [PATCH 8/8] gitlab: disable FF_SCRIPT_SECTIONS on msys jobs

[Thomas Huth]

On 01/08/2023 15.04, Daniel P. Berrangé wrote:
> The FF_SCRIPT_SECTIONS=1 variable should ordinarily cause output from
> each line of the job script to be presented in a collapsible section
> with execution time listed.
> 
> While it works on Linux shared runners, when used with Windows runners
> with PowerShell, this option does not create any sections, and actually
> causes echo'ing of commands to be disabled, making it even worse to
> debug the jobs.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/windows.yml | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> index 552e3b751d..cd7622a761 100644
> --- a/.gitlab-ci.d/windows.yml
> +++ b/.gitlab-ci.d/windows.yml
> @@ -12,6 +12,10 @@
>     needs: []
>     stage: build
>     timeout: 80m
> +  variables:
> +    # This feature doesn't (currently) work with PowerShell, it stops
> +    # the echo'ing of commands being run and doesn't show any timing
> +    FF_SCRIPT_SECTIONS: 0
>     artifacts:
>       name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
>       expire_in: 7 days

Acked-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 0/8] gitlab: speed up msys windows jobs with GCC

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> This is an alternative and/or complementary to Thomas' proposal
> to use CLang with msys:
>
>   https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html
>
> First of all, the current msys installer we're using is over 12
> months out of date. Thus after running the install, pacman then
> replaces most of what we've just installed with new downloaded
> content. Using the most update installer cuts 3+1/2 minutes off
> the msys install time - 7 minutes becomes 3+1/2.
>
> Secondly, QEMU defaults to compiling with -O2 and this is more
> computationally expensive for GCC. Switching to -O0 drops the
> build time from 60 minutes down to 45 minutes.

From the fine manual[*]: "The effectiveness of some warnings depends on
optimizations also being enabled.  For example '-Wsuggest-final-types'
is more effective with link-time optimization and some instances of
other warnings may not be issued at all unless optimization is enabled.
While optimization in general improves the efficacy of control and data
flow sensitive warnings, in some cases it may also cause false
positives."  Do we care?

[...]


[*] https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Warning-Options.html

Re: [PATCH 0/8] gitlab: speed up msys windows jobs with GCC

[Daniel P. Berrangé]

On Tue, Aug 01, 2023 at 03:53:22PM +0200, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > This is an alternative and/or complementary to Thomas' proposal
> > to use CLang with msys:
> >
> >   https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html
> >
> > First of all, the current msys installer we're using is over 12
> > months out of date. Thus after running the install, pacman then
> > replaces most of what we've just installed with new downloaded
> > content. Using the most update installer cuts 3+1/2 minutes off
> > the msys install time - 7 minutes becomes 3+1/2.
> >
> > Secondly, QEMU defaults to compiling with -O2 and this is more
> > computationally expensive for GCC. Switching to -O0 drops the
> > build time from 60 minutes down to 45 minutes.
> 
> From the fine manual[*]: "The effectiveness of some warnings depends on
> optimizations also being enabled.  For example '-Wsuggest-final-types'
> is more effective with link-time optimization and some instances of
> other warnings may not be issued at all unless optimization is enabled.
> While optimization in general improves the efficacy of control and data
> flow sensitive warnings, in some cases it may also cause false
> positives."  Do we care?

In general, yes, we do care.

In this specific case though, we're battling to figure out the lesser
of multiple evils.

Right now we configure with:

  --target-list=x86_64-softmmu --without-default-devices 

and so with optimization enabled, we'll get good warning coverage of
a small amount of code, except we don't because people started
ignoring the msys jobs as they timeout too frequently.

If we can use Thomas' clang switch or my -O0 patches, we can get
within the timeout, so people can trust the job once again. If we
can do both ideas and cut the time even more, then we can enable
more features (perhaps drop --without-default-devices).

So the warnings might not be quite as good, but we'll have the
warnings across a larger amount of code.

Alot of the warnings from the Linux/macOS builds will also apply
in the Windows builds. I think on balance I'd probably prefer us
to build a larger amount of code for Windows. This is in context
of free shared runners at least.

As a more drastic option, we might need to consider using the
Azure credits for Windows  runners too. If we could have bigger
VMs for Windows CI, we can build more and have better warnings
at the same time. 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/8] gitlab: speed up msys windows jobs with GCC

[Thomas Huth]

On 01/08/2023 16.35, Daniel P. Berrangé wrote:
> On Tue, Aug 01, 2023 at 03:53:22PM +0200, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>>
>>> This is an alternative and/or complementary to Thomas' proposal
>>> to use CLang with msys:
>>>
>>>    https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html
>>>
>>> First of all, the current msys installer we're using is over 12
>>> months out of date. Thus after running the install, pacman then
>>> replaces most of what we've just installed with new downloaded
>>> content. Using the most update installer cuts 3+1/2 minutes off
>>> the msys install time - 7 minutes becomes 3+1/2.
>>>
>>> Secondly, QEMU defaults to compiling with -O2 and this is more
>>> computationally expensive for GCC. Switching to -O0 drops the
>>> build time from 60 minutes down to 45 minutes.
>>
>>  From the fine manual[*]: "The effectiveness of some warnings depends on
>> optimizations also being enabled.  For example '-Wsuggest-final-types'
>> is more effective with link-time optimization and some instances of
>> other warnings may not be issued at all unless optimization is enabled.
>> While optimization in general improves the efficacy of control and data
>> flow sensitive warnings, in some cases it may also cause false
>> positives."  Do we care?
> 
> In general, yes, we do care.
> 
> In this specific case though, we're battling to figure out the lesser
> of multiple evils.

I agree. Additionally, we also test compiling for Windows with the MinGW 
cross compiler suite in a Fedora container, and we still use the default 
optimization there, so we should have that covered.

  Thomas

Re: [PATCH 0/8] gitlab: speed up msys windows jobs with GCC

[Thomas Huth]

On 01/08/2023 15.03, Daniel P. Berrangé wrote:
> This is an alternative and/or complementary to Thomas' proposal
> to use CLang with msys:
> 
>    https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05402.html
> 
> First of all, the current msys installer we're using is over 12
> months out of date. Thus after running the install, pacman then
> replaces most of what we've just installed with new downloaded
> content. Using the most update installer cuts 3+1/2 minutes off
> the msys install time - 7 minutes becomes 3+1/2.
> 
> Secondly, QEMU defaults to compiling with -O2 and this is more
> computationally expensive for GCC. Switching to -O0 drops the
> build time from 60 minutes down to 45 minutes.
> 
> Thirdly, including debug symbols also has an overhead, and turning
> that off reduces time still further down to 38 minutes.
> 
> IOW, between all three changes, we can cut approx 25-26 minutes
> off the job execution time, bringing it nicely within the job
> timeout.
> 
> The actually phase of installing the mingw deps still accounts
> for about 10 minutes and has not been optimized.
> 
> Possibly the same trick of -O0 and skipping -g would also help
> the clang alternative Thomas' proposed. If so, that could be
> enough to let us enable more features / targets during the
> msys build.

I really like the idea! And I guess my idea with Clang needs some more work 
'til it is acceptable, so let's go with your idea for now to fix the timeout 
problem in the CI ... we can still optimize later with Clang in case we 
found a good solution for that ms_struct problem...

  Thomas