[RFC PATCH] gitlab-ci: Test compilation on Windows with MSYS2

[RFC PATCH] gitlab-ci: Test compilation on Windows with MSYS2

[Thomas Huth]

Gitlab also provides runners with Windows, we can use them to
test compilation with MSYS2.

However, it takes quite a long time to set up the VM, so to
stay in the 1h time frame, we can only compile and check one
target here. And there is also still a problem with compiling
the multiboot.bin in pc-bios/optionrom/, so this is currently
disabled, too.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 I'm sending this out as an RFC in case somebody else wants to play
 with this. Considering that the corresponding job in Cirrus-CI compiles
 way more targets within 1 hour, we might rather want to keep using
 the Cirrus-CI job instead?

 .gitlab-ci.d/qemu-project.yml |  1 +
 .gitlab-ci.d/windows.yml      | 59 +++++++++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+)
 create mode 100644 .gitlab-ci.d/windows.yml

diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index 64cb2ba1da..f1a34038af 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -9,3 +9,4 @@ include:
   - local: '/.gitlab-ci.d/crossbuilds.yml'
   - local: '/.gitlab-ci.d/buildtest.yml'
   - local: '/.gitlab-ci.d/static_checks.yml'
+  - local: '/.gitlab-ci.d/windows.yml'
diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
new file mode 100644
index 0000000000..1874a06739
--- /dev/null
+++ b/.gitlab-ci.d/windows.yml
@@ -0,0 +1,59 @@
+.shared_windows_runners:
+  tags:
+  - shared-windows
+  - windows
+  - windows-1809
+
+msys2-build:
+  extends: .shared_windows_runners
+  stage: build
+  before_script:
+  - Invoke-WebRequest
+    "https://github.com/msys2/msys2-installer/releases/download/nightly-x86_64/msys2-base-x86_64-latest.sfx.exe"
+    -outfile "msys2.exe"
+  - .\msys2.exe -y -oC:\
+  - del msys2.exe
+  - ((Get-Content -path C:\msys64\etc\\post-install\\07-pacman-key.post -Raw)
+      -replace '--refresh-keys', '--version') |
+     Set-Content -Path C:\msys64\etc\\post-install\\07-pacman-key.post
+  - C:\msys64\usr\bin\bash -lc "sed -i 's/^CheckSpace/#CheckSpace/g' /etc/pacman.conf"
+  - C:\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu'  # Core update
+  - C:\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu'  # Normal update
+  - C:\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
+      diffutils git grep make pkg-config sed
+      mingw-w64-x86_64-capstone
+      mingw-w64-x86_64-curl
+      mingw-w64-x86_64-cyrus-sasl
+      mingw-w64-x86_64-gcc
+      mingw-w64-x86_64-glib2
+      mingw-w64-x86_64-gnutls
+      mingw-w64-x86_64-gtk3
+      mingw-w64-x86_64-libgcrypt
+      mingw-w64-x86_64-libjpeg-turbo
+      mingw-w64-x86_64-libnfs
+      mingw-w64-x86_64-libpng
+      mingw-w64-x86_64-libssh
+      mingw-w64-x86_64-libtasn1
+      mingw-w64-x86_64-libusb
+      mingw-w64-x86_64-libxml2
+      mingw-w64-x86_64-lzo2
+      mingw-w64-x86_64-make
+      mingw-w64-x86_64-nettle
+      mingw-w64-x86_64-ninja
+      mingw-w64-x86_64-pixman
+      mingw-w64-x86_64-pkgconf
+      mingw-w64-x86_64-python
+      mingw-w64-x86_64-SDL2
+      mingw-w64-x86_64-SDL2_image
+      mingw-w64-x86_64-snappy
+      mingw-w64-x86_64-usbredir
+      mingw-w64-x86_64-zstd "
+  - taskkill /F /FI "MODULES eq msys-2.0.dll"
+  script:
+  - $env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
+  - $env:MSYSTEM = 'MINGW64'     # Start a 64 bit Mingw environment
+  - C:\msys64\usr\bin\bash -lc './configure --target-list=x86_64-softmmu
+      --enable-capstone=system --disable-fdt'
+  - C:\msys64\usr\bin\bash -lc "sed -i '/^ROMS=/d' build/config-host.mak"
+  - C:\msys64\usr\bin\bash -lc 'make -j2'
+  - C:\msys64\usr\bin\bash -lc 'make V=1 check'
-- 
2.27.0

Re: [RFC PATCH] gitlab-ci: Test compilation on Windows with MSYS2

[Daniel P. Berrangé]

On Fri, Jul 09, 2021 at 09:52:18AM +0200, Thomas Huth wrote:
> Gitlab also provides runners with Windows, we can use them to
> test compilation with MSYS2.
> 
> However, it takes quite a long time to set up the VM, so to
> stay in the 1h time frame, we can only compile and check one
> target here. And there is also still a problem with compiling
> the multiboot.bin in pc-bios/optionrom/, so this is currently
> disabled, too.

THe gitlab VM provisioning seems quite fast (2+1/2 minutes),
so I guess you mean the time is sunk in intsalling msys
packages.

In the crurent Cirrus job there's a bunch of code that is
done to cache all the msys install, so that 90% of the time
we're merely extracting a local zipfile.

I wonder if we can do similar caching here. Perhaps even
have 2 seprate jobs running on the windows runner. The
first job can just build a zip file and store it as an
artifact and cache it in gitlab for speed. The second
job can do the actual build ?

> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  I'm sending this out as an RFC in case somebody else wants to play
>  with this. Considering that the corresponding job in Cirrus-CI compiles
>  way more targets within 1 hour, we might rather want to keep using
>  the Cirrus-CI job instead?

Yes, unless we can do the same kind of caching tricks to speed it
all up.

> 
>  .gitlab-ci.d/qemu-project.yml |  1 +
>  .gitlab-ci.d/windows.yml      | 59 +++++++++++++++++++++++++++++++++++
>  2 files changed, 60 insertions(+)
>  create mode 100644 .gitlab-ci.d/windows.yml

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] gitlab-ci: Test compilation on Windows with MSYS2

[Thomas Huth]

On 09/07/2021 10.20, Daniel P. Berrangé wrote:
> On Fri, Jul 09, 2021 at 09:52:18AM +0200, Thomas Huth wrote:
>> Gitlab also provides runners with Windows, we can use them to
>> test compilation with MSYS2.
>>
>> However, it takes quite a long time to set up the VM, so to
>> stay in the 1h time frame, we can only compile and check one
>> target here. And there is also still a problem with compiling
>> the multiboot.bin in pc-bios/optionrom/, so this is currently
>> disabled, too.
> 
> THe gitlab VM provisioning seems quite fast (2+1/2 minutes),
> so I guess you mean the time is sunk in intsalling msys
> packages.
> 
> In the crurent Cirrus job there's a bunch of code that is
> done to cache all the msys install, so that 90% of the time
> we're merely extracting a local zipfile.
> 
> I wonder if we can do similar caching here.

Maybe, yes. It might save 10 - 15 minutes or so, I guess.

> Perhaps even
> have 2 seprate jobs running on the windows runner. The
> first job can just build a zip file and store it as an
> artifact and cache it in gitlab for speed. The second
> job can do the actual build ?

I'm not sure ... the Windows builders are likely single-threaded on Gitlab, 
too, just like the Linux containers. So that the main problem - while we can 
compile with -j8 on Cirrus, it's 8 times slower on Gitlab.

  Thomas

Re: [RFC PATCH] gitlab-ci: Test compilation on Windows with MSYS2

[Daniel P. Berrangé]

On Fri, Jul 09, 2021 at 04:34:54PM +0200, Thomas Huth wrote:
> On 09/07/2021 10.20, Daniel P. Berrangé wrote:
> > On Fri, Jul 09, 2021 at 09:52:18AM +0200, Thomas Huth wrote:
> > > Gitlab also provides runners with Windows, we can use them to
> > > test compilation with MSYS2.
> > > 
> > > However, it takes quite a long time to set up the VM, so to
> > > stay in the 1h time frame, we can only compile and check one
> > > target here. And there is also still a problem with compiling
> > > the multiboot.bin in pc-bios/optionrom/, so this is currently
> > > disabled, too.
> > 
> > THe gitlab VM provisioning seems quite fast (2+1/2 minutes),
> > so I guess you mean the time is sunk in intsalling msys
> > packages.
> > 
> > In the crurent Cirrus job there's a bunch of code that is
> > done to cache all the msys install, so that 90% of the time
> > we're merely extracting a local zipfile.
> > 
> > I wonder if we can do similar caching here.
> 
> Maybe, yes. It might save 10 - 15 minutes or so, I guess.
> 
> > Perhaps even
> > have 2 seprate jobs running on the windows runner. The
> > first job can just build a zip file and store it as an
> > artifact and cache it in gitlab for speed. The second
> > job can do the actual build ?
> 
> I'm not sure ... the Windows builders are likely single-threaded on Gitlab,
> too, just like the Linux containers. So that the main problem - while we can
> compile with -j8 on Cirrus, it's 8 times slower on Gitlab.

Ah yes, good point.

Since we have cirrus integration for freebsd/macos now, we ought todo
the same for Windows too.  I auto-generated the freebsd/mac logic
with libvirt-ci, but I ignored msys since it was somewhat special
in its build commands. We can still set it up manually though to
integrate

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|