* [PATCH] gitlab-ci: Test compilation on Windows with MSYS2
@ 2021-11-15 14:06 Thomas Huth
2021-11-15 14:29 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 6+ messages in thread
From: Thomas Huth @ 2021-11-15 14:06 UTC (permalink / raw)
To: qemu-devel, Alex Bennée; +Cc: Yonggang Luo, Philippe Mathieu-Daudé
Gitlab also provides runners with Windows, we can use them to
test compilation with MSYS2, in both, 64-bit and 32-bit.
However, it takes quite a long time to set up the VM, so to
stay in the 1h time frame, we can only compile and check one
target here.
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
"make check" recently broke on MSYS2, and nobody noticed since apparently
hardly anybody looks at the cirrus-CI output ... so here's another try
to get some more test coverage in this area in the gitlab-CI instead.
Patch needs to be applied after the "tests/unit/test-smp-parse:
Make an unified name for the tested machine" patch to get "make check"
fixed first.
RFC -> v1:
- Use cache to speed up installation a little bit
- Add a 32-bit builder, too
.gitlab-ci.d/qemu-project.yml | 1 +
.gitlab-ci.d/windows.yml | 98 +++++++++++++++++++++++++++++++++++
2 files changed, 99 insertions(+)
create mode 100644 .gitlab-ci.d/windows.yml
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index b3d79bc429..871262fe0e 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -11,3 +11,4 @@ include:
- local: '/.gitlab-ci.d/static_checks.yml'
- local: '/.gitlab-ci.d/custom-runners.yml'
- local: '/.gitlab-ci.d/cirrus.yml'
+ - local: '/.gitlab-ci.d/windows.yml'
diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
new file mode 100644
index 0000000000..309f7e7fb8
--- /dev/null
+++ b/.gitlab-ci.d/windows.yml
@@ -0,0 +1,98 @@
+.shared_msys2_builder:
+ tags:
+ - shared-windows
+ - windows
+ - windows-1809
+ cache:
+ key: "${CI_JOB_NAME}-cache"
+ paths:
+ - ${CI_PROJECT_DIR}/msys64/var/cache
+ needs: []
+ stage: build
+ timeout: 70m
+ before_script:
+ - If ( !(Test-Path -Path msys64\var\cache ) ) {
+ mkdir msys64\var\cache
+ }
+ - If ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
+ Invoke-WebRequest
+ "https://github.com/msys2/msys2-installer/releases/download/2021-07-25/msys2-base-x86_64-20210725.sfx.exe"
+ -outfile "msys64\var\cache\msys2.exe"
+ }
+ - msys64\var\cache\msys2.exe -y
+ - ((Get-Content -path .\msys64\etc\\post-install\\07-pacman-key.post -Raw)
+ -replace '--refresh-keys', '--version') |
+ Set-Content -Path ${CI_PROJECT_DIR}\msys64\etc\\post-install\\07-pacman-key.post
+ - .\msys64\usr\bin\bash -lc "sed -i 's/^CheckSpace/#CheckSpace/g' /etc/pacman.conf"
+ - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu' # Core update
+ - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu' # Normal update
+ - taskkill /F /FI "MODULES eq msys-2.0.dll"
+
+msys2-64bit:
+ extends: .shared_msys2_builder
+ script:
+ - .\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
+ diffutils git grep make sed
+ mingw-w64-x86_64-capstone
+ mingw-w64-x86_64-curl
+ mingw-w64-x86_64-cyrus-sasl
+ mingw-w64-x86_64-gcc
+ mingw-w64-x86_64-glib2
+ mingw-w64-x86_64-gnutls
+ mingw-w64-x86_64-libnfs
+ mingw-w64-x86_64-libpng
+ mingw-w64-x86_64-libssh
+ mingw-w64-x86_64-libtasn1
+ mingw-w64-x86_64-libusb
+ mingw-w64-x86_64-libxml2
+ mingw-w64-x86_64-nettle
+ mingw-w64-x86_64-ninja
+ mingw-w64-x86_64-pixman
+ mingw-w64-x86_64-pkgconf
+ mingw-w64-x86_64-python
+ mingw-w64-x86_64-SDL2
+ mingw-w64-x86_64-SDL2_image
+ mingw-w64-x86_64-snappy
+ mingw-w64-x86_64-usbredir
+ mingw-w64-x86_64-zstd "
+ - $env:CHERE_INVOKING = 'yes' # Preserve the current working directory
+ - $env:MSYSTEM = 'MINGW64' # Start a 64 bit Mingw environment
+ - .\msys64\usr\bin\bash -lc './configure --target-list=x86_64-softmmu
+ --enable-capstone=system'
+ - .\msys64\usr\bin\bash -lc "sed -i '/^ROMS=/d' build/config-host.mak"
+ - .\msys64\usr\bin\bash -lc 'make -j2'
+ - .\msys64\usr\bin\bash -lc 'make check'
+
+msys2-32bit:
+ extends: .shared_msys2_builder
+ script:
+ - .\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
+ diffutils git grep make sed
+ mingw-w64-i686-capstone
+ mingw-w64-i686-curl
+ mingw-w64-i686-cyrus-sasl
+ mingw-w64-i686-gcc
+ mingw-w64-i686-glib2
+ mingw-w64-i686-gnutls
+ mingw-w64-i686-gtk3
+ mingw-w64-i686-libgcrypt
+ mingw-w64-i686-libjpeg-turbo
+ mingw-w64-i686-libssh
+ mingw-w64-i686-libtasn1
+ mingw-w64-i686-libusb
+ mingw-w64-i686-libxml2
+ mingw-w64-i686-lzo2
+ mingw-w64-i686-ninja
+ mingw-w64-i686-pixman
+ mingw-w64-i686-pkgconf
+ mingw-w64-i686-python
+ mingw-w64-i686-snappy
+ mingw-w64-i686-usbredir "
+ - $env:CHERE_INVOKING = 'yes' # Preserve the current working directory
+ - $env:MSYSTEM = 'MINGW32' # Start a 32-bit MinG environment
+ - mkdir output
+ - cd output
+ - ..\msys64\usr\bin\bash -lc "../configure --target-list=ppc64-softmmu
+ --enable-capstone=system"
+ - ..\msys64\usr\bin\bash -lc 'make -j2'
+ - ..\msys64\usr\bin\bash -lc 'make check'
--
2.27.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] gitlab-ci: Test compilation on Windows with MSYS2
2021-11-15 14:06 [PATCH] gitlab-ci: Test compilation on Windows with MSYS2 Thomas Huth
@ 2021-11-15 14:29 ` Philippe Mathieu-Daudé
2021-11-16 7:05 ` Marc-André Lureau
0 siblings, 1 reply; 6+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-11-15 14:29 UTC (permalink / raw)
To: Thomas Huth, qemu-devel, Alex Bennée; +Cc: Yonggang Luo
On 11/15/21 15:06, Thomas Huth wrote:
> Gitlab also provides runners with Windows, we can use them to
> test compilation with MSYS2, in both, 64-bit and 32-bit.
>
> However, it takes quite a long time to set up the VM, so to
> stay in the 1h time frame, we can only compile and check one
> target here.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> "make check" recently broke on MSYS2, and nobody noticed since apparently
> hardly anybody looks at the cirrus-CI output ... so here's another try
> to get some more test coverage in this area in the gitlab-CI instead.
> Patch needs to be applied after the "tests/unit/test-smp-parse:
> Make an unified name for the tested machine" patch to get "make check"
> fixed first.
>
> RFC -> v1:
> - Use cache to speed up installation a little bit
> - Add a 32-bit builder, too
>
> .gitlab-ci.d/qemu-project.yml | 1 +
> .gitlab-ci.d/windows.yml | 98 +++++++++++++++++++++++++++++++++++
> 2 files changed, 99 insertions(+)
> create mode 100644 .gitlab-ci.d/windows.yml
>
> diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
> index b3d79bc429..871262fe0e 100644
> --- a/.gitlab-ci.d/qemu-project.yml
> +++ b/.gitlab-ci.d/qemu-project.yml
> @@ -11,3 +11,4 @@ include:
> - local: '/.gitlab-ci.d/static_checks.yml'
> - local: '/.gitlab-ci.d/custom-runners.yml'
> - local: '/.gitlab-ci.d/cirrus.yml'
> + - local: '/.gitlab-ci.d/windows.yml'
> diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> new file mode 100644
> index 0000000000..309f7e7fb8
> --- /dev/null
> +++ b/.gitlab-ci.d/windows.yml
> @@ -0,0 +1,98 @@
> +.shared_msys2_builder:
> + tags:
> + - shared-windows
> + - windows
> + - windows-1809
> + cache:
> + key: "${CI_JOB_NAME}-cache"
It would be nice to cache the shared 'before_script' part,
but it doesn't seems trivial; meanwhile this patch works and
is KISS, so:
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> + paths:
> + - ${CI_PROJECT_DIR}/msys64/var/cache
> + needs: []
> + stage: build
> + timeout: 70m
> + before_script:
> + - If ( !(Test-Path -Path msys64\var\cache ) ) {
> + mkdir msys64\var\cache
> + }
> + - If ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
> + Invoke-WebRequest
> + "https://github.com/msys2/msys2-installer/releases/download/2021-07-25/msys2-base-x86_64-20210725.sfx.exe"
> + -outfile "msys64\var\cache\msys2.exe"
> + }
> + - msys64\var\cache\msys2.exe -y
> + - ((Get-Content -path .\msys64\etc\\post-install\\07-pacman-key.post -Raw)
> + -replace '--refresh-keys', '--version') |
> + Set-Content -Path ${CI_PROJECT_DIR}\msys64\etc\\post-install\\07-pacman-key.post
> + - .\msys64\usr\bin\bash -lc "sed -i 's/^CheckSpace/#CheckSpace/g' /etc/pacman.conf"
> + - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu' # Core update
> + - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu' # Normal update
> + - taskkill /F /FI "MODULES eq msys-2.0.dll"
> +
> +msys2-64bit:
> + extends: .shared_msys2_builder
> + script:
> + - .\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
> + diffutils git grep make sed
> + mingw-w64-x86_64-capstone
[...]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gitlab-ci: Test compilation on Windows with MSYS2
2021-11-15 14:29 ` Philippe Mathieu-Daudé
@ 2021-11-16 7:05 ` Marc-André Lureau
2021-11-16 8:26 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 6+ messages in thread
From: Marc-André Lureau @ 2021-11-16 7:05 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Yonggang Luo, Thomas Huth, Alex Bennée, QEMU
[-- Attachment #1: Type: text/plain, Size: 3838 bytes --]
Hi
On Mon, Nov 15, 2021 at 6:31 PM Philippe Mathieu-Daudé <f4bug@amsat.org>
wrote:
> On 11/15/21 15:06, Thomas Huth wrote:
> > Gitlab also provides runners with Windows, we can use them to
> > test compilation with MSYS2, in both, 64-bit and 32-bit.
> >
> > However, it takes quite a long time to set up the VM, so to
> > stay in the 1h time frame, we can only compile and check one
> > target here.
>
I wonder why gitlab does not offer the docker executor. On the freedesktop
gitlab instance, they have windows docker executor, which speeds up the
build time. Maybe we could also have our own Windows runner for qemu?
>
> > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > ---
> > "make check" recently broke on MSYS2, and nobody noticed since
> apparently
> > hardly anybody looks at the cirrus-CI output ... so here's another try
> > to get some more test coverage in this area in the gitlab-CI instead.
> > Patch needs to be applied after the "tests/unit/test-smp-parse:
> > Make an unified name for the tested machine" patch to get "make check"
> > fixed first.
> >
> > RFC -> v1:
> > - Use cache to speed up installation a little bit
> > - Add a 32-bit builder, too
> >
> > .gitlab-ci.d/qemu-project.yml | 1 +
> > .gitlab-ci.d/windows.yml | 98 +++++++++++++++++++++++++++++++++++
> > 2 files changed, 99 insertions(+)
> > create mode 100644 .gitlab-ci.d/windows.yml
> >
> > diff --git a/.gitlab-ci.d/qemu-project.yml
> b/.gitlab-ci.d/qemu-project.yml
> > index b3d79bc429..871262fe0e 100644
> > --- a/.gitlab-ci.d/qemu-project.yml
> > +++ b/.gitlab-ci.d/qemu-project.yml
> > @@ -11,3 +11,4 @@ include:
> > - local: '/.gitlab-ci.d/static_checks.yml'
> > - local: '/.gitlab-ci.d/custom-runners.yml'
> > - local: '/.gitlab-ci.d/cirrus.yml'
> > + - local: '/.gitlab-ci.d/windows.yml'
> > diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
> > new file mode 100644
> > index 0000000000..309f7e7fb8
> > --- /dev/null
> > +++ b/.gitlab-ci.d/windows.yml
> > @@ -0,0 +1,98 @@
> > +.shared_msys2_builder:
> > + tags:
> > + - shared-windows
> > + - windows
> > + - windows-1809
> > + cache:
> > + key: "${CI_JOB_NAME}-cache"
>
> It would be nice to cache the shared 'before_script' part,
> but it doesn't seems trivial; meanwhile this patch works and
> is KISS, so:
>
> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>
> > + paths:
> > + - ${CI_PROJECT_DIR}/msys64/var/cache
> > + needs: []
> > + stage: build
> > + timeout: 70m
> > + before_script:
> > + - If ( !(Test-Path -Path msys64\var\cache ) ) {
> > + mkdir msys64\var\cache
> > + }
> > + - If ( !(Test-Path -Path msys64\var\cache\msys2.exe ) ) {
> > + Invoke-WebRequest
> > + "
> https://github.com/msys2/msys2-installer/releases/download/2021-07-25/msys2-base-x86_64-20210725.sfx.exe
> "
> > + -outfile "msys64\var\cache\msys2.exe"
> > + }
> > + - msys64\var\cache\msys2.exe -y
> > + - ((Get-Content -path .\msys64\etc\\post-install\\07-pacman-key.post
> -Raw)
> > + -replace '--refresh-keys', '--version') |
> > + Set-Content -Path ${CI_PROJECT_DIR}\msys64\etc\\post-install\\
> 07-pacman-key.post
> > + - .\msys64\usr\bin\bash -lc "sed -i 's/^CheckSpace/#CheckSpace/g'
> /etc/pacman.conf"
> > + - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu' # Core update
> > + - .\msys64\usr\bin\bash -lc 'pacman --noconfirm -Syuu' # Normal
> update
> > + - taskkill /F /FI "MODULES eq msys-2.0.dll"
> > +
> > +msys2-64bit:
> > + extends: .shared_msys2_builder
> > + script:
> > + - .\msys64\usr\bin\bash -lc "pacman -Sy --noconfirm --needed
> > + diffutils git grep make sed
> > + mingw-w64-x86_64-capstone
> [...]
>
>
--
Marc-André Lureau
[-- Attachment #2: Type: text/html, Size: 5424 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gitlab-ci: Test compilation on Windows with MSYS2
2021-11-16 7:05 ` Marc-André Lureau
@ 2021-11-16 8:26 ` Philippe Mathieu-Daudé
2021-11-17 13:52 ` Alex Bennée
0 siblings, 1 reply; 6+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-11-16 8:26 UTC (permalink / raw)
To: Marc-André Lureau; +Cc: Yonggang Luo, Alex Bennée, Thomas Huth, QEMU
On 11/16/21 08:05, Marc-André Lureau wrote:
> Hi
>
> On Mon, Nov 15, 2021 at 6:31 PM Philippe Mathieu-Daudé <f4bug@amsat.org
> <mailto:f4bug@amsat.org>> wrote:
>
> On 11/15/21 15:06, Thomas Huth wrote:
> > Gitlab also provides runners with Windows, we can use them to
> > test compilation with MSYS2, in both, 64-bit and 32-bit.
> >
> > However, it takes quite a long time to set up the VM, so to
> > stay in the 1h time frame, we can only compile and check one
> > target here.
>
>
> I wonder why gitlab does not offer the docker executor. On the
> freedesktop gitlab instance, they have windows docker executor, which
> speeds up the build time. Maybe we could also have our own Windows
> runner for qemu?
We could, foss.org provides the QEMU project with x86 VMs resources
we are not using. What we miss is a sysadmin willing to setup &
maintain a such runner.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gitlab-ci: Test compilation on Windows with MSYS2
2021-11-16 8:26 ` Philippe Mathieu-Daudé
@ 2021-11-17 13:52 ` Alex Bennée
2021-11-17 14:44 ` Daniel P. Berrangé
0 siblings, 1 reply; 6+ messages in thread
From: Alex Bennée @ 2021-11-17 13:52 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Yonggang Luo, Marc-André Lureau, Thomas Huth, QEMU
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> On 11/16/21 08:05, Marc-André Lureau wrote:
>> Hi
>>
>> On Mon, Nov 15, 2021 at 6:31 PM Philippe Mathieu-Daudé <f4bug@amsat.org
>> <mailto:f4bug@amsat.org>> wrote:
>>
>> On 11/15/21 15:06, Thomas Huth wrote:
>> > Gitlab also provides runners with Windows, we can use them to
>> > test compilation with MSYS2, in both, 64-bit and 32-bit.
>> >
>> > However, it takes quite a long time to set up the VM, so to
>> > stay in the 1h time frame, we can only compile and check one
>> > target here.
>>
>>
>> I wonder why gitlab does not offer the docker executor. On the
>> freedesktop gitlab instance, they have windows docker executor, which
>> speeds up the build time. Maybe we could also have our own Windows
>> runner for qemu?
>
> We could, foss.org provides the QEMU project with x86 VMs resources
> we are not using. What we miss is a sysadmin willing to setup &
> maintain a such runner.
I think we might also have Azure credits from MS, but the same issues
about admin and setup probably exist.
--
Alex Bennée
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] gitlab-ci: Test compilation on Windows with MSYS2
2021-11-17 13:52 ` Alex Bennée
@ 2021-11-17 14:44 ` Daniel P. Berrangé
0 siblings, 0 replies; 6+ messages in thread
From: Daniel P. Berrangé @ 2021-11-17 14:44 UTC (permalink / raw)
To: Alex Bennée
Cc: Thomas Huth, Yonggang Luo, Marc-André Lureau,
Philippe Mathieu-Daudé, QEMU
On Wed, Nov 17, 2021 at 01:52:57PM +0000, Alex Bennée wrote:
>
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>
> > On 11/16/21 08:05, Marc-André Lureau wrote:
> >> Hi
> >>
> >> On Mon, Nov 15, 2021 at 6:31 PM Philippe Mathieu-Daudé <f4bug@amsat.org
> >> <mailto:f4bug@amsat.org>> wrote:
> >>
> >> On 11/15/21 15:06, Thomas Huth wrote:
> >> > Gitlab also provides runners with Windows, we can use them to
> >> > test compilation with MSYS2, in both, 64-bit and 32-bit.
> >> >
> >> > However, it takes quite a long time to set up the VM, so to
> >> > stay in the 1h time frame, we can only compile and check one
> >> > target here.
> >>
> >>
> >> I wonder why gitlab does not offer the docker executor. On the
> >> freedesktop gitlab instance, they have windows docker executor, which
> >> speeds up the build time. Maybe we could also have our own Windows
> >> runner for qemu?
> >
> > We could, foss.org provides the QEMU project with x86 VMs resources
> > we are not using. What we miss is a sysadmin willing to setup &
> > maintain a such runner.
>
> I think we might also have Azure credits from MS, but the same issues
> about admin and setup probably exist.
I've never explored this in any way, but IIUC, Azure should have
ability to run *windows* containers. If that's possible, then we
could avoid the admin burden of a VM and just use throwaway windows
containers as we do for Linux.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-11-17 14:46 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-11-15 14:06 [PATCH] gitlab-ci: Test compilation on Windows with MSYS2 Thomas Huth
2021-11-15 14:29 ` Philippe Mathieu-Daudé
2021-11-16 7:05 ` Marc-André Lureau
2021-11-16 8:26 ` Philippe Mathieu-Daudé
2021-11-17 13:52 ` Alex Bennée
2021-11-17 14:44 ` Daniel P. Berrangé
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).