[RFC PATCH 0/3] .github/workflows: Add GitHub action for Windows WHPX build

[RFC PATCH 0/3] .github/workflows: Add GitHub action for Windows WHPX build

[Philippe Mathieu-Daudé]

Reposting Stefan's patch in case we want to merge it in
(I 'd rather not, adding them as GitLab jobs instead):
- add MAINTAINERS entry
- improve speed by parallelizing jobs

Philippe Mathieu-Daudé (2):
  MAINTAINERS: Add entry for GitHub 'Actions' config files
  .github/workflows: Use matrix strategy to parallelize builds

Stefan Weil (1):
  .github/workflows: Add GitHub action for Windows build

 .github/workflows/build.sh                | 112 ++++++++++++++++++++++
 .github/workflows/pkg-config-crosswrapper |  37 +++++++
 .github/workflows/win.yml                 |  25 +++++
 MAINTAINERS                               |   6 ++
 4 files changed, 180 insertions(+)
 create mode 100755 .github/workflows/build.sh
 create mode 100755 .github/workflows/pkg-config-crosswrapper
 create mode 100644 .github/workflows/win.yml

-- 
2.21.3

[RFC PATCH 1/3] .github/workflows: Add GitHub action for Windows build

[Philippe Mathieu-Daudé]

From: Stefan Weil <sw@weilnetz.de>

The GitHub action is restricted to https://github.com/qemu/qemu.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
Message-Id: <20200803202042.1869013-1-sw@weilnetz.de>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .github/workflows/build.sh                | 112 ++++++++++++++++++++++
 .github/workflows/pkg-config-crosswrapper |  37 +++++++
 .github/workflows/win.yml                 |  34 +++++++
 3 files changed, 183 insertions(+)
 create mode 100755 .github/workflows/build.sh
 create mode 100755 .github/workflows/pkg-config-crosswrapper
 create mode 100644 .github/workflows/win.yml

diff --git a/.github/workflows/build.sh b/.github/workflows/build.sh
new file mode 100755
index 0000000000..c430a367be
--- /dev/null
+++ b/.github/workflows/build.sh
@@ -0,0 +1,112 @@
+#!/bin/bash
+
+# GitHub actions - Create QEMU installer for Windows
+
+# Author: Stefan Weil (2020)
+
+#~ set -e
+set -x
+
+ARCH=$1
+DLLS="libgcc_s_sjlj-1.dll libgomp-1.dll libstdc++-6.dll"
+
+if test "$ARCH" != "i686"; then
+  ARCH=x86_64
+  DLLS="libgcc_s_seh-1.dll libgomp-1.dll libstdc++-6.dll"
+fi
+
+ROOTDIR=$PWD
+DISTDIR=$ROOTDIR/dist
+HOST=$ARCH-w64-mingw32
+BUILDDIR=bin/ndebug/$HOST
+PKG_ARCH=mingw64-${ARCH/_/-}
+
+# Install cygwin key and add cygwin sources.
+curl -s https://qemu.weilnetz.de/debian/gpg.key | sudo apt-key add -
+echo deb https://qemu.weilnetz.de/debian/ testing contrib | \
+  sudo tee /etc/apt/sources.list.d/cygwin.list
+
+# Install packages.
+sudo apt-get update
+sudo apt-get install --no-install-recommends \
+  mingw-w64-tools nsis \
+  gcc libc6-dev \
+  g++-mingw-w64-${ARCH/_/-} gcc-mingw-w64-${ARCH/_/-} \
+  bison flex gettext python3-sphinx texinfo \
+  $PKG_ARCH-adwaita-icon-theme $PKG_ARCH-cogl $PKG_ARCH-curl \
+  $PKG_ARCH-gmp $PKG_ARCH-gnutls $PKG_ARCH-gtk3 $PKG_ARCH-icu \
+  $PKG_ARCH-libxml2 $PKG_ARCH-ncurses $PKG_ARCH-sdl2 $PKG_ARCH-usbredir
+
+# Workaround for buggy cross pkg-config.
+sudo ln -sf $PWD/.github/workflows/pkg-config-crosswrapper \
+  /usr/bin/$HOST-pkg-config
+
+# Get header files for WHPX API from Mingw-w64 git master.
+(
+sudo mkdir -p /usr/$HOST/sys-include
+cd /usr/$HOST/sys-include
+SF_URLBASE=https://sourceforge.net
+URL=$SF_URLBASE/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include
+sudo curl -s -o winhvemulation.h $URL/winhvemulation.h?format=raw
+sudo curl -s -o winhvplatform.h $URL/winhvplatform.h?format=raw
+sudo curl -s -o winhvplatformdefs.h $URL/winhvplatformdefs.h?format=raw
+sudo ln -s winhvemulation.h WinHvEmulation.h
+sudo ln -s winhvplatform.h WinHvPlatform.h
+sudo ln -s winhvplatformdefs.h WinHvPlatformDefs.h
+)
+
+DLL_PATH=$PWD/dll/$HOST
+
+mkdir -p $DISTDIR
+mkdir -p $DLL_PATH
+
+for dll in $DLLS; do
+  ln -sf /usr/lib/gcc/$HOST/*-win32/$dll $DLL_PATH
+done
+
+DLLS="iconv.dll libatk-1.0-0.dll libbz2-1.dll"
+DLLS="$DLLS libcairo-2.dll libcairo-gobject-2.dll libcurl-4.dll"
+DLLS="$DLLS libeay32.dll libepoxy-0.dll libexpat-1.dll"
+DLLS="$DLLS libffi-6.dll libfontconfig-1.dll libfreetype-6.dll"
+DLLS="$DLLS libgdk-3-0.dll libgdk_pixbuf-2.0-0.dll"
+DLLS="$DLLS libgio-2.0-0.dll libglib-2.0-0.dll libgmodule-2.0-0.dll"
+DLLS="$DLLS libgmp-10.dll libgnutls-30.dll libgobject-2.0-0.dll libgtk-3-0.dll"
+DLLS="$DLLS libharfbuzz-0.dll libhogweed-4.dll libidn2-0.dll libintl-8.dll"
+DLLS="$DLLS libjpeg-8.dll liblzo2-2.dll"
+DLLS="$DLLS libncursesw6.dll libnettle-6.dll libnghttp2-14.dll"
+DLLS="$DLLS libp11-kit-0.dll libpango-1.0-0.dll libpangocairo-1.0-0.dll"
+DLLS="$DLLS libpangoft2-1.0-0.dll libpangowin32-1.0-0.dll libpcre-1.dll"
+DLLS="$DLLS libpixman-1-0.dll libpng16-16.dll libssh2-1.dll libtasn1-6.dll"
+DLLS="$DLLS libunistring-2.dll libusb-1.0.dll libusbredirparser-1.dll"
+DLLS="$DLLS SDL2.dll ssleay32.dll zlib1.dll"
+
+for dll in $DLLS; do
+  ln -sf /usr/$HOST/sys-root/mingw/bin/$dll $DLL_PATH
+done
+
+ln -sf /usr/$HOST/lib/libwinpthread-1.dll $DLL_PATH
+
+# Build QEMU installer.
+
+echo Building $HOST...
+mingw=/usr/$HOST/sys-root/mingw
+mkdir -p $BUILDDIR && cd $BUILDDIR
+
+# Run configure.
+../../../configure --cross-prefix=$HOST- --disable-guest-agent-msi \
+    --disable-werror --enable-whpx \
+    --extra-cflags="-I $mingw/include" \
+    --extra-ldflags="-L $mingw/lib"
+
+# Add config.log to build artifacts.
+cp config.log $DISTDIR/
+
+make
+
+echo Building installers...
+date=$(date +%Y%m%d)
+INSTALLER=$DISTDIR/qemu-$ARCH-setup-$date.exe
+make installer DLL_PATH=$DLL_PATH SIGNCODE=true INSTALLER=$INSTALLER
+
+echo Calculate SHA-512 checksum...
+(cd $DISTDIR; exe=$(basename $INSTALLER); sha512sum $exe >${exe/exe/sha512})
diff --git a/.github/workflows/pkg-config-crosswrapper b/.github/workflows/pkg-config-crosswrapper
new file mode 100755
index 0000000000..768e554194
--- /dev/null
+++ b/.github/workflows/pkg-config-crosswrapper
@@ -0,0 +1,37 @@
+#! /bin/sh
+# pkg-config wrapper for cross-building
+# Sets pkg-config search path to search multiarch and historical cross-compiling paths.
+
+# If the user has already set PKG_CONFIG_LIBDIR, believe it (even if empty):
+# it's documented to be an override
+if [ x"${PKG_CONFIG_LIBDIR+set}" = x ]; then
+  # GNU triplet for the compiler, e.g. i486-linux-gnu for Debian i386,
+  # i686-linux-gnu for Ubuntu i386
+  basename="${0##*/}"
+  triplet="${basename%-pkg-config}"
+  # Normalized multiarch path if any, e.g. i386-linux-gnu for i386
+  multiarch="`dpkg-architecture -t"${triplet}" -qDEB_HOST_MULTIARCH 2>/dev/null`"
+
+  PKG_CONFIG_LIBDIR="/usr/local/${triplet}/lib/pkgconfig"
+  # For a native build we would also want to append /usr/local/lib/pkgconfig
+  # at this point; but this is a cross-building script, so don't
+  PKG_CONFIG_LIBDIR="$PKG_CONFIG_LIBDIR:/usr/local/share/pkgconfig"
+
+  if [ -n "$multiarch" ]; then
+    PKG_CONFIG_LIBDIR="/usr/local/lib/${multiarch}/pkgconfig:$PKG_CONFIG_LIBDIR"
+    PKG_CONFIG_LIBDIR="$PKG_CONFIG_LIBDIR:/usr/lib/${multiarch}/pkgconfig"
+  fi
+
+  PKG_CONFIG_LIBDIR="$PKG_CONFIG_LIBDIR:/usr/${triplet}/lib/pkgconfig"
+  # For a native build we would also want to append /usr/lib/pkgconfig
+  # at this point; but this is a cross-building script, so don't
+  # If you want to allow use of un-multiarched -dev packages for crossing
+  # (at the risk of finding build-arch stuff you didn't want, if not in a clean chroot)
+  # Uncomment the next line:
+  # PKG_CONFIG_LIBDIR="$PKG_CONFIG_LIBDIR:/usr/lib/pkgconfig"
+  PKG_CONFIG_LIBDIR="$PKG_CONFIG_LIBDIR:/usr/share/pkgconfig"
+
+  export PKG_CONFIG_LIBDIR
+fi
+
+exec pkg-config "$@"
diff --git a/.github/workflows/win.yml b/.github/workflows/win.yml
new file mode 100644
index 0000000000..81cf48530f
--- /dev/null
+++ b/.github/workflows/win.yml
@@ -0,0 +1,34 @@
+# GitHub actions - Create QEMU installers for Windows
+
+# The action is restricted to https://github.com/qemu/qemu.
+# That avoids an unnecessary waste of resources when each fork
+# runs the action, too.
+
+name: Cross build for Windows
+
+on: [push]
+
+jobs:
+  build32:
+    if: github.repository == 'qemu/qemu'
+    runs-on: [ubuntu-20.04]
+    steps:
+    - uses: actions/checkout@v2
+    - name: Build QEMU installer (32 bit)
+      run: .github/workflows/build.sh i686
+    - uses: actions/upload-artifact@v1
+      with:
+        name: QEMU Installer Windows 32 bit
+        path: dist
+
+  build64:
+    if: github.repository == 'qemu/qemu'
+    runs-on: [ubuntu-20.04]
+    steps:
+    - uses: actions/checkout@v2
+    - name: Build QEMU installer (64 bit)
+      run: .github/workflows/build.sh x86_64
+    - uses: actions/upload-artifact@v1
+      with:
+        name: QEMU Installer Windows 64 bit
+        path: dist
-- 
2.21.3

[RFC PATCH 2/3] MAINTAINERS: Add entry for GitHub 'Actions' config files

[Philippe Mathieu-Daudé]

Add an entry for the GitHub 'Actions' configuration files
used to build the WHPX Windows installer.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 MAINTAINERS | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 0886eb3d2b..075728421d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3068,6 +3068,12 @@ R: Wainer dos Santos Moschetta <wainersm@redhat.com>
 S: Maintained
 F: .gitlab-ci.yml
 
+GitHub Actions (build Windows installer)
+M: Stefan Weil <sw@weilnetz.de>
+M: Sunil Muthuswamy <sunilmut@microsoft.com>
+S: Maintained
+F: .github/workflows/
+
 Guest Test Compilation Support
 M: Alex Bennée <alex.bennee@linaro.org>
 R: Philippe Mathieu-Daudé <f4bug@amsat.org>
-- 
2.21.3

[RFC PATCH 3/3] .github/workflows: Use matrix strategy to parallelize builds

[Philippe Mathieu-Daudé]

Using sequencial builds, both jobs take almost 2h to build.
By using the matrix strategy we can build the jobs in parallel,
reducing the total build time to 1h12m (as of v5.1.0-rc2).

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .github/workflows/build.sh |  8 ++++----
 .github/workflows/win.yml  | 23 +++++++----------------
 2 files changed, 11 insertions(+), 20 deletions(-)

diff --git a/.github/workflows/build.sh b/.github/workflows/build.sh
index c430a367be..4f2c6b56f3 100755
--- a/.github/workflows/build.sh
+++ b/.github/workflows/build.sh
@@ -7,10 +7,10 @@
 #~ set -e
 set -x
 
-ARCH=$1
-DLLS="libgcc_s_sjlj-1.dll libgomp-1.dll libstdc++-6.dll"
-
-if test "$ARCH" != "i686"; then
+if test "$1" == "32"; then
+  ARCH=i686
+  DLLS="libgcc_s_sjlj-1.dll libgomp-1.dll libstdc++-6.dll"
+else
   ARCH=x86_64
   DLLS="libgcc_s_seh-1.dll libgomp-1.dll libstdc++-6.dll"
 fi
diff --git a/.github/workflows/win.yml b/.github/workflows/win.yml
index 81cf48530f..afd827db8e 100644
--- a/.github/workflows/win.yml
+++ b/.github/workflows/win.yml
@@ -9,26 +9,17 @@ name: Cross build for Windows
 on: [push]
 
 jobs:
-  build32:
+  build:
+    strategy:
+      matrix:
+        arch: [32, 64]
     if: github.repository == 'qemu/qemu'
     runs-on: [ubuntu-20.04]
     steps:
     - uses: actions/checkout@v2
-    - name: Build QEMU installer (32 bit)
-      run: .github/workflows/build.sh i686
+    - name: Build QEMU installer (${{ matrix.arch }} bit)
+      run: .github/workflows/build.sh ${{ matrix.arch }}
     - uses: actions/upload-artifact@v1
       with:
-        name: QEMU Installer Windows 32 bit
-        path: dist
-
-  build64:
-    if: github.repository == 'qemu/qemu'
-    runs-on: [ubuntu-20.04]
-    steps:
-    - uses: actions/checkout@v2
-    - name: Build QEMU installer (64 bit)
-      run: .github/workflows/build.sh x86_64
-    - uses: actions/upload-artifact@v1
-      with:
-        name: QEMU Installer Windows 64 bit
+        name: QEMU Installer Windows ${{ matrix.arch }} bit
         path: dist
-- 
2.21.3

Re: [RFC PATCH 3/3] .github/workflows: Use matrix strategy to parallelize builds

[Stefan Weil]

Hi Philippe,

just a side note: GitHub action runs parallel jobs by default. I use two
jobs (32 and 64 bit builds), so both are built at the same time, see
https://github.com/stweil/qemu/actions.

Regards,

Stefan

Am 04.08.20 um 10:48 schrieb Philippe Mathieu-Daudé:
> Using sequencial builds, both jobs take almost 2h to build.
> By using the matrix strategy we can build the jobs in parallel,
> reducing the total build time to 1h12m (as of v5.1.0-rc2).
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  .github/workflows/build.sh |  8 ++++----
>  .github/workflows/win.yml  | 23 +++++++----------------
>  2 files changed, 11 insertions(+), 20 deletions(-)
>
> diff --git a/.github/workflows/build.sh b/.github/workflows/build.sh
> index c430a367be..4f2c6b56f3 100755
> --- a/.github/workflows/build.sh
> +++ b/.github/workflows/build.sh
> @@ -7,10 +7,10 @@
>  #~ set -e
>  set -x
>  
> -ARCH=$1
> -DLLS="libgcc_s_sjlj-1.dll libgomp-1.dll libstdc++-6.dll"
> -
> -if test "$ARCH" != "i686"; then
> +if test "$1" == "32"; then
> +  ARCH=i686
> +  DLLS="libgcc_s_sjlj-1.dll libgomp-1.dll libstdc++-6.dll"
> +else
>    ARCH=x86_64
>    DLLS="libgcc_s_seh-1.dll libgomp-1.dll libstdc++-6.dll"
>  fi
> diff --git a/.github/workflows/win.yml b/.github/workflows/win.yml
> index 81cf48530f..afd827db8e 100644
> --- a/.github/workflows/win.yml
> +++ b/.github/workflows/win.yml
> @@ -9,26 +9,17 @@ name: Cross build for Windows
>  on: [push]
>  
>  jobs:
> -  build32:
> +  build:
> +    strategy:
> +      matrix:
> +        arch: [32, 64]
>      if: github.repository == 'qemu/qemu'
>      runs-on: [ubuntu-20.04]
>      steps:
>      - uses: actions/checkout@v2
> -    - name: Build QEMU installer (32 bit)
> -      run: .github/workflows/build.sh i686
> +    - name: Build QEMU installer (${{ matrix.arch }} bit)
> +      run: .github/workflows/build.sh ${{ matrix.arch }}
>      - uses: actions/upload-artifact@v1
>        with:
> -        name: QEMU Installer Windows 32 bit
> -        path: dist
> -
> -  build64:
> -    if: github.repository == 'qemu/qemu'
> -    runs-on: [ubuntu-20.04]
> -    steps:
> -    - uses: actions/checkout@v2
> -    - name: Build QEMU installer (64 bit)
> -      run: .github/workflows/build.sh x86_64
> -    - uses: actions/upload-artifact@v1
> -      with:
> -        name: QEMU Installer Windows 64 bit
> +        name: QEMU Installer Windows ${{ matrix.arch }} bit
>          path: dist

Re: [RFC PATCH 3/3] .github/workflows: Use matrix strategy to parallelize builds

[Philippe Mathieu-Daudé]

On 8/4/20 11:04 AM, Stefan Weil wrote:
> Hi Philippe,
> 
> just a side note: GitHub action runs parallel jobs by default. I use two
> jobs (32 and 64 bit builds), so both are built at the same time, see
> https://github.com/stweil/qemu/actions.

Indeed... Not sure why my tests ran serialized, maybe I had too many
jobs scheduled.

> 
> Regards,
> 
> Stefan
> 
> Am 04.08.20 um 10:48 schrieb Philippe Mathieu-Daudé:
>> Using sequencial builds, both jobs take almost 2h to build.
>> By using the matrix strategy we can build the jobs in parallel,
>> reducing the total build time to 1h12m (as of v5.1.0-rc2).
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>>  .github/workflows/build.sh |  8 ++++----
>>  .github/workflows/win.yml  | 23 +++++++----------------
>>  2 files changed, 11 insertions(+), 20 deletions(-)
>>
>> diff --git a/.github/workflows/build.sh b/.github/workflows/build.sh
>> index c430a367be..4f2c6b56f3 100755
>> --- a/.github/workflows/build.sh
>> +++ b/.github/workflows/build.sh
>> @@ -7,10 +7,10 @@
>>  #~ set -e
>>  set -x
>>  
>> -ARCH=$1
>> -DLLS="libgcc_s_sjlj-1.dll libgomp-1.dll libstdc++-6.dll"
>> -
>> -if test "$ARCH" != "i686"; then
>> +if test "$1" == "32"; then
>> +  ARCH=i686
>> +  DLLS="libgcc_s_sjlj-1.dll libgomp-1.dll libstdc++-6.dll"
>> +else
>>    ARCH=x86_64
>>    DLLS="libgcc_s_seh-1.dll libgomp-1.dll libstdc++-6.dll"
>>  fi
>> diff --git a/.github/workflows/win.yml b/.github/workflows/win.yml
>> index 81cf48530f..afd827db8e 100644
>> --- a/.github/workflows/win.yml
>> +++ b/.github/workflows/win.yml
>> @@ -9,26 +9,17 @@ name: Cross build for Windows
>>  on: [push]
>>  
>>  jobs:
>> -  build32:
>> +  build:
>> +    strategy:
>> +      matrix:
>> +        arch: [32, 64]
>>      if: github.repository == 'qemu/qemu'
>>      runs-on: [ubuntu-20.04]
>>      steps:
>>      - uses: actions/checkout@v2
>> -    - name: Build QEMU installer (32 bit)
>> -      run: .github/workflows/build.sh i686
>> +    - name: Build QEMU installer (${{ matrix.arch }} bit)
>> +      run: .github/workflows/build.sh ${{ matrix.arch }}
>>      - uses: actions/upload-artifact@v1
>>        with:
>> -        name: QEMU Installer Windows 32 bit
>> -        path: dist
>> -
>> -  build64:
>> -    if: github.repository == 'qemu/qemu'
>> -    runs-on: [ubuntu-20.04]
>> -    steps:
>> -    - uses: actions/checkout@v2
>> -    - name: Build QEMU installer (64 bit)
>> -      run: .github/workflows/build.sh x86_64
>> -    - uses: actions/upload-artifact@v1
>> -      with:
>> -        name: QEMU Installer Windows 64 bit
>> +        name: QEMU Installer Windows ${{ matrix.arch }} bit
>>          path: dist
> 
> 

Re: [RFC PATCH 0/3] .github/workflows: Add GitHub action for Windows WHPX build

[Thomas Huth]

On 04/08/2020 10.48, Philippe Mathieu-Daudé wrote:
> Reposting Stefan's patch in case we want to merge it in
> (I 'd rather not, adding them as GitLab jobs instead):

Yeah, we should rather try to consolidate our CI zoo instead of adding
yet another one...

As mentioned in another mail, I'm already working on using our
containers for cross-compiling on gitlab-CI:

 https://gitlab.com/huth/qemu/-/jobs/669580489#L39

... not quite there yet, but it should be doable. I hope it will be
sufficient to simply also add the WinHv* headers there to get the
compilation coverage for WHXP.

 Thomas