[PATCH v5 0/2] Improve cirrus msys2

[PATCH v5 0/2] Improve cirrus msys2

[Yonggang Luo]

Fixes the broken msys2/mingw ci and speed it up.

V4-V5
Now the cache are usefull by using 7zip to archive msys64
Saved about 18min, compare
https://cirrus-ci.com/task/5093551157542912
https://cirrus-ci.com/task/6177196538593280

V3-V4
Using cirrus cache to speed up msys2 ci instead of downloading archive file

V2-V3
Add one more patch:
cirrus: msys2/mingw speed is up, add excluded target back
Do not build sphinx on windows, that's failing
set the number of parallel count to fixed number 8

V1-V2
Resolve the cirrus conflict

Yonggang Luo (2):
  cirrus: Fixing and speedup the msys2/mingw CI
  cirrus: msys2/mingw speed is up, add excluded target back

 .cirrus.yml | 113 +++++++++++++++++++++++++++++++---------------------
 1 file changed, 67 insertions(+), 46 deletions(-)

-- 
2.28.0.windows.1

[PATCH v5 1/2] cirrus: Fixing and speedup the msys2/mingw CI

[Yonggang Luo]

Use cache of cirrus caching msys2
The install of msys2 are refer to https://github.com/msys2/setup-msys2
The first time install msys2 would be time consuming, so increase timeout_in to 90m
according to https://cirrus-ci.org/faq/#instance-timed-out

Apply patch of https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg00072.html

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
 .cirrus.yml | 110 +++++++++++++++++++++++++++++++---------------------
 1 file changed, 66 insertions(+), 44 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index d58782ce67..a1145d0366 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -46,64 +46,86 @@ macos_xcode_task:
     - gmake check
 
 windows_msys2_task:
+  timeout_in: 90m
   windows_container:
-    image: cirrusci/windowsservercore:cmake
+    image: cirrusci/windowsservercore:2019
     os_version: 2019
     cpu: 8
     memory: 8G
   env:
+    CIRRUS_SHELL: powershell
     MSYS: winsymlinks:nativestrict
     MSYSTEM: MINGW64
     CHERE_INVOKING: 1
   printenv_script:
-    - C:\tools\msys64\usr\bin\bash.exe -lc 'printenv'
+    - choco install -y --no-progress 7zip
+    - Write-Output $env:PATH
+  msys2_cache:
+    folder: C:\tools\archive
+    reupload_on_changes: false
+    fingerprint_script: cat .cirrus.yml
+    populate_script:
+      - |
+        md C:\tools
+        md C:\tools\archive
+        $start_time = Get-Date
+        cd C:\tools
+        bitsadmin /transfer msys_download /dynamic /download /priority FOREGROUND https://github.com/msys2/msys2-installer/releases/download/2020-09-03/msys2-base-x86_64-20200903.sfx.exe C:\tools\base.exe
+        Write-Output "Download time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
+        C:\tools\base.exe -y
+        ((Get-Content -path C:\tools\msys64\etc\\post-install\\07-pacman-key.post -Raw) -replace '--refresh-keys', '--version') | Set-Content -Path C:\tools\msys64\etc\\post-install\\07-pacman-key.post
+        C:\tools\msys64\usr\bin\bash.exe -lc "sed -i 's/^CheckSpace/#CheckSpace/g' /etc/pacman.conf"
+        C:\tools\msys64\usr\bin\bash.exe -lc "export"
+        C:\tools\msys64\usr\bin\bash.exe -lc "grep -rl 'repo.msys2.org/' /etc/pacman.d/mirrorlist.* | xargs sed -i 's/repo.msys2.org\//mirrors.tuna.tsinghua.edu.cn\/msys2\//g'"
+        C:\tools\msys64\usr\bin\pacman.exe --noconfirm -Sy
+        echo Y | C:\tools\msys64\usr\bin\pacman.exe --noconfirm -Suu --overwrite=*
+        taskkill /F /FI "MODULES eq msys-2.0.dll"
+        tasklist
+        C:\tools\msys64\usr\bin\bash.exe -lc "mv -f /etc/pacman.conf.pacnew /etc/pacman.conf || true"
+        C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Suu --overwrite=*"
+        C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed \
+          diffutils git grep make pkg-config sed \
+          mingw-w64-x86_64-python \
+          mingw-w64-x86_64-python-setuptools \
+          mingw-w64-x86_64-toolchain \
+          mingw-w64-x86_64-SDL2 \
+          mingw-w64-x86_64-SDL2_image \
+          mingw-w64-x86_64-gtk3 \
+          mingw-w64-x86_64-glib2 \
+          mingw-w64-x86_64-ninja \
+          mingw-w64-x86_64-jemalloc \
+          mingw-w64-x86_64-lzo2 \
+          mingw-w64-x86_64-zstd \
+          mingw-w64-x86_64-libjpeg-turbo \
+          mingw-w64-x86_64-pixman \
+          mingw-w64-x86_64-libgcrypt \
+          mingw-w64-x86_64-libpng \
+          mingw-w64-x86_64-libssh \
+          mingw-w64-x86_64-libxml2 \
+          mingw-w64-x86_64-snappy \
+          mingw-w64-x86_64-libusb \
+          mingw-w64-x86_64-usbredir \
+          mingw-w64-x86_64-libtasn1 \
+          mingw-w64-x86_64-nettle \
+          mingw-w64-x86_64-cyrus-sasl \
+          mingw-w64-x86_64-curl \
+          mingw-w64-x86_64-gnutls \
+          "
+        C:\tools\msys64\usr\bin\bash.exe -lc "rm -rf /var/cache/pacman/pkg/*"
+        cd C:\tools\msys64
+        echo "Start archive"
+        cmd /C "7z a -ttar . -so | 7z a -txz -simsys2-x86_64.tar C:\tools\archive\msys2-x86_64.tar.xz"
   install_script:
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
-        curl -O http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
-        curl -O http://repo.msys2.org/msys/x86_64/msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz.sig"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd /c/tools &&
-        pacman -U --noconfirm msys2-keyring-r21.b39fb11-1-any.pkg.tar.xz"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Sy"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed
-        bash pacman pacman-mirrors msys2-runtime"
-    - taskkill /F /IM gpg-agent.exe
-    - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -Su"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "pacman --noconfirm -S --needed
-        base-devel
-        git
-        mingw-w64-x86_64-python
-        mingw-w64-x86_64-python-setuptools
-        mingw-w64-x86_64-toolchain
-        mingw-w64-x86_64-capstone
-        mingw-w64-x86_64-SDL2
-        mingw-w64-x86_64-SDL2_image
-        mingw-w64-x86_64-gtk3
-        mingw-w64-x86_64-glib2
-        mingw-w64-x86_64-ninja
-        mingw-w64-x86_64-make
-        mingw-w64-x86_64-lzo2
-        mingw-w64-x86_64-zstd
-        mingw-w64-x86_64-libjpeg-turbo
-        mingw-w64-x86_64-pixman
-        mingw-w64-x86_64-libgcrypt
-        mingw-w64-x86_64-libpng
-        mingw-w64-x86_64-libssh
-        mingw-w64-x86_64-libxml2
-        mingw-w64-x86_64-snappy
-        mingw-w64-x86_64-libusb
-        mingw-w64-x86_64-usbredir
-        mingw-w64-x86_64-libtasn1
-        mingw-w64-x86_64-nettle
-        mingw-w64-x86_64-cyrus-sasl
-        mingw-w64-x86_64-curl
-        mingw-w64-x86_64-gnutls
-        mingw-w64-x86_64-zstd"
+    - |
+      cd C:\tools
+      cmd /C "7z x C:\tools\archive\msys2-x86_64.tar.xz -so | 7z x -aoa -simsys2-x86_64.tar -ttar -omsys64"
+      C:\tools\msys64\usr\bin\bash.exe -lc "export"
+
   script:
     - C:\tools\msys64\usr\bin\bash.exe -lc "mkdir build"
     - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && ../configure
       --python=python3 --ninja=ninja
       --target-list-exclude=i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j$NUMBER_OF_PROCESSORS"
+    - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j8"
   test_script:
     - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make V=1 check"
-- 
2.28.0.windows.1

Re: [PATCH v5 1/2] cirrus: Fixing and speedup the msys2/mingw CI

[Thomas Huth]

On 07/10/2020 16.52, Yonggang Luo wrote:
> Use cache of cirrus caching msys2
> The install of msys2 are refer to https://github.com/msys2/setup-msys2
> The first time install msys2 would be time consuming, so increase timeout_in to 90m
> according to https://cirrus-ci.org/faq/#instance-timed-out
> 
> Apply patch of https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg00072.html
> 
> Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
> ---
>  .cirrus.yml | 110 +++++++++++++++++++++++++++++++---------------------
>  1 file changed, 66 insertions(+), 44 deletions(-)
> 
> diff --git a/.cirrus.yml b/.cirrus.yml
> index d58782ce67..a1145d0366 100644
> --- a/.cirrus.yml
> +++ b/.cirrus.yml
> @@ -46,64 +46,86 @@ macos_xcode_task:
>      - gmake check
>  
>  windows_msys2_task:
> +  timeout_in: 90m
>    windows_container:
> -    image: cirrusci/windowsservercore:cmake
> +    image: cirrusci/windowsservercore:2019
>      os_version: 2019
>      cpu: 8
>      memory: 8G
>    env:
> +    CIRRUS_SHELL: powershell
>      MSYS: winsymlinks:nativestrict
>      MSYSTEM: MINGW64
>      CHERE_INVOKING: 1
>    printenv_script:
> -    - C:\tools\msys64\usr\bin\bash.exe -lc 'printenv'
> +    - choco install -y --no-progress 7zip

Since you've now removed the 'printenv' line, I think the section should
maybe not be called "printenv_script" anymore, should it?

 Thomas

Re: [PATCH v5 1/2] cirrus: Fixing and speedup the msys2/mingw CI

[罗勇刚(Yonggang Luo)]

[-- Attachment #1: Type: text/plain, Size: 1718 bytes --]

On Wed, Oct 7, 2020 at 11:12 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 07/10/2020 16.52, Yonggang Luo wrote:
> > Use cache of cirrus caching msys2
> > The install of msys2 are refer to https://github.com/msys2/setup-msys2
> > The first time install msys2 would be time consuming, so increase
timeout_in to 90m
> > according to https://cirrus-ci.org/faq/#instance-timed-out
> >
> > Apply patch of
https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg00072.html
> >
> > Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
> > ---
> >  .cirrus.yml | 110 +++++++++++++++++++++++++++++++---------------------
> >  1 file changed, 66 insertions(+), 44 deletions(-)
> >
> > diff --git a/.cirrus.yml b/.cirrus.yml
> > index d58782ce67..a1145d0366 100644
> > --- a/.cirrus.yml
> > +++ b/.cirrus.yml
> > @@ -46,64 +46,86 @@ macos_xcode_task:
> >      - gmake check
> >
> >  windows_msys2_task:
> > +  timeout_in: 90m
> >    windows_container:
> > -    image: cirrusci/windowsservercore:cmake
> > +    image: cirrusci/windowsservercore:2019
> >      os_version: 2019
> >      cpu: 8
> >      memory: 8G
> >    env:
> > +    CIRRUS_SHELL: powershell
> >      MSYS: winsymlinks:nativestrict
> >      MSYSTEM: MINGW64
> >      CHERE_INVOKING: 1
> >    printenv_script:
> > -    - C:\tools\msys64\usr\bin\bash.exe -lc 'printenv'
> > +    - choco install -y --no-progress 7zip
>
> Since you've now removed the 'printenv' line, I think the section should
> maybe not be called "printenv_script" anymore, should it?
May I name it install_7zip?
Cause that's a deps need installed first.
>
>  Thomas
>


--
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo

[-- Attachment #2: Type: text/html, Size: 2471 bytes --]

Re: [PATCH v5 1/2] cirrus: Fixing and speedup the msys2/mingw CI

[Alex Bennée]

罗勇刚(Yonggang Luo) <luoyonggang@gmail.com> writes:

> On Wed, Oct 7, 2020 at 11:12 PM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 07/10/2020 16.52, Yonggang Luo wrote:
>> > Use cache of cirrus caching msys2
>> > The install of msys2 are refer to https://github.com/msys2/setup-msys2
>> > The first time install msys2 would be time consuming, so increase
> timeout_in to 90m
>> > according to https://cirrus-ci.org/faq/#instance-timed-out
>> >
>> > Apply patch of
> https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg00072.html
>> >
>> > Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
>> > ---
>> >  .cirrus.yml | 110 +++++++++++++++++++++++++++++++---------------------
>> >  1 file changed, 66 insertions(+), 44 deletions(-)
>> >
>> > diff --git a/.cirrus.yml b/.cirrus.yml
>> > index d58782ce67..a1145d0366 100644
>> > --- a/.cirrus.yml
>> > +++ b/.cirrus.yml
>> > @@ -46,64 +46,86 @@ macos_xcode_task:
>> >      - gmake check
>> >
>> >  windows_msys2_task:
>> > +  timeout_in: 90m
>> >    windows_container:
>> > -    image: cirrusci/windowsservercore:cmake
>> > +    image: cirrusci/windowsservercore:2019
>> >      os_version: 2019
>> >      cpu: 8
>> >      memory: 8G
>> >    env:
>> > +    CIRRUS_SHELL: powershell
>> >      MSYS: winsymlinks:nativestrict
>> >      MSYSTEM: MINGW64
>> >      CHERE_INVOKING: 1
>> >    printenv_script:
>> > -    - C:\tools\msys64\usr\bin\bash.exe -lc 'printenv'
>> > +    - choco install -y --no-progress 7zip
>>
>> Since you've now removed the 'printenv' line, I think the section should
>> maybe not be called "printenv_script" anymore, should it?
> May I name it install_7zip?
> Cause that's a deps need installed first.

I renamed it setup_script: when I merged it.

>>
>>  Thomas
>>


-- 
Alex Bennée

[PATCH v5 2/2] cirrus: msys2/mingw speed is up, add excluded target back

[Yonggang Luo]

The following target are add back:
i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
---
 .cirrus.yml | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index a1145d0366..639f0b51c4 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -124,8 +124,7 @@ windows_msys2_task:
   script:
     - C:\tools\msys64\usr\bin\bash.exe -lc "mkdir build"
     - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && ../configure
-      --python=python3 --ninja=ninja
-      --target-list-exclude=i386-softmmu,arm-softmmu,ppc-softmmu,mips-softmmu"
+      --python=python3 --ninja=ninja"
     - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j8"
   test_script:
     - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make V=1 check"
-- 
2.28.0.windows.1

Re: [PATCH v5 0/2] Improve cirrus msys2

[罗勇刚(Yonggang Luo)]

[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]

CC Alex, now it's really fixed

On Wed, Oct 7, 2020 at 10:53 PM Yonggang Luo <luoyonggang@gmail.com> wrote:
>
> Fixes the broken msys2/mingw ci and speed it up.
>
> V4-V5
> Now the cache are usefull by using 7zip to archive msys64
> Saved about 18min, compare
> https://cirrus-ci.com/task/5093551157542912
> https://cirrus-ci.com/task/6177196538593280
>
> V3-V4
> Using cirrus cache to speed up msys2 ci instead of downloading archive
file
>
> V2-V3
> Add one more patch:
> cirrus: msys2/mingw speed is up, add excluded target back
> Do not build sphinx on windows, that's failing
> set the number of parallel count to fixed number 8
>
> V1-V2
> Resolve the cirrus conflict
>
> Yonggang Luo (2):
>   cirrus: Fixing and speedup the msys2/mingw CI
>   cirrus: msys2/mingw speed is up, add excluded target back
>
>  .cirrus.yml | 113 +++++++++++++++++++++++++++++++---------------------
>  1 file changed, 67 insertions(+), 46 deletions(-)
>
> --
> 2.28.0.windows.1
>


--
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo

[-- Attachment #2: Type: text/html, Size: 1466 bytes --]

Re: [PATCH v5 0/2] Improve cirrus msys2

[Alex Bennée]

Yonggang Luo <luoyonggang@gmail.com> writes:

> Fixes the broken msys2/mingw ci and speed it up.

Queued to prepr/misc-testing-gitdm-plugin-fixes, thanks. I renamed the
script to setup_script.

-- 
Alex Bennée