Re: [PATCH] gitlab-ci: migrate Windows builds away from Chocolatey

[Justin Tobler <jltobler@gmail.com>]

On 26/06/15 02:21PM, Patrick Steinhardt wrote:
> The Windows builds in GitLab CI use Chocolatey to install dependencies.
> Unfortunately, Chocolatey seems to be very unreliable, which causes the
> jobs to fail very regularly. This is a limitation that seems to be
> somewhat known [1]:
> 
>   As an organization, you want 100% reliability (or at least that
>   potential), and you may want full trust and control as well. This is
>   something you can get with internally hosted packages, and you are
>   unlikely to achieve from use of the Community Package Repository.
> 
> So using the Community Package Repository is kind of discouraged in case
> one wants reliability. We _do_ want reliability though, and we cannot
> easily switch to an enterprise license to fix this issue.

Make sense.

> Introduce a new script that downloads and installs dependencies
> directly. This has a couple of benefits:
> 
>   - We can drop our dependency on Chocolatey completely, thus improving
>     reliability.
> 
>   - We can easily cache the installers.
> 
>   - We get direct control over the exact versions we install.

Naive question: Do we expect to have to update the pinned versions
often?

>   - Installing dependencies is sped up from roundabout 3 minutes to 1
>     minute.

Is fetching the dependencides directly just plain faster? Or is this due
to the caching?

> [1]: https://docs.chocolatey.org/en-us/community-repository/community-packages-disclaimer/#summary
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
> Hi
> 
> I've been quite annoyed recently because our Windows builds in GitLab CI
> are extremely flakey. All of those flakes come from Chocolatey, which is
> why this patch moves away from it.
> 
> Thanks!
> 
> Patrick
> ---
>  .gitlab-ci.yml              | 11 ++++++---
>  ci/install-dependencies.ps1 | 55 +++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 63 insertions(+), 3 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index e0b9a0d82b..87a5343a94 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -161,11 +161,16 @@ test:mingw64:
>      TEST_OUTPUT_DIRECTORY: "C:/Git-Test"
>    tags:
>      - saas-windows-medium-amd64
> +  cache:
> +    key:
> +      files:
> +        - ci/install-dependencies.ps1
> +    paths:
> +      - .dependencies

Nice that we can cache the installers now.

>    before_script:
>      - *windows_before_script
> -    - choco install -y git meson ninja rust-ms
> -    - Import-Module $env:ChocolateyInstall\helpers\chocolateyProfile.psm1
> -    - refreshenv
> +    - ./ci/install-dependencies.ps1
> +    - $env:Path = "C:\Meson;C:\Rust\bin;$env:Path"

I assume Git is already discoverable on the path?

>      - New-Item -Path $env:TEST_OUTPUT_DIRECTORY -ItemType Directory
>  
>  build:msvc-meson:
> diff --git a/ci/install-dependencies.ps1 b/ci/install-dependencies.ps1
> new file mode 100755
> index 0000000000..e3b367fa54
> --- /dev/null
> +++ b/ci/install-dependencies.ps1
> @@ -0,0 +1,55 @@
> +param(
> +    [string]$DownloadDirectory = '.dependencies'
> +)
> +
> +$ErrorActionPreference = 'Stop'
> +$ProgressPreference = 'SilentlyContinue'
> +
> +$GitVersion = '2.54.0.windows.1'
> +$MesonVersion = '1.11.0'
> +$RustVersion = '1.96.0'
> +
> +New-Item -Path $DownloadDirectory -ItemType Directory -Force | Out-Null
> +New-Item -Path .git/info -ItemType Directory -Force | Out-Null
> +New-Item -Path .git/info/exclude -ItemType File -Force | Out-Null
> +Add-Content -Path .git/info/exclude -Value "/$DownloadDirectory"

Here we create the ".dependencies" directory and add it to
".git/info/exclude" to be ignored.

> +function Get-Installer {
> +    param(
> +        [Parameter(Mandatory = $true)][string]$Name,
> +        [Parameter(Mandatory = $true)][string]$Url
> +    )
> +
> +    $path = Join-Path $DownloadDirectory $Name
> +    if (-not (Test-Path $path)) {
> +        Write-Host "Downloading $Url"
> +        Invoke-WebRequest $Url -OutFile $path -TimeoutSec 300

We only download the installer if it is not already cached. Makes sense.

> +    }
> +    return $path
> +}
> +
> +function Invoke-Installer {
> +    param(
> +        [Parameter(Mandatory = $true)][string]$FilePath,
> +        [Parameter(Mandatory = $true)][string[]]$ArgumentList
> +    )
> +
> +    Write-Host "Running $FilePath $($ArgumentList -join ' ')"
> +    $process = Start-Process -Wait -PassThru -FilePath $FilePath -ArgumentList $ArgumentList
> +    if ($process.ExitCode -ne 0) {
> +        throw "$FilePath failed with exit code $($process.ExitCode)"
> +    }
> +}
> +
> +$gitAssetVersion = $GitVersion -replace '\.windows\.\d+$', ''
> +$gitInstaller = Get-Installer "Git-Installer.exe" `
> +    "https://github.com/git-for-windows/git/releases/download/v$GitVersion/PortableGit-$gitAssetVersion-64-bit.7z.exe"
> +Invoke-Installer $gitInstaller @('-y', '-o"C:\Program Files\Git"')
> +
> +$mesonMsi = Get-Installer "meson.msi" `
> +    "https://github.com/mesonbuild/meson/releases/download/$MesonVersion/meson-$MesonVersion-64.msi"
> +Invoke-Installer msiexec.exe @('/i', $mesonMsi, 'INSTALLDIR=C:\Meson', '/quiet', '/norestart')
> +
> +$rustMsi = Get-Installer "rust.msi" `
> +    "https://static.rust-lang.org/dist/rust-$RustVersion-x86_64-pc-windows-msvc.msi"
> +Invoke-Installer msiexec.exe @('/i', $rustMsi, 'INSTALLDIR=C:\Rust', 'ADDLOCAL=Rustc,Cargo,Std', '/quiet', '/norestart')

Here is actually invoke the helpers to fetch and install the
dependencies. Looks good. I also validated that this job is working on
GitLab CI.

-Justin