Re: [PATCH] ci: unset GITLAB_FEATURES envvar to not bust xargs(1) limits

[Justin Tobler <jltobler@gmail.com>]

On 26/03/02 12:55PM, Patrick Steinhardt wrote:
> We have started to see the following assert happen in our GitLab CI
> pipelines for jobs that use Windows with Meson:
> 
>   assertion "bc_ctl.arg_max >= LINE_MAX" failed: file "xargs.c", line 512, function: main
> 
> The assert in question verifies that we have enough room available to
> pass at least `LINE_MAX` many bytes via the command line. The xargs(1)
> binary in those jobs comes from Git for Windows, which in turn sources
> the binaries from MSYS2, and has the following limits in place:
> 
>   $ & "C:/Program Files/Git/usr/bin/bash.exe" -l -c 'xargs --show-limits </dev/null'
>   Your environment variables take up 17373 bytes
>   POSIX upper limit on argument length (this system): 12579
>   POSIX smallest allowable upper limit on argument length (all systems): 4096
>   Maximum length of command we could actually use: 18446744073709546822
>   Size of command buffer we are actually using: 12579
>   Maximum parallelism (--max-procs must be no greater): 2147483647
> 
> What's interesting to see is the limit of 16 exabits for the maximum
> command line length. This value might seem a bit high, and it is indeed
> the result of an underflow: our environment is larger than the POSIX
> upper limit on argument length, and the value is computed by subtracting
> the former from the latter. So what we get is the result of `2^64 -
> (17373 - 12579)`.

Ok so the problem here is that the environment variables are taking up
17373 bytes which exceeds the upper limit here of 12579.

> This makes it clear that the problem here is the size of our environment
> variables. A listing sorted by length yields the following result:
> 
>   $ Get-ChildItem "Env:" |
>       Sort-Object { $_.Value.Length } -Descending |
>       Select-Object Name, @{Name="Length"; Expression={$_.Value.Length}}
>   Name                                          Length
>   ----                                          ------
>   GITLAB_FEATURES                                 6386
>   Path                                             706
>   PSModulePath                                     229
> 
> The GITLAB_FEATURES environment variable makes up for roughly a third of
> the complete environment. This variable is a comma-separated list of
> features available for the GitLab instance, and seemingly it has been
> growing over time as GitLab added more and more features.
> 
> Fix the issue by unsetting the environment variable in "ci/lib.sh". This
> ensures that the environment variables are now smaller than the upper
> limit on argument length again, and that in turn fixes the assert in
> xargs(1).

So if we unset GITLAB_FEATURES, that puts us at 10987 bytes (17373 -
6386) which would be under the upper limit. Unsetting this environment
variable seems like a reasonable means to mitigate this problem. Naive
question: is the upper limit something we could increase for the
environment? 

[snip]
>  ci/lib.sh | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/ci/lib.sh b/ci/lib.sh
> index 3ecbf147db..42a2b6a318 100755
> --- a/ci/lib.sh
> +++ b/ci/lib.sh
> @@ -231,6 +231,10 @@ then
>  	distro=$(echo "$CI_JOB_IMAGE" | tr : -)
>  elif test true = "$GITLAB_CI"
>  then
> +	# This environment is multiple kB in size and may cause us to exceed
> +	# xargs(1) limits on Windows.
> +	unset GITLAB_FEATURES

Now when running in GitLab CI we unset the large envvar. Looks good.

-Justin