[PATCH] ci: unset GITLAB_FEATURES envvar to not bust xargs(1) limits

[PATCH] ci: unset GITLAB_FEATURES envvar to not bust xargs(1) limits

[Patrick Steinhardt]

We have started to see the following assert happen in our GitLab CI
pipelines for jobs that use Windows with Meson:

  assertion "bc_ctl.arg_max >= LINE_MAX" failed: file "xargs.c", line 512, function: main

The assert in question verifies that we have enough room available to
pass at least `LINE_MAX` many bytes via the command line. The xargs(1)
binary in those jobs comes from Git for Windows, which in turn sources
the binaries from MSYS2, and has the following limits in place:

  $ & "C:/Program Files/Git/usr/bin/bash.exe" -l -c 'xargs --show-limits </dev/null'
  Your environment variables take up 17373 bytes
  POSIX upper limit on argument length (this system): 12579
  POSIX smallest allowable upper limit on argument length (all systems): 4096
  Maximum length of command we could actually use: 18446744073709546822
  Size of command buffer we are actually using: 12579
  Maximum parallelism (--max-procs must be no greater): 2147483647

What's interesting to see is the limit of 16 exabits for the maximum
command line length. This value might seem a bit high, and it is indeed
the result of an underflow: our environment is larger than the POSIX
upper limit on argument length, and the value is computed by subtracting
the former from the latter. So what we get is the result of `2^64 -
(17373 - 12579)`.

This makes it clear that the problem here is the size of our environment
variables. A listing sorted by length yields the following result:

  $ Get-ChildItem "Env:" |
      Sort-Object { $_.Value.Length } -Descending |
      Select-Object Name, @{Name="Length"; Expression={$_.Value.Length}}
  Name                                          Length
  ----                                          ------
  GITLAB_FEATURES                                 6386
  Path                                             706
  PSModulePath                                     229

The GITLAB_FEATURES environment variable makes up for roughly a third of
the complete environment. This variable is a comma-separated list of
features available for the GitLab instance, and seemingly it has been
growing over time as GitLab added more and more features.

Fix the issue by unsetting the environment variable in "ci/lib.sh". This
ensures that the environment variables are now smaller than the upper
limit on argument length again, and that in turn fixes the assert in
xargs(1).

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
Hi,

this patch series fixes another issue that we saw creeping into GitLab
CI jobs for MSVC+Windows. The root cause is that our environment
variables have grown too large, and thus xargs(1) was hitting an assert.

A test run of this can be found at [1].

Thanks!

Patrick

[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/514
---
 ci/lib.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/ci/lib.sh b/ci/lib.sh
index 3ecbf147db..42a2b6a318 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -231,6 +231,10 @@ then
 	distro=$(echo "$CI_JOB_IMAGE" | tr : -)
 elif test true = "$GITLAB_CI"
 then
+	# This environment is multiple kB in size and may cause us to exceed
+	# xargs(1) limits on Windows.
+	unset GITLAB_FEATURES
+
 	CI_TYPE=gitlab-ci
 	CI_BRANCH="$CI_COMMIT_REF_NAME"
 	CI_COMMIT="$CI_COMMIT_SHA"

---
base-commit: 2cc71917514657b93014134350864f4849edfc83
change-id: 20260302-pks-msvc-meson-xargs-c4bc35496078

Re: [PATCH] ci: unset GITLAB_FEATURES envvar to not bust xargs(1) limits

[Justin Tobler]

On 26/03/02 12:55PM, Patrick Steinhardt wrote:
> We have started to see the following assert happen in our GitLab CI
> pipelines for jobs that use Windows with Meson:
> 
>   assertion "bc_ctl.arg_max >= LINE_MAX" failed: file "xargs.c", line 512, function: main
> 
> The assert in question verifies that we have enough room available to
> pass at least `LINE_MAX` many bytes via the command line. The xargs(1)
> binary in those jobs comes from Git for Windows, which in turn sources
> the binaries from MSYS2, and has the following limits in place:
> 
>   $ & "C:/Program Files/Git/usr/bin/bash.exe" -l -c 'xargs --show-limits </dev/null'
>   Your environment variables take up 17373 bytes
>   POSIX upper limit on argument length (this system): 12579
>   POSIX smallest allowable upper limit on argument length (all systems): 4096
>   Maximum length of command we could actually use: 18446744073709546822
>   Size of command buffer we are actually using: 12579
>   Maximum parallelism (--max-procs must be no greater): 2147483647
> 
> What's interesting to see is the limit of 16 exabits for the maximum
> command line length. This value might seem a bit high, and it is indeed
> the result of an underflow: our environment is larger than the POSIX
> upper limit on argument length, and the value is computed by subtracting
> the former from the latter. So what we get is the result of `2^64 -
> (17373 - 12579)`.

Ok so the problem here is that the environment variables are taking up
17373 bytes which exceeds the upper limit here of 12579.

> This makes it clear that the problem here is the size of our environment
> variables. A listing sorted by length yields the following result:
> 
>   $ Get-ChildItem "Env:" |
>       Sort-Object { $_.Value.Length } -Descending |
>       Select-Object Name, @{Name="Length"; Expression={$_.Value.Length}}
>   Name                                          Length
>   ----                                          ------
>   GITLAB_FEATURES                                 6386
>   Path                                             706
>   PSModulePath                                     229
> 
> The GITLAB_FEATURES environment variable makes up for roughly a third of
> the complete environment. This variable is a comma-separated list of
> features available for the GitLab instance, and seemingly it has been
> growing over time as GitLab added more and more features.
> 
> Fix the issue by unsetting the environment variable in "ci/lib.sh". This
> ensures that the environment variables are now smaller than the upper
> limit on argument length again, and that in turn fixes the assert in
> xargs(1).

So if we unset GITLAB_FEATURES, that puts us at 10987 bytes (17373 -
6386) which would be under the upper limit. Unsetting this environment
variable seems like a reasonable means to mitigate this problem. Naive
question: is the upper limit something we could increase for the
environment? 

[snip]
>  ci/lib.sh | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/ci/lib.sh b/ci/lib.sh
> index 3ecbf147db..42a2b6a318 100755
> --- a/ci/lib.sh
> +++ b/ci/lib.sh
> @@ -231,6 +231,10 @@ then
>  	distro=$(echo "$CI_JOB_IMAGE" | tr : -)
>  elif test true = "$GITLAB_CI"
>  then
> +	# This environment is multiple kB in size and may cause us to exceed
> +	# xargs(1) limits on Windows.
> +	unset GITLAB_FEATURES

Now when running in GitLab CI we unset the large envvar. Looks good.

-Justin

Re: [PATCH] ci: unset GITLAB_FEATURES envvar to not bust xargs(1) limits

[Patrick Steinhardt]

On Mon, Mar 02, 2026 at 11:11:52AM -0600, Justin Tobler wrote:
> On 26/03/02 12:55PM, Patrick Steinhardt wrote:
[snip]
> > The GITLAB_FEATURES environment variable makes up for roughly a third of
> > the complete environment. This variable is a comma-separated list of
> > features available for the GitLab instance, and seemingly it has been
> > growing over time as GitLab added more and more features.
> > 
> > Fix the issue by unsetting the environment variable in "ci/lib.sh". This
> > ensures that the environment variables are now smaller than the upper
> > limit on argument length again, and that in turn fixes the assert in
> > xargs(1).
> 
> So if we unset GITLAB_FEATURES, that puts us at 10987 bytes (17373 -
> 6386) which would be under the upper limit. Unsetting this environment
> variable seems like a reasonable means to mitigate this problem. Naive
> question: is the upper limit something we could increase for the
> environment?

Unfortunately not. Under normal Linux systems you'd be able to do that,
but in MSYS2 the limits are hardcoded as far as I could see.

Thanks!

Patrick

Re: [PATCH] ci: unset GITLAB_FEATURES envvar to not bust xargs(1) limits

[Justin Tobler]

On 26/03/03 07:15AM, Patrick Steinhardt wrote:
> On Mon, Mar 02, 2026 at 11:11:52AM -0600, Justin Tobler wrote:
> > On 26/03/02 12:55PM, Patrick Steinhardt wrote:
> [snip]
> > > The GITLAB_FEATURES environment variable makes up for roughly a third of
> > > the complete environment. This variable is a comma-separated list of
> > > features available for the GitLab instance, and seemingly it has been
> > > growing over time as GitLab added more and more features.
> > > 
> > > Fix the issue by unsetting the environment variable in "ci/lib.sh". This
> > > ensures that the environment variables are now smaller than the upper
> > > limit on argument length again, and that in turn fixes the assert in
> > > xargs(1).
> > 
> > So if we unset GITLAB_FEATURES, that puts us at 10987 bytes (17373 -
> > 6386) which would be under the upper limit. Unsetting this environment
> > variable seems like a reasonable means to mitigate this problem. Naive
> > question: is the upper limit something we could increase for the
> > environment?
> 
> Unfortunately not. Under normal Linux systems you'd be able to do that,
> but in MSYS2 the limits are hardcoded as far as I could see.

Ah ok, good to know. I was wondering if there would be value in trying
to increase the headroom we have in case the overall size of the
environment variables increases over time again, but it sounds like this
wouldn't be possible. Hopefully we should be good though. :)

Thanks,
-Justin