Intermittent meson failures on msys2

Intermittent meson failures on msys2

[Richard Henderson]

Hi guys,

There's an occasional failure on msys2, where meson fails to capture the output of a build 
script.  E.g.

https://gitlab.com/qemu-project/qemu/-/jobs/2642051161

FAILED: ui/input-keymap-qcode-to-linux.c.inc
"C:/GitLab-Runner/builds/qemu-project/qemu/msys64/mingw64/bin/python3.exe" 
"C:/GitLab-Runner/builds/qemu-project/qemu/meson/meson.py" "--internal" "exe" "--capture" 
"ui/input-keymap-qcode-to-linux.c.inc" "--" 
"C:/GitLab-Runner/builds/qemu-project/qemu/msys64/mingw64/bin/python3.exe" 
"../ui/keycodemapdb/tools/keymap-gen" "code-map" "--lang" "glib2" "--varname" 
"qemu_input_map_qcode_to_linux" "../ui/keycodemapdb/data/keymaps.csv" "qcode" "linux"
[301/1665] Generating input-keymap-qcode-to-qnum.c.inc with a custom command (wrapped by 
meson to capture output)
ninja: build stopped: subcommand failed.


https://gitlab.com/qemu-project/qemu/-/jobs/2625836697

FAILED: ui/shader/texture-blit-frag.h
"C:/GitLab-Runner/builds/qemu-project/qemu/msys64/mingw64/bin/python3.exe" 
"C:/GitLab-Runner/builds/qemu-project/qemu/meson/meson.py" "--internal" "exe" "--capture" 
"ui/shader/texture-blit-frag.h" "--" "perl" 
"C:/GitLab-Runner/builds/qemu-project/qemu/scripts/shaderinclude.pl" 
"../ui/shader/texture-blit.frag"
[313/1663] Generating texture-blit-vert.h with a custom command (wrapped by meson to 
capture output)
ninja: build stopped: subcommand failed.


Could you have a look please?


r~

Re: Intermittent meson failures on msys2

[Marc-André Lureau]

[-- Attachment #1: Type: text/plain, Size: 2101 bytes --]

Hi

On Mon, Jun 27, 2022 at 6:41 AM Richard Henderson <
richard.henderson@linaro.org> wrote:

> Hi guys,
>
> There's an occasional failure on msys2, where meson fails to capture the
> output of a build
> script.  E.g.
>
> https://gitlab.com/qemu-project/qemu/-/jobs/2642051161
>
> FAILED: ui/input-keymap-qcode-to-linux.c.inc
> "C:/GitLab-Runner/builds/qemu-project/qemu/msys64/mingw64/bin/python3.exe"
> "C:/GitLab-Runner/builds/qemu-project/qemu/meson/meson.py" "--internal"
> "exe" "--capture"
> "ui/input-keymap-qcode-to-linux.c.inc" "--"
> "C:/GitLab-Runner/builds/qemu-project/qemu/msys64/mingw64/bin/python3.exe"
> "../ui/keycodemapdb/tools/keymap-gen" "code-map" "--lang" "glib2"
> "--varname"
> "qemu_input_map_qcode_to_linux" "../ui/keycodemapdb/data/keymaps.csv"
> "qcode" "linux"
> [301/1665] Generating input-keymap-qcode-to-qnum.c.inc with a custom
> command (wrapped by
> meson to capture output)
> ninja: build stopped: subcommand failed.
>
>
> https://gitlab.com/qemu-project/qemu/-/jobs/2625836697
>
> FAILED: ui/shader/texture-blit-frag.h
> "C:/GitLab-Runner/builds/qemu-project/qemu/msys64/mingw64/bin/python3.exe"
> "C:/GitLab-Runner/builds/qemu-project/qemu/meson/meson.py" "--internal"
> "exe" "--capture"
> "ui/shader/texture-blit-frag.h" "--" "perl"
> "C:/GitLab-Runner/builds/qemu-project/qemu/scripts/shaderinclude.pl"
> "../ui/shader/texture-blit.frag"
> [313/1663] Generating texture-blit-vert.h with a custom command (wrapped
> by meson to
> capture output)
> ninja: build stopped: subcommand failed.
>
>
> Could you have a look please?
>
>
>
Ah, we don't have artifacts for msys2 builds it seems, that would perhaps
help. It would make sense to at least take meson-logs/*.txt. I'll work on a
patch.

My guess is that CI randomly fails with "too many opened files", as I have
seen that regularly on various projects with Windows runners. And here,
it's probably reaching limits when running python/perl scripts
simultaneously... I don't see an easy way to solve that if that's the issue.

-- 
Marc-André Lureau

[-- Attachment #2: Type: text/html, Size: 3191 bytes --]

Re: Intermittent meson failures on msys2

[Richard Henderson]

On 7/8/22 18:11, Marc-André Lureau wrote:
> My guess is that CI randomly fails with "too many opened files", as I have seen that 
> regularly on various projects with Windows runners. And here, it's probably reaching 
> limits when running python/perl scripts simultaneously... I don't see an easy way to solve 
> that if that's the issue.

If that's really the issue, with no solution, then we should turn these jobs off.


r~

Re: Intermittent meson failures on msys2

[Daniel P. Berrangé]

On Fri, Jul 08, 2022 at 04:41:48PM +0400, Marc-André Lureau wrote:
> Hi
> 
> On Mon, Jun 27, 2022 at 6:41 AM Richard Henderson <
> richard.henderson@linaro.org> wrote:
> 
> > Hi guys,
> >
> > There's an occasional failure on msys2, where meson fails to capture the
> > output of a build
> > script.  E.g.
> >
> > https://gitlab.com/qemu-project/qemu/-/jobs/2642051161
> >
> > FAILED: ui/input-keymap-qcode-to-linux.c.inc
> > "C:/GitLab-Runner/builds/qemu-project/qemu/msys64/mingw64/bin/python3.exe"
> > "C:/GitLab-Runner/builds/qemu-project/qemu/meson/meson.py" "--internal"
> > "exe" "--capture"
> > "ui/input-keymap-qcode-to-linux.c.inc" "--"
> > "C:/GitLab-Runner/builds/qemu-project/qemu/msys64/mingw64/bin/python3.exe"
> > "../ui/keycodemapdb/tools/keymap-gen" "code-map" "--lang" "glib2"
> > "--varname"
> > "qemu_input_map_qcode_to_linux" "../ui/keycodemapdb/data/keymaps.csv"
> > "qcode" "linux"
> > [301/1665] Generating input-keymap-qcode-to-qnum.c.inc with a custom
> > command (wrapped by
> > meson to capture output)
> > ninja: build stopped: subcommand failed.
> >
> >
> > https://gitlab.com/qemu-project/qemu/-/jobs/2625836697
> >
> > FAILED: ui/shader/texture-blit-frag.h
> > "C:/GitLab-Runner/builds/qemu-project/qemu/msys64/mingw64/bin/python3.exe"
> > "C:/GitLab-Runner/builds/qemu-project/qemu/meson/meson.py" "--internal"
> > "exe" "--capture"
> > "ui/shader/texture-blit-frag.h" "--" "perl"
> > "C:/GitLab-Runner/builds/qemu-project/qemu/scripts/shaderinclude.pl"
> > "../ui/shader/texture-blit.frag"
> > [313/1663] Generating texture-blit-vert.h with a custom command (wrapped
> > by meson to
> > capture output)
> > ninja: build stopped: subcommand failed.
> >
> >
> > Could you have a look please?
> >
> >
> >
> Ah, we don't have artifacts for msys2 builds it seems, that would perhaps
> help. It would make sense to at least take meson-logs/*.txt. I'll work on a
> patch.
> 
> My guess is that CI randomly fails with "too many opened files", as I have
> seen that regularly on various projects with Windows runners. And here,
> it's probably reaching limits when running python/perl scripts
> simultaneously... I don't see an easy way to solve that if that's the issue.

There shouldn't be very much parallelism even taking place, because

https://docs.gitlab.com/ee/ci/runners/saas/windows_saas_runner.html

says  "Windows runners execute your CI/CD jobs on n1-standard-2 
       instances with 2 vCPUs and 7.5 GB RAM. "

unless ninja is setting a parellism much higher than nCPUs ?


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Intermittent meson failures on msys2

[Thomas Huth]

On 08/07/2022 15.13, Daniel P. Berrangé wrote:
> On Fri, Jul 08, 2022 at 04:41:48PM +0400, Marc-André Lureau wrote:
>> Hi
>>
>> On Mon, Jun 27, 2022 at 6:41 AM Richard Henderson <
>> richard.henderson@linaro.org> wrote:
>>
>>> Hi guys,
>>>
>>> There's an occasional failure on msys2, where meson fails to capture the
>>> output of a build
>>> script.  E.g.
>>>
>>> https://gitlab.com/qemu-project/qemu/-/jobs/2642051161
>>>
>>> FAILED: ui/input-keymap-qcode-to-linux.c.inc
>>> "C:/GitLab-Runner/builds/qemu-project/qemu/msys64/mingw64/bin/python3.exe"
>>> "C:/GitLab-Runner/builds/qemu-project/qemu/meson/meson.py" "--internal"
>>> "exe" "--capture"
>>> "ui/input-keymap-qcode-to-linux.c.inc" "--"
>>> "C:/GitLab-Runner/builds/qemu-project/qemu/msys64/mingw64/bin/python3.exe"
>>> "../ui/keycodemapdb/tools/keymap-gen" "code-map" "--lang" "glib2"
>>> "--varname"
>>> "qemu_input_map_qcode_to_linux" "../ui/keycodemapdb/data/keymaps.csv"
>>> "qcode" "linux"
>>> [301/1665] Generating input-keymap-qcode-to-qnum.c.inc with a custom
>>> command (wrapped by
>>> meson to capture output)
>>> ninja: build stopped: subcommand failed.
>>>
>>>
>>> https://gitlab.com/qemu-project/qemu/-/jobs/2625836697
>>>
>>> FAILED: ui/shader/texture-blit-frag.h
>>> "C:/GitLab-Runner/builds/qemu-project/qemu/msys64/mingw64/bin/python3.exe"
>>> "C:/GitLab-Runner/builds/qemu-project/qemu/meson/meson.py" "--internal"
>>> "exe" "--capture"
>>> "ui/shader/texture-blit-frag.h" "--" "perl"
>>> "C:/GitLab-Runner/builds/qemu-project/qemu/scripts/shaderinclude.pl"
>>> "../ui/shader/texture-blit.frag"
>>> [313/1663] Generating texture-blit-vert.h with a custom command (wrapped
>>> by meson to
>>> capture output)
>>> ninja: build stopped: subcommand failed.
>>>
>>>
>>> Could you have a look please?
>>>
>>>
>>>
>> Ah, we don't have artifacts for msys2 builds it seems, that would perhaps
>> help. It would make sense to at least take meson-logs/*.txt. I'll work on a
>> patch.
>>
>> My guess is that CI randomly fails with "too many opened files", as I have
>> seen that regularly on various projects with Windows runners. And here,
>> it's probably reaching limits when running python/perl scripts
>> simultaneously... I don't see an easy way to solve that if that's the issue.
> 
> There shouldn't be very much parallelism even taking place, because
> 
> https://docs.gitlab.com/ee/ci/runners/saas/windows_saas_runner.html
> 
> says  "Windows runners execute your CI/CD jobs on n1-standard-2
>         instances with 2 vCPUs and 7.5 GB RAM. "
> 
> unless ninja is setting a parellism much higher than nCPUs ?

We're compiling with "make -j2" there, see .gitlab-ci.d/windows.yml ... so I 
don't think that it's about too many things going on in parallel. 
Additionally, the problem hasn't been there a couple of weeks ago, so it's 
either something new that we merged recently, or the Windows containers or 
MSYS2 environment has been changed recently?

  Thomas