* [Buildroot] [PATCH] support/misc/gitlab-ci.yml.in: include Branches workflow
@ 2020-08-30 15:59 Arnout Vandecappelle
2020-08-31 12:43 ` Yann E. MORIN
2020-10-06 20:42 ` Yann E. MORIN
0 siblings, 2 replies; 7+ messages in thread
From: Arnout Vandecappelle @ 2020-08-30 15:59 UTC (permalink / raw)
To: buildroot
Recently, gitlab-ci has gained a lot more flexibility in when pipelines
are created. As a side-effect of this, double pipelines may be created
when a `when` clause is used in some job. Avoid this by adding a
workflow that launches it only on branches. See [1] and [2].
Note that in reality, the duplicate pipelines only occur for merge
requests. Since we don't have merge requests, we don't have this
problem. Howver, gitlab also displays an annoying error message on the
pipeline page [3], and it sends an error mail to the triggerer (i.e.,
Arnout), so it's still useful to do this.
[1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines
[2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates
[3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
I'd like to get this one in master, because the e-mails are annoying me
:-)
---
support/misc/gitlab-ci.yml.in | 3 +++
1 file changed, 3 insertions(+)
diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in
index dddebf09e9..8a031898ef 100644
--- a/support/misc/gitlab-ci.yml.in
+++ b/support/misc/gitlab-ci.yml.in
@@ -1,6 +1,9 @@
# Configuration for Gitlab-CI.
# Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
+include:
+ - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'
+
image: buildroot/base:20200814.2228
.check_base:
--
2.26.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Buildroot] [PATCH] support/misc/gitlab-ci.yml.in: include Branches workflow
2020-08-30 15:59 [Buildroot] [PATCH] support/misc/gitlab-ci.yml.in: include Branches workflow Arnout Vandecappelle
@ 2020-08-31 12:43 ` Yann E. MORIN
2020-08-31 12:58 ` Arnout Vandecappelle
2020-10-06 20:42 ` Yann E. MORIN
1 sibling, 1 reply; 7+ messages in thread
From: Yann E. MORIN @ 2020-08-31 12:43 UTC (permalink / raw)
To: buildroot
Arnout, All,
+Romain and +Jugurtha, as we've had some discussions on that topic
publicly and privately before.
On 2020-08-30 17:59 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly:
> Recently, gitlab-ci has gained a lot more flexibility in when pipelines
> are created. As a side-effect of this, double pipelines may be created
> when a `when` clause is used in some job. Avoid this by adding a
> workflow that launches it only on branches. See [1] and [2].
>
> Note that in reality, the duplicate pipelines only occur for merge
> requests. Since we don't have merge requests, we don't have this
> problem. Howver, gitlab also displays an annoying error message on the
> pipeline page [3], and it sends an error mail to the triggerer (i.e.,
> Arnout), so it's still useful to do this.
>
> [1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines
> [2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates
> [3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds
>
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
> I'd like to get this one in master, because the e-mails are annoying me
> :-)
I'd like to suggest an alternative: move all the conditions into the
generating script, now that we do have a script that generates the
pipeline description.
Having the conditions in the script will help handle such cases, by only
ever emitting those jobs we actually want to run. This would allow for
more flexibility than the limited micro-language used by gitlab-ci.yml.
Also, this will allow us to not care about the evolutions of that micro-
language.
To that effect, I already started workign on that two weeks ago, and I
have omething that works as before (even better I think):
https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/gitlab-ci-cond
I just need to cleanup that last commit (and remember why I started
doing it. meh...). I hope I'll be able to send it tonight.
Regards,
Yann E. MORIN.
> ---
> support/misc/gitlab-ci.yml.in | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in
> index dddebf09e9..8a031898ef 100644
> --- a/support/misc/gitlab-ci.yml.in
> +++ b/support/misc/gitlab-ci.yml.in
> @@ -1,6 +1,9 @@
> # Configuration for Gitlab-CI.
> # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
>
> +include:
> + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'
> +
> image: buildroot/base:20200814.2228
>
> .check_base:
> --
> 2.26.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] [PATCH] support/misc/gitlab-ci.yml.in: include Branches workflow
2020-08-31 12:43 ` Yann E. MORIN
@ 2020-08-31 12:58 ` Arnout Vandecappelle
2020-08-31 19:52 ` Yann E. MORIN
2020-08-31 21:49 ` Yann E. MORIN
0 siblings, 2 replies; 7+ messages in thread
From: Arnout Vandecappelle @ 2020-08-31 12:58 UTC (permalink / raw)
To: buildroot
On 31/08/2020 14:43, Yann E. MORIN wrote:
> Arnout, All,
>
> +Romain and +Jugurtha, as we've had some discussions on that topic
> publicly and privately before.
>
> On 2020-08-30 17:59 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly:
>> Recently, gitlab-ci has gained a lot more flexibility in when pipelines
>> are created. As a side-effect of this, double pipelines may be created
>> when a `when` clause is used in some job. Avoid this by adding a
>> workflow that launches it only on branches. See [1] and [2].
>>
>> Note that in reality, the duplicate pipelines only occur for merge
>> requests. Since we don't have merge requests, we don't have this
>> problem. Howver, gitlab also displays an annoying error message on the
>> pipeline page [3], and it sends an error mail to the triggerer (i.e.,
>> Arnout), so it's still useful to do this.
>>
>> [1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines
>> [2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates
>> [3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds
>>
>> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
>> ---
>> I'd like to get this one in master, because the e-mails are annoying me
>> :-)
>
> I'd like to suggest an alternative: move all the conditions into the
> generating script, now that we do have a script that generates the
> pipeline description.
Note that it's not the complicated conditions on the defconfig and test jobs
that are the issue. The warning is also issued for check-flake8... So I think
the workflow will still be needed (or alternatively, every job should have a
condition to run only on branches).
Also, I would like to move the "constant" bits back out to the top-level
gitlab-ci.yml, so the first page already shows the quick test results.
> Having the conditions in the script will help handle such cases, by only
> ever emitting those jobs we actually want to run. This would allow for
> more flexibility than the limited micro-language used by gitlab-ci.yml.
>
> Also, this will allow us to not care about the evolutions of that micro-
> language.
>
> To that effect, I already started workign on that two weeks ago, and I
> have omething that works as before (even better I think):
>
> https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/gitlab-ci-cond
>
> I just need to cleanup that last commit (and remember why I started
> doing it. meh...). I hope I'll be able to send it tonight.
I think that's a great idea. However, that sounds like a bigger change which
probably shouldn't go into master. So I'd prefer to still merge my patch as well.
Regards,
Arnout
>
> Regards,
> Yann E. MORIN.
>
>> ---
>> support/misc/gitlab-ci.yml.in | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in
>> index dddebf09e9..8a031898ef 100644
>> --- a/support/misc/gitlab-ci.yml.in
>> +++ b/support/misc/gitlab-ci.yml.in
>> @@ -1,6 +1,9 @@
>> # Configuration for Gitlab-CI.
>> # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
>>
>> +include:
>> + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'
>> +
>> image: buildroot/base:20200814.2228
>>
>> .check_base:
>> --
>> 2.26.2
>>
>> _______________________________________________
>> buildroot mailing list
>> buildroot at busybox.net
>> http://lists.busybox.net/mailman/listinfo/buildroot
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] [PATCH] support/misc/gitlab-ci.yml.in: include Branches workflow
2020-08-31 12:58 ` Arnout Vandecappelle
@ 2020-08-31 19:52 ` Yann E. MORIN
2020-09-01 6:52 ` Arnout Vandecappelle
2020-08-31 21:49 ` Yann E. MORIN
1 sibling, 1 reply; 7+ messages in thread
From: Yann E. MORIN @ 2020-08-31 19:52 UTC (permalink / raw)
To: buildroot
Arnout, All,
On 2020-08-31 14:58 +0200, Arnout Vandecappelle spake thusly:
> Note that it's not the complicated conditions on the defconfig and test jobs
> that are the issue. The warning is also issued for check-flake8... So I think
> the workflow will still be needed (or alternatively, every job should have a
> condition to run only on branches).
Sorry, but I don't follow: if the jobs are only generated when we want
them, then they no longer have conditions in the generated gitlab-ci.yml
to begin with; that's the whole point of moving the conditional handling
into the script.
> On 31/08/2020 14:43, Yann E. MORIN wrote:
> > I'd like to suggest an alternative: move all the conditions into the
> > generating script, now that we do have a script that generates the
> > pipeline description.
[--SNIP--]
> Also, I would like to move the "constant" bits back out to the top-level
> gitlab-ci.yml, so the first page already shows the quick test results.
Sorry, but again they are not "constant". For example, check-flake8,
check-package and check-DEVELOPERS are not run when one pushes a
*_defconfig or *_test branch (or however they are named) to trigger the
build of jsut that defconfig or just that runtime test.
> > Having the conditions in the script will help handle such cases, by only
> > ever emitting those jobs we actually want to run. This would allow for
> > more flexibility than the limited micro-language used by gitlab-ci.yml.
> >
> > Also, this will allow us to not care about the evolutions of that micro-
> > language.
> I think that's a great idea. However, that sounds like a bigger change which
> probably shouldn't go into master. So I'd prefer to still merge my patch as well.
See an actual comment on that patch, below...
> >> diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in
> >> index dddebf09e9..8a031898ef 100644
> >> --- a/support/misc/gitlab-ci.yml.in
> >> +++ b/support/misc/gitlab-ci.yml.in
> >> @@ -1,6 +1,9 @@
> >> # Configuration for Gitlab-CI.
> >> # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
> >>
> >> +include:
> >> + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'
Where does that file come from? How are we guaranteed that its content
does not change and break our pipelines?
If that template is interesting, then we should carry it rather than
depend on an external, un-managed entity...
Regards,
Yann E. MORIN.
> >> image: buildroot/base:20200814.2228
> >>
> >> .check_base:
> >> --
> >> 2.26.2
> >>
> >> _______________________________________________
> >> buildroot mailing list
> >> buildroot at busybox.net
> >> http://lists.busybox.net/mailman/listinfo/buildroot
> >
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] [PATCH] support/misc/gitlab-ci.yml.in: include Branches workflow
2020-08-31 12:58 ` Arnout Vandecappelle
2020-08-31 19:52 ` Yann E. MORIN
@ 2020-08-31 21:49 ` Yann E. MORIN
1 sibling, 0 replies; 7+ messages in thread
From: Yann E. MORIN @ 2020-08-31 21:49 UTC (permalink / raw)
To: buildroot
Arnout, Jugurtha, All,
On 2020-08-31 14:58 +0200, Arnout Vandecappelle spake thusly:
> On 31/08/2020 14:43, Yann E. MORIN wrote:
> > +Romain and +Jugurtha, as we've had some discussions on that topic
> > publicly and privately before.
[--SNIP--]
> > To that effect, I already started workign on that two weeks ago, and I
> > have omething that works as before (even better I think):
> > https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/gitlab-ci-cond
> > I just need to cleanup that last commit (and remember why I started
> > doing it. meh...). I hope I'll be able to send it tonight.
And of course, as usual, I did not have time to properly finish that
branch...
Still, it is very close to be ready, with all the test cases now working
as expected:
Random branch, only basic tests
https://gitlab.com/ymorin/buildroot/-/commits/yem/gitlab-ci-cond
https://gitlab.com/ymorin/buildroot/-/pipelines/184068345
All defconfigs:
https://gitlab.com/ymorin/buildroot/-/commits/FOO-defconfigs
https://gitlab.com/ymorin/buildroot/-/pipelines/184069188
One defconfig:
https://gitlab.com/ymorin/buildroot/-/commits/FOO-warp7_defconfig
https://gitlab.com/ymorin/buildroot/-/pipelines/184068523
All runtime tests:
https://gitlab.com/ymorin/buildroot/-/commits/FOO-runtime-tests
https://gitlab.com/ymorin/buildroot/-/pipelines/184070380
One runtime test:
https://gitlab.com/ymorin/buildroot/-/commits/FOO-tests.package.test_openssh
https://gitlab.com/ymorin/buildroot/-/pipelines/184068633
I still have to test a maintenance branch (by faking one with a branch
named 'fooo.no.x' to match the pattern), but I'm afraid I've reached my
allowed number of jobs for now...
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] [PATCH] support/misc/gitlab-ci.yml.in: include Branches workflow
2020-08-31 19:52 ` Yann E. MORIN
@ 2020-09-01 6:52 ` Arnout Vandecappelle
0 siblings, 0 replies; 7+ messages in thread
From: Arnout Vandecappelle @ 2020-09-01 6:52 UTC (permalink / raw)
To: buildroot
On 31/08/2020 21:52, Yann E. MORIN wrote:
> Arnout, All,
>
> On 2020-08-31 14:58 +0200, Arnout Vandecappelle spake thusly:
>> Note that it's not the complicated conditions on the defconfig and test jobs
>> that are the issue. The warning is also issued for check-flake8... So I think
>> the workflow will still be needed (or alternatively, every job should have a
>> condition to run only on branches).
>
> Sorry, but I don't follow: if the jobs are only generated when we want
> them, then they no longer have conditions in the generated gitlab-ci.yml
> to begin with; that's the whole point of moving the conditional handling
> into the script.
Ah, OK, I thought it was just about generating the defconfig and tests parts.
My bad.
>> On 31/08/2020 14:43, Yann E. MORIN wrote:
[snip]
>
> See an actual comment on that patch, below...
>
>>>> diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in
>>>> index dddebf09e9..8a031898ef 100644
>>>> --- a/support/misc/gitlab-ci.yml.in
>>>> +++ b/support/misc/gitlab-ci.yml.in
>>>> @@ -1,6 +1,9 @@
>>>> # Configuration for Gitlab-CI.
>>>> # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
>>>>
>>>> +include:
>>>> + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'
>
> Where does that file come from? How are we guaranteed that its content
> does not change and break our pipelines?
>
> If that template is interesting, then we should carry it rather than
> depend on an external, un-managed entity...
It is part of the gitlab installation. That means that indeed it is not under
our control, but neither is the rest of gitlab (e.g., the part that generates
those warnings :-). In other words: it's about as reliable as the rules clauses
we have now.
Note that the workflow file is indeed very simple, and we could simply
instantiate it here.
Again, though, in order to expedite the 2020.08 release, I propose to apply
this simple patch to master, and soon after merge your series that makes it
script-based - possibly backporting it to the stable branches, though I don't
really think that is needed.
Regards,
Arnout
^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] [PATCH] support/misc/gitlab-ci.yml.in: include Branches workflow
2020-08-30 15:59 [Buildroot] [PATCH] support/misc/gitlab-ci.yml.in: include Branches workflow Arnout Vandecappelle
2020-08-31 12:43 ` Yann E. MORIN
@ 2020-10-06 20:42 ` Yann E. MORIN
1 sibling, 0 replies; 7+ messages in thread
From: Yann E. MORIN @ 2020-10-06 20:42 UTC (permalink / raw)
To: buildroot
Arnout, All,
On 2020-08-30 17:59 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly:
> Recently, gitlab-ci has gained a lot more flexibility in when pipelines
> are created. As a side-effect of this, double pipelines may be created
> when a `when` clause is used in some job. Avoid this by adding a
> workflow that launches it only on branches. See [1] and [2].
>
> Note that in reality, the duplicate pipelines only occur for merge
> requests. Since we don't have merge requests, we don't have this
> problem. Howver, gitlab also displays an annoying error message on the
> pipeline page [3], and it sends an error mail to the triggerer (i.e.,
> Arnout), so it's still useful to do this.
>
> [1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines
> [2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates
> [3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds
>
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
We've no merged a better script that contains all the logic about
generating the child pipeline, so I believe this patch of yours is no
longer needed.
Regards,
Yann E. MORIN.
> ---
> I'd like to get this one in master, because the e-mails are annoying me
> :-)
> ---
> support/misc/gitlab-ci.yml.in | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in
> index dddebf09e9..8a031898ef 100644
> --- a/support/misc/gitlab-ci.yml.in
> +++ b/support/misc/gitlab-ci.yml.in
> @@ -1,6 +1,9 @@
> # Configuration for Gitlab-CI.
> # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines
>
> +include:
> + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml'
> +
> image: buildroot/base:20200814.2228
>
> .check_base:
> --
> 2.26.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-10-06 20:42 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-08-30 15:59 [Buildroot] [PATCH] support/misc/gitlab-ci.yml.in: include Branches workflow Arnout Vandecappelle
2020-08-31 12:43 ` Yann E. MORIN
2020-08-31 12:58 ` Arnout Vandecappelle
2020-08-31 19:52 ` Yann E. MORIN
2020-09-01 6:52 ` Arnout Vandecappelle
2020-08-31 21:49 ` Yann E. MORIN
2020-10-06 20:42 ` Yann E. MORIN
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox