[XEN PATCH 0/3] automation/eclair: do not allow failure for triggered analyses

[XEN PATCH 0/3] automation/eclair: do not allow failure for triggered analyses

[Federico Serafini]

Patch 1/3 does some preparation work.

Patch 2/3, as the title says, removes allow_failure = true for triggered
analyses.

Patch 3/3 makes explicit that initally no files are tagged as adopted, this
is needed by the scheduled analysis.

Federico Serafini (3):
  automation/eclair: tag Rule 7.2 as clean and temporarily remove Rules
    1.1 and 8.2
  automation: do not allow failure for triggered analyses
  automation/eclair: make explicit there are no adopted files by default

 automation/eclair_analysis/ECLAIR/analysis.ecl | 4 ++++
 automation/eclair_analysis/ECLAIR/tagging.ecl  | 2 +-
 automation/gitlab-ci/analyze.yaml              | 1 -
 3 files changed, 5 insertions(+), 2 deletions(-)

-- 
2.34.1

[XEN PATCH 1/3] automation/eclair: tag Rule 7.2 as clean and temporarily remove Rules 1.1 and 8.2

[Federico Serafini]

Update ECLAIR configuration to consider Rule 7.2 as clean.

Temporarily remove the clean tag from Rules 1.1 and 8.2:
when violations of such rules will be addressed, the clean tag will be
reintroduced.

Signed-off-by: Federico Serafini <federico.serafini@bugseng.com>
---
 automation/eclair_analysis/ECLAIR/tagging.ecl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/automation/eclair_analysis/ECLAIR/tagging.ecl b/automation/eclair_analysis/ECLAIR/tagging.ecl
index d609b470eb..bdf94ed996 100644
--- a/automation/eclair_analysis/ECLAIR/tagging.ecl
+++ b/automation/eclair_analysis/ECLAIR/tagging.ecl
@@ -19,7 +19,7 @@
 
 -doc_begin="Clean guidelines: new violations for these guidelines are not accepted."
 
--service_selector={clean_guidelines_common,"MC3R1.D1.1||MC3R1.D2.1||MC3R1.D4.1||MC3R1.D4.11||MC3R1.D4.14||MC3R1.R1.1||MC3R1.R11.7||MC3R1.R11.9||MC3R1.R12.5||MC3R1.R1.3||MC3R1.R1.4||MC3R1.R14.1||MC3R1.R16.7||MC3R1.R17.1||MC3R1.R17.3||MC3R1.R17.4||MC3R1.R17.5||MC3R1.R17.6||MC3R1.R20.13||MC3R1.R20.14||MC3R1.R20.4||MC3R1.R20.9||MC3R1.R21.13||MC3R1.R21.19||MC3R1.R21.21||MC3R1.R2.2||MC3R1.R22.2||MC3R1.R22.4||MC3R1.R22.5||MC3R1.R22.6||MC3R1.R2.6||MC3R1.R3.1||MC3R1.R3.2||MC3R1.R4.1||MC3R1.R4.2||MC3R1.R5.1||MC3R1.R5.2||MC3R1.R5.4||MC3R1.R5.6||MC3R1.R6.1||MC3R1.R6.2||MC3R1.R7.1||MC3R1.R7.4||MC3R1.R8.1||MC3R1.R8.10||MC3R1.R8.12||MC3R1.R8.14||MC3R1.R8.2||MC3R1.R8.5||MC3R1.R8.6||MC3R1.R8.8||MC3R1.R9.2||MC3R1.R9.3||MC3R1.R9.4||MC3R1.R9.5"
+-service_selector={clean_guidelines_common,"MC3R1.D1.1||MC3R1.D2.1||MC3R1.D4.1||MC3R1.D4.11||MC3R1.D4.14||MC3R1.R11.7||MC3R1.R11.9||MC3R1.R12.5||MC3R1.R1.3||MC3R1.R1.4||MC3R1.R14.1||MC3R1.R16.7||MC3R1.R17.1||MC3R1.R17.3||MC3R1.R17.4||MC3R1.R17.5||MC3R1.R17.6||MC3R1.R20.13||MC3R1.R20.14||MC3R1.R20.4||MC3R1.R20.9||MC3R1.R21.13||MC3R1.R21.19||MC3R1.R21.21||MC3R1.R2.2||MC3R1.R22.2||MC3R1.R22.4||MC3R1.R22.5||MC3R1.R22.6||MC3R1.R2.6||MC3R1.R3.1||MC3R1.R3.2||MC3R1.R4.1||MC3R1.R4.2||MC3R1.R5.1||MC3R1.R5.2||MC3R1.R5.4||MC3R1.R5.6||MC3R1.R6.1||MC3R1.R6.2||MC3R1.R7.1||MC3R1.R7.2||MC3R1.R7.4||MC3R1.R8.1||MC3R1.R8.10||MC3R1.R8.12||MC3R1.R8.14||MC3R1.R8.5||MC3R1.R8.6||MC3R1.R8.8||MC3R1.R9.2||MC3R1.R9.3||MC3R1.R9.4||MC3R1.R9.5"
 }
 
 -setq=target,getenv("XEN_TARGET_ARCH")
-- 
2.34.1

Re: [XEN PATCH 1/3] automation/eclair: tag Rule 7.2 as clean and temporarily remove Rules 1.1 and 8.2

[Stefano Stabellini]

On Mon, 29 Apr 2024, Federico Serafini wrote:
> Update ECLAIR configuration to consider Rule 7.2 as clean.
> 
> Temporarily remove the clean tag from Rules 1.1 and 8.2:
> when violations of such rules will be addressed, the clean tag will be
> reintroduced.
> 
> Signed-off-by: Federico Serafini <federico.serafini@bugseng.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  automation/eclair_analysis/ECLAIR/tagging.ecl | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/automation/eclair_analysis/ECLAIR/tagging.ecl b/automation/eclair_analysis/ECLAIR/tagging.ecl
> index d609b470eb..bdf94ed996 100644
> --- a/automation/eclair_analysis/ECLAIR/tagging.ecl
> +++ b/automation/eclair_analysis/ECLAIR/tagging.ecl
> @@ -19,7 +19,7 @@
>  
>  -doc_begin="Clean guidelines: new violations for these guidelines are not accepted."
>  
> --service_selector={clean_guidelines_common,"MC3R1.D1.1||MC3R1.D2.1||MC3R1.D4.1||MC3R1.D4.11||MC3R1.D4.14||MC3R1.R1.1||MC3R1.R11.7||MC3R1.R11.9||MC3R1.R12.5||MC3R1.R1.3||MC3R1.R1.4||MC3R1.R14.1||MC3R1.R16.7||MC3R1.R17.1||MC3R1.R17.3||MC3R1.R17.4||MC3R1.R17.5||MC3R1.R17.6||MC3R1.R20.13||MC3R1.R20.14||MC3R1.R20.4||MC3R1.R20.9||MC3R1.R21.13||MC3R1.R21.19||MC3R1.R21.21||MC3R1.R2.2||MC3R1.R22.2||MC3R1.R22.4||MC3R1.R22.5||MC3R1.R22.6||MC3R1.R2.6||MC3R1.R3.1||MC3R1.R3.2||MC3R1.R4.1||MC3R1.R4.2||MC3R1.R5.1||MC3R1.R5.2||MC3R1.R5.4||MC3R1.R5.6||MC3R1.R6.1||MC3R1.R6.2||MC3R1.R7.1||MC3R1.R7.4||MC3R1.R8.1||MC3R1.R8.10||MC3R1.R8.12||MC3R1.R8.14||MC3R1.R8.2||MC3R1.R8.5||MC3R1.R8.6||MC3R1.R8.8||MC3R1.R9.2||MC3R1.R9.3||MC3R1.R9.4||MC3R1.R9.5"
> +-service_selector={clean_guidelines_common,"MC3R1.D1.1||MC3R1.D2.1||MC3R1.D4.1||MC3R1.D4.11||MC3R1.D4.14||MC3R1.R11.7||MC3R1.R11.9||MC3R1.R12.5||MC3R1.R1.3||MC3R1.R1.4||MC3R1.R14.1||MC3R1.R16.7||MC3R1.R17.1||MC3R1.R17.3||MC3R1.R17.4||MC3R1.R17.5||MC3R1.R17.6||MC3R1.R20.13||MC3R1.R20.14||MC3R1.R20.4||MC3R1.R20.9||MC3R1.R21.13||MC3R1.R21.19||MC3R1.R21.21||MC3R1.R2.2||MC3R1.R22.2||MC3R1.R22.4||MC3R1.R22.5||MC3R1.R22.6||MC3R1.R2.6||MC3R1.R3.1||MC3R1.R3.2||MC3R1.R4.1||MC3R1.R4.2||MC3R1.R5.1||MC3R1.R5.2||MC3R1.R5.4||MC3R1.R5.6||MC3R1.R6.1||MC3R1.R6.2||MC3R1.R7.1||MC3R1.R7.2||MC3R1.R7.4||MC3R1.R8.1||MC3R1.R8.10||MC3R1.R8.12||MC3R1.R8.14||MC3R1.R8.5||MC3R1.R8.6||MC3R1.R8.8||MC3R1.R9.2||MC3R1.R9.3||MC3R1.R9.4||MC3R1.R9.5"
>  }
>  
>  -setq=target,getenv("XEN_TARGET_ARCH")
> -- 
> 2.34.1
> 

[XEN PATCH 2/3] automation: do not allow failure for triggered analyses

[Federico Serafini]

Do not allow_failure for triggered analyses:
introducing regressions of clean guidelines will cause a CI failure.

Signed-off-by: Federico Serafini <federico.serafini@bugseng.com>
---
 automation/gitlab-ci/analyze.yaml | 1 -
 1 file changed, 1 deletion(-)

diff --git a/automation/gitlab-ci/analyze.yaml b/automation/gitlab-ci/analyze.yaml
index 46c9d8e2e5..32bf570149 100644
--- a/automation/gitlab-ci/analyze.yaml
+++ b/automation/gitlab-ci/analyze.yaml
@@ -26,7 +26,6 @@
 
 .eclair-analysis:triggered:
   extends: .eclair-analysis
-  allow_failure: true
   rules:
     - if: $CI_PIPELINE_SOURCE == "schedule"
       when: never
-- 
2.34.1

Re: [XEN PATCH 2/3] automation: do not allow failure for triggered analyses

[Stefano Stabellini]

On Mon, 29 Apr 2024, Federico Serafini wrote:
> Do not allow_failure for triggered analyses:
> introducing regressions of clean guidelines will cause a CI failure.
> 
> Signed-off-by: Federico Serafini <federico.serafini@bugseng.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  automation/gitlab-ci/analyze.yaml | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/automation/gitlab-ci/analyze.yaml b/automation/gitlab-ci/analyze.yaml
> index 46c9d8e2e5..32bf570149 100644
> --- a/automation/gitlab-ci/analyze.yaml
> +++ b/automation/gitlab-ci/analyze.yaml
> @@ -26,7 +26,6 @@
>  
>  .eclair-analysis:triggered:
>    extends: .eclair-analysis
> -  allow_failure: true
>    rules:
>      - if: $CI_PIPELINE_SOURCE == "schedule"
>        when: never
> -- 
> 2.34.1
> 

[XEN PATCH 3/3] automation/eclair: make explicit there are no adopted files by default

[Federico Serafini]

Update ECLAIR configuration to consider no adopted files by default.

Signed-off-by: Federico Serafini <federico.serafini@bugseng.com>
---
 automation/eclair_analysis/ECLAIR/analysis.ecl | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/automation/eclair_analysis/ECLAIR/analysis.ecl b/automation/eclair_analysis/ECLAIR/analysis.ecl
index a604582da3..66ed7f952c 100644
--- a/automation/eclair_analysis/ECLAIR/analysis.ecl
+++ b/automation/eclair_analysis/ECLAIR/analysis.ecl
@@ -21,6 +21,10 @@ map_strings("scheduled-analysis",analysis_kind)
 
 -eval_file=toolchain.ecl
 -eval_file=public_APIs.ecl
+
+-doc="Initially, there are no files tagged as adopted."
+-file_tag+={adopted,"none()"}
+
 if(not(scheduled_analysis),
     eval_file("adopted.ecl")
 )
-- 
2.34.1

Re: [XEN PATCH 3/3] automation/eclair: make explicit there are no adopted files by default

[Stefano Stabellini]

On Mon, 29 Apr 2024, Federico Serafini wrote:
> Update ECLAIR configuration to consider no adopted files by default.
> 
> Signed-off-by: Federico Serafini <federico.serafini@bugseng.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  automation/eclair_analysis/ECLAIR/analysis.ecl | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/automation/eclair_analysis/ECLAIR/analysis.ecl b/automation/eclair_analysis/ECLAIR/analysis.ecl
> index a604582da3..66ed7f952c 100644
> --- a/automation/eclair_analysis/ECLAIR/analysis.ecl
> +++ b/automation/eclair_analysis/ECLAIR/analysis.ecl
> @@ -21,6 +21,10 @@ map_strings("scheduled-analysis",analysis_kind)
>  
>  -eval_file=toolchain.ecl
>  -eval_file=public_APIs.ecl
> +
> +-doc="Initially, there are no files tagged as adopted."
> +-file_tag+={adopted,"none()"}
> +
>  if(not(scheduled_analysis),
>      eval_file("adopted.ecl")
>  )
> -- 
> 2.34.1
> 

[REGRESSION] Re: [XEN PATCH 0/3] automation/eclair: do not allow failure for triggered analyses

[Andrew Cooper]

On 29/04/2024 4:21 pm, Federico Serafini wrote:
> Patch 1/3 does some preparation work.
>
> Patch 2/3, as the title says, removes allow_failure = true for triggered
> analyses.
>
> Patch 3/3 makes explicit that initally no files are tagged as adopted, this
> is needed by the scheduled analysis.

I'm afraid that something in this series is broken.

Since these patches went in, all pipelines are now getting a status of
blocked rather than passed.

If I manually start the Eclair jobs, then eventually the pipeline gets
to Passed.

~Andrew

Re: [REGRESSION] Re: [XEN PATCH 0/3] automation/eclair: do not allow failure for triggered analyses

[Federico Serafini]

On 03/05/24 21:14, Andrew Cooper wrote:
> On 29/04/2024 4:21 pm, Federico Serafini wrote:
>> Patch 1/3 does some preparation work.
>>
>> Patch 2/3, as the title says, removes allow_failure = true for triggered
>> analyses.
>>
>> Patch 3/3 makes explicit that initally no files are tagged as adopted, this
>> is needed by the scheduled analysis.
> 
> I'm afraid that something in this series is broken.
> 
> Since these patches went in, all pipelines are now getting a status of
> blocked rather than passed.
> 
> If I manually start the Eclair jobs, then eventually the pipeline gets
> to Passed.

Can you provide us a link to those failures?
I am looking at gitlab xen-project/xen and xen-project/patchew
and everything seems ok.

-- 
Federico Serafini, M.Sc.

Software Engineer, BUGSENG (http://bugseng.com)

Re: [REGRESSION] Re: [XEN PATCH 0/3] automation/eclair: do not allow failure for triggered analyses

[Andrew Cooper]

On 03/05/2024 8:44 pm, Federico Serafini wrote:
> On 03/05/24 21:14, Andrew Cooper wrote:
>> On 29/04/2024 4:21 pm, Federico Serafini wrote:
>>> Patch 1/3 does some preparation work.
>>>
>>> Patch 2/3, as the title says, removes allow_failure = true for
>>> triggered
>>> analyses.
>>>
>>> Patch 3/3 makes explicit that initally no files are tagged as
>>> adopted, this
>>> is needed by the scheduled analysis.
>>
>> I'm afraid that something in this series is broken.
>>
>> Since these patches went in, all pipelines are now getting a status of
>> blocked rather than passed.
>>
>> If I manually start the Eclair jobs, then eventually the pipeline gets
>> to Passed.
>
> Can you provide us a link to those failures?
> I am looking at gitlab xen-project/xen and xen-project/patchew
> and everything seems ok.
>

https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1276081658
is the first one I noticed as blocked, and I manually ran.  That ended
up working.

https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1277724847
is still in the blocked state.  The build-each-commit failure is unrelated.

~Andrew

Re: [REGRESSION] Re: [XEN PATCH 0/3] automation/eclair: do not allow failure for triggered analyses

[Federico Serafini]

On 03/05/24 21:46, Andrew Cooper wrote:
> On 03/05/2024 8:44 pm, Federico Serafini wrote:
>> On 03/05/24 21:14, Andrew Cooper wrote:
>>> On 29/04/2024 4:21 pm, Federico Serafini wrote:
>>>> Patch 1/3 does some preparation work.
>>>>
>>>> Patch 2/3, as the title says, removes allow_failure = true for
>>>> triggered
>>>> analyses.
>>>>
>>>> Patch 3/3 makes explicit that initally no files are tagged as
>>>> adopted, this
>>>> is needed by the scheduled analysis.
>>>
>>> I'm afraid that something in this series is broken.
>>>
>>> Since these patches went in, all pipelines are now getting a status of
>>> blocked rather than passed.
>>>
>>> If I manually start the Eclair jobs, then eventually the pipeline gets
>>> to Passed.
>>
>> Can you provide us a link to those failures?
>> I am looking at gitlab xen-project/xen and xen-project/patchew
>> and everything seems ok.
>>
> 
> https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1276081658
> is the first one I noticed as blocked, and I manually ran.  That ended
> up working.
> 
> https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1277724847
> is still in the blocked state.  The build-each-commit failure is unrelated.

This is intentional and was introduced by
commit 7c1bf8661db5c00bd8c9a25015fe8678b2ff9ac6

The ECLAIR analysis under people/* need to be activated
manually.

Is this causing some problems to the CI?

-- 
Federico Serafini, M.Sc.

Software Engineer, BUGSENG (http://bugseng.com)

Re: [REGRESSION] Re: [XEN PATCH 0/3] automation/eclair: do not allow failure for triggered analyses

[Andrew Cooper]

On 03/05/2024 9:01 pm, Federico Serafini wrote:
> On 03/05/24 21:46, Andrew Cooper wrote:
>> On 03/05/2024 8:44 pm, Federico Serafini wrote:
>>> On 03/05/24 21:14, Andrew Cooper wrote:
>>>> On 29/04/2024 4:21 pm, Federico Serafini wrote:
>>>>> Patch 1/3 does some preparation work.
>>>>>
>>>>> Patch 2/3, as the title says, removes allow_failure = true for
>>>>> triggered
>>>>> analyses.
>>>>>
>>>>> Patch 3/3 makes explicit that initally no files are tagged as
>>>>> adopted, this
>>>>> is needed by the scheduled analysis.
>>>>
>>>> I'm afraid that something in this series is broken.
>>>>
>>>> Since these patches went in, all pipelines are now getting a status of
>>>> blocked rather than passed.
>>>>
>>>> If I manually start the Eclair jobs, then eventually the pipeline gets
>>>> to Passed.
>>>
>>> Can you provide us a link to those failures?
>>> I am looking at gitlab xen-project/xen and xen-project/patchew
>>> and everything seems ok.
>>>
>>
>> https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1276081658
>> is the first one I noticed as blocked, and I manually ran.  That ended
>> up working.
>>
>> https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1277724847
>> is still in the blocked state.  The build-each-commit failure is
>> unrelated.
>
> This is intentional and was introduced by
> commit 7c1bf8661db5c00bd8c9a25015fe8678b2ff9ac6
>
> The ECLAIR analysis under people/* need to be activated
> manually.

Yes.  I know, and that matches the behaviour I saw.

>
> Is this causing some problems to the CI?
>

Yes.

See https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines

Prior to this series, the manual actions were not used but the pipeline
was overall in the Passed state.  Specifically, they ended up being skipped.

After this series, the manual actions are now blocking the pipeline, not
letting it complete, and not marking it as passed.

~Andrew

Re: [REGRESSION] Re: [XEN PATCH 0/3] automation/eclair: do not allow failure for triggered analyses

[Nicola Vetrini]

On 2024-05-03 22:06, Andrew Cooper wrote:
> On 03/05/2024 9:01 pm, Federico Serafini wrote:
>> On 03/05/24 21:46, Andrew Cooper wrote:
>>> On 03/05/2024 8:44 pm, Federico Serafini wrote:
>>>> On 03/05/24 21:14, Andrew Cooper wrote:
>>>>> On 29/04/2024 4:21 pm, Federico Serafini wrote:
>>>>>> Patch 1/3 does some preparation work.
>>>>>> 

Hi,

>>>>>> Patch 2/3, as the title says, removes allow_failure = true for
>>>>>> triggered
>>>>>> analyses.
>>>>>> 
>>>>>> Patch 3/3 makes explicit that initally no files are tagged as
>>>>>> adopted, this
>>>>>> is needed by the scheduled analysis.
>>>>> 
>>>>> I'm afraid that something in this series is broken.
>>>>> 
>>>>> Since these patches went in, all pipelines are now getting a status 
>>>>> of
>>>>> blocked rather than passed.
>>>>> 
>>>>> If I manually start the Eclair jobs, then eventually the pipeline 
>>>>> gets
>>>>> to Passed.
>>>> 
>>>> Can you provide us a link to those failures?
>>>> I am looking at gitlab xen-project/xen and xen-project/patchew
>>>> and everything seems ok.
>>>> 
>>> 
>>> https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1276081658
>>> is the first one I noticed as blocked, and I manually ran.  That 
>>> ended
>>> up working.
>>> 
>>> https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1277724847
>>> is still in the blocked state.  The build-each-commit failure is
>>> unrelated.
>> 
>> This is intentional and was introduced by
>> commit 7c1bf8661db5c00bd8c9a25015fe8678b2ff9ac6
>> 
>> The ECLAIR analysis under people/* need to be activated
>> manually.
> 
> Yes.  I know, and that matches the behaviour I saw.
> 
>> 
>> Is this causing some problems to the CI?
>> 
> 
> Yes.
> 
> See https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines
> 
> Prior to this series, the manual actions were not used but the pipeline
> was overall in the Passed state.  Specifically, they ended up being 
> skipped.
> 
> After this series, the manual actions are now blocking the pipeline, 
> not
> letting it complete, and not marking it as passed.

one way to fix this can be to set allow_failure when in the people/* 
branches, which will cause the stage to be skipped and thus mark the 
pipeline as Passed when all other steps have completed. See 
https://gitlab.com/xen-project/people/bugseng/xen/-/pipelines/1278550868


-- 
Nicola Vetrini, BSc
Software Engineer, BUGSENG srl (https://bugseng.com)