Re: tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)

Re: tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)

[Guillaume Tucker]

Hi Ingo,

On 07/02/2023 09:53, Ingo Molnar wrote:
> 
> * kernelci.org bot <bot@kernelci.org> wrote:
> 
>> tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)
>>
>> Full Build Summary: https://kernelci.org/build/tip/branch/master/kernel/v6.2-rc7-273-gd67c17ddc899/
>>
>> Tree: tip
>> Branch: master
> 
> Is anyone maintaining this CI build bot? Because it's either unmaintained, 
> or is blatantly lying about regressions.
> 
> Let's see one of the 'errors' it reported today:

Yes, it's being maintained.  However there's no way for the
KernelCI core team to manually verify all the errors reported.
So kernel developers following up on such errors is the best way
to handle them and get them resolved.  Thanks for replying to the
email report :)

>> Errors summary:
>>
>>     4    cc1: error: ‘-mloongson-mmi’ must be used with ‘-mhard-float’
> 
> ... but this exact same error was 'reported' a year ago on January 22:
> 
>>    2    cc1: error: ‘-mloongson-mmi’ must be used with ‘-mhard-float’
> 
> So these regression reports are useless in this form and they clutter 
> people's inboxes. Is any person reading them and acting on them to make 
> sure these emails are sensible?

They are not reported as regressions but as build errors.  So it
doesn't matter when it started failing or if this has always
failed, any build error will show up in the report.  It's
definitely something to improve, the runtime test reports
differentiate regressions or "new failures" and errors that have
always been there.  This will also apply to builds at some point,
it's on the roadmap for the next few months.  Please bear with us
in the meantime.

About the actual kernel build error, I guess it's a shame nobody
has fixed this yet.  I'll take a look what the root cause might
be.  If it's a KernelCI build configuration issue e.g. using
wrong compiler flags then we can fix that, otherwise it's
probably something to report more directly to some MIPS
maintainers.

Thanks,
Guillaume

Re: tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)

[Ingo Molnar]

* Guillaume Tucker <guillaume.tucker@collabora.com> wrote:

> Hi Ingo,
> 
> On 07/02/2023 09:53, Ingo Molnar wrote:
> > 
> > * kernelci.org bot <bot@kernelci.org> wrote:
> > 
> >> tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)
> >>
> >> Full Build Summary: https://kernelci.org/build/tip/branch/master/kernel/v6.2-rc7-273-gd67c17ddc899/
> >>
> >> Tree: tip
> >> Branch: master
> > 
> > Is anyone maintaining this CI build bot? Because it's either unmaintained, 
> > or is blatantly lying about regressions.
> > 
> > Let's see one of the 'errors' it reported today:
> 
> Yes, it's being maintained.

Good, and thanks for the quick response. :-)

> However there's no way for the KernelCI core team to manually verify all 
> the errors reported. So kernel developers following up on such errors is 
> the best way to handle them and get them resolved.  Thanks for replying 
> to the email report :)
> 
> >> Errors summary:
> >>
> >>     4    cc1: error: ‘-mloongson-mmi’ must be used with ‘-mhard-float’
> > 
> > ... but this exact same error was 'reported' a year ago on January 22:
> > 
> >>    2    cc1: error: ‘-mloongson-mmi’ must be used with ‘-mhard-float’
> > 
> > So these regression reports are useless in this form and they clutter 
> > people's inboxes. Is any person reading them and acting on them to make 
> > sure these emails are sensible?
> 
> They are not reported as regressions but as build errors.  So it doesn't 
> matter when it started failing or if this has always failed, any build 
> error will show up in the report.  It's definitely something to improve, 
> the runtime test reports differentiate regressions or "new failures" and 
> errors that have always been there.  This will also apply to builds at 
> some point, it's on the roadmap for the next few months.  Please bear 
> with us in the meantime.

At least the title of the email should report the 'delta' status of any 
tree it is testing: the difference to Linus's tree.

For example if Linus's tree is:

   linus/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-xxx-xxxxxxxxxxx)

   [ mockup. ]

Then the tip/master email title should not be:

   tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)

But something like this 'delta' report:

   tip/master build: 205 builds: PASS (v6.2-rc7-273-gd67c17ddc899)

Or if tip/master introduced new build failures/warnings, it should not 
report this:

   tip/master build: 205 builds: 6 failed, 199 passed, 10 errors, 24 warnings (v6.2-rc7-273-gd67c17ddc899)

... but should report something like this:

   tip/master build: 205 builds: FAIL [+1 errors, +2 warnings] (v6.2-rc7-273-gd67c17ddc899)

BTW., if I got such 'delta' emails I could guarantee to handle each and 
every one of them. With the current format it's the occasional spot check 
of these emails and a frustrating excercise of trying to figure out the 
delta to Linus's tree - and failing.

Yes, if tip/master has an older base then these might not be entirely 
accurate deltas, but *that* is something maintainers can fix in their 
routine flow: merge in Linus's latest or fix a bug newly introduced 
upstream.

But prominently displaying years-old bugs as 'errors' in the title is not 
just misleading, it's actively counterproductive. The primary output of 
continuous integration reports must always be 'delta' based.

> About the actual kernel build error, I guess it's a shame nobody has 
> fixed this yet. [...]

The main problem with the reports here is that there's no way to determine 
'at a glance' whether there's any regressions compared to Linus's tree - 
which is really what maintainers of non-Linus trees care about.

The figures the bot sends out as email titles are fine for Linus's tree I 
suspect, but are misleading in this form for any maintainer tree.

Thanks,

	Ingo

Re: tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1069 bytes --]

On Tue, Feb 07, 2023 at 10:56:04AM +0100, Guillaume Tucker wrote:
> On 07/02/2023 09:53, Ingo Molnar wrote:

> >>     4    cc1: error: ‘-mloongson-mmi’ must be used with ‘-mhard-float’
> > 
> > ... but this exact same error was 'reported' a year ago on January 22:
> > 
> >>    2    cc1: error: ‘-mloongson-mmi’ must be used with ‘-mhard-float’

> > So these regression reports are useless in this form and they clutter 
> > people's inboxes. Is any person reading them and acting on them to make 
> > sure these emails are sensible?

> About the actual kernel build error, I guess it's a shame nobody
> has fixed this yet.  I'll take a look what the root cause might
> be.  If it's a KernelCI build configuration issue e.g. using
> wrong compiler flags then we can fix that, otherwise it's
> probably something to report more directly to some MIPS
> maintainers.

Honestly at this point I think we should just drop the affected MIPS
configurations at this point, as Ingo says they've been failing for
so long with nobody caring.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)

[Guillaume Tucker]

+Thomas Bogendoerfer +linux-mips

On 07/02/2023 14:40, Mark Brown wrote:
> On Tue, Feb 07, 2023 at 10:56:04AM +0100, Guillaume Tucker wrote:
>> On 07/02/2023 09:53, Ingo Molnar wrote:
> 
>>>>     4    cc1: error: ‘-mloongson-mmi’ must be used with ‘-mhard-float’
>>>
>>> ... but this exact same error was 'reported' a year ago on January 22:
>>>
>>>>    2    cc1: error: ‘-mloongson-mmi’ must be used with ‘-mhard-float’
> 
>>> So these regression reports are useless in this form and they clutter 
>>> people's inboxes. Is any person reading them and acting on them to make 
>>> sure these emails are sensible?
> 
>> About the actual kernel build error, I guess it's a shame nobody
>> has fixed this yet.  I'll take a look what the root cause might
>> be.  If it's a KernelCI build configuration issue e.g. using
>> wrong compiler flags then we can fix that, otherwise it's
>> probably something to report more directly to some MIPS
>> maintainers.
> 
> Honestly at this point I think we should just drop the affected MIPS
> configurations at this point, as Ingo says they've been failing for
> so long with nobody caring.

Maybe the MIPS people just haven't seen this.  I've added Thomas
and the linux-mips list to confirm.

After some investigation, it turns out the error happens when
doing "make modules_install".  Here's the issue:

* modules_install is listed in "no-compiler-targets" in the
top-level Makefile

* as a result, scripts/Makefile.compiler is not included

* arch/mips/loongson64/Platform requires the "cc-option" function
  to add -mnon-loongson-mmi

* since "cc-option" is not defined when just doing "make
  modules_install", the flag is not added and the error mentioned
  above occurs

GitHub issue: https://github.com/kernelci/kernelci-project/issues/176

Here's a hack to prove this point, need-compiler is defined in
the top-level Makefile so it shouldn't be used here but
this "fixes" the problem:



diff --git a/arch/mips/Makefile b/arch/mips/Makefile
index 490dea07d4e0..024f62dbef76 100644
--- a/arch/mips/Makefile
+++ b/arch/mips/Makefile
@@ -317,10 +317,12 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
 KBUILD_LDFLAGS         += -m $(ld-emul)
 
 ifdef CONFIG_MIPS
+ifdef need-compiler
 CHECKFLAGS += $(shell $(CC) $(KBUILD_CFLAGS) -dM -E -x c /dev/null | \
        grep -E -vw '__GNUC_(MINOR_|PATCHLEVEL_)?_' | \
        sed -e "s/^\#define /-D'/" -e "s/ /'='/" -e "s/$$/'/" -e 's/\$$/&&/g')
 endif
+endif
 
 OBJCOPYFLAGS           += --remove-section=.reginfo
 


I guess another way would be to unconditionally add the options
to the cflags, in fact there are other places where this appears
to be done.  I'm not sure which GCC or Clang versions support it
or not, so that may not work in practice.



diff --git a/arch/mips/loongson64/Platform b/arch/mips/loongson64/Platform
index 473404cae1c4..b7b2db13f1a2 100644
--- a/arch/mips/loongson64/Platform
+++ b/arch/mips/loongson64/Platform
@@ -12,7 +12,7 @@ endif
 
 # Some -march= flags enable MMI instructions, and GCC complains about that
 # support being enabled alongside -msoft-float. Thus explicitly disable MMI.
-cflags-y += $(call cc-option,-mno-loongson-mmi)
+cflags-y += -mno-loongson-mmi
 
 #
 # Loongson Machines' Support



This was reproduced using GCC 10 with the latest
kernelci/gcc-10:mips-kselftest-kernelci Docker image running as
root.


Is this something someone familiar with MIPS would like to fix
properly?  If not then please confirm that we should just drop
the loongson2k_defconfig builds from KernelCI.  Is this kernel
config actually still maintained in mainline?

Thanks,
Guillaume

Re: tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)

[Nick Desaulniers]

On Tue, Feb 7, 2023 at 6:18 AM Guillaume Tucker
<guillaume.tucker@collabora.com> wrote:
>
> +Thomas Bogendoerfer +linux-mips
>
> On 07/02/2023 14:40, Mark Brown wrote:
> > On Tue, Feb 07, 2023 at 10:56:04AM +0100, Guillaume Tucker wrote:
> >> On 07/02/2023 09:53, Ingo Molnar wrote:
> >
> >>>>     4    cc1: error: ‘-mloongson-mmi’ must be used with ‘-mhard-float’
> >>>
> >>> ... but this exact same error was 'reported' a year ago on January 22:
> >>>
> >>>>    2    cc1: error: ‘-mloongson-mmi’ must be used with ‘-mhard-float’
> >
> >>> So these regression reports are useless in this form and they clutter
> >>> people's inboxes. Is any person reading them and acting on them to make
> >>> sure these emails are sensible?

FWIW, I read these reports daily. I only address the issues that look
specific to clang though.

> >
> >> About the actual kernel build error, I guess it's a shame nobody
> >> has fixed this yet.  I'll take a look what the root cause might
> >> be.  If it's a KernelCI build configuration issue e.g. using
> >> wrong compiler flags then we can fix that, otherwise it's
> >> probably something to report more directly to some MIPS
> >> maintainers.
> >
> > Honestly at this point I think we should just drop the affected MIPS
> > configurations at this point, as Ingo says they've been failing for
> > so long with nobody caring.
>
> Maybe the MIPS people just haven't seen this.  I've added Thomas
> and the linux-mips list to confirm.
>
> After some investigation, it turns out the error happens when
> doing "make modules_install".  Here's the issue:
>
> * modules_install is listed in "no-compiler-targets" in the
> top-level Makefile
>
> * as a result, scripts/Makefile.compiler is not included
>
> * arch/mips/loongson64/Platform requires the "cc-option" function
>   to add -mnon-loongson-mmi
>
> * since "cc-option" is not defined when just doing "make
>   modules_install", the flag is not added and the error mentioned
>   above occurs

+ Masahiro

>
> GitHub issue: https://github.com/kernelci/kernelci-project/issues/176
>
> Here's a hack to prove this point, need-compiler is defined in
> the top-level Makefile so it shouldn't be used here but
> this "fixes" the problem:
>
>
>
> diff --git a/arch/mips/Makefile b/arch/mips/Makefile
> index 490dea07d4e0..024f62dbef76 100644
> --- a/arch/mips/Makefile
> +++ b/arch/mips/Makefile
> @@ -317,10 +317,12 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
>  KBUILD_LDFLAGS         += -m $(ld-emul)
>
>  ifdef CONFIG_MIPS
> +ifdef need-compiler
>  CHECKFLAGS += $(shell $(CC) $(KBUILD_CFLAGS) -dM -E -x c /dev/null | \
>         grep -E -vw '__GNUC_(MINOR_|PATCHLEVEL_)?_' | \
>         sed -e "s/^\#define /-D'/" -e "s/ /'='/" -e "s/$$/'/" -e 's/\$$/&&/g')
>  endif
> +endif
>
>  OBJCOPYFLAGS           += --remove-section=.reginfo
>
>
>
> I guess another way would be to unconditionally add the options
> to the cflags, in fact there are other places where this appears
> to be done.  I'm not sure which GCC or Clang versions support it
> or not, so that may not work in practice.
>
>
>
> diff --git a/arch/mips/loongson64/Platform b/arch/mips/loongson64/Platform
> index 473404cae1c4..b7b2db13f1a2 100644
> --- a/arch/mips/loongson64/Platform
> +++ b/arch/mips/loongson64/Platform
> @@ -12,7 +12,7 @@ endif
>
>  # Some -march= flags enable MMI instructions, and GCC complains about that
>  # support being enabled alongside -msoft-float. Thus explicitly disable MMI.
> -cflags-y += $(call cc-option,-mno-loongson-mmi)
> +cflags-y += -mno-loongson-mmi

Nack. Clang doesn't support this flag. See also
commit 0e96ea5c3eb5 ("MIPS: Loongson64: Clean up use of cc-ifversion")


>
>  #
>  # Loongson Machines' Support
>
>
>
> This was reproduced using GCC 10 with the latest
> kernelci/gcc-10:mips-kselftest-kernelci Docker image running as
> root.
>
>
> Is this something someone familiar with MIPS would like to fix
> properly? If not then please confirm that we should just drop
> the loongson2k_defconfig builds from KernelCI.  Is this kernel
> config actually still maintained in mainline?
>
> Thanks,
> Guillaume
>
>


-- 
Thanks,
~Nick Desaulniers

Re: tip/master build: 205 builds: 5 failed, 200 passed, 9 errors, 22 warnings (v6.2-rc7-273-gd67c17ddc899)

[Maciej W. Rozycki]

On Tue, 7 Feb 2023, Guillaume Tucker wrote:

> After some investigation, it turns out the error happens when
> doing "make modules_install".  Here's the issue:
> 
> * modules_install is listed in "no-compiler-targets" in the
> top-level Makefile
> 
> * as a result, scripts/Makefile.compiler is not included
> 
> * arch/mips/loongson64/Platform requires the "cc-option" function
>   to add -mnon-loongson-mmi
> 
> * since "cc-option" is not defined when just doing "make
>   modules_install", the flag is not added and the error mentioned
>   above occurs
> 
> GitHub issue: https://github.com/kernelci/kernelci-project/issues/176
> 
> Here's a hack to prove this point, need-compiler is defined in
> the top-level Makefile so it shouldn't be used here but
> this "fixes" the problem:
> 
> 
> 
> diff --git a/arch/mips/Makefile b/arch/mips/Makefile
> index 490dea07d4e0..024f62dbef76 100644
> --- a/arch/mips/Makefile
> +++ b/arch/mips/Makefile
> @@ -317,10 +317,12 @@ KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
>  KBUILD_LDFLAGS         += -m $(ld-emul)
>  
>  ifdef CONFIG_MIPS
> +ifdef need-compiler
>  CHECKFLAGS += $(shell $(CC) $(KBUILD_CFLAGS) -dM -E -x c /dev/null | \
>         grep -E -vw '__GNUC_(MINOR_|PATCHLEVEL_)?_' | \
>         sed -e "s/^\#define /-D'/" -e "s/ /'='/" -e "s/$$/'/" -e 's/\$$/&&/g')
>  endif
> +endif

 I think it is actually the proper way, and if this gets a go-ahead, then 
all you need is to replace `CONFIG_MIPS' with `need-compiler' in the 
existing conditional, as the very purpose of this conditional is to guard 
the assignment against being invoked, except for `no-dot-config-targets' 
only (as discussed here: 
<https://lore.kernel.org/all/1675328702-8328-1-git-send-email-yangtiezhu@loongson.cn/> 
as recently as last week) rather than all of `no-compiler-targets'.  
Evidently that is not enough, but please mind that with embedded targets 
people often do not use modules and therefore do not trip over issues with 
`make modules_install'.

 This should probably be marked with:

Fixes: 13ceb48bc19c ("MIPS: Loongson2ef: Remove unnecessary 
{as,cc}-option calls")

as it's when `-march=loongson2f' (which implies `-mloongson-mmi') started 
being added unconditionally (and then `-march=loongson3a' followed with 
commit 0e96ea5c3eb5 ("MIPS: Loongson64: Clean up use of cc-ifversion")).
Nick, I'm afraid these were both your stuff.

 Also Loongson targets have their dedicated platform maintainers (cc-ed) 
and general MIPS maintainers may pay less attention to such issues.

 FWIW and HTH,

  Maciej