linux-arm-kernel.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* ❌ FAIL: Test report for kernel 5.5.0-rc6-40f39e8.cki (arm-next)
@ 2020-01-16 13:19 CKI Project
  2020-01-16 13:55 ` Rachel Sibley
       [not found] ` <CAKv+Gu-CEGOdTvektx_pqD=WqUXFi3YKXmj=pka5CiHc-6dCSQ@mail.gmail.com>
  0 siblings, 2 replies; 6+ messages in thread
From: CKI Project @ 2020-01-16 13:19 UTC (permalink / raw)
  To: will, catalin.marinas, linux-arm-kernel
  Cc: Jianlin Shi, Jianwen Ji, Hangbin Liu


Hello,

We ran automated tests on a recent commit from this kernel tree:

       Kernel repo: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
            Commit: 40f39e8372fe - Merge branch 'for-next/core' into for-kernelci

The results of these automated tests are provided below.

    Overall result: FAILED (see details below)
             Merge: OK
           Compile: OK
             Tests: FAILED

All kernel binaries, config files, and logs are available for download here:

  https://artifacts.cki-project.org/pipelines/385274

One or more kernel tests failed:

    aarch64:
     ❌ Networking tunnel: gre basic
     ❌ Networking tunnel: vxlan basic

We hope that these logs can help you find the problem quickly. For the full
detail on our testing procedures, please scroll to the bottom of this message.

Please reply to this email if you have any questions about the tests that we
ran or if you have any suggestions on how to make future tests more effective.

        ,-.   ,-.
       ( C ) ( K )  Continuous
        `-',-.`-'   Kernel
          ( I )     Integration
           `-'
______________________________________________________________________________

Compile testing
---------------

We compiled the kernel for 1 architecture:

    aarch64:
      make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg


Hardware testing
----------------
We booted each kernel and ran the following tests:

  aarch64:
    Host 1:
       ✅ Boot test
       ✅ Podman system integration test (as root)
       ✅ Podman system integration test (as user)
       ✅ LTP
       ✅ Loopdev Sanity
       ✅ Memory function: memfd_create
       ✅ AMTU (Abstract Machine Test Utility)
       ✅ Networking bridge: sanity
       ✅ Ethernet drivers sanity
       ✅ Networking MACsec: sanity
       ✅ Networking socket: fuzz
       ✅ Networking sctp-auth: sockopts test
       ✅ Networking: igmp conformance test
       ✅ Networking route: pmtu
       ✅ Networking route_func: local
       ✅ Networking route_func: forward
       ✅ Networking TCP: keepalive test
       ✅ Networking UDP: socket
       ✅ Networking tunnel: geneve basic test
       ❌ Networking tunnel: gre basic
       ✅ L2TP basic test
       ❌ Networking tunnel: vxlan basic
       ✅ Networking ipsec: basic netns transport
       ✅ Networking ipsec: basic netns tunnel
       ✅ audit: audit testsuite test
       ✅ httpd: mod_ssl smoke sanity
       ✅ tuned: tune-processes-through-perf
       ✅ ALSA PCM loopback test
       ✅ ALSA Control (mixer) Userspace Element test
       ✅ storage: SCSI VPD
       ✅ trace: ftrace/tracer
       🚧 ✅ CIFS Connectathon
       🚧 ✅ POSIX pjd-fstest suites
       ⏱  jvm test suite
       ⏱  Memory function: kaslr
       ⏱  LTP: openposix test suite
       ⏱  Networking vnic: ipvlan/basic
       ⏱  iotop: sanity
       ⏱  Usex - version 1.9-29
       ⏱  storage: dm/common

    Host 2:

       ⚡ Internal infrastructure issues prevented one or more tests (marked
       with ⚡⚡⚡) from running on this architecture.
       This is not the fault of the kernel that was tested.

       ✅ Boot test
       ✅ xfstests: ext4
       ✅ xfstests: xfs
       ✅ selinux-policy: serge-testsuite
       ✅ lvm thinp sanity
       ✅ storage: software RAID testing
       ✅ stress: stress-ng
       🚧 ⚡⚡⚡ IPMI driver test
       🚧 ✅ IPMItool loop stress test
       🚧 ✅ Storage blktests

  Test sources: https://github.com/CKI-project/tests-beaker
    💚 Pull requests are welcome for new tests or improvements to existing tests!

Waived tests
------------
If the test run included waived tests, they are marked with 🚧. Such tests are
executed but their results are not taken into account. Tests are waived when
their results are not reliable enough, e.g. when they're just introduced or are
being fixed.

Testing timeout
---------------
We aim to provide a report within reasonable timeframe. Tests that haven't
finished running are marked with ⏱. Reports for non-upstream kernels have
a Beaker recipe linked to next to each host.


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: ❌ FAIL: Test report for kernel 5.5.0-rc6-40f39e8.cki (arm-next)
  2020-01-16 13:19 ❌ FAIL: Test report for kernel 5.5.0-rc6-40f39e8.cki (arm-next) CKI Project
@ 2020-01-16 13:55 ` Rachel Sibley
       [not found] ` <CAKv+Gu-CEGOdTvektx_pqD=WqUXFi3YKXmj=pka5CiHc-6dCSQ@mail.gmail.com>
  1 sibling, 0 replies; 6+ messages in thread
From: Rachel Sibley @ 2020-01-16 13:55 UTC (permalink / raw)
  To: CKI Project, will, catalin.marinas, linux-arm-kernel
  Cc: Jianlin Shi, Jianwen Ji, Hangbin Liu



On 1/16/20 8:19 AM, CKI Project wrote:
> 
> Hello,
> 
> We ran automated tests on a recent commit from this kernel tree:
> 
>         Kernel repo: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
>              Commit: 40f39e8372fe - Merge branch 'for-next/core' into for-kernelci
> 
> The results of these automated tests are provided below.
> 
>      Overall result: FAILED (see details below)
>               Merge: OK
>             Compile: OK
>               Tests: FAILED
> 
> All kernel binaries, config files, and logs are available for download here:
> 
>    https://artifacts.cki-project.org/pipelines/385274
> 
> One or more kernel tests failed:
> 
>      aarch64:
>       ❌ Networking tunnel: gre basic
>       ❌ Networking tunnel: vxlan basic


Please disregard the networking failures, they are caused by an infrastructure bug
related to our test harness being too chatty when extracting the test repo, this leads
to test timeouts and aborts. We have a workaround in place which was merged this morning
and you should hopefully stop seeing these failures soon. Sorry about the false positives,
and hope the explanation makes sense.

-Rachel

> 
> We hope that these logs can help you find the problem quickly. For the full
> detail on our testing procedures, please scroll to the bottom of this message.
> 
> Please reply to this email if you have any questions about the tests that we
> ran or if you have any suggestions on how to make future tests more effective.
> 
>          ,-.   ,-.
>         ( C ) ( K )  Continuous
>          `-',-.`-'   Kernel
>            ( I )     Integration
>             `-'
> ______________________________________________________________________________
> 
> Compile testing
> ---------------
> 
> We compiled the kernel for 1 architecture:
> 
>      aarch64:
>        make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
> 
> 
> Hardware testing
> ----------------
> We booted each kernel and ran the following tests:
> 
>    aarch64:
>      Host 1:
>         ✅ Boot test
>         ✅ Podman system integration test (as root)
>         ✅ Podman system integration test (as user)
>         ✅ LTP
>         ✅ Loopdev Sanity
>         ✅ Memory function: memfd_create
>         ✅ AMTU (Abstract Machine Test Utility)
>         ✅ Networking bridge: sanity
>         ✅ Ethernet drivers sanity
>         ✅ Networking MACsec: sanity
>         ✅ Networking socket: fuzz
>         ✅ Networking sctp-auth: sockopts test
>         ✅ Networking: igmp conformance test
>         ✅ Networking route: pmtu
>         ✅ Networking route_func: local
>         ✅ Networking route_func: forward
>         ✅ Networking TCP: keepalive test
>         ✅ Networking UDP: socket
>         ✅ Networking tunnel: geneve basic test
>         ❌ Networking tunnel: gre basic
>         ✅ L2TP basic test
>         ❌ Networking tunnel: vxlan basic
>         ✅ Networking ipsec: basic netns transport
>         ✅ Networking ipsec: basic netns tunnel
>         ✅ audit: audit testsuite test
>         ✅ httpd: mod_ssl smoke sanity
>         ✅ tuned: tune-processes-through-perf
>         ✅ ALSA PCM loopback test
>         ✅ ALSA Control (mixer) Userspace Element test
>         ✅ storage: SCSI VPD
>         ✅ trace: ftrace/tracer
>         🚧 ✅ CIFS Connectathon
>         🚧 ✅ POSIX pjd-fstest suites
>         ⏱  jvm test suite
>         ⏱  Memory function: kaslr
>         ⏱  LTP: openposix test suite
>         ⏱  Networking vnic: ipvlan/basic
>         ⏱  iotop: sanity
>         ⏱  Usex - version 1.9-29
>         ⏱  storage: dm/common
> 
>      Host 2:
> 
>         ⚡ Internal infrastructure issues prevented one or more tests (marked
>         with ⚡⚡⚡) from running on this architecture.
>         This is not the fault of the kernel that was tested.
> 
>         ✅ Boot test
>         ✅ xfstests: ext4
>         ✅ xfstests: xfs
>         ✅ selinux-policy: serge-testsuite
>         ✅ lvm thinp sanity
>         ✅ storage: software RAID testing
>         ✅ stress: stress-ng
>         🚧 ⚡⚡⚡ IPMI driver test
>         🚧 ✅ IPMItool loop stress test
>         🚧 ✅ Storage blktests
> 
>    Test sources: https://github.com/CKI-project/tests-beaker
>      💚 Pull requests are welcome for new tests or improvements to existing tests!
> 
> Waived tests
> ------------
> If the test run included waived tests, they are marked with 🚧. Such tests are
> executed but their results are not taken into account. Tests are waived when
> their results are not reliable enough, e.g. when they're just introduced or are
> being fixed.
> 
> Testing timeout
> ---------------
> We aim to provide a report within reasonable timeframe. Tests that haven't
> finished running are marked with ⏱. Reports for non-upstream kernels have
> a Beaker recipe linked to next to each host.
> 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: ❌ FAIL: Test report for kernel 5.5.0-rc6-40f39e8.cki (arm-next)
       [not found] ` <CAKv+Gu-CEGOdTvektx_pqD=WqUXFi3YKXmj=pka5CiHc-6dCSQ@mail.gmail.com>
@ 2020-01-16 14:14   ` Veronika Kabatova
  2020-01-16 14:33     ` Will Deacon
       [not found]     ` <CAKv+Gu_Q5tpf1UakSHuDKq7_3BgMavpwucK6cz_k=VcDD0y8dw@mail.gmail.com>
  0 siblings, 2 replies; 6+ messages in thread
From: Veronika Kabatova @ 2020-01-16 14:14 UTC (permalink / raw)
  To: Ard Biesheuvel
  Cc: Jianwen Ji, Hangbin Liu, Catalin Marinas, Jianlin Shi,
	CKI Project, Will Deacon, linux-arm-kernel

----- Original Message ----- 

> From: "Ard Biesheuvel" <ard.biesheuvel@linaro.org>
> To: "CKI Project" <cki-project@redhat.com>
> Cc: "Jianwen Ji" <jiji@redhat.com>, "Hangbin Liu" <haliu@redhat.com>,
> "Catalin Marinas" <catalin.marinas@arm.com>, "Jianlin Shi"
> <jishi@redhat.com>, "Will Deacon" <will@kernel.org>, "linux-arm-kernel"
> <linux-arm-kernel@lists.infradead.org>
> Sent: Thursday, January 16, 2020 2:30:06 PM
> Subject: Re: ❌ FAIL: Test report for kernel 5.5.0-rc6-40f39e8.cki (arm-next)

> On Thu, 16 Jan 2020 at 14:20, CKI Project < cki-project@redhat.com > wrote:

> > Hello,
> 

> > We ran automated tests on a recent commit from this kernel tree:
> 

> > Kernel repo: git:// git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
> 
> > Commit: 40f39e8372fe - Merge branch 'for-next/core' into for-kernelci
> 

> > The results of these automated tests are provided below.
> 

> > Overall result: FAILED (see details below)
> 
> > Merge: OK
> 
> > Compile: OK
> 
> > Tests: FAILED
> 

> > All kernel binaries, config files, and logs are available for download
> > here:
> 

> > https://artifacts.cki-project.org/pipelines/385274
> 

> > One or more kernel tests failed:
> 

> > aarch64:
> 
> > ❌ Networking tunnel: gre basic
> 
> > ❌ Networking tunnel: vxlan basic
> 

> > We hope that these logs can help you find the problem quickly. For the full
> 
> > detail on our testing procedures, please scroll to the bottom of this
> > message.
> 

> > Please reply to this email if you have any questions about the tests that
> > we
> 
> > ran or if you have any suggestions on how to make future tests more
> > effective.
> 

> May I kindly suggest that you stop cc'ing the mailing list on automated,
> unmoderated CI results?

> It's great that you run a CI system, but you should really have the minimal
> courtesy to double check failure reports before sending them out into the
> world.

Hi Ard,

we are sorry about that. The testing and emails to the list were requested
by Will a while ago. If he or Catalin change their minds about them we will
stop sending them, in the meanwhile you can always set up a filter.

We are not paid by ARM so the testing is a "strive for the best" effort,
as we have a bunch of ARM machines we're willing to lend to community for
extra coverage. We always cc the test maintainers and also us to take a
look at the issues if they do occur. As Rachel mentioned, we already put
a workaround in place to prevent these issues from happening further so
hopefully you won't get more emails because of these Beaker bugs.


Thanks,
Veronika

> > ,-. ,-.
> 
> > ( C ) ( K ) Continuous
> 
> > `-',-.`-' Kernel
> 
> > ( I ) Integration
> 
> > `-'
> 
> > ______________________________________________________________________________
> 

> > Compile testing
> 
> > ---------------
> 

> > We compiled the kernel for 1 architecture:
> 

> > aarch64:
> 
> > make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
> 

> > Hardware testing
> 
> > ----------------
> 
> > We booted each kernel and ran the following tests:
> 

> > aarch64:
> 
> > Host 1:
> 
> > ✅ Boot test
> 
> > ✅ Podman system integration test (as root)
> 
> > ✅ Podman system integration test (as user)
> 
> > ✅ LTP
> 
> > ✅ Loopdev Sanity
> 
> > ✅ Memory function: memfd_create
> 
> > ✅ AMTU (Abstract Machine Test Utility)
> 
> > ✅ Networking bridge: sanity
> 
> > ✅ Ethernet drivers sanity
> 
> > ✅ Networking MACsec: sanity
> 
> > ✅ Networking socket: fuzz
> 
> > ✅ Networking sctp-auth: sockopts test
> 
> > ✅ Networking: igmp conformance test
> 
> > ✅ Networking route: pmtu
> 
> > ✅ Networking route_func: local
> 
> > ✅ Networking route_func: forward
> 
> > ✅ Networking TCP: keepalive test
> 
> > ✅ Networking UDP: socket
> 
> > ✅ Networking tunnel: geneve basic test
> 
> > ❌ Networking tunnel: gre basic
> 
> > ✅ L2TP basic test
> 
> > ❌ Networking tunnel: vxlan basic
> 
> > ✅ Networking ipsec: basic netns transport
> 
> > ✅ Networking ipsec: basic netns tunnel
> 
> > ✅ audit: audit testsuite test
> 
> > ✅ httpd: mod_ssl smoke sanity
> 
> > ✅ tuned: tune-processes-through-perf
> 
> > ✅ ALSA PCM loopback test
> 
> > ✅ ALSA Control (mixer) Userspace Element test
> 
> > ✅ storage: SCSI VPD
> 
> > ✅ trace: ftrace/tracer
> 
> > 🚧 ✅ CIFS Connectathon
> 
> > 🚧 ✅ POSIX pjd-fstest suites
> 
> > ⏱ jvm test suite
> 
> > ⏱ Memory function: kaslr
> 
> > ⏱ LTP: openposix test suite
> 
> > ⏱ Networking vnic: ipvlan/basic
> 
> > ⏱ iotop: sanity
> 
> > ⏱ Usex - version 1.9-29
> 
> > ⏱ storage: dm/common
> 

> > Host 2:
> 

> > ⚡ Internal infrastructure issues prevented one or more tests (marked
> 
> > with ⚡⚡⚡) from running on this architecture.
> 
> > This is not the fault of the kernel that was tested.
> 

> > ✅ Boot test
> 
> > ✅ xfstests: ext4
> 
> > ✅ xfstests: xfs
> 
> > ✅ selinux-policy: serge-testsuite
> 
> > ✅ lvm thinp sanity
> 
> > ✅ storage: software RAID testing
> 
> > ✅ stress: stress-ng
> 
> > 🚧 ⚡⚡⚡ IPMI driver test
> 
> > 🚧 ✅ IPMItool loop stress test
> 
> > 🚧 ✅ Storage blktests
> 

> > Test sources: https://github.com/CKI-project/tests-beaker
> 
> > 💚 Pull requests are welcome for new tests or improvements to existing
> > tests!
> 

> > Waived tests
> 
> > ------------
> 
> > If the test run included waived tests, they are marked with 🚧. Such tests
> > are
> 
> > executed but their results are not taken into account. Tests are waived
> > when
> 
> > their results are not reliable enough, e.g. when they're just introduced or
> > are
> 
> > being fixed.
> 

> > Testing timeout
> 
> > ---------------
> 
> > We aim to provide a report within reasonable timeframe. Tests that haven't
> 
> > finished running are marked with ⏱. Reports for non-upstream kernels have
> 
> > a Beaker recipe linked to next to each host.
> 

> > _______________________________________________
> 
> > linux-arm-kernel mailing list
> 
> > linux-arm-kernel@lists.infradead.org
> 
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: ❌ FAIL: Test report for kernel 5.5.0-rc6-40f39e8.cki (arm-next)
  2020-01-16 14:14   ` Veronika Kabatova
@ 2020-01-16 14:33     ` Will Deacon
       [not found]       ` <CAKv+Gu_BqEz_Q1H9CXe1ZTCkExEnAtsrdEQcPXZthsgGhWxK9A@mail.gmail.com>
       [not found]     ` <CAKv+Gu_Q5tpf1UakSHuDKq7_3BgMavpwucK6cz_k=VcDD0y8dw@mail.gmail.com>
  1 sibling, 1 reply; 6+ messages in thread
From: Will Deacon @ 2020-01-16 14:33 UTC (permalink / raw)
  To: Veronika Kabatova
  Cc: Jianwen Ji, Hangbin Liu, Ard Biesheuvel, Catalin Marinas,
	Jianlin Shi, CKI Project, linux-arm-kernel

Hi Veronika,

On Thu, Jan 16, 2020 at 09:14:49AM -0500, Veronika Kabatova wrote:
> > On Thu, 16 Jan 2020 at 14:20, CKI Project < cki-project@redhat.com > wrote:
> > > One or more kernel tests failed:
> > 
> 
> > > aarch64:
> > 
> > > ❌ Networking tunnel: gre basic
> > 
> > > ❌ Networking tunnel: vxlan basic
> > 
> > May I kindly suggest that you stop cc'ing the mailing list on automated,
> > unmoderated CI results?
> 
> > It's great that you run a CI system, but you should really have the minimal
> > courtesy to double check failure reports before sending them out into the
> > world.
> 
> we are sorry about that. The testing and emails to the list were requested
> by Will a while ago. If he or Catalin change their minds about them we will
> stop sending them, in the meanwhile you can always set up a filter.

The results are usually very useful (thanks!), and I think it's good to
have them on the list so that people can see them and reply to them, if
necessary. I don't /think/ anybody is disagreeing with that.

However, in this specific case, the results aren't very useful because the
infrastructure is playing up, so I think that's what Ard is getting at when
he says "unmoderated".

> We are not paid by ARM so the testing is a "strive for the best" effort,
> as we have a bunch of ARM machines we're willing to lend to community for
> extra coverage. We always cc the test maintainers and also us to take a
> look at the issues if they do occur. As Rachel mentioned, we already put
> a workaround in place to prevent these issues from happening further so
> hopefully you won't get more emails because of these Beaker bugs.

Thanks. If you get a chance, then it would be good to stop the test emails
going out to the mailing list while the infrastructure is known to be
broken, otherwise it can distract people into investigating failures that
aren't actually caused by changes to the kernel.

Cheers,

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: ❌ FAIL: Test report for kernel 5.5.0-rc6-40f39e8.cki (arm-next)
       [not found]     ` <CAKv+Gu_Q5tpf1UakSHuDKq7_3BgMavpwucK6cz_k=VcDD0y8dw@mail.gmail.com>
@ 2020-01-16 15:31       ` Jakub Racek
  0 siblings, 0 replies; 6+ messages in thread
From: Jakub Racek @ 2020-01-16 15:31 UTC (permalink / raw)
  To: Ard Biesheuvel
  Cc: Jianwen Ji, Veronika Kabatova, Hangbin Liu, Catalin Marinas,
	Jianlin Shi, CKI Project, Will Deacon, linux-arm-kernel

Hi Ard,
+++ Ard Biesheuvel [16/01/20 15:27 +0100]:
>4017059 Warn Running cki@gitlab:385274 5.5.0-rc6-40f39e8.cki@arm aarch64
>
>On Thu, 16 Jan 2020 at 15:14, Veronika Kabatova <vkabatov@redhat.com> wrote:
>>
>> ----- Original Message -----
>>
>> > From: "Ard Biesheuvel" <ard.biesheuvel@linaro.org>
>> > To: "CKI Project" <cki-project@redhat.com>
>> > Cc: "Jianwen Ji" <jiji@redhat.com>, "Hangbin Liu" <haliu@redhat.com>,
>> > "Catalin Marinas" <catalin.marinas@arm.com>, "Jianlin Shi"
>> > <jishi@redhat.com>, "Will Deacon" <will@kernel.org>, "linux-arm-kernel"
>> > <linux-arm-kernel@lists.infradead.org>
>> > Sent: Thursday, January 16, 2020 2:30:06 PM
>> > Subject: Re: ❌ FAIL: Test report for kernel 5.5.0-rc6-40f39e8.cki (arm-next)
>>
>> > On Thu, 16 Jan 2020 at 14:20, CKI Project < cki-project@redhat.com > wrote:
>>
>> > > Hello,
>> >
>>
>> > > We ran automated tests on a recent commit from this kernel tree:
>> >
>>
>> > > Kernel repo: git:// git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
>> >
>> > > Commit: 40f39e8372fe - Merge branch 'for-next/core' into for-kernelci
>> >
>>
>> > > The results of these automated tests are provided below.
>> >
>>
>> > > Overall result: FAILED (see details below)
>> >
>> > > Merge: OK
>> >
>> > > Compile: OK
>> >
>> > > Tests: FAILED
>> >
>>
>> > > All kernel binaries, config files, and logs are available for download
>> > > here:
>> >
>>
>> > > https://artifacts.cki-project.org/pipelines/385274
>> >
>>
>> > > One or more kernel tests failed:
>> >
>>
>> > > aarch64:
>> >
>> > > ❌ Networking tunnel: gre basic
>> >
>> > > ❌ Networking tunnel: vxlan basic
>> >
>>
>> > > We hope that these logs can help you find the problem quickly. For the full
>> >
>> > > detail on our testing procedures, please scroll to the bottom of this
>> > > message.
>> >
>>
>> > > Please reply to this email if you have any questions about the tests that
>> > > we
>> >
>> > > ran or if you have any suggestions on how to make future tests more
>> > > effective.
>> >
>>
>> > May I kindly suggest that you stop cc'ing the mailing list on automated,
>> > unmoderated CI results?
>>
>> > It's great that you run a CI system, but you should really have the minimal
>> > courtesy to double check failure reports before sending them out into the
>> > world.
>>
>> Hi Ard,
>>
>> we are sorry about that. The testing and emails to the list were requested
>> by Will a while ago. If he or Catalin change their minds about them we will
>> stop sending them, in the meanwhile you can always set up a filter.
>>
>
>Fair enough. I wasn't aware that they explicitly asked you to cc the
>mailing list on these emails.
>
>> We are not paid by ARM so the testing is a "strive for the best" effort,
>> as we have a bunch of ARM machines we're willing to lend to community for
>> extra coverage. We always cc the test maintainers and also us to take a
>> look at the issues if they do occur. As Rachel mentioned, we already put
>> a workaround in place to prevent these issues from happening further so
>> hopefully you won't get more emails because of these Beaker bugs.
>>
>
>I do appreciate the testing effort, it just seems to be a waste of not
>just your time to send out half baked false positive reports to such a
>wide audience.

As mentioned above, we were asked to send these reports, because we've caught
bugs in the kernel before. I honestly believe we can do a lot of good,
especially if we continue working with the community and opensourcing
our work for everyone's benefit.

I don't recall any complaints for quite some time, so I don't think it's fair
to call it half-baked because a false positive was sent.
I think that just sways hearts and minds of people in the wrong way and 
prevents cooperation. Does that make sense?

Sorry, but as one of the people working on the project, I feel the
need to step out as well and make a comment.

Best regards,
Jakub
>
>


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: ❌ FAIL: Test report for kernel 5.5.0-rc6-40f39e8.cki (arm-next)
       [not found]       ` <CAKv+Gu_BqEz_Q1H9CXe1ZTCkExEnAtsrdEQcPXZthsgGhWxK9A@mail.gmail.com>
@ 2020-01-16 16:26         ` Veronika Kabatova
  0 siblings, 0 replies; 6+ messages in thread
From: Veronika Kabatova @ 2020-01-16 16:26 UTC (permalink / raw)
  To: Ard Biesheuvel, Will Deacon
  Cc: Jianwen Ji, Hangbin Liu, Catalin Marinas, Jianlin Shi,
	CKI Project, linux-arm-kernel



----- Original Message -----
> From: "Ard Biesheuvel" <ard.biesheuvel@linaro.org>
> To: "Will Deacon" <will@kernel.org>
> Cc: "Jianwen Ji" <jiji@redhat.com>, "Veronika Kabatova" <vkabatov@redhat.com>, "Hangbin Liu" <haliu@redhat.com>,
> "Catalin Marinas" <catalin.marinas@arm.com>, "Jianlin Shi" <jishi@redhat.com>, "CKI Project"
> <cki-project@redhat.com>, "linux-arm-kernel" <linux-arm-kernel@lists.infradead.org>
> Sent: Thursday, January 16, 2020 3:56:05 PM
> Subject: Re: ❌ FAIL: Test report for kernel 5.5.0-rc6-40f39e8.cki (arm-next)
> 
> On Thu, 16 Jan 2020 at 15:33, Will Deacon <will@kernel.org> wrote:
> >
> > Hi Veronika,
> >
> > On Thu, Jan 16, 2020 at 09:14:49AM -0500, Veronika Kabatova wrote:
> > > > On Thu, 16 Jan 2020 at 14:20, CKI Project < cki-project@redhat.com >
> > > > wrote:
> > > > > One or more kernel tests failed:
> > > >
> > >
> > > > > aarch64:
> > > >
> > > > > ❌ Networking tunnel: gre basic
> > > >
> > > > > ❌ Networking tunnel: vxlan basic
> > > >
> > > > May I kindly suggest that you stop cc'ing the mailing list on
> > > > automated,
> > > > unmoderated CI results?
> > >
> > > > It's great that you run a CI system, but you should really have the
> > > > minimal
> > > > courtesy to double check failure reports before sending them out into
> > > > the
> > > > world.
> > >
> > > we are sorry about that. The testing and emails to the list were
> > > requested
> > > by Will a while ago. If he or Catalin change their minds about them we
> > > will
> > > stop sending them, in the meanwhile you can always set up a filter.
> >
> > The results are usually very useful (thanks!), and I think it's good to
> > have them on the list so that people can see them and reply to them, if
> > necessary. I don't /think/ anybody is disagreeing with that.
> >

We're glad to hear that!

> > However, in this specific case, the results aren't very useful because the
> > infrastructure is playing up, so I think that's what Ard is getting at when
> > he says "unmoderated".
> >
> 
> Apologies for the kneejerk reaction, but there are quite a number of
> bots helpfully cc'ing the world on their reports these days, and the
> fact that these particular ones are known to be false positives
> triggered my unfiltered response.
> 
> 

No worries!

We're working with KernelCI to streamline the upstream testing efforts.
As everything is moving forward, you should see less bots on the list
(at least from people who are also working with the project).

These particular problems we ran into were harder to debug without extra
information (and not caused by our code but a project we depend on) so
we weren't able to filter them out automatically.

> 
> > > We are not paid by ARM so the testing is a "strive for the best" effort,
> > > as we have a bunch of ARM machines we're willing to lend to community for
> > > extra coverage. We always cc the test maintainers and also us to take a
> > > look at the issues if they do occur. As Rachel mentioned, we already put
> > > a workaround in place to prevent these issues from happening further so
> > > hopefully you won't get more emails because of these Beaker bugs.
> >
> > Thanks. If you get a chance, then it would be good to stop the test emails
> > going out to the mailing list while the infrastructure is known to be
> > broken, otherwise it can distract people into investigating failures that
> > aren't actually caused by changes to the kernel.
> >

I agree.

We do have a few things in mind to reduce the false positives (which would
also filter out these ones) and pipeline stabilization in general. This
includes a switch to stop the automatic reports when we detect a reoccurring
infra problem and can't easily work around it.


Thanks,
Veronika

> > Cheers,
> >
> > Will
> 
> 
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2020-01-16 16:26 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-01-16 13:19 ❌ FAIL: Test report for kernel 5.5.0-rc6-40f39e8.cki (arm-next) CKI Project
2020-01-16 13:55 ` Rachel Sibley
     [not found] ` <CAKv+Gu-CEGOdTvektx_pqD=WqUXFi3YKXmj=pka5CiHc-6dCSQ@mail.gmail.com>
2020-01-16 14:14   ` Veronika Kabatova
2020-01-16 14:33     ` Will Deacon
     [not found]       ` <CAKv+Gu_BqEz_Q1H9CXe1ZTCkExEnAtsrdEQcPXZthsgGhWxK9A@mail.gmail.com>
2020-01-16 16:26         ` Veronika Kabatova
     [not found]     ` <CAKv+Gu_Q5tpf1UakSHuDKq7_3BgMavpwucK6cz_k=VcDD0y8dw@mail.gmail.com>
2020-01-16 15:31       ` Jakub Racek

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).