Re: ❌ FAIL (SKIPPED 191 of 328): Test report for master (6.5.0-rc3, mainline.kernel.org-clang, 57012c57)

Re: ❌ FAIL (SKIPPED 191 of 328): Test report for master (6.5.0-rc3, mainline.kernel.org-clang, 57012c57)

[Baoquan He]

Hi CKI team,

On 07/28/23 at 04:40am, cki-project@redhat.com wrote:
> Hi, we tested your kernel and here are the results:
> 
>     Overall result: FAILED
>              Merge: OK
>            Compile: OK
>               Test: FAILED
> 
> 
> Kernel information:
>     Commit message: Merge tag 'net-6.5-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
> 
> You can find all the details about the test run at
>     https://datawarehouse.cki-project.org/kcidb/checkouts/97648
> 
> One or more kernel tests failed:
>     Unrecognized or new issues:
>           x86_64 - kdump - sysrq-c
>                    Logs: https://datawarehouse.cki-project.org/kcidb/tests/8928580
>                    Non-passing subtests:
>                        ❌ FAIL /kdump/crash-sysrq-c
>           x86_64 - kdump - file-load
>                    Logs: https://datawarehouse.cki-project.org/kcidb/tests/8928581
>                    Non-passing subtests:
>                        ❌ FAIL /kdump/crash-sysrq-c
>           x86_64 (debug) - Boot test
>                    Logs: https://datawarehouse.cki-project.org/kcidb/tests/8928644
>                    Non-passing subtests:
>                        ❌ FAIL distribution/kpkginstall/journalctl-check

All these failure cases have empty console.log, there's no way to check
what's going on there. The empty console.log is related to 1st kernel,
there must something wrong withih it.

Thanks
Baoquan

Re: ❌ FAIL (SKIPPED 191 of 328): Test report for master (6.5.0-rc3, mainline.kernel.org-clang, 57012c57)

[Bruno Goncalves]

On Fri, 28 Jul 2023 at 06:57, Baoquan He <bhe@redhat.com> wrote:
>
> Hi CKI team,
>
> On 07/28/23 at 04:40am, cki-project@redhat.com wrote:
> > Hi, we tested your kernel and here are the results:
> >
> >     Overall result: FAILED
> >              Merge: OK
> >            Compile: OK
> >               Test: FAILED
> >
> >
> > Kernel information:
> >     Commit message: Merge tag 'net-6.5-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
> >
> > You can find all the details about the test run at
> >     https://datawarehouse.cki-project.org/kcidb/checkouts/97648
> >
> > One or more kernel tests failed:
> >     Unrecognized or new issues:
> >           x86_64 - kdump - sysrq-c
> >                    Logs: https://datawarehouse.cki-project.org/kcidb/tests/8928580
> >                    Non-passing subtests:
> >                        ❌ FAIL /kdump/crash-sysrq-c
> >           x86_64 - kdump - file-load
> >                    Logs: https://datawarehouse.cki-project.org/kcidb/tests/8928581
> >                    Non-passing subtests:
> >                        ❌ FAIL /kdump/crash-sysrq-c
> >           x86_64 (debug) - Boot test
> >                    Logs: https://datawarehouse.cki-project.org/kcidb/tests/8928644
> >                    Non-passing subtests:
> >                        ❌ FAIL distribution/kpkginstall/journalctl-check
>
> All these failure cases have empty console.log, there's no way to check
> what's going on there. The empty console.log is related to 1st kernel,
> there must something wrong withih it.

Thanks, yes it is a pain. Sometimes the serial console in the machines
don't work at all, that's why the console.log is empty.
I wonder if the kdump tests would upload journalctl logs, we would be
able to use it to check for failures instead of relying only on
console.log.

Bruno
>
> Thanks
> Baoquan
>

Re: ❌ FAIL (SKIPPED 191 of 328): Test report for master (6.5.0-rc3, mainline.kernel.org-clang, 57012c57)

[Baoquan He]

On 07/28/23 at 08:54am, Bruno Goncalves wrote:
> On Fri, 28 Jul 2023 at 06:57, Baoquan He <bhe@redhat.com> wrote:
> >
> > Hi CKI team,
> >
> > On 07/28/23 at 04:40am, cki-project@redhat.com wrote:
> > > Hi, we tested your kernel and here are the results:
> > >
> > >     Overall result: FAILED
> > >              Merge: OK
> > >            Compile: OK
> > >               Test: FAILED
> > >
> > >
> > > Kernel information:
> > >     Commit message: Merge tag 'net-6.5-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net
> > >
> > > You can find all the details about the test run at
> > >     https://datawarehouse.cki-project.org/kcidb/checkouts/97648
> > >
> > > One or more kernel tests failed:
> > >     Unrecognized or new issues:
> > >           x86_64 - kdump - sysrq-c
> > >                    Logs: https://datawarehouse.cki-project.org/kcidb/tests/8928580
> > >                    Non-passing subtests:
> > >                        ❌ FAIL /kdump/crash-sysrq-c
> > >           x86_64 - kdump - file-load
> > >                    Logs: https://datawarehouse.cki-project.org/kcidb/tests/8928581
> > >                    Non-passing subtests:
> > >                        ❌ FAIL /kdump/crash-sysrq-c
> > >           x86_64 (debug) - Boot test
> > >                    Logs: https://datawarehouse.cki-project.org/kcidb/tests/8928644
> > >                    Non-passing subtests:
> > >                        ❌ FAIL distribution/kpkginstall/journalctl-check
> >
> > All these failure cases have empty console.log, there's no way to check
> > what's going on there. The empty console.log is related to 1st kernel,
> > there must something wrong withih it.
> 
> Thanks, yes it is a pain. Sometimes the serial console in the machines
> don't work at all, that's why the console.log is empty.
> I wonder if the kdump tests would upload journalctl logs, we would be
> able to use it to check for failures instead of relying only on
> console.log.

Thanks, Bruno.

It could be a little helpful to have journalctl logs. However, it may
not help much in kdump case. When I check these kdump failure case of CKI, I
usually check test_console.log firstly. If test_console.log is
incomplete, I will check console.log because console.log contains all
printed log on console.

As for journalctrl logs, it can only get the boot log of 1st kernel. As
for kdump boot log, e.g kdump boot could hang in early stage or the
middle of bootup, the journalctl log doesn't make any sense.

So for kdump, console log is very important.

Thanks
Baoquan

❌ FAIL (SKIPPED 191 of 328): Test report for master (6.5.0-rc3, mainline.kernel.org-clang, 57012c57)

[cki-project]

Hi, we tested your kernel and here are the results:

    Overall result: FAILED
             Merge: OK
           Compile: OK
              Test: FAILED


Kernel information:
    Commit message: Merge tag 'net-6.5-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net

You can find all the details about the test run at
    https://datawarehouse.cki-project.org/kcidb/checkouts/97648

One or more kernel tests failed:
    Unrecognized or new issues:
          x86_64 - kdump - sysrq-c
                   Logs: https://datawarehouse.cki-project.org/kcidb/tests/8928580
                   Non-passing subtests:
                       ❌ FAIL /kdump/crash-sysrq-c
          x86_64 - kdump - file-load
                   Logs: https://datawarehouse.cki-project.org/kcidb/tests/8928581
                   Non-passing subtests:
                       ❌ FAIL /kdump/crash-sysrq-c
          x86_64 (debug) - Boot test
                   Logs: https://datawarehouse.cki-project.org/kcidb/tests/8928644
                   Non-passing subtests:
                       ❌ FAIL distribution/kpkginstall/journalctl-check

    We also see the following known issues which are not related to your changes:
        Issue: kdump - kexec_boot: image_arm64_probe: Bad arm64 image header
            URL: https://issues.redhat.com/browse/RHELPLAN-138156
            Affected tests:
                 aarch64 - kdump - kexec_boot
                 aarch64 - kdump - kexec-file-load
        Issue: Firmware test suite: mv: cannot stat 'dtcompilerparser.tab.c': No such file or directory
            URL: https://gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/issues/1544
            Affected tests:
                 aarch64 - Hardware - Firmware test suite
        Issue: xfstests generic/173 failing on ark-eln aarch64
            URL: https://bugzilla.redhat.com/show_bug.cgi?id=1989409
            Affected tests:
                 aarch64 - xfstests - btrfs



If you find a failure unrelated to your changes, please ask the test maintainer to review it.
This will prevent the failures from being incorrectly reported in the future.

Please reply to this email if you have any questions about the tests that we
ran or if you have any suggestions on how to make future tests more effective.

        ,-.   ,-.
       ( C ) ( K )  Continuous
        `-',-.`-'   Kernel
          ( I )     Integration
           `-'
______________________________________________________________________________