Re: 💥 PANICKED: Test report for?kernel 5.9.0-rc3-020ad03.cki (block)

[Ming Lei <ming.lei@redhat.com>]

Hi Veronika,

On Fri, Sep 04, 2020 at 07:06:25AM -0400, Veronika Kabatova wrote:
> 
> 
> ----- Original Message -----
> > From: "Ming Lei" <ming.lei@redhat.com>
> > To: "CKI Project" <cki-project@redhat.com>
> > Cc: linux-block@vger.kernel.org, axboe@kernel.dk, "Changhui Zhong" <czhong@redhat.com>
> > Sent: Friday, September 4, 2020 3:02:33 AM
> > Subject: Re: 💥 PANICKED: Test report for	kernel 5.9.0-rc3-020ad03.cki (block)
> > 
> > On Thu, Sep 03, 2020 at 05:07:57PM -0000, CKI Project wrote:
> > > 
> > > Hello,
> > > 
> > > We ran automated tests on a recent commit from this kernel tree:
> > > 
> > >        Kernel repo:
> > >        https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git
> > >             Commit: 020ad0333b03 - Merge branch 'for-5.10/block' into
> > >             for-next
> > > 
> > > The results of these automated tests are provided below.
> > > 
> > >     Overall result: FAILED (see details below)
> > >              Merge: OK
> > >            Compile: OK
> > >              Tests: PANICKED
> > > 
> > > All kernel binaries, config files, and logs are available for download
> > > here:
> > > 
> > >   https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/09/02/613166
> > > 
> > > One or more kernel tests failed:
> > > 
> > >     ppc64le:
> > >      💥 storage: software RAID testing
> > > 
> > >     aarch64:
> > >      💥 storage: software RAID testing
> > > 
> > >     x86_64:
> > >      💥 storage: software RAID testing
> > > 
> > > We hope that these logs can help you find the problem quickly. For the full
> > > detail on our testing procedures, please scroll to the bottom of this
> > > message.
> > > 
> > > Please reply to this email if you have any questions about the tests that
> > > we
> > > ran or if you have any suggestions on how to make future tests more
> > > effective.
> > > 
> > >         ,-.   ,-.
> > >        ( C ) ( K )  Continuous
> > >         `-',-.`-'   Kernel
> > >           ( I )     Integration
> > >            `-'
> > > ______________________________________________________________________________
> > > 
> > > Compile testing
> > > ---------------
> > > 
> > > We compiled the kernel for 4 architectures:
> > > 
> > >     aarch64:
> > >       make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg
> > > 
> > >     ppc64le:
> > >       make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg
> > > 
> > >     s390x:
> > >       make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg
> > > 
> > >     x86_64:
> > >       make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg
> > > 
> > > 
> > > 
> > > Hardware testing
> > > ----------------
> > > We booted each kernel and ran the following tests:
> > > 
> > >   aarch64:
> > >     Host 1:
> > >        ✅ Boot test
> > >        ✅ ACPI table test
> > >        ✅ LTP
> > >        ✅ Loopdev Sanity
> > >        ✅ Memory function: memfd_create
> > >        ✅ AMTU (Abstract Machine Test Utility)
> > >        ✅ Ethernet drivers sanity
> > >        ✅ storage: SCSI VPD
> > >        🚧 ✅ CIFS Connectathon
> > >        🚧 ✅ POSIX pjd-fstest suites
> > > 
> > >     Host 2:
> > > 
> > >        ⚡ Internal infrastructure issues prevented one or more tests (marked
> > >        with ⚡⚡⚡) from running on this architecture.
> > >        This is not the fault of the kernel that was tested.
> > > 
> > >        ⚡⚡⚡ Boot test
> > >        ⚡⚡⚡ xfstests - ext4
> > >        ⚡⚡⚡ xfstests - xfs
> > >        ⚡⚡⚡ storage: software RAID testing
> > >        ⚡⚡⚡ stress: stress-ng
> > >        🚧 ⚡⚡⚡ xfstests - btrfs
> > >        🚧 ⚡⚡⚡ Storage blktests
> > > 
> > >     Host 3:
> > >        ✅ Boot test
> > >        ✅ xfstests - ext4
> > >        ✅ xfstests - xfs
> > >        💥 storage: software RAID testing
> > >        ⚡⚡⚡ stress: stress-ng
> > >        🚧 ⚡⚡⚡ xfstests - btrfs
> > >        🚧 ⚡⚡⚡ Storage blktests
> > > 
> > >   ppc64le:
> > >     Host 1:
> > >        ✅ Boot test
> > >        🚧 ✅ kdump - sysrq-c
> > > 
> > >     Host 2:
> > >        ✅ Boot test
> > >        ✅ xfstests - ext4
> > >        ✅ xfstests - xfs
> > >        💥 storage: software RAID testing
> > >        🚧 ⚡⚡⚡ xfstests - btrfs
> > >        🚧 ⚡⚡⚡ Storage blktests
> > > 
> > >     Host 3:
> > > 
> > >        ⚡ Internal infrastructure issues prevented one or more tests (marked
> > >        with ⚡⚡⚡) from running on this architecture.
> > >        This is not the fault of the kernel that was tested.
> > > 
> > >        ✅ Boot test
> > >        ⚡⚡⚡ LTP
> > >        ⚡⚡⚡ Loopdev Sanity
> > >        ⚡⚡⚡ Memory function: memfd_create
> > >        ⚡⚡⚡ AMTU (Abstract Machine Test Utility)
> > >        ⚡⚡⚡ Ethernet drivers sanity
> > >        🚧 ⚡⚡⚡ CIFS Connectathon
> > >        🚧 ⚡⚡⚡ POSIX pjd-fstest suites
> > > 
> > >   s390x:
> > >     Host 1:
> > >        ✅ Boot test
> > >        ✅ stress: stress-ng
> > >        🚧 ✅ Storage blktests
> > > 
> > >     Host 2:
> > >        ✅ Boot test
> > >        ✅ LTP
> > >        ✅ Loopdev Sanity
> > >        ✅ Memory function: memfd_create
> > >        ✅ AMTU (Abstract Machine Test Utility)
> > >        ✅ Ethernet drivers sanity
> > >        🚧 ✅ CIFS Connectathon
> > >        🚧 ✅ POSIX pjd-fstest suites
> > > 
> > >   x86_64:
> > >     Host 1:
> > >        ✅ Boot test
> > >        ✅ Storage SAN device stress - qedf driver
> > > 
> > >     Host 2:
> > >        ⏱  Boot test
> > >        ⏱  Storage SAN device stress - mpt3sas_gen1
> > > 
> > >     Host 3:
> > >        ✅ Boot test
> > >        ✅ xfstests - ext4
> > >        ✅ xfstests - xfs
> > >        💥 storage: software RAID testing
> > >        ⚡⚡⚡ stress: stress-ng
> > >        🚧 ⚡⚡⚡ xfstests - btrfs
> > >        🚧 ⚡⚡⚡ Storage blktests
> > > 
> > >     Host 4:
> > >        ✅ Boot test
> > >        ✅ Storage SAN device stress - lpfc driver
> > > 
> > >     Host 5:
> > >        ✅ Boot test
> > >        🚧 ✅ kdump - sysrq-c
> > > 
> > >     Host 6:
> > >        ✅ Boot test
> > >        ✅ ACPI table test
> > >        ✅ LTP
> > >        ✅ Loopdev Sanity
> > >        ✅ Memory function: memfd_create
> > >        ✅ AMTU (Abstract Machine Test Utility)
> > >        ✅ Ethernet drivers sanity
> > >        ✅ kernel-rt: rt_migrate_test
> > >        ✅ kernel-rt: rteval
> > >        ✅ kernel-rt: sched_deadline
> > >        ✅ kernel-rt: smidetect
> > >        ✅ storage: SCSI VPD
> > >        🚧 ✅ CIFS Connectathon
> > >        🚧 ✅ POSIX pjd-fstest suites
> > > 
> > >     Host 7:
> > >        ✅ Boot test
> > >        ✅ kdump - sysrq-c - megaraid_sas
> > > 
> > >     Host 8:
> > >        ✅ Boot test
> > >        ✅ Storage SAN device stress - qla2xxx driver
> > > 
> > >     Host 9:
> > >        ⏱  Boot test
> > >        ⏱  kdump - sysrq-c - mpt3sas_gen1
> > > 
> > >   Test sources: https://gitlab.com/cki-project/kernel-tests
> > 
> > Hello,
> > 
> 
> Hi Ming,
> 
> first the good news: Both issues detected by LTP and RAID test are
> officially gone after the revert. There's some x86_64 testing still
> running but the results look good so far!
> 
> > Can you share us the exact commands for setting up xfstests over
> > 'software RAID testing' from the above tree?
> > 
> 
> It's this test (which seeing your @redhat email, you can also trigger
> via internal Brew testing if you use the "stor" test set):
> 
> https://gitlab.com/cki-project/kernel-tests/-/tree/master/storage/swraid/trim
> 
> The important part of the test is:
> 
> https://gitlab.com/cki-project/kernel-tests/-/blob/master/storage/swraid/trim/main.sh#L27
> 
> The test maintainer (Changhui) is cced on this thread in case you need
> any help or have questions about the test.
> 
> 
> 
> I'll just quickly mention, please be careful if you're planning on
> testing LTP/msgstress04 on ppc64le in Beaker, as the conserver overload
> is causing issues to lab owners.
> 
> 
> Let us know if we can help you with something else,

I have verified the revised patches does fix kernel oops in 'software
RAID storage test'. However, I can't reproduce the OOM in LTP/msgstress04.

Could you help to check if LTP/msgstress04 can pass with the following
tree(top three patches) which is against the latest for-5.10/block:

	https://github.com/ming1/linux/commits/v5.9-rc-block-test

Thanks,
Ming