Re: 💥 PANICKED: Test report for kernel 5.8.0-rc2-c698ae9.cki (block)

[Rachel Sibley <rasibley@redhat.com>]

Hi, we're seeing multiple panics across all arches, I included a snippet of the call trace for both
xfstests and boot test.

You should be able to inspect in more detail by viewing the console.log under each build/tests directory:
https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/06/30/609250

Thanks,
Rachel

On 7/1/20 9:06 AM, CKI Project wrote:
> 
> Hello,
> 
> We ran automated tests on a recent commit from this kernel tree:
> 
>         Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git
>              Commit: c698ae90fb5e - Merge branch 'for-5.9/block' into for-next
> 
> The results of these automated tests are provided below.
> 
>      Overall result: FAILED (see details below)
>               Merge: OK
>             Compile: OK
>               Tests: PANICKED
> 
> All kernel binaries, config files, and logs are available for download here:
> 
>    https://cki-artifacts.s3.us-east-2.amazonaws.com/index.html?prefix=datawarehouse/2020/06/30/609250
> 
> One or more kernel tests failed:
> 
>      s390x:
>       ❌ Boot test
>       ❌ Boot test
>       ❌ Boot test
> 
>      ppc64le:
>       ❌ Boot test
>       ❌ Boot test
>       💥 xfstests - ext4

https://cki-artifacts.s3.us-east-2.amazonaws.com/datawarehouse/2020/06/30/609250/build_ppc64le_redhat%3A926155/tests/8501352/ppc64le_3_console.log

[  890.198174] run fstests generic/040 at 2020-06-30 12:03:02
[  891.055910] BUG: Unable to handle kernel instruction fetch (NULL pointer?)
[  891.055942] Faulting instruction address: 0x00000000
[  891.055956] Oops: Kernel access of bad area, sig: 11 [#1]
[  891.055969] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA PowerNV
[  891.055982] Modules linked in: dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rfkill joydev i40e at24 sunrpc ses 
enclosure scsi_transport_sas regmap_i2c ofpart powernv_flash mtd crct10dif_vpmsum ipmi_powernv ipmi_devintf opal_prd ipmi_msghandler rtc_opal i2c_opal 
ip_tables xfs libcrc32c ast i2c_algo_bit drm_vram_helper drm_kms_helper syscopyarea vmx_crypto sysfillrect sysimgblt crc32c_vpmsum fb_sys_fops 
drm_ttm_helper ttm drm i2c_core aacraid drm_panel_orientation_quirks
[  891.056077] CPU: 25 PID: 84211 Comm: systemd-udevd Kdump: loaded Not tainted 5.8.0-rc2-c698ae9.cki #1
[  891.056095] NIP:  0000000000000000 LR: c00000000070eef0 CTR: 0000000000000000
[  891.056110] REGS: c0000007c25474e0 TRAP: 0400   Not tainted  (5.8.0-rc2-c698ae9.cki)
[  891.056125] MSR:  9000000040009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 24488248  XER: 20040000
[  891.056145] CFAR: c00000000070eeec IRQMASK: 0
[  891.056145] GPR00: c00000000070f050 c0000007c2547770 c000000001cb7f00 c0000002b6059af8
[  891.056145] GPR04: 0000000000000000 c0000007dcf6f000 c0000002b6059af8 0000000000000000
[  891.056145] GPR08: 00000007fc940000 c000000001c97d78 0000000000000000 0000000000000000
[  891.056145] GPR12: 0000000000000000 c0000007fffe2e00 c0000007c2344400 0000000000000000
[  891.056145] GPR16: 0000000000000000 00007fffc9b7cb50 c0000007c2344400 0000000000000000
[  891.056145] GPR20: c0000002b307bdd8 0000000000000000 c0000007c2547ca8 c0000007dcf6f000
[  891.056145] GPR24: 000000000000000c 000000000000000a c0000007c2547790 0000000000000001
[  891.056145] GPR28: 0000000000000000 0000000000000000 00000000ffffffff c0000002b6059af8
[  891.056260] NIP [0000000000000000] 0x0
[  891.056272] LR [c00000000070eef0] submit_bio_noacct+0x2f0/0x5c0
[  891.056285] Call Trace:
[  891.056294] [c0000007c2547770] [c00000000070f050] submit_bio_noacct+0x450/0x5c0 (unreliable)
[  891.056312] [c0000007c2547800] [c00000000070f228] submit_bio+0x68/0x2d0
[  891.056328] [c0000007c25478c0] [c000000000505fe8] mpage_readahead+0x1c8/0x290
[  891.056345] [c0000007c25479a0] [c0000000004fd6f8] blkdev_readahead+0x28/0x40
[  891.056362] [c0000007c25479c0] [c000000000383980] read_pages+0xb0/0x4a0
[  891.056376] [c0000007c2547a40] [c000000000384474] page_cache_readahead_unbounded+0x244/0x300
[  891.056395] [c0000007c2547b00] [c00000000037445c] generic_file_buffered_read+0x9bc/0x1120
[  891.056411] [c0000007c2547c50] [c0000000004fddc0] blkdev_read_iter+0x50/0x80
[  891.056428] [c0000007c2547c70] [c000000000493c64] new_sync_read+0x124/0x1a0
[  891.056443] [c0000007c2547d10] [c000000000496e30] vfs_read+0x100/0x200
[  891.056471] [c0000007c2547d70] [c000000000497368] ksys_read+0x78/0x130
[  891.056487] [c0000007c2547dc0] [c000000000030564] system_call_exception+0xe4/0x170
[  891.056504] [c0000007c2547e20] [c00000000000ca70] system_call_common+0xf0/0x278
[  891.056518] Instruction dump:
[  891.056529] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[  891.056545] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[  891.056564] ---[ end trace 14197a45ec121b51 ]---
> 
>      aarch64:
>       💥 Boot test
>       💥 xfstests - ext4
> 
>      x86_64:
>       💥 Boot test
>       💥 xfstests - ext4
>       💥 Boot test
>       💥 Boot test
>       💥 Boot test

https://cki-artifacts.s3.us-east-2.amazonaws.com/datawarehouse/2020/06/30/609250/build_x86_64_redhat%3A926153/tests/8501344/x86_64_6_console.log

[   12.160102] Call Trace:
[   12.162838]  submit_bio_noacct+0x1f4/0x3d0
[   12.167414]  mpage_readahead+0x159/0x1b0
[   12.171795]  ? __blkdev_direct_IO_simple+0x2b0/0x2b0
[   12.177337]  read_pages+0x5d/0x300
[   12.181132]  page_cache_readahead_unbounded+0x1aa/0x230
[   12.186965]  force_page_cache_readahead+0xda/0x140
[   12.192313]  generic_file_buffered_read+0x647/0xc00
[   12.197761]  new_sync_read+0x102/0x180
[   12.201946]  vfs_read+0x9d/0x150
[   12.205549]  ksys_read+0x4f/0xc0
[   12.209153]  do_syscall_64+0x4d/0x90
[   12.213145]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   12.218781] RIP: 0033:0x7f263ae70542
[   12.222767] Code: Bad RIP value.
[   12.226367] RSP: 002b:00007fff55c02f38 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[   12.234817] RAX: ffffffffffffffda RBX: 0000557dff1219e8 RCX: 00007f263ae70542
[   12.242780] RDX: 0000000000000040 RSI: 0000557dff1219f8 RDI: 0000000000000006
[   12.250742] RBP: 0000557dff1170c0 R08: 0000557dff1219d0 R09: 00007f263af41a40
[   12.258705] R10: 0000000000000010 R11: 0000000000000246 R12: 00000003bfff0000
[   12.266668] R13: 0000000000000040 R14: 0000557dff1219d0 R15: 0000557dff117110
[   12.274631] Modules linked in: mgag200 drm_vram_helper drm_kms_helper lpfc drm_ttm_helper ttm crct10dif_pclmul crc32_pclmul crc32c_intel nvmet_fc 
drm nvmet ghash_clmulni_intel nvme_fc nvme_fabrics i2c_algo_bit nvme_core scsi_transport_fc wmi scsi_dh_alua scsi_dh_rdac scsi_dh_emc
[   12.302196] CR2: 0000000000000000
[   12.305931] ---[ end trace 4b2a7525c30bbc3f ]---

> 
> We hope that these logs can help you find the problem quickly. For the full
> detail on our testing procedures, please scroll to the bottom of this message.
> 
> Please reply to this email if you have any questions about the tests that we
> ran or if you have any suggestions on how to make future tests more effective.
> 
>          ,-.   ,-.
>         ( C ) ( K )  Continuous
>          `-',-.`-'   Kernel
>            ( I )     Integration
>             `-'
> ______________________________________________________________________________
> 
> Compile testing
> ---------------
> 
> We compiled the kernel for 4 architectures:
> 
>      aarch64:
>        make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
> 
>      ppc64le:
>        make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
> 
>      s390x:
>        make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
> 
>      x86_64:
>        make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
> 
> 
> 
> Hardware testing
> ----------------
> We booted each kernel and ran the following tests:
> 
>    aarch64:
>      Host 1:
>         ❌ Boot test
>         ⚡⚡⚡ LTP
>         ⚡⚡⚡ Loopdev Sanity
>         ⚡⚡⚡ Memory function: memfd_create
>         ⚡⚡⚡ AMTU (Abstract Machine Test Utility)
>         ⚡⚡⚡ Ethernet drivers sanity
>         ⚡⚡⚡ storage: SCSI VPD
>         🚧 ⚡⚡⚡ CIFS Connectathon
>         🚧 ⚡⚡⚡ POSIX pjd-fstest suites
> 
>      Host 2:
>         ✅ Boot test
>         💥 xfstests - ext4
>         ⚡⚡⚡ xfstests - xfs
>         ⚡⚡⚡ storage: software RAID testing
>         ⚡⚡⚡ stress: stress-ng
>         🚧 ⚡⚡⚡ Storage blktests
> 
>    ppc64le:
>      Host 1:
>         ❌ Boot test
>         🚧 ⚡⚡⚡ kdump - sysrq-c
> 
>      Host 2:
>         ❌ Boot test
>         ⚡⚡⚡ LTP
>         ⚡⚡⚡ Loopdev Sanity
>         ⚡⚡⚡ Memory function: memfd_create
>         ⚡⚡⚡ AMTU (Abstract Machine Test Utility)
>         ⚡⚡⚡ Ethernet drivers sanity
>         🚧 ⚡⚡⚡ CIFS Connectathon
>         🚧 ⚡⚡⚡ POSIX pjd-fstest suites
> 
>      Host 3:
>         ✅ Boot test
>         💥 xfstests - ext4
>         ⚡⚡⚡ xfstests - xfs
>         ⚡⚡⚡ storage: software RAID testing
>         🚧 ⚡⚡⚡ Storage blktests
> 
>    s390x:
>      Host 1:
>         ❌ Boot test
>         ⚡⚡⚡ LTP
>         ⚡⚡⚡ Loopdev Sanity
>         ⚡⚡⚡ Memory function: memfd_create
>         ⚡⚡⚡ Ethernet drivers sanity
>         🚧 ⚡⚡⚡ CIFS Connectathon
>         🚧 ⚡⚡⚡ POSIX pjd-fstest suites
> 
>      Host 2:
>         ❌ Boot test
>         ⚡⚡⚡ stress: stress-ng
>         🚧 ⚡⚡⚡ Storage blktests
> 
>      Host 3:
>         ❌ Boot test
>         🚧 ⚡⚡⚡ kdump - sysrq-c
> 
>    x86_64:
>      Host 1:
> 
>         ⚡ Internal infrastructure issues prevented one or more tests (marked
>         with ⚡⚡⚡) from running on this architecture.
>         This is not the fault of the kernel that was tested.
> 
>         ⚡⚡⚡ Boot test
>         🚧 ⚡⚡⚡ kdump - sysrq-c - mpt3sas_gen1
> 
>      Host 2:
>         💥 Boot test
>         ⚡⚡⚡ Storage SAN device stress - lpfc driver
> 
>      Host 3:
> 
>         ⚡ Internal infrastructure issues prevented one or more tests (marked
>         with ⚡⚡⚡) from running on this architecture.
>         This is not the fault of the kernel that was tested.
> 
>         ⚡⚡⚡ Boot test
>         🚧 ⚡⚡⚡ kdump - sysrq-c
> 
>      Host 4:
>         ✅ Boot test
>         💥 xfstests - ext4
>         ⚡⚡⚡ xfstests - xfs
>         ⚡⚡⚡ storage: software RAID testing
>         ⚡⚡⚡ stress: stress-ng
>         🚧 ⚡⚡⚡ Storage blktests
> 
>      Host 5:
> 
>         ⚡ Internal infrastructure issues prevented one or more tests (marked
>         with ⚡⚡⚡) from running on this architecture.
>         This is not the fault of the kernel that was tested.
> 
>         ⚡⚡⚡ Boot test
>         ⚡⚡⚡ kdump - sysrq-c - megaraid_sas
> 
>      Host 6:
>         💥 Boot test
>         ⚡⚡⚡ LTP
>         ⚡⚡⚡ Loopdev Sanity
>         ⚡⚡⚡ Memory function: memfd_create
>         ⚡⚡⚡ AMTU (Abstract Machine Test Utility)
>         ⚡⚡⚡ Ethernet drivers sanity
>         ⚡⚡⚡ storage: SCSI VPD
>         🚧 ⚡⚡⚡ CIFS Connectathon
>         🚧 ⚡⚡⚡ POSIX pjd-fstest suites
> 
>      Host 7:
>         💥 Boot test
>         ⚡⚡⚡ Storage SAN device stress - qla2xxx driver
> 
>      Host 8:
>         💥 Boot test
>         ⚡⚡⚡ Storage SAN device stress - qedf driver
> 
>      Host 9:
>         ⏱  Boot test
>         ⏱  Storage SAN device stress - mpt3sas_gen1
> 
>    Test sources: https://github.com/CKI-project/tests-beaker
>      💚 Pull requests are welcome for new tests or improvements to existing tests!
> 
> Aborted tests
> -------------
> Tests that didn't complete running successfully are marked with ⚡⚡⚡.
> If this was caused by an infrastructure issue, we try to mark that
> explicitly in the report.
> 
> Waived tests
> ------------
> If the test run included waived tests, they are marked with 🚧. Such tests are
> executed but their results are not taken into account. Tests are waived when
> their results are not reliable enough, e.g. when they're just introduced or are
> being fixed.
> 
> Testing timeout
> ---------------
> We aim to provide a report within reasonable timeframe. Tests that haven't
> finished running yet are marked with ⏱.
> 
>