Re: ❌ FAIL: Test report for kernel 5.3.13-015654b.cki (stable-queue)

[Veronika Kabatova <vkabatov@redhat.com>]

----- Original Message -----
> From: "CKI Project" <cki-project@redhat.com>
> To: "Linux Stable maillist" <stable@vger.kernel.org>
> Cc: "Major Hayden" <major@redhat.com>
> Sent: Thursday, November 28, 2019 11:11:26 AM
> Subject: ❌ FAIL: Test report for kernel 5.3.13-015654b.cki (stable-queue)
> 
> 
> Hello,
> 
> We ran automated tests on a recent commit from this kernel tree:
> 
>        Kernel repo:
>        git://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git
>             Commit: 015654bdddc1 - media: mceusb: fix out of bounds read in
>             MCE receiver buffer
> 
> The results of these automated tests are provided below.
> 
>     Overall result: FAILED (see details below)
>              Merge: OK
>            Compile: OK
>              Tests: FAILED
> 
> All kernel binaries, config files, and logs are available for download here:
> 
>   https://artifacts.cki-project.org/pipelines/311282
> 
> One or more kernel tests failed:
> 
>     ppc64le:
>      ❌ Boot test
>      ❌ Boot test
> 
>     aarch64:
>      ❌ Podman system integration test (as root)
>      ❌ Boot test
> 
>     x86_64:
>      ❌ Podman system integration test (as root)
>      ❌ Boot test
> 

Hi,

there have been multiple call traces during boots such as the one
below. You can find more information and full console logs in the
link in the summary above.


[   38.534343] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000070 
[   38.536528] Mem abort info: 
[   38.537188]   ESR = 0x96000004 
[   38.537909]   Exception class = DABT (current EL), IL = 32 bits 
[   38.539323]   SET = 0, FnV = 0 
[   38.540077]   EA = 0, S1PTW = 0 
[   38.540838] Data abort info: 
[   38.541545]   ISV = 0, ISS = 0x00000004 
[   38.542455]   CM = 0, WnR = 0 
[   38.543152] user pgtable: 4k pages, 48-bit VAs, pgdp=00000001b525b000 
[   38.544753] [0000000000000070] pgd=0000000000000000 
[   38.545957] Internal error: Oops: 96000004 [#1] SMP 
[   38.547238] Modules linked in: sunrpc vfat fat crct10dif_ce virtio_net ghash_ce net_failover failover ip_tables xfs libcrc32c virtio_console virtio_blk qemu_fw_cfg virtio_mmio 
[   38.552298] CPU: 2 PID: 635 Comm: kworker/u8:3 Not tainted 5.3.13-015654b.cki #1 
[   38.554055] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 
[   38.555801] Workqueue: netns cleanup_net 
[   38.556788] pstate: 60400005 (nZCv daif +PAN -UAO) 
[   38.557985] pc : kernfs_find_ns+0x24/0x128 
[   38.558987] lr : kernfs_find_and_get_ns+0x44/0x68 
[   38.560168] sp : ffff0000102dba90 
[   38.561003] x29: ffff0000102dba90 x28: 0000000000000000  
[   38.562531] x27: 00000000ffffffff x26: ffff5f51c319c000  
[   38.564281] x25: ffff5f51c27ee0c8 x24: ffff5f51c31b6473  
[   38.564283] x23: 0000000000000140 x22: 0000000000000000  
[   38.564285] x21: ffff5f51c290edc8 x20: 0000000000000000  
[[      3388..556548298164]]  x1r9e:s t0r0a0i0n0t0d00[0804040]0:0 00* *x 1I8n:s t0a0l0l0i0n0g0 0d0e0p0e0n0d0e0n6c i e
s[   
 38.564287] x17: 0000000000000000 x16: ffff5f51c245eef8  
[   38.564289] x15: 0000000000000000 x14: 0000000000000000  
[   38.573279] x13: ffffde0cbd92e310 x12: ffffde0cbc5784e8  
[   38.574606] x11: ffffde0cbc578490 x10: 0000000000000040  
[   38.575931] x9 : ffffde0cbd92e318 x8 : 0000000000210d00  
[   38.577237] x7 : 0000000000000008 x6 : ffffde0ca8874380  
[   38.578562] x5 : 54df216eecf59a86 x4 : ffff7f7832a21d20  
[   38.579877] x3 : 000000008020001c x2 : 0000000000000000  
[   38.581324] x1 : ffff5f51c290edc8 x0 : ffff5f51c1ca47cc  
[   38.582655] Call trace: 
[   38.583284]  kernfs_find_ns+0x24/0x128 
[   38.584215]  kernfs_find_and_get_ns+0x44/0x68 
[   38.585311]  sysfs_remove_group+0x38/0xa0 
[   38.586315]  netdev_queue_update_kobjects+0xfc/0x1a0 
[   38.587574]  netdev_unregister_kobject+0x64/0x90 
[   38.588747]  rollback_registered_many+0x25c/0x540 
[   38.589948]  unregister_netdevice_many.part.0+0x20/0x88 
[   38.591296]  default_device_exit_batch+0x154/0x170 
[   38.592541]  ops_exit_list.isra.0+0x6c/0x80 
[   38.593632]  cleanup_net+0x290/0x3c0 
[   38.594573]  process_one_work+0x1bc/0x3e8 
[   38.595578]  worker_thread+0x54/0x440 
[   38.596558]  kthread+0x104/0x130 
[   38.597396]  ret_from_fork+0x10/0x18 
[   38.598362] Code: aa0103f5 aa1e03e0 aa0203f6 d503201f (7940e260)  



Veronika

> We hope that these logs can help you find the problem quickly. For the full
> detail on our testing procedures, please scroll to the bottom of this
> message.
> 
> Please reply to this email if you have any questions about the tests that we
> ran or if you have any suggestions on how to make future tests more
> effective.
> 
>         ,-.   ,-.
>        ( C ) ( K )  Continuous
>         `-',-.`-'   Kernel
>           ( I )     Integration
>            `-'
> ______________________________________________________________________________
> 
> Compile testing
> ---------------
> 
> We compiled the kernel for 3 architectures:
> 
>     aarch64:
>       make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
> 
>     ppc64le:
>       make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
> 
>     x86_64:
>       make options: -j30 INSTALL_MOD_STRIP=1 targz-pkg
> 
> 
> Hardware testing
> ----------------
> We booted each kernel and ran the following tests:
> 
>   aarch64:
>     Host 1:
>        ✅ Boot test
>        ❌ Podman system integration test (as root)
>        ⚡⚡⚡ Podman system integration test (as user)
>        ⚡⚡⚡ LTP
>        ⚡⚡⚡ Loopdev Sanity
>        ⚡⚡⚡ Memory function: memfd_create
>        ⚡⚡⚡ Memory function: kaslr
>        ⚡⚡⚡ AMTU (Abstract Machine Test Utility)
>        ⚡⚡⚡ LTP: openposix test suite
>        ⚡⚡⚡ Networking bridge: sanity
>        ⚡⚡⚡ Ethernet drivers sanity
>        ⚡⚡⚡ Networking MACsec: sanity
>        ⚡⚡⚡ Networking socket: fuzz
>        ⚡⚡⚡ Networking sctp-auth: sockopts test
>        ⚡⚡⚡ Networking: igmp conformance test
>        ⚡⚡⚡ Networking route: pmtu
>        ⚡⚡⚡ Networking route_func: local
>        ⚡⚡⚡ Networking route_func: forward
>        ⚡⚡⚡ Networking TCP: keepalive test
>        ⚡⚡⚡ Networking UDP: socket
>        ⚡⚡⚡ Networking tunnel: geneve basic test
>        ⚡⚡⚡ Networking tunnel: gre basic
>        ⚡⚡⚡ L2TP basic test
>        ⚡⚡⚡ Networking tunnel: vxlan basic
>        ⚡⚡⚡ Networking ipsec: basic netns transport
>        ⚡⚡⚡ Networking ipsec: basic netns tunnel
>        ⚡⚡⚡ audit: audit testsuite test
>        ⚡⚡⚡ httpd: mod_ssl smoke sanity
>        ⚡⚡⚡ tuned: tune-processes-through-perf
>        ⚡⚡⚡ ALSA PCM loopback test
>        ⚡⚡⚡ ALSA Control (mixer) Userspace Element test
>        ⚡⚡⚡ storage: SCSI VPD
>        ⚡⚡⚡ stress: stress-ng
>        ⚡⚡⚡ trace: ftrace/tracer
>        🚧 ⚡⚡⚡ CIFS Connectathon
>        🚧 ⚡⚡⚡ POSIX pjd-fstest suites
>        🚧 ⚡⚡⚡ jvm test suite
>        🚧 ⚡⚡⚡ Networking vnic: ipvlan/basic
>        🚧 ⚡⚡⚡ iotop: sanity
>        🚧 ⚡⚡⚡ Usex - version 1.9-29
>        🚧 ⚡⚡⚡ storage: dm/common
> 
>     Host 2:
>        ❌ Boot test
>        ⚡⚡⚡ xfstests: ext4
>        ⚡⚡⚡ xfstests: xfs
>        ⚡⚡⚡ lvm thinp sanity
>        ⚡⚡⚡ storage: software RAID testing
>        🚧 ⚡⚡⚡ IPMI driver test
>        🚧 ⚡⚡⚡ IPMItool loop stress test
>        🚧 ⚡⚡⚡ selinux-policy: serge-testsuite
>        🚧 ⚡⚡⚡ Storage blktests
> 
>   ppc64le:
>     Host 1:
>        ❌ Boot test
>        ⚡⚡⚡ xfstests: ext4
>        ⚡⚡⚡ xfstests: xfs
>        ⚡⚡⚡ lvm thinp sanity
>        ⚡⚡⚡ storage: software RAID testing
>        🚧 ⚡⚡⚡ IPMI driver test
>        🚧 ⚡⚡⚡ IPMItool loop stress test
>        🚧 ⚡⚡⚡ selinux-policy: serge-testsuite
>        🚧 ⚡⚡⚡ Storage blktests
> 
>     Host 2:
>        ❌ Boot test
>        ⚡⚡⚡ Podman system integration test (as root)
>        ⚡⚡⚡ Podman system integration test (as user)
>        ⚡⚡⚡ LTP
>        ⚡⚡⚡ Loopdev Sanity
>        ⚡⚡⚡ Memory function: memfd_create
>        ⚡⚡⚡ Memory function: kaslr
>        ⚡⚡⚡ AMTU (Abstract Machine Test Utility)
>        ⚡⚡⚡ LTP: openposix test suite
>        ⚡⚡⚡ Networking bridge: sanity
>        ⚡⚡⚡ Ethernet drivers sanity
>        ⚡⚡⚡ Networking MACsec: sanity
>        ⚡⚡⚡ Networking socket: fuzz
>        ⚡⚡⚡ Networking sctp-auth: sockopts test
>        ⚡⚡⚡ Networking route: pmtu
>        ⚡⚡⚡ Networking route_func: local
>        ⚡⚡⚡ Networking route_func: forward
>        ⚡⚡⚡ Networking TCP: keepalive test
>        ⚡⚡⚡ Networking UDP: socket
>        ⚡⚡⚡ Networking tunnel: geneve basic test
>        ⚡⚡⚡ Networking tunnel: gre basic
>        ⚡⚡⚡ L2TP basic test
>        ⚡⚡⚡ Networking tunnel: vxlan basic
>        ⚡⚡⚡ Networking ipsec: basic netns tunnel
>        ⚡⚡⚡ audit: audit testsuite test
>        ⚡⚡⚡ httpd: mod_ssl smoke sanity
>        ⚡⚡⚡ tuned: tune-processes-through-perf
>        ⚡⚡⚡ ALSA PCM loopback test
>        ⚡⚡⚡ ALSA Control (mixer) Userspace Element test
>        ⚡⚡⚡ trace: ftrace/tracer
>        🚧 ⚡⚡⚡ CIFS Connectathon
>        🚧 ⚡⚡⚡ POSIX pjd-fstest suites
>        🚧 ⚡⚡⚡ jvm test suite
>        🚧 ⚡⚡⚡ Networking vnic: ipvlan/basic
>        🚧 ⚡⚡⚡ iotop: sanity
>        🚧 ⚡⚡⚡ Usex - version 1.9-29
>        🚧 ⚡⚡⚡ storage: dm/common
> 
>   x86_64:
>     Host 1:
>        ⏱  Boot test
>        ⏱  Storage SAN device stress - megaraid_sas
> 
>     Host 2:
>        ✅ Boot test
>        ❌ Podman system integration test (as root)
>        ⚡⚡⚡ Podman system integration test (as user)
>        ⚡⚡⚡ LTP
>        ⚡⚡⚡ Loopdev Sanity
>        ⚡⚡⚡ Memory function: memfd_create
>        ⚡⚡⚡ Memory function: kaslr
>        ⚡⚡⚡ AMTU (Abstract Machine Test Utility)
>        ⚡⚡⚡ LTP: openposix test suite
>        ⚡⚡⚡ Networking bridge: sanity
>        ⚡⚡⚡ Ethernet drivers sanity
>        ⚡⚡⚡ Networking MACsec: sanity
>        ⚡⚡⚡ Networking socket: fuzz
>        ⚡⚡⚡ Networking sctp-auth: sockopts test
>        ⚡⚡⚡ Networking: igmp conformance test
>        ⚡⚡⚡ Networking route: pmtu
>        ⚡⚡⚡ Networking route_func: local
>        ⚡⚡⚡ Networking route_func: forward
>        ⚡⚡⚡ Networking TCP: keepalive test
>        ⚡⚡⚡ Networking UDP: socket
>        ⚡⚡⚡ Networking tunnel: geneve basic test
>        ⚡⚡⚡ Networking tunnel: gre basic
>        ⚡⚡⚡ L2TP basic test
>        ⚡⚡⚡ Networking tunnel: vxlan basic
>        ⚡⚡⚡ Networking ipsec: basic netns transport
>        ⚡⚡⚡ Networking ipsec: basic netns tunnel
>        ⚡⚡⚡ audit: audit testsuite test
>        ⚡⚡⚡ httpd: mod_ssl smoke sanity
>        ⚡⚡⚡ tuned: tune-processes-through-perf
>        ⚡⚡⚡ pciutils: sanity smoke test
>        ⚡⚡⚡ ALSA PCM loopback test
>        ⚡⚡⚡ ALSA Control (mixer) Userspace Element test
>        ⚡⚡⚡ storage: SCSI VPD
>        ⚡⚡⚡ stress: stress-ng
>        ⚡⚡⚡ trace: ftrace/tracer
>        🚧 ⚡⚡⚡ CIFS Connectathon
>        🚧 ⚡⚡⚡ POSIX pjd-fstest suites
>        🚧 ⚡⚡⚡ jvm test suite
>        🚧 ⚡⚡⚡ Networking vnic: ipvlan/basic
>        🚧 ⚡⚡⚡ iotop: sanity
>        🚧 ⚡⚡⚡ Usex - version 1.9-29
>        🚧 ⚡⚡⚡ storage: dm/common
> 
>     Host 3:
>        ❌ Boot test
>        ⚡⚡⚡ xfstests: ext4
>        ⚡⚡⚡ xfstests: xfs
>        ⚡⚡⚡ lvm thinp sanity
>        ⚡⚡⚡ storage: software RAID testing
>        🚧 ⚡⚡⚡ IOMMU boot test
>        🚧 ⚡⚡⚡ IPMI driver test
>        🚧 ⚡⚡⚡ IPMItool loop stress test
>        🚧 ⚡⚡⚡ selinux-policy: serge-testsuite
>        🚧 ⚡⚡⚡ power-management: cpupower/sanity test
>        🚧 ⚡⚡⚡ Storage blktests
> 
>     Host 4:
>        ⏱  Boot test
>        ⏱  Storage SAN device stress - mpt3sas driver
> 
>   Test sources: https://github.com/CKI-project/tests-beaker
>     💚 Pull requests are welcome for new tests or improvements to existing
>     tests!
> 
> Waived tests
> ------------
> If the test run included waived tests, they are marked with 🚧. Such tests
> are
> executed but their results are not taken into account. Tests are waived when
> their results are not reliable enough, e.g. when they're just introduced or
> are
> being fixed.
> 
> Testing timeout
> ---------------
> We aim to provide a report within reasonable timeframe. Tests that haven't
> finished running are marked with ⏱. Reports for non-upstream kernels have
> a Beaker recipe linked to next to each host.
> 
>