Re: centos 7.2，I got some oops form my production line

[Xishi Qiu <qiuxishi@huawei.com>]

On 2017/6/29 16:22, Xishi Qiu wrote:

> centos 7.2，I got some oops form my production line,
> Anybody has seen these errors before?
> 

Here is another one

[  703.025737] BUG: unable to handle kernel NULL pointer dereference at 0000000000000d68
[  703.026008] IP: [<ffffffffa02a46c2>] mlx4_en_QUERY_PORT+0xa2/0x190 [mlx4_en]
[  703.026008] PGD 377f2a067 PUD 379df4067 PMD 0 
[  703.026008] Oops: 0002 [#1] SMP 
[  703.033019] Modules linked in: sch_htb haek(OVE) squashfs loop binfmt_misc phram mtdblock mtd_blkdevs mtd zlib_deflate nf_log_ipv4 nf_log_common xt_LOG ipmi_watchdog ipmi_devintf ipmi_si ipmi_msghandler vfat fat bonding tipc kboxdriver(O) kbox(O) ipt_REJECT iptable_filter signo_catch(O) mlx4_ib(OVE) ib_sa(OVE) ib_mad(OVE) ib_core(OVE) mlx4_en(OVE) ib_addr(OVE) ib_netlink(OVE) vxlan ip6_udp_tunnel udp_tunnel ptp pps_core mlx4_core(OVE) compat(OVE) isofs crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev dm_mod parport_pc sg parport pcspkr i2c_piix4 i2c_core ip_tables ext3 mbcache jbd sr_mod cdrom ata_generic pata_acpi virtio_blk(OVE) virtio_console(OVE) kvm_ivshmem(OVE) crct10dif_pclmul crct10dif_common ata_piix crc32c_intel serio_raw libata pv_channel(OVE)
[  703.055064] mlx4_core 0000:00:07.0: mlx4_dec_port_macs removed mac, port: 1, now: 0
[  703.033019]  virtio_pci(OVE) virtio_ring(OVE) virtio(OVE) floppy monitor_netdev(OE)
[  703.033019] CPU: 3 PID: 3038 Comm: kworker/3:2 Tainted: G        W  OE  ----V-------   3.10.0-327.49.58.52.x86_64 #1
[  703.033019] Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.8.1-0-g4adadbd-20161111_105425-HGH1000008200 04/01/2014
[  703.033019] Workqueue: events linkwatch_event
[  703.033019] task: ffff88041a9bf300 ti: ffff880412134000 task.ti: ffff880412134000
[  703.066565] RIP: 0010:[<ffffffffa02a46c2>]  [<ffffffffa02a46c2>] mlx4_en_QUERY_PORT+0xa2/0x190 [mlx4_en]
[  703.066565] RSP: 0018:ffff880412137bd0  EFLAGS: 00010a03
[  703.066565] RAX: ffff8800ba5bc000 RBX: ffff880410cd1000 RCX: 0000000000000038
[  703.066565] RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffff88041472046c
[  703.074752] RBP: ffff880412137c10 R08: ffffffff81668be0 R09: ffffffff81dc63c0
[  703.074752] R10: 0000000000000400 R11: 0000000000000017 R12: ffff8803773eea20
[  703.074752] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88041b5eb000
[  703.074752] FS:  0000000000000000(0000) GS:ffff880434ac0000(0000) knlGS:0000000000000000
[  703.074752] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  703.074752] CR2: 0000000000000d68 CR3: 000000036ff69000 CR4: 00000000001407e0
[  703.086770] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  703.086770] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  703.086770] Stack:
[  703.086770]  ffff880400000043 000000000000ea60 ffff880400000000 00000000ba5bc000
[  703.086770]  ffff880410ce0000 ffff880412137cac ffff88041b5eb8c0 ffff880410ce08c0
[  703.086770]  ffff880412137c78 ffffffffa02a29a4 dead000000200200 000000004ff29e7e
[  703.086770] Call Trace:
[  703.086770]  [<ffffffffa02a29a4>] mlx4_en_get_settings+0x34/0x540 [mlx4_en]
[  703.086770]  [<ffffffff815383b6>] __ethtool_get_settings+0x86/0x140
[  703.104149]  [<ffffffffa032abad>] bond_update_speed_duplex+0x3d/0x90 [bonding]
[  703.104149]  [<ffffffffa0330e57>] bond_netdev_event+0x137/0x360 [bonding]
[  703.104149]  [<ffffffff8164ba3c>] notifier_call_chain+0x4c/0x70
[  703.104149]  [<ffffffff810ad426>] raw_notifier_call_chain+0x16/0x20
[  703.104149]  [<ffffffff8152fbfd>] call_netdevice_notifiers+0x2d/0x60
[  703.104149]  [<ffffffff81531913>] netdev_state_change+0x23/0x40
[  703.104149]  [<ffffffff81548600>] linkwatch_do_dev+0x40/0x60
[  703.104149]  [<ffffffff815488ef>] __linkwatch_run_queue+0xef/0x200
[  703.104149]  [<ffffffff81548a25>] linkwatch_event+0x25/0x30
[  703.104149]  [<ffffffff8109eb6b>] process_one_work+0x17b/0x470
[  703.104149]  [<ffffffff8109f93b>] worker_thread+0x11b/0x400
[  703.104149]  [<ffffffff8109f820>] ? rescuer_thread+0x400/0x400
[  703.104149]  [<ffffffff810a707f>] kthread+0xcf/0xe0
[  703.104149]  [<ffffffff810a6fb0>] ? kthread_create_on_node+0x140/0x140
[  703.104149]  [<ffffffff8164ffd8>] ret_from_fork+0x58/0x90
[  703.104149]  [<ffffffff810a6fb0>] ? kthread_create_on_node+0x140/0x140
[  703.134274] Code: 48 8b 3b 4c 89 e6 e8 7e 7e f4 ff 48 83 c4 20 44 89 f0 5b 41 5c 41 5d 41 5e 5d c3 66 0f 1f 44 00 00 49 8b 04 24 0f be 10 c1 ea 1f <41> 89 95 68 0d 00 00 0f b6 50 05 83 e2 6f 80 fa 40 0f 87 b7 00 
[  703.134274] RIP  [<ffffffffa02a46c2>] mlx4_en_QUERY_PORT+0xa2/0x190 [mlx4_en]
[  703.134274]  RSP <ffff880412137bd0>
[  703.134274] CR2: 0000000000000d68
[  703.134274] ---[ end trace 76a7da47a517c30b ]---
[  703.134274] Kernel panic - not syncing: Fatal exception


> 
> 1)
> 2017-06-28T02:18:16.461384+08:00[880983.488036] do nothing after die!
> 2017-06-28T02:18:16.462068+08:00[880983.488723] Modules linked in: fuse iptable_filter sha512_generic icp_qa_al_vf(OVE) vfat fat isofs ext4 jbd2 xfs libcrc32c kboxdriver(O) ipmi_devintf ipmi_si ipmi_msghandler kbox(O) signo_catch(O) mlx4_core(OVE) compat(OVE) ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr parport_pc i2c_piix4 parport i2c_core ip_tables ext3 mbcache jbd ata_generic pata_acpi virtio_console(OVE) virtio_balloon(OVE) virtio_blk(OVE) virtio_net(OVE) kvm_ivshmem(OVE) crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_piix pv_channel(OVE) virtio_pci(OVE) virtio_ring(OVE) libata virtio(OVE) floppy bonding
> 2017-06-28T02:18:16.473941+08:00[880983.500597] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G           OE  ----V-------   3.10.0-327.44.58.28.x86_64 #1
> 2017-06-28T02:18:16.475784+08:00[880983.502440] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-20170515_131417-build9a64a246a230 04/01/2014
> 2017-06-28T02:18:16.477993+08:00[880983.504643] task: ffff880138b62280 ti: ffff880138b74000 task.ti: ffff880138b74000
> 2017-06-28T02:18:16.486759+08:00[880983.513413] RIP: 0010:[<ffffffff8108ea50>]  [<ffffffff8108ea50>] get_next_timer_interrupt+0x1c0/0x270
> 2017-06-28T02:18:16.490230+08:00[880983.516874] RSP: 0018:ffff880138b77dd8  EFLAGS: 00010093
> 2017-06-28T02:18:16.492066+08:00[880983.518720] RAX: d8fe6d5832a10700 RBX: 0003213fc510c9c0 RCX: 0000000030303031
> 2017-06-28T02:18:16.494646+08:00[880983.521299] RDX: 0000000030303031 RSI: ffff880138ba5298 RDI: 0000000001347e27
> 2017-06-28T02:18:16.497232+08:00[880983.523877] RBP: ffff880138b77e30 R08: fffffffefbb208f1 R09: 0000000000000001
> 2017-06-28T02:18:16.499848+08:00[880983.526500] R10: 0000000000000027 R11: 0000000000000027 R12: 00000001347e26d0
> 2017-06-28T02:18:16.502272+08:00[880983.528917] R13: ffff880138ba5028 R14: ffff880138ba4000 R15: ffff880138b77de8
> 2017-06-28T02:18:16.504845+08:00[880983.531498] FS:  0000000000000000(0000) GS:ffff88013ed00000(0000) knlGS:0000000000000000
> 2017-06-28T02:18:16.507583+08:00[880983.534237] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> 2017-06-28T02:18:16.509242+08:00[880983.535898] CR2: 000000000ad7a168 CR3: 000000013587f000 CR4: 00000000001407e0
> 2017-06-28T02:18:16.511736+08:00[880983.538391] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> 2017-06-28T02:18:16.514088+08:00[880983.540742] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> 2017-06-28T02:18:16.516363+08:00[880983.543019] Stack:
> 2017-06-28T02:18:16.517237+08:00[880983.543895]  ffff880138b77e00 ffff880138ba5028 ffff880138ba5428 ffff880138ba5828
> 2017-06-28T02:18:16.519793+08:00[880983.546451]  ffff880138ba5c28 cdf899bbcb229fc8 ffff88013ed0fbc0 0003213fc510c9c0
> 2017-06-28T02:18:16.522327+08:00[880983.548982]  0000000000000001 ffff88013ed0cf00 00000001347e26d0 ffff880138b77e88
> 2017-06-28T02:18:16.525118+08:00[880983.551772] Call Trace:
> 2017-06-28T02:18:16.526205+08:00[880983.552858]  [<ffffffff810e2df8>] tick_nohz_stop_sched_tick+0x1e8/0x2e0
> 2017-06-28T02:18:16.528063+08:00[880983.554716]  [<ffffffff81058b2f>] ? kvm_sched_clock_read+0x1f/0x30
> 2017-06-28T02:18:16.529842+08:00[880983.556496]  [<ffffffff810e2f8e>] __tick_nohz_idle_enter+0x9e/0x150
> 2017-06-28T02:18:16.531547+08:00[880983.558199]  [<ffffffff810e34ad>] tick_nohz_idle_enter+0x3d/0x70
> 2017-06-28T02:18:16.533235+08:00[880983.559890]  [<ffffffff810d702e>] cpu_startup_entry+0x9e/0x290
> 2017-06-28T02:18:16.534843+08:00[880983.561498]  [<ffffffff81047c5a>] start_secondary+0x1ba/0x230
> 2017-06-28T02:18:16.536488+08:00[880983.563141] Code: 45 a8 41 89 fb 41 83 e3 3f 45 89 da 0f 1f 80 00 00 00 00 49 63 f2 48 89 ca 48 c1 e6 04 4c 01 ee 48 8b 06 48 39 f0 74 2c 0f 1f 00 <f6> 40 18 01 48 89 ca 75 18 48 8b 50 10 41 b9 01 00 00 00 49 89 
> 2017-06-28T02:18:16.543574+08:00[880983.570229] RIP  [<ffffffff8108ea50>] get_next_timer_interrupt+0x1c0/0x270
> 2017-06-28T02:18:16.545377+08:00[880983.572032]  RSP <ffff880138b77dd8>
> 
> 
> 2)
> [ 6401.956939] BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
> [ 6401.958982] IP: [<ffffffff81218739>] seq_escape+0x19/0x120
> [ 6401.960325] PGD 5c4d3067 PUD 5c4d2067 PMD 0 
> [ 6401.961458] Oops: 0000 [#1] SMP 
> [ 6401.962347] Modules linked in: klp_0001_tty_io_c(OE) zram(C) veth loop regmap_i2c rfkill binfmt_misc ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack bridge stp llc overlay() rtos_kbox_panic(OE) signo_catch(O) coretemp vmwgfx ttm crc32_pclmul drm_kms_helper ghash_clmulni_intel aesni_intel drm lrw gf128mul glue_helper ppdev ablk_helper cryptd pcspkr serio_raw vmw_balloon sg parport_pc parport i2c_piix4 floppy vmw_vmci i2c_core shpchp pata_acpi dm_mod sha512_generic ip_tables sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ata_piix vmxnet3 libata vmw_pvscsi ext4 mbcache jbd2 [last unloaded: klp_0001_tty_io_c]
> [ 6401.981457] CPU: 3 PID: 15090 Comm: cat Tainted: G         C OE K----V------- T 3.10.0-327.36.58.4.x86_64 #1
> [ 6401.984677] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
> [ 6401.987115] task: ffff8801a45a8000 ti: ffff8800191e8000 task.ti: ffff8800191e8000
> [ 6401.988887] RIP: 0010:[<ffffffff81218739>]  [<ffffffff81218739>] seq_escape+0x19/0x120
> [ 6401.990897] RSP: 0018:ffff8800191ebe10  EFLAGS: 00010282
> [ 6401.992212] RAX: 0000000000000000 RBX: ffffffffa03b0010 RCX: 0000000000000000
> [ 6401.993895] RDX: ffffffff818c6c1d RSI: 0000000000000001 RDI: ffff8800ad843b00
> [ 6401.995485] RBP: ffff8800191ebe48 R08: 0000000000000022 R09: 0000000000000022
> [ 6401.997014] R10: 0000000000000000 R11: ffff8800191ebcee R12: ffff8800ad843b00
> [ 6401.998608] R13: ffff8800ac9ffec0 R14: ffff8800191ebf48 R15: 0000000000000001
> [ 6402.000155] FS:  00007f6a3a74d740(0000) GS:ffff8801bed80000(0000) knlGS:0000000000000000
> [ 6402.001873] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 6402.003131] CR2: 0000000000000001 CR3: 0000000060752000 CR4: 00000000000407e0
> [ 6402.004644] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 6402.006141] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 6402.007675] Stack:
> [ 6402.008221]  00000000b6458b7d ffffffff818c6c1d ffffffffa03b0010 ffff8800ad843b00
> [ 6402.010274]  ffff8800ac9ffec0 ffff8800191ebf48 ffff8800ad843b00 ffff8800191ebe90
> [ 6402.012347]  ffffffff81331857 ffff8800191ebe66 666dffff8165a4be ffff8800191e006c
> [ 6402.015506] Call Trace:
> [ 6402.017168]  [<ffffffff81331857>] ddebug_proc_show+0x87/0xf0
> [ 6402.019780]  [<ffffffff812185b8>] seq_read+0x238/0x3a0
> [ 6402.022289]  [<ffffffff811f3bdc>] vfs_read+0x9c/0x170
> [ 6402.024775]  [<ffffffff811f472f>] SyS_read+0x7f/0xe0
> [ 6402.027247]  [<ffffffff81668273>] system_call_fastpath+0x16/0x1b
> [ 6402.029952] Code: fc ff ff 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 53 48 83 ec 10 <44> 0f b6 26 48 8b 07 48 89 7d c8 48 89 55 d0 49 89 c5 48 89 c3 
> [ 6402.040474] RIP  [<ffffffff81218739>] seq_escape+0x19/0x120
> [ 6402.043137]  RSP <ffff8800191ebe10>
> [ 6402.045108] CR2: 0000000000000001
> [ 6402.048707] ---[ end trace 56a06232addee1f6 ]---
> [ 6402.051063] Kernel panic - not syncing: Fatal exception
> [ 6402.054623] CPU: 3 PID: 15090 Comm: cat Tainted: G      D  C OE K----V------- T 3.10.0-327.36.58.4.x86_64 #1
> [ 6402.054626] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
> [ 6402.054628] task: ffff8801a45a8000 ti: ffff8800191e8000 task.ti: ffff8800191e8000
> [ 6402.054652] RIP: 0033:[<00007f6a3a25f480>]  [<00007f6a3a25f480>] 0x7f6a3a25f47f
> [ 6402.054654] RSP: 002b:00007ffe395c4c68  EFLAGS: 00010206
> [ 6402.054655] RAX: 0000000000000000 RBX: ffffffff81668273 RCX: 00000000020ab030
> [ 6402.054656] RDX: 0000000000010000 RSI: 00000000020ac000 RDI: 0000000000000004
> [ 6402.054658] RBP: 00000000020ac000 R08: 0000000000000000 R09: 0000000000010fff
> [ 6402.054659] R10: 00007ffe395c4990 R11: 0000000000000246 R12: 0000000000000000
> [ 6402.054660] R13: 0000000000000004 R14: 00000000020ac000 R15: 0000000000010000
> [ 6402.054662] FS:  00007f6a3a74d740(0000) GS:ffff8801bed80000(0000) knlGS:0000000000000000
> [ 6402.054663] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 6402.054665] CR2: 0000000000000001 CR3: 0000000060752000 CR4: 00000000000407e0
> [ 6402.054670] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 6402.054674] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 6402.054675] 
> 
> 
> 
> .
>