From mboxrd@z Thu Jan 1 00:00:00 1970 From: swise@opengridcomputing.com (Steve Wise) Date: Tue, 12 Jul 2016 11:38:58 -0500 Subject: crash on device removal In-Reply-To: <00cc01d1dc5b$51c7fa90$f557efb0$@opengridcomputing.com> References: <00cc01d1dc5b$51c7fa90$f557efb0$@opengridcomputing.com> Message-ID: <00cd01d1dc5b$e374c3b0$aa5e4b10$@opengridcomputing.com> > > Hey Christoph, > > I see a crash when shutting down a nvme host node via 'reboot' that has 1 > target device attached. The shutdown causes iw_cxgb4 to be removed which > triggers the device removal logic in the nvmf rdma transport. The crash > is here: > > (gdb) list *nvme_rdma_free_qe+0x18 > 0x1e8 is in nvme_rdma_free_qe (drivers/nvme/host/rdma.c:196). > 191 } > 192 > 193 static void nvme_rdma_free_qe(struct ib_device *ibdev, struct > nvme_rdma_qe *qe, > 194 size_t capsule_size, enum dma_data_direction dir) > 195 { > 196 ib_dma_unmap_single(ibdev, qe->dma, capsule_size, dir); > 197 kfree(qe->data); > 198 } > 199 > 200 static int nvme_rdma_alloc_qe(struct ib_device *ibdev, struct > nvme_rdma_qe *qe, > > Apparently qe is NULL. > > Looking at the device removal path, the logic appears correct (see > nvme_rdma_device_unplug() and the nice function comment :) ). I'm > wondering if concurrently to the host device removal path cleaning up > queues, the target is disconnecting all of its queues due to the first > disconnect event from the host causing some cleanup race on the host side? > Although since the removal path executing in the cma event handler upcall, > I don't think another thread would be handling a disconnect event. Maybe > the qp async event handler flow? > I see the async event handler, nvme_rdma_qp_event() does nothing but a pr_debug(), so no race with the cm event handler thread and the async event handler thread... > Thoughts? > > Here is the Oops: > > [ 710.929451] iw_cxgb4:0000:83:00.4: Detach > [ 711.242989] iw_cxgb4:0000:82:00.4: Detach > [ 711.247039] nvme nvme1: Got rdma device removal event, deleting ctrl > [ 711.298244] BUG: unable to handle kernel NULL pointer dereference at > 0000000000000010 > [ 711.306162] IP: [] nvme_rdma_free_qe+0x18/0x80 > [nvme_rdma] > [ 711.313286] PGD 0 > [ 711.315348] Oops: 0000 [#1] SMP > [ 711.318519] Modules linked in: nvme_rdma nvme_fabrics brd iw_cxgb4 > cxgb4 ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE > nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 > nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT nf_reject_ipv4 xt_CHECKSUM > iptable_mangle iptable_filter ip_tables bridge 8021q mrp garp stp llc > cachefiles fscache rdma_ucm rdma_cm iw_cm ib_ipoib ib_cm ib_uverbs ib_umad > ocrdma be2net iw_nes libcrc32c iw_cxgb3 cxgb3 mdio ib_qib rdmavt mlx5_ib > mlx5_core mlx4_en ib_mthca binfmt_misc dm_mirror dm_region_hash dm_log > vhost_net macvtap macvlan vhost tun kvm irqbypass uinput iTCO_wdt > iTCO_vendor_support mxm_wmi pcspkr mlx4_ib ib_core mlx4_core dm_mod > i2c_i801 sg ipmi_ssif ipmi_si ipmi_msghandler nvme nvme_core lpc_ich > mfd_core mei_me mei igb dca ptp pps_core wmi ext4(E) mbcache(E) jbd2(E) > sd_mod(E) ahci(E) libahci(E) libata(E) mgag200(E) ttm(E) drm_kms_helper(E) > drm(E) fb_sys_fops(E) sysimgblt(E) sysfillrect(E) syscopyarea(E) > i2c_algo_bit(E) i2c_core(E) [last unloaded: cxgb4] > [ 711.412158] CPU: 0 PID: 4213 Comm: reboot Tainted: G E > 4.7.0-rc2-block-for-next+ #77 > [ 711.421064] Hardware name: Supermicro X9DR3-F/X9DR3-F, BIOS 3.2a > 07/09/2015 > [ 711.428058] task: ffff881033b495c0 ti: ffff88100fc24000 task.ti: > ffff88100fc24000 > [ 711.435563] RIP: 0010:[] [] > nvme_rdma_free_qe+0x18/0x80 [nvme_rdma] > [ 711.445104] RSP: 0018:ffff88100fc279a8 EFLAGS: 00010292 > [ 711.450442] RAX: 0000000000000000 RBX: 0000000000000000 RCX: > 0000000000000002 > [ 711.457608] RDX: 0000000000000010 RSI: 0000000000000000 RDI: > ffff881034168000 > [ 711.464775] RBP: ffff88100fc279b8 R08: 0000000000000001 R09: > ffffea0001e51d10 > [ 711.471943] R10: ffffea0001e51d18 R11: 0000000000000000 R12: > 0000000000000000 > [ 711.479112] R13: 0000000000000020 R14: ffff881034168000 R15: > ffff8810345b8140 > [ 711.486285] FS: 00007feac7042700(0000) GS:ffff88103ee00000(0000) > knlGS:0000000000000000 > [ 711.494405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 711.500175] CR2: 0000000000000010 CR3: 00000010229d7000 CR4: > 00000000000406f0 > [ 711.507341] Stack: > [ 711.509367] ffff881034285000 0000000000000001 ffff88100fc279f8 > ffffffffa039adcf > [ 711.516868] ffff88100fc279d8 ffff881034285000 ffff881037f9f000 > ffff881034272c00 > [ 711.524384] ffff88100fc27b18 ffff881034272dd8 ffff88100fc27a88 > ffffffffa039c8f5 > [ 711.531897] Call Trace: > [ 711.534371] [] nvme_rdma_destroy_queue_ib+0x5f/0x90 > [nvme_rdma] > [ 711.541972] [] nvme_rdma_cm_handler+0x2c5/0x340 > [nvme_rdma] > [ 711.549228] [] ? kmem_cache_free+0x1dd/0x200 > [ 711.555177] [] ? cma_comp+0x49/0x60 [rdma_cm] > [ 711.561217] [] cma_remove_id_dev+0x8f/0xa0 [rdma_cm] > [ 711.567860] [] cma_process_remove+0xb7/0x100 > [rdma_cm] > [ 711.574678] [] ? __kernfs_remove+0x114/0x1d0 > [ 711.580626] [] cma_remove_one+0x3e/0x60 [rdma_cm] > [ 711.587015] [] ib_unregister_device+0xb0/0x150 > [ib_core] > [ 711.595252] [] c4iw_unregister_device+0x64/0x90 > [iw_cxgb4] > [ 711.603648] [] c4iw_remove+0x27/0x60 [iw_cxgb4] > [ 711.611069] [] c4iw_uld_state_change+0x111/0x250 > [iw_cxgb4] > [ 711.619532] [] ? _cond_resched+0x1d/0x30 > [ 711.626317] [] ? list_del+0x11/0x40 > [ 711.632678] [] detach_ulds+0x4a/0xf0 [cxgb4] > [ 711.639822] [] remove_one+0x18d/0x1b0 [cxgb4] > [ 711.647060] [] pci_device_shutdown+0x41/0x90 > [ 711.654189] [] device_shutdown+0x45/0x1b0 > [ 711.661051] [] kernel_restart_prepare+0x36/0x40 > [ 711.668414] [] kernel_restart+0x16/0x60 > [ 711.675084] [] SYSC_reboot+0x1a5/0x230 > [ 711.681645] [] ? mntput+0x21/0x30 > [ 711.687738] [] ? __fput+0x177/0x240 > [ 711.693964] [] ? ____fput+0xe/0x10 > [ 711.700097] [] ? do_audit_syscall_entry+0x66/0x70 > [ 711.707481] [] ? > syscall_trace_enter_phase1+0xf8/0x120 > [ 711.715273] [] ? exit_to_usermode_loop+0x74/0xf0 > [ 711.722514] [] SyS_reboot+0xe/0x10 > [ 711.728517] [] do_syscall_64+0x78/0x1d0 > [ 711.734931] [] ? do_page_fault+0x37/0x90 > [ 711.741410] [] entry_SYSCALL64_slow_path+0x25/0x25 > [ 711.748731] Code: 01 00 00 c9 c3 0f 0b eb fe 66 2e 0f 1f 84 00 00 00 00 > 00 55 48 89 e5 53 48 83 ec 08 66 66 66 66 90 48 8b 87 f0 02 00 00 48 89 f3 > <48> 8b 76 10 48 85 c0 74 13 ff 50 10 48 8b 7b 08 e8 93 4d e6 e0 > [ 711.770832] RIP [] nvme_rdma_free_qe+0x18/0x80 > [nvme_rdma] > [ 711.778904] RSP > [ 711.783290] CR2: 0000000000000010