Re: SRPt oops with 4.5-rc3-ish

[Bart Van Assche <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>]

On 02/14/16 08:09, Doug Ledford wrote:
> While testing with my latest kernel (rc3 plus pending RDMA patches), I
> ran across this oops:
>
> [dledford@linux-ws ~]$ console rdma-storage-04
> Enter dledford-Qj3k6FK1/F6RgOCG8Jv1mWcM3YSXpimQh7FX57BcuXVWk0Htik3J/w@public.gmane.org's password:
> [Enter `^Ec?' for help]
> [-- MOTD -- https://home.corp.redhat.com/wiki/conserver]
> [playback]
> [160605.947614]  [<ffffffff81150545>] ? call_rcu_sched+0x25/0x30
> [160605.954074]  [<ffffffffc0b3dd84>]
> target_fabric_nacl_base_release+0x64/0x70]
> [160605.963731]  [<ffffffff813ccc6f>] config_item_release+0x9f/0x1c0
> [160605.970579]  [<ffffffff813ccdf2>] config_item_put+0x62/0x80
> [160605.976936]  [<ffffffff813c97d3>] configfs_rmdir+0x343/0x500
> [160605.983396]  [<ffffffff8131287a>] vfs_rmdir+0x13a/0x220
> [160605.989375]  [<ffffffff813197db>] do_rmdir+0x1fb/0x260
> [160605.995244]  [<ffffffff8131adde>] SyS_rmdir+0x1e/0x30
> [160606.001019]  [<ffffffff81a0922e>] entry_SYSCALL_64_fastpath+0x12/0x71
> [160606.009586] ---[ end trace 820588f5ef5f6148 ]---
> [160607.051593] ib_srpt Received SRP_LOGIN_REQ with i_port_id
> 0x7f0ee700032d1de)
> [160607.078225] ib_srpt rejected SRP_LOGIN_REQ because the target port
> has not d
> [160611.228909] ib_srpt Received IB DREQ ERROR event.
> [160613.276862] ib_srpt Received IB TimeWait exit for cm_id
> ffff881cc9dc7a00.
> [160613.290322] BUG: unable to handle kernel paging request at
> 0000000000018630
> [160613.301470] IP: [<ffffffff81125694>]
> native_queued_spin_lock_slowpath+0x2e40
> [160613.313112] PGD 0
> [160613.318577] Oops: 0002 [#1] SMP
> [160613.325358] Modules linked in: nfnetlink(+) ip6t_rpfilter 8021q garp
> ip6t_R]
> [160613.492357] CPU: 1 PID: 44982 Comm: kworker/1:1 Tainted: G        W
> I     44
> [160613.505978] Hardware name: Dell Inc. PowerEdge R730xd/0599V5, BIOS
> 1.0.4 084
> [160613.517697] Workqueue: events srpt_release_channel_work [ib_srpt]
> [160613.527634] task: ffff881d01099000 ti: ffff881d02014000 task.ti:
> ffff881d020
> [160613.539130] RIP: 0010:[<ffffffff81125694>]  [<ffffffff81125694>]
> native_que0
> [160613.553326] RSP: 0018:ffff881d02017d90  EFLAGS: 00010006
> [160613.562332] RAX: 00000000000000ea RBX: 0000000000000206 RCX:
> 000000000001860
> [160613.573401] RDX: 0000000000080000 RSI: ffff881d4c818600 RDI:
> ffff880f2d7c7d8
> [160613.584472] RBP: ffff881d02017d90 R08: 0000000000000023 R09:
> 000000000000000
> [160613.595491] R10: 00000000ffffffd8 R11: 00000000000211c0 R12:
> ffff880f2d7c7d0
> [160613.606568] R13: ffff881ce426d000 R14: ffff881cca702a00 R15:
> 000000000000000
> [160613.617643] FS:  0000000000000000(0000) GS:ffff881d4c800000(0000)
> knlGS:0000
> [160613.629793] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [160613.639315] CR2: 0000000000018630 CR3: 0000000001ca9000 CR4:
> 000000000014060
> [160613.650471] Stack:
> [160613.655843]  ffff881d02017da0 ffffffff8122ac4c ffff881d02017db8
> ffffffff81a7
> [160613.667361]  ffff880f2d7c7d18 ffff881d02017de0 ffffffff81121255
> ffff881cca70
> [160613.678885]  ffff881ce426d058 ffff881ce426d000 ffff881d02017e10
> ffffffffc070
> [160613.690366] Call Trace:
> [160613.696195]  [<ffffffff8122ac4c>] queued_spin_lock_slowpath+0x12/0x1d
> [160613.706533]  [<ffffffff81a08ea7>] _raw_spin_lock_irqsave+0x87/0xa0
> [160613.716586]  [<ffffffff81121255>] complete+0x25/0x70
> [160613.725318]  [<ffffffffc07e7e80>]
> srpt_release_channel_work+0x180/0x210 [ib]
> [160613.736889]  [<ffffffff810e6dd8>] process_one_work+0x228/0x650
> [160613.746616]  [<ffffffff810e79be>] worker_thread+0x21e/0x800
> [160613.756047]  [<ffffffff81a02035>] ? __schedule+0x4b5/0xe6a
> [160613.765371]  [<ffffffff810e77a0>] ? kzalloc+0x30/0x30
> [160613.774203]  [<ffffffff810efc38>] kthread+0x118/0x150
> [160613.783000]  [<ffffffff810efb20>] ? flush_kthread_worker+0xd0/0xd0
> [160613.792932]  [<ffffffff81a0958f>] ret_from_fork+0x3f/0x70
> [160613.801994]  [<ffffffff810efb20>] ? flush_kthread_worker+0xd0/0xd0
> [160613.811897] Code: 01 00 00 74 ec e9 d7 fd ff ff 48 89 c1 c1 e8 12 48
> c1 e9
> [160613.840260] RIP  [<ffffffff81125694>]
> native_queued_spin_lock_slowpath+0x2e0
> [160613.851846]  RSP <ffff881d02017d90>
> [160613.858812] CR2: 0000000000018630
> [160613.874762] ---[ end trace 820588f5ef5f6149 ]---
> [160613.937225] Kernel panic - not syncing: Fatal exception
> [160613.946167] Kernel Offset: disabled
> [160614.004693] ---[ end Kernel panic - not syncing: Fatal exception
> [-- MARK -- Sun Feb 14 15:50:00 2016]
> [-- dledford-CKb8VAQLn9hXrIkS9f7CXA@public.gmane.org@ovpn-116-26.rdu2.redhat.com attached -- Sun Feb
> 14 15:5]
>
>
>
> Basic description of situation that cause the oops:
>
> Server with 30+ SRPt luns, 2 SRP devices, 1 active client busy beating
> away on 1 lun via two paths (active/passive setup)
>
> Run dnf upgrade (dnf is yum's replacement, so just a system wide
> software update).
>
> Get to the cleanup for targetcli/target-restore and it invokes an
> attempt to reload the target service while still in use.  During the
> process of deconfiguring the luns that are in use, this oops occurred.
> Sending the report to you because it appears to involve the
> multi-channel support.

Hello Doug,

As far as I know the session shutdown code in the LIO core has never 
worked reliably in the presence of active I/O in any upstream kernel 
version. All my tests of the ib_srpt patch series I submitted recently 
have been performed on top of a long series of bug fixes for the LIO 
core. The tree I have been testing is available at 
https://github.com/bvanassche/linux/tree/lio-tmf-fixes-2016-01-13. I 
have tried a few times to submit the LIO core patches to Nic Bellinger 
(making TMF handling synchronous + several fixes for race conditions 
related to session shutdown). Apparently Nic is trying to fix the 
existing approach for TMF handling (handling TMF from another context 
than the regular command execution context) but so far without success 
(see e.g. http://www.spinics.net/lists/target-devel/index.html#11822).

Bart.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html