Re: rdma provider module references

Steve Wise
2010-12-16T15:34:22Z

On 12/15/2010 11:09 AM, Roland Dreier wrote:
> > I notice that if I have a user rdma application running that has an
> > rdma connection using iw_cxgb3, then the iw_cxgb3 module reference
> > count is bumped and thus it cannot be unloaded. However when I have
> > an NFSRDMA connection that utilizes iw_cxgb3, the module reference
> > count is not bumped, and iw_cxgb3 can erroneously be unloaded while
> > the NFSRDMA connection is still active, causing a crash.
>
> What is supposed to happen is that as the HW driver is unloading, it
> calls ib_unregister_device() first, and this calls each client's
> .remove() method to have it release everything related to that device.
>
> However I guess NFS/RDMA is behind the RDMA CM, which is supposed to
> handle device removal. In that code it seems to end up in
> cma_process_remove(), which appears at first glance to do the right
> things to destroy all connections etc.
>
> The idea is that RDMA devices should be like net devices, ie you can
> remove them even if they're in use -- things should just clean up,
> rather than blocking the module removal. The uverbs case is a bit of a
> hack because we don't have a way to handle revoking the mmap regions
> etc yet.
>
> What goes wrong with NFS/RDMA in this scheme? It looks like it should work.
>

Here's one stack. From this I assume the offload connection was still active after iw_cxgb3 was unloaded...

Call Trace:
<IRQ> [<ffffffff80037136>] kref_get+0x38/0x3d
[<ffffffff885fb5b1>] :iw_cxgb3:sched+0x17/0x49
[<ffffffff8824cf37>] :cxgb3:process_rx+0x37/0x8b
[<ffffffff8824a3e7>] :cxgb3:process_responses+0xc09/0xc63
[<ffffffff8824ac65>] :cxgb3:napi_rx_handler+0x36/0xa4
[<ffffffff8000c88a>] net_rx_action+0xac/0x1e0
[<ffffffff8824ac15>] :cxgb3:t3_sge_intr_msix_napi+0x173/0x18d
[<ffffffff80012409>] __do_softirq+0x89/0x133
[<ffffffff8005f2fc>] call_softirq+0x1c/0x28
[<ffffffff8006dba8>] do_softirq+0x2c/0x85
[<ffffffff8006da30>] do_IRQ+0xec/0xf5
[<ffffffff800575d0>] mwait_idle+0x0/0x4a
[<ffffffff8005e615>] ret_from_intr+0x0/0xa
<EOI> [<ffffffff80057606>] mwait_idle+0x36/0x4a
[<ffffffff800497be>] cpu_idle+0x95/0xb8
[<ffffffff80078997>] start_secondary+0x498/0x4a7