Re: rdma provider module references

Steve Wise
2010-12-15T18:53:40Z

On 12/15/2010 11:09 AM, Roland Dreier wrote:
> > I notice that if I have a user rdma application running that has an
> > rdma connection using iw_cxgb3, then the iw_cxgb3 module reference
> > count is bumped and thus it cannot be unloaded. However when I have
> > an NFSRDMA connection that utilizes iw_cxgb3, the module reference
> > count is not bumped, and iw_cxgb3 can erroneously be unloaded while
> > the NFSRDMA connection is still active, causing a crash.
>
> What is supposed to happen is that as the HW driver is unloading, it
> calls ib_unregister_device() first, and this calls each client's
> .remove() method to have it release everything related to that device.
>
> However I guess NFS/RDMA is behind the RDMA CM, which is supposed to
> handle device removal. In that code it seems to end up in
> cma_process_remove(), which appears at first glance to do the right
> things to destroy all connections etc.
>
> The idea is that RDMA devices should be like net devices, ie you can
> remove them even if they're in use -- things should just clean up,
> rather than blocking the module removal. The uverbs case is a bit of a
> hack because we don't have a way to handle revoking the mmap regions
> etc yet.
>

Thanks for the description of how it should work.

> What goes wrong with NFS/RDMA in this scheme? It looks like it should work.
>

I'm still investigating exactly what happens. I'll follow up once I know more.

Thanks!