Re: Problem with RDMA device removal architecture

Steve Wise
2010-03-26T17:42:50Z

Sean Hefty wrote:
>> In other words, I think we want the ucma context to stay around until
>> the application destroys it (via explicit means or via exit). But the
>> rdma_cm_id gets destroyed immediately upon receiving a DEVICE_REMOVE event.
>>
>
> Yes - this is what I was trying to say.
>
> The problem is trying to destroy the rdma_cm_id 'immediately' upon receiving a
> device remove event. The user could be in a separate downcall into the rdma_cm
> at the time the device removal occurs, and we cannot call rdma_destroy_id while
> calling rdma_foo() for the same id. Plus, 'foo' may be 'destroy_id', and we
> can't destroy the id twice.
>
> For the trivial case, if the rdma_cm_id is not in use, we just return -1 to the
> remove device callback to destroy the id. But if the rdma_cm_id is in use, then
> we must schedule the destruction of the rdma_cm_id to a separate thread and
> synchronize that thread against user space destroying the id before the thread
> can run.
>
>

I think you don't want to schedule another thread. The semantics of the
device removal and eeh events is that when the driver returns, all usage
of the device is stopped and it can be removed, reset, etc. Because of
this, these events are run on a thread so they can block if needed.

Steve.