RE: [RFC] XRC upstream merge reboot

Hefty, Sean
2011-06-23T18:03:50Z

> If you are thinking of using this for tracking, there is no way that I can
> see except reference counting to know when all users of the TGT have
> finished with it. Indicating usage by "tapping" the TGT QP whenever an
> SRQ receives a packet via the TGT is not a good idea (this is data
> path!), not to mention that quiescence is not a good indicator of whether
> an app is done using the target (the amount of time that things
> are quiet depends on the app).

I mentioned the event as a standard way for communicating the tgt qp to other processes using the xrcd. Currently, we have something like this:

Process 1:
create tgt qp
for each relevant process
send out of band message with tgt qpn

Process 2..n:
receive tgt qpn
kernel call to register with tgt qp
kernel searches for tgt qpn, bumps ref count
<later>
kernel call to unregister with tgt qp
kernel searches for tgt qpn, decrements ref count

The use of an event is simply an (incomplete) idea to standardize what is currently done by sending/receiving the out of band messages with the tgt qpn. On this note, I can't imagine that this usage model is better performing that simply creating a new xrcd for each job...

I was thinking about this more last night, in more generic terms, what we want is a way for an application to allocate a QP in the kernel and share it with other apps. I think it would be best to reuse the existing ibverbs API and kern-abi as much as possible. (Eventually someone may find a use for this other than just TGT QPs.) I just don't know how we handle sharing or transferring ownership of the CM data.

- Sean