Re: [PATCH][RFC] Linux VM hooks for advanced RDMA NICs

[Brice Goglin <Brice.Goglin@ens-lyon.org>]

David Addison a écrit :
> Hi,
> 
> here is a patch we use to integrate the Quadrics NICs into the Linux 
> kernel.
> The patch adds hooks to the Linux VM subsystem so that registered 'IOPROC'
> devices can be informed of page table changes.
> This allows the Quadrics NICs to perform user RDMAs safely, without 
> requiring
> page pinning. Looking through some of the recent IB and Ammasso 
> discussions,
> it may also prove useful to those NICs too.

Hi,

I worked on a similar patch to help updating a registration cache on
Myrinet. I came to the problem of deciding between registering ioproc
to the entire address space (1) or only to some VMA (2).
You're doing (1), I tried (2).

(2) avoids calling ioproc hooks for all pages that are never involved
in any communication. This might be good if the amount of pages that
are involved is not too high and if the coproc_ops cost is a little bit
high.
Do you have any numbers about this in real applications on QsNet ?

I see two drawback in (2).
First, it requires to play with the list of ioproc_ops when VMA are
merged or split. Actually, it's not that bad since the list often
contains only 1 ioproc_ops.
Secondly, you have to add the ioproc to all involved VMA at some point.
It's easy when the API asks the application to register, you just add
the ioproc_ops to the target VMA during registration. But, I guess it's
not easy with Quadrics, right ?


I see in your patch that ioproc are not inherited during fork.
How do you support fork in your driver/lib then ?
What if a COW page is given to the son and the copy to the father
while some IO are being processed ? Do you require the application to
call a specific routine after forking ?
Don't you think it might be good to add a hook in the fork code
so that ioproc are inherited or duplicated pages are invalidated
in the card ?

Regards,
Brice