Re: [RFC PATCH 2/4] uverbs: Add common ib_iomem_get service

[Tom Tucker <tom-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>]

On 7/29/10 3:13 PM, Jason Gunthorpe wrote:
> On Thu, Jul 29, 2010 at 02:07:11PM -0500, Tom Tucker wrote:
>    
>>>> Add an ib_iomem_get service that converts a vma to an array of
>>>> physical addresses. This makes it easier for each device driver to
>>>> add support for the reg_io_mr provider method.
>>>>          
> Tom, could you say a few words on what this is for?
>
>    

The proposed interface is there to allow a user-mode application to mmap 
kernel memory and/or device memory as a data source or sync for RDMA.

This is useful for a number of applications, for example:

- An application that wishes to provide kernel statistics to a 
monitoring host with zero overhead. For example, a set of compute nodes 
that are being monitored by a statistics gathering application (Ovis), 
but don't want to introduce scheduler jitter into the compute node.  The 
stats are provided in a kernel allocated buffer and the monitoring app 
'gathers' those statistics with RDMA_READ at zero cost to the monitored 
host.

- An application that wishes to provide network (RDMA) access to a large 
static memory hung off of it's PCI-e bus, thereby creating a distributed 
and persistent memory

Those are the only two that I've written so far that use these verbs.


> Also, I'd like to see a strong defence of this new user space API
> particularly:
>   1) Why can't this be done with the existing ibv_reg_mr, like huge
>      pages are.
>    

The ibv_reg_mr API assumes that the memory being registered was 
allocated in user mode and is part of the current->mm VMA. It uses 
get_user_pages which will scoff and jeer at kernel memory.

>   2) How is it possible for userspace to know when it should use
>      ibv_reg_mr vs ibv_reg_io_mr?
>    

By virtue of the device that it is mmap'ing. If I mmap my_vmstat_driver, 
I know that the memory I am mapping is a kernel buffer.

>   3) Why can't ibv_dereg_io_mr be entirely handled by ibv_dereg_mr?
>
>    
Probably could.

> On first glance, this seems like a hugely bad API to me :)
>
>    

Well hopefully now that it's purpose is revealed you will change your 
view and we can collaboratively make it better :-)

> Also, if new user space APIs are really needed, they do need man pages
> too ...
>
>    
Of course.

> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html