ibv_reg_mr call failed

ibv_reg_mr call failed

[Liu Ginhann]

We are running into an unexpected problem while registering memory for IB operation. Ibv_reg_mr() call return EFAULT - bad address. 

The pointer passed into ibv_reg_mr call is mmap back from memory allocated in our kernel driver. Our test code is able to use this mapped back virtual address to peek and poke without issue; the size is expected. This memory is a fixed size FIFO buffer designed to be used as our FPGA device buffer and IB transport buffer. There is host software to manage buffer usage. If we just malloc memory from app and pass it into ibv_reg_mr call, memory register call return ok.

It seems like for some reason, ibv_reg_mr doesn't like the pointer we passed in. Here is how we allocate and map the buffer inside the driver, 
. 4M memory is created and allocated using the kmem_cache_create, kmem_cache_alloc call during driver initialization.
. When mmap is called from user application at run-time, remap_pfn_range is called to map the buffer into the process's virtual address space. page_to_pfn(virt_to_page(buffer) is used to convert kernel virtual address into page and pfn.

We tried to change flags during allocation or mapping, but none of them work.

Any suggestion how to solve this problem?

Here is version information of our test chassis,
64-bit Suse Linux: 3.0.35-rt58-0.5-rt
OFED Info: MLNX_OFED_LINUX-1.5.3-3.1.0 (OFED-1.5.3-3.1.0)

Regards, 



Hank Liu



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ibv_reg_mr call failed

[Or Gerlitz]

On 24/05/2013 20:43, Liu Ginhann wrote:
> Here is version information of our test chassis,
what driver / card  are you using?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ibv_reg_mr call failed

[Liu Ginhann]

Mellanox connectx-3 mlx4. 


On May 25, 2013, at 11:59 PM, "Or Gerlitz" <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:

> On 24/05/2013 20:43, Liu Ginhann wrote:
>> Here is version information of our test chassis,
> what driver / card  are you using?
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ibv_reg_mr call failed

[Or Gerlitz]

On 24/05/2013 20:43, Liu Ginhann wrote:
> Ibv_reg_mr() call return EFAULT - bad address

Basically, AFAIK, the IB stack should support what you are trying to do, 
If you tell from which code e.g in libibverbs/libmlx4 or the kernel this 
value is returned, it might be helpful for further assisting you.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ibv_reg_mr call failed

[Liu Ginhann]

Will dig into it.Thanks.



On May 26, 2013, at 6:36 AM, "Or Gerlitz" <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:

> On 24/05/2013 20:43, Liu Ginhann wrote:
>> Ibv_reg_mr() call return EFAULT - bad address
> 
> Basically, AFAIK, the IB stack should support what you are trying to do, If you tell from which code e.g in libibverbs/libmlx4 or the kernel this value is returned, it might be helpful for further assisting you.
> 
> Or.
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ibv_reg_mr call failed

[Or Gerlitz]

On Sun, May 26, 2013 at 4:52 PM, Liu Ginhann
<Ginhann.Liu-Omk3PdybHIaN9aS15agKxg@public.gmane.org> wrote:
> Will dig into it.Thanks.



does this works if you use get_free_pages in the kernel instead of kmem_cache?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: ibv_reg_mr call failed

[Liu Ginhann]

Or,

Tried __get_free_pages, kmalloc, all behave the same - fail in ibv_reg_mr call but the va mapped back is fine for peek and poke. 

Any other suggestion? 

Hank 

> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Or Gerlitz
> Sent: Monday, May 27, 2013 12:53 AM
> To: Liu Ginhann
> Cc: Or Gerlitz; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: Re: ibv_reg_mr call failed
> 
> On Sun, May 26, 2013 at 4:52 PM, Liu Ginhann
> <Ginhann.Liu-Omk3PdybHIaN9aS15agKxg@public.gmane.org> wrote:
> > Will dig into it.Thanks.
> 
> 
> 
> does this works if you use get_free_pages in the kernel instead of
> kmem_cache?
> 
> Or.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the
> body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ibv_reg_mr call failed

[Or Gerlitz]

On Fri, May 31, 2013 at 2:05 AM, Liu Ginhann
<Ginhann.Liu-Omk3PdybHIaN9aS15agKxg@public.gmane.org> wrote:
> Or,
>
> Tried __get_free_pages, kmalloc, all behave the same - fail in ibv_reg_mr call but the va mapped back is fine for peek and poke.
>
> Any other suggestion?

Yes, tell from where the EFAULT  error originates

>
> Hank
>
>> -----Original Message-----
>> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
>> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Or Gerlitz
>> Sent: Monday, May 27, 2013 12:53 AM
>> To: Liu Ginhann
>> Cc: Or Gerlitz; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> Subject: Re: ibv_reg_mr call failed
>>
>> On Sun, May 26, 2013 at 4:52 PM, Liu Ginhann
>> <Ginhann.Liu-Omk3PdybHIaN9aS15agKxg@public.gmane.org> wrote:
>> > Will dig into it.Thanks.
>>
>>
>>
>> does this works if you use get_free_pages in the kernel instead of
>> kmem_cache?
>>
>> Or.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the
>> body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at
>> http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ibv_reg_mr call failed

[Or Gerlitz]

On 08/06/2013 19:42, Liu Ginhann wrote:
>> does this works if you use get_free_pages in the kernel instead of
>> kmem_cache?
>>
> I tried get_free_pages, kmalloc, kmem_cache_alloc. None of them work, it failed with the same error - EFAULT bad pointer.
>
> After code walk through, I believe it failed in ib_umem_get routine get_user_pages call. Below is the code snippet and it seems like the address point to is expect to be user space memory. If the comment is true, then that may be able to explain why ibv_reg_mr is not happy with remap address from kernel allocated memory but perfectly fine with malloc from user space.

Guys,  do you agree, will ib_umem_get always fail when provided memory 
which wasn't allocated @ user-space? why?

Or.

>
> If this is true. Do you think there is another way to accomplish this? It got to have some way to do this.
>
> You help is appreciated.
>
> Hank
>
> /**
>   * ib_umem_get - Pin and DMA map userspace memory.
>   * @context: userspace context to pin memory for
>   * @addr: userspace virtual address to start at
>   * @size: length of region to pin
>   * @access: IB_ACCESS_xxx flags for memory being pinned
>   * @dmasync: flush in-flight DMA when the memory region is written
>   */
> struct ib_umem *ib_umem_get(struct ib_ucontext *context, unsigned long addr,
> 			    size_t size, int access, int dmasync)
> {
> 	ret = 0;
> 	while (npages) {
> 		ret = get_user_pages(current, current->mm, cur_base,
> 				     min_t(unsigned long, npages,
> 					   PAGE_SIZE / sizeof (struct page *)),
> 				     1, !umem->writable, page_list, vma_list);
>
> 		if (ret < 0)
> 			goto out;
>
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html