Problem Pinning Physical Memory

Problem Pinning Physical Memory

[Brian]

ibv_reg_mr works fine as long as the shared region is in RAM. However, if I
change the memory region to the physical memory on a PCI device the call to
ibv_reg_mr fails. What should I be doing to register a physical memory region?

In searching through this forum I've seen some patches relating to
reg_io_mr/dereg_io_mr. However, I don't see those changes in the 1.5.2 ofed
sources. I had hoped to try applying the patches myself but the structure of the
unchanged code in the patches appears to be different than the 1.5.2 ofed
sources. Is there another, more recent, code base that I should be using?

Thank you in advance for your time and help! 

-Brian

PS. My apologies if this appears posted twice. I waited a couple of hours for
the original to post and it didn't - despite seeing later posts appear. 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem Pinning Physical Memory

[Eli Cohen]

On Mon, Nov 22, 2010 at 06:28:47PM +0000, Brian wrote:
> ibv_reg_mr works fine as long as the shared region is in RAM. However, if I
> change the memory region to the physical memory on a PCI device the call to
> ibv_reg_mr fails. What should I be doing to register a physical memory region?
> 
> In searching through this forum I've seen some patches relating to
> reg_io_mr/dereg_io_mr. However, I don't see those changes in the 1.5.2 ofed
> sources. I had hoped to try applying the patches myself but the structure of the
> unchanged code in the patches appears to be different than the 1.5.2 ofed
> sources. Is there another, more recent, code base that I should be using?
> 

You can't regiter IO memory using ibv_reg_mr() - you can only do that
by registering that memory using physical memory registration which is
available only for kernel consumers. You can't allow userspace to
register physical pages as this is a security breach.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem Pinning Physical Memory

[Jason Gunthorpe]

On Wed, Nov 24, 2010 at 03:55:07PM +0200, Eli Cohen wrote:
> On Mon, Nov 22, 2010 at 06:28:47PM +0000, Brian wrote:
> > ibv_reg_mr works fine as long as the shared region is in RAM. However, if I
> > change the memory region to the physical memory on a PCI device the call to
> > ibv_reg_mr fails. What should I be doing to register a physical memory region?
> > 
> > In searching through this forum I've seen some patches relating to
> > reg_io_mr/dereg_io_mr. However, I don't see those changes in the 1.5.2 ofed
> > sources. I had hoped to try applying the patches myself but the structure of the
> > unchanged code in the patches appears to be different than the 1.5.2 ofed
> > sources. Is there another, more recent, code base that I should be using?
> > 
> 
> You can't regiter IO memory using ibv_reg_mr() - you can only do that
> by registering that memory using physical memory registration which is
> available only for kernel consumers. 

> You can't allow userspace to register physical pages as this is a
> security breach.

Eh? If it is mmap'd into my processes address space then letting it be
used for RDMA makes no difference to the security of the system.

The last time this came up I said that the kernel side of ibv_reg_mr
should do the right thing for all types of memory that are mmap'd into
a process and I still think that is true. RDMA to device memory could
be very useful and with things like GEM managing the allocation of
device (video) memory to userspace, so it can be done safely.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem Pinning Physical Memory

[Steve Wise]

On 11/24/2010 11:42 AM, Jason Gunthorpe wrote:
>
> The last time this came up I said that the kernel side of ibv_reg_mr
> should do the right thing for all types of memory that are mmap'd into
> a process and I still think that is true. RDMA to device memory could
> be very useful and with things like GEM managing the allocation of
> device (video) memory to userspace, so it can be done safely.
>
> Jason
>    

Tom posted changes to support this a while back.  Tom, do you have an 
updated patch series for this support?

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem Pinning Physical Memory

[Tom Tucker]

On 11/29/10 11:10 AM, Steve Wise wrote:
>
>
> On 11/24/2010 11:42 AM, Jason Gunthorpe wrote:
>>
>> The last time this came up I said that the kernel side of ibv_reg_mr
>> should do the right thing for all types of memory that are mmap'd into
>> a process and I still think that is true. RDMA to device memory could
>> be very useful and with things like GEM managing the allocation of
>> device (video) memory to userspace, so it can be done safely.
>>
>> Jason
>
> Tom posted changes to support this a while back.  Tom, do you have an 
> updated patch series for this support?
>
Yes. I removed the new verb and followed Jason's recommendation of adding 
this support to the core reg_mr support. I used the type bits in the vma 
struct to determine the type of memory being registered and just did the 
"right thing."

I'll repost in the the next day or two.

Tom

> Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem Pinning Physical Memory

[Alan Cook]

> Tom Tucker <tom@...> writes:
> 
> Yes. I removed the new verb and followed Jason's recommendation of adding 
> this support to the core reg_mr support. I used the type bits in the vma 
> struct to determine the type of memory being registered and just did the 
> "right thing."
> 
> I'll repost in the the next day or two.
> 
> Tom
> 

Tom,

Couple of questions:

I noticed that OFED 1.5.3 was released last week.  Are the changes you speak of
part of that release? If not, is there an alternate branch/project that I should
be looking at or into to for the mentioned changes?

Also, I am inferring that the changes allowing the registering of physical
memory will only happen if my application is running in kernel space.  Is this
correct? or will I be able to register the physical memory from user space now
as well?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem Pinning Physical Memory

[Tom Tucker]

On 11/30/10 9:24 AM, Alan Cook wrote:
>> Tom Tucker<tom@...>  writes:
>>
>> Yes. I removed the new verb and followed Jason's recommendation of adding
>> this support to the core reg_mr support. I used the type bits in the vma
>> struct to determine the type of memory being registered and just did the
>> "right thing."
>>
>> I'll repost in the the next day or two.
>>
>> Tom
>>
> Tom,
>
> Couple of questions:
>
> I noticed that OFED 1.5.3 was released last week.  Are the changes you speak of
> part of that release?
No.
> If not, is there an alternate branch/project that I should
> be looking at or into to for the mentioned changes?
The patch will be against the top-of-tree Linux kernel.
> Also, I am inferring that the changes allowing the registering of physical
> memory will only happen if my application is running in kernel space.

Actually, no.

>   Is this
> correct? or will I be able to register the physical memory from user space now
> as well?
What I implemented was support for mmap'd memory. In practical terms for 
your application you would write a driver that supported the mmap file op. 
The driver's mmap routine would ioremap the pci memory of interest and 
stuff it in the provided vma. The user-mode app then ibv_reg_mr the 
address and length returned by mmap.

Make sense?
Tom

> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem Pinning Physical Memory

[Alan Cook]

Tom Tucker <tom@...> writes:

> What I implemented was support for mmap'd memory. In practical terms for 
> your application you would write a driver that supported the mmap file op. 
> The driver's mmap routine would ioremap the pci memory of interest and 
> stuff it in the provided vma. The user-mode app then ibv_reg_mr the 
> address and length returned by mmap.
> 
> Make sense?

I believe so.  Something along these lines has already been implemented for the
device with which I am working, but was failing due to lacking your patch.

I am not able to find the patch that you submitted (bad search terms?).  I also
am having a difficult time tracking down (with any certainty) the sources that
your patch would be made against.

If you could provide lines to both I would greatly appreciate it.

Thanks.

-- Alan


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem Pinning Physical Memory

[Or Gerlitz]

On 12/1/2010 5:30 PM, Alan Cook wrote:
> I am not able to find the patch that you submitted (bad search 
> terms?). I also am having a difficult time tracking down (with any 
> certainty) the sources that your patch would be made against.

Alan, kernel patches posted to this mailing list are against the 
mainline (upstream) Linux kernel, which you can get hold off through 
kernel.org, the mailing list is archived in various places, 
http://marc.info/?l=linux-rdma
from where I easily got to http://marc.info/?l=linux-rdma&m=128042070114898

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem Pinning Physical Memory

[Alan Cook]

On Thu, Dec 2, 2010 at 2:19 AM, Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
> Alan, kernel patches posted to this mailing list are against the mainline (upstream) Linux kernel

Thanks for pointing me to the correct kernel.

I found Tom's patch from July, but was unsure if that was the patch in
question or not.  It seems that he will have an updated version with
ibv_reg_mr() and not ibv_reg_io_mr() doing the registration, so I will
just wait for the update.

-- Alan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problem Pinning Physical Memory

[Brian]

Eli Cohen <eli@...> writes:
> You can't regiter IO memory using ibv_reg_mr() - you can only do that
> by registering that memory using physical memory registration which is
> available only for kernel consumers. You can't allow userspace to
> register physical pages as this is a security breach.

So, would ibv_reg_mr work with physical memory if run from a kernel module? (or
would something else have to be used instead). I have no requirement that this
run in userspace.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html