Re: Status of "ummunot" branch?

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Wed, Jun 05, 2013 at 06:45:13PM +0000, Jeff Squyres (jsquyres) wrote:

> Hum.  I was under the impression that with today's code (i.e., not ODP), if you
> 
> a = malloc(N);
> ibv_reg_mr(..., a, N, ...);
> free(a);
> 
> (assuming that the memory actually left the process at free)
> 
> Then the relevant kernel verbs driver was notified, and would
> unregister that device.  ...but I'm an MPI guy, not a kernel guy --
> it seems like you're saying that my impression was wrong (which
> doesn't currently matter because we intercept free/sbrk and
> unregister such memory, anyway).

Sadly no, what happens is that once you do ibv_reg_mr that 'HCA
virtual address' is forever tied to the physical memory under the
'process virtual address' *at that moment* forever.

So in the case above, RDMA can continue after the free, and it
continues to hit the same *physical* memory that it always hit, but
due to the free the process has lost access to that memory (the kernel
keeps the physical memory reserved for RDMA purposes until unreg
though).

This is fundamentally why you need to intercept mmap/munmap/sbrk - if
the process's VM mapping is changed through those syscalls then the
HCA's VM and the process VM becomes de-synchronized.

> > 'magically be registered' is the wrong way to think about it - the
> > registration of VA=0x100 is simply kept, and any change to the
> > underlying physical mapping of the VA is synchronized with the HCA.
> 
> What happens if you:
> 
> a = malloc(N * page_size);
> ibv_reg_mr(..., a, N * page_size, ...);
> free(a);
> // incoming RDMA arrives targeted at buffer a

Haggai should comment on this, but my impression/expectation was
you'll get a remote protection fault/

> Or if you:
> 
> a = malloc(N * page_size);
> ibv_reg_mr(..., a, N * page_size, ...);
> free(a);
> a = malloc(N / 2 * page_size);
> // incoming RDMA arrives targeted at buffer a that is of length (N*page_size)

again, I expect a remote protection fault.

Noting of course, both of these cases are only true if the underlying
VM is manipulated in a way that makes the pages unmapped (eg
mmap/munmap, not free)

I would also assume that attempts to RDMA write read only pages
protection fault as well.

> It does seem quite odd, abstractly speaking, that a registration
> would survive a free/re-malloc (which is arguably a "different"
> buffer).

Not at all: the purpose of the registration is to allow access via
RDMA to a portion of the process's address space. The address space
doesn't change, but what it is mapped to can vary.

So - the ODP semantics make much more sense, so much so I'm not sure
we need a ODP flag at all, but that can be discussed when the patches
are proposed...

> That being said, it still seems like MPI needs a registration cache.
> It is several good steps forward if we don't need to intercept
> free/sbrk/whatever, but when MPI_Send(buf, ...) is invoked, we still
> have to check that the entire buf is registered.  If ibv_reg_mr(...,
> 0, 2^64, ...) was supported, that would obviate the entire need for
> registration caches.  That would be wonderful.

Yes, except that this shifts around where the registration overhead
ends up. Basically the HCA driver now has the registration cache you
had in MPI, and all the same overheads still exist. No free lunch
here :(

Haggai: A verb to resize a registration would probably be a helpful
step. MPI could maintain one registration that covers the sbrk
region and one registration that covers the heap, much easier than
searching tables and things.

Also bear in mind that all RDMA access protections will be disabled if
you register the entire process VM, the remote(s) can scribble/read
everything..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html