Re: [ofa-general] Re: Demand paging for memory regions

Patrick Geoffray
2008-02-13T03:59:26Z

Jason,

Jason Gunthorpe wrote:
> I don't know much about Quadrics, but I would be hesitant to lump it
> in too much with these RDMA semantics. Christian's comments sound like
> they operate closer to what you described and that is why the have an
> existing patch set. I don't know :)

The Quadrics folks have been doing RDMA for 10 years, there is a reason
why they maintained a patch.

> What it boils down to is that to implement true removal of pages in a
> general way the kernel and HCA must either drop packets or stall
> incoming packets, both are big performance problems - and I can't see
> many users wanting this. Enterprise style people using SCSI, NFS, etc
> already have short pin periods and HPC MPI users probably won't care
> about the VM issues enough to warrent the performance overhead.

This is not true, HPC people do care about the VM issues a lot. Memory
registration (pinning and translating) is usually too expensive to be
performed in the critical path before and after each send or receive. So
they factor it out by registering a buffer the first time it is used,
and keeping it registered in a registration cache. However, the
application may free() a buffer that is in the registration cache, so
HPC people provide their own malloc to catch free(). They also try to
catch sbrk() and munmap() to deregister memory before it is released to
the OS. This is a Major pain that a VM notifier would easily solve.
Being able to swap registered pages to disk or migrate them in a NUMA
system is a welcome bonus.

Patrick
--
Patrick Geoffray
Myricom, Inc.