Naïve question wrt ibv_reg_mr(), pinned pages and Linux VM

[Tim Wright <tim.wright-c+vHNXOUHMmEK/hMebVsMw@public.gmane.org>]

Hello everybody,
I have a question regarding behavior of the memory pinning using
ibv_reg_mr() on RHEL/Centos 5.3/OFED-1.4.1. This may be more a Linux
feature/issue, and I apologize if that is the case. Here is the scenario

1. An application runs which allocates a significant fraction of the system
memory as RDMA buffers (e.g. 3GB of ram on a 4GB system). These are setup
using ibv_reg_mr(). It is clear that the pages are pinned from the kernel
perspective. With just this program running, the resident set size of the
program approaches the allocation size.
2. If other memory-intensive processes are now started, the resident set
size of the RDMA-using program shrinks dramatically.
3. Even if the other memory-intensive programs are stopped, and the
RDMA-using program is forced to read its memory, the resident set never
grows to a ³reasonable² size again.

My potentially foolish assumptions are/were that:
i) Since the memory is pinned anyway, it would be locked into the process
address space, and
ii) even if that were not the case, that the process would be able to regain
a large RSS when any competing processes stopped.

For ii), it almost seems that the VM doesn¹t realize that the pages it would
be grabbing back are already resident and therefore won¹t actually take any
more memory.

Clearly, I can use mlock() to avoid the issue, but I was wondering if I have
missed something obvious here. Any clues/brickbats gratefully received!

Regards,

Tim Wright

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html