Re: [PATCH] Document Linux Memory Policy - V2

Christoph Lameter
2007-07-31T19:10:37Z

On Tue, 31 Jul 2007, Lee Schermerhorn wrote:

> > Is it worth mentioning numactl here?
>
> Actually, I tried not to mention numactl by name--just that that APIs
> and headers reside in an "out of tree" package. This is a kernel doc
> and I wasn't sure about referencing out of tree "stuff".. Andi
> suggested that I not try to describe the syscalls in any detail [thus my
> updates to the man pages], and I removed that. But, I'll figure out a
> way to forward reference the brief API descriptions later in the doc.

numactl definitely must be mentioned because it is the user space API for
these things.

> > This appears to contradict the previous paragram. The last paragraph
> > would imply that the policy is applied to mappings that are mmaped
> > MAP_SHARED where they really only apply to shmem mappings.
>
> Conceptually, shared policies apply to shared "memory objects".
> However, the implementation is incomplete--only shmem/shm object
> currently support this concept. [I'd REALLY like to fix this, but am
> getting major push back... :-(]

The shmem implementation has bad semantics (affects other processes
that are unaware of another process redirecting its memory accesses) and
should not be extended to other types of object.

> > It's sufficent to say that MPOL_BIND will restrict the process to allocating
> > pages within a set of nodes specified by a nodemask because the end result
> > from the external observer will be similar.
>
> OK. But, I don't want to lose the idea that, with the BIND policy,
> pages will be allocated first from one of the nodes [lowest #] and then
> from the next and so on. This is important, because I've had colleagues
> complain to me that it was broken. They thought that if they bound a
> multithread application to cpus on several nodes and to the same nodes
> memories, they would get local allocation with fall back only to the
> nodes they specified. They really wanted cpuset semantics, but these
> were not available at the time.

Right. That is something that would be fixed if we could pass a nodemask
to alloc_pages.

> OK. I'll rework this entire section. Again, I don't want to lose what
> I think are important semantics for a user. And, maybe by documenting
> ugly behavior for all to see, we'll do something about it?

Correct. I hope you include the ugly shared shmem semantics with the
effect on unsuspecting processes?