Re: [RFC 0/4] RFC - Coherent Device Memory (Not for inclusion)

[Balbir Singh <bsingharora@gmail.com>]

On Thu, 2017-04-20 at 10:29 -0500, Christoph Lameter wrote:
> On Thu, 20 Apr 2017, Balbir Singh wrote:
> > Couple of things are needed
> > 
> > 1. Isolation of allocation
> 
> cgroups, memory policy and cpuset provide that
> 

Yes and we are building on top of mempolicies. The problem becomes a little
worse when the coherent device memory node is seen as CPUless node. I
was trying to solve 1 and 2 with the same approach.

> > 2. Isolation of certain algorithms like kswapd/auto-numa balancing
> 
> Ok that may mean adding some generic functionality to limit those

As in per-algorithm tunables? I think it would be definitely good to have
that. I do not know how well that would scale?

> 
> > > The approach sounds pretty invasive to me.
> > 
> > Could you please elaborate, you mean the user space programming bits?
> 
> No I mean the modification of the memory policies in particular. We are
> adding more exceptions to an already complex and fragile system.
> 
> Can we do this in a generic way just using hotplug nodes and some of the
> existing isolation mechanisms?
>

Yes, that was the first approach we tried and we are reusing whatever
we can -- HMM for driver driven migration, mempolicies for allocation
control and N_COHERENT_MEMORY for isolation because of 1 and 2 above
combined.
 
> 
> > Ideally we need the following:
> > 
> > 1. Transparency about being able to allocate memory anywhere and the ability
> > to migrate memory between coherent device memory and normal system memory
> 
> If it is a memory node then you have that already.
> 
> > 2. The ability to explictly allocate memory from coherent device memory
> 
> Ditto
> 
> > 3. Isolation of normal allocations from coherent device memory unless
> > explictly stated, same as (2) above
> 
> memory policies etc do that.
> 
> > 4. The ability to hotplug in and out the memory at run-time
> 
> hotplug code does that.
> 
> 
> > 5. Exchange pointers between coherent device memory and normal memory
> > for the compute on the coherent device memory to use

> 
> I dont see anything preventing that from occurring right now. Thats a
> device issue with doing proper virtual to physical mapping right?
> 

Some of these requirements come from whether we use NUMA or HMM-CDM.
We prefer NUMA and it meets the above requirements quite well.

> > I could list further things, but largely coherent device memory is like
> > system memory except that we believe that things like auto-numa balancing
> > and kswapd will not work well due to lack of information about references
> > and faults.
> 
> Ok so far I do not see that we need coherent nodes at all.
>

I presume you are suggesting this based on the fact that we add additional
infrastructure for auto-numa/kswapd/etc isolation?
 
> > Some of the mm-summit notes are at https://lwn.net/Articles/717601/
> > The goals align with HMM, except that the device memory is coherent. HMM
> > has a CDM variation as well.
> 
> I was at the presentation but at that point you were interested in a
> different approach it seems.

I do remember you were present, I don't think things have changed since then.

> 
> > We've been using the term coherent device memory (CDM). I could rephrase the
> > text and documentation for consistency. Would you prefer a different term?
> 
> Hotplug memory node?
> 

Normal memory is hotpluggable too.. but I'd be fine as long as everyone agrees

Balbir

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>