Re: [LSF/MM/BPF TOPIC] BoF VM live migration over CXL memory​

[Gregory Price <gregory.price@memverge.com>]

On Wed, Apr 12, 2023 at 10:38:04AM +0200, David Hildenbrand wrote:
> On 12.04.23 04:54, Huang, Ying wrote:
> > Gregory Price <gregory.price@memverge.com> writes:
> > 
> > > On Tue, Apr 11, 2023 at 02:37:50PM +0800, Huang, Ying wrote:
> > > > Gregory Price <gregory.price@memverge.com> writes:
> > > > 
> > > > [snip]
> > > > 
> > > > > 2. During the migration process, the memory needs to be forced not to be
> > > > >     migrated to another node by other means (tiering software, swap,
> > > > >     etc).  The obvious way of doing this would be to migrate and
> > > > >     temporarily pin the page... but going back to problem #1 we see that
> > > > >     ZONE_MOVABLE and Pinning are mutually exclusive.  So that's
> > > > >     troublesome.
> > > > 
> > > > Can we use memory policy (cpusets, mbind(), set_mempolicy(), etc.) to
> > > > avoid move pages out of CXL.mem node?  Now, there are gaps in tiering,
> > > > but I think it is fixable.
> > > > 
> > > > Best Regards,
> > > > Huang, Ying
> > > > 
> > > > [snip]
> > > 
> > > That feels like a hack/bodge rather than a proper solution to me.
> > > 
> > > Maybe this is an affirmative argument for the creation of an EXMEM
> > > zone.
> > 
> > Let's start with requirements.  What is the requirements for a new zone
> > type?
> 
> I'm stills scratching my head regarding this. I keep hearing all different
> kind of statements that just add more confusions "we want it to be
> hotunpluggable" "we want to allow for long-term pinning memory" "but we
> still want it to be movable" "we want to place some unmovable allocations on
> it". Huh?
> 
> Just to clarify: ZONE_MOVABLE allows for pinning. It just doesn't allow for
> long-term pinning of memory.
> 

I apologize for the confusion, this is my fault.  I had assumed that
since dax regions can't be pinned, subsequent nodes backed by a dax
device could not be pinned.  In testing this, this is not the case.

Re: long-term pinning, can you be more explicit as to what is considered
long-term?  Minutes? hours? days? etc

If a migration operation is considered short term, then pinning VM
memory during migration deals with this issue cleanly.

So walking back my statement - give my testing, i don't believe there's
a reason for a new zone.

> For good reason, because long-term pinning of memory is just the worst
> (memory waste, fragmentation, overcommit) and instead of finding new ways to
> *avoid* long-term pinnings, we're coming up with advanced concepts to
> work-around the fundamental property of long-term pinnings.
> 
> We want all memory to be long-term pinnable and we want all memory to be
> movable/hotunpluggable. That's not going to work.
> 
> If you'd ask me today, my prediction is that ZONE_EXMEM is not going to
> happen.
> 
> -- 
> Thanks,
> 
> David / dhildenb
>