Re: [LSF/MM/BPF TOPIC][RFC PATCH v4 00/27] Private Memory Nodes (w/ Compressed RAM)

Gregory Price
2026-03-03T20:36:23Z

On Thu, Feb 26, 2026 at 02:27:24PM +1100, Alistair Popple wrote:
> On 2026-02-25 at 02:17 +1100, Gregory Price <gourry@gourry.net> wrote...
> >
> > If your service only allocates movable pages - your ZONE_NORMAL is
> > effectively ZONE_MOVABLE.
>
> This is interesting - it sounds like the conclusion of this is ZONE_* is just a
> bad abstraction and should be replaced with something else maybe some like this?
>
> And FWIW I'm not tied to the ZONE_DEVICE as being a good abstraction, it's just
> what we seem to have today for determing page types. It almost sounds like what
> we want is just a bunch of hooks that can be associated with a range of pages,
> and then you just get rid of ZONE_DEVICE and instead install hooks appropriate
> for each page a driver manages. I have to think more about that though, this
> is just what popped into my head when you start saying ZONE_MOVABLE could also
> disappear :-)
>
... snip ...
> >
> > You don't have to squint because it was deliberate :]
>
> Nice.
>

I've had some time to chew on this a bit more.

Adding a node-scope `struct dev_pagemap` produces some interesting
(arguably useful / valuable) effects.

The invariant would be clamping the entire node to ZONE_DEVICE
(more on this below).

So if we think about it this way - we could just view this whole thing
as another variant of ZONE_DEVICE - but without needing the memremap
infrastructure (you can use normal hotplug to achieve it).

0. pgdat->private becomes pgdat->dev_pagemap
N_MEMORY_PRIVATE -> N_MEMORY_DEVICE ?

As a start, do a direct conversion, and use the existing
infrastructure. then expand hooks as needed (and as-is reasonable)

Some of the `struct dev_pagemap {}` fields become dead at the node
scope, but this is a plumbing issue.

There's already an similar split between the dev_pagemap and the ops
structure, so it might map very cleanly.

1. "Clamping the entire node to ZONE_DEVICE"

When we do this, the *actual* ZONE becomes completely irrelevant.
The allocation path is entirely controlled, so you might actually end
up freeing up the folio flags that track the zone:

static inline enum zone_type memdesc_zonenum(memdesc_flags_t flags)
{
ASSERT_EXCLUSIVE_BITS(flags.f, ZONES_MASK << ZONES_PGSHIFT);
return (flags.f >> ZONES_PGSHIFT) & ZONES_MASK;
}

becomes:

folio_is_zone_device(folio) {
return node_is_device_node(folio_nid(folio)) ||
memdesc_is_zone_device(folio->flags);
}

Kind of an interesting. You still need these flags for traditional
ZONE_DEVICE, so you can't evict it completely, but you can start to
see a path here.

2. One dev_pagemap per node or multiple w/ pagemap range searching

Checking membership is always cheap:

node_is_device_node()

Getting ops can be cheap if 1:1 mappings exists:

pgdat->device_ops->callback()

Or may be expensive if range-based matching is required:

node_device_op(folio, ...) {
ops = node_ops_lookup(folio); /* pfn-range binary search */
ops->callback(folio, ...)
}

pgmap already has an embedded range:

struct dev_pagemap {
...
int nr_range;
union {
struct range range;
DECLARE_FLEX_ARRAY(struct range, ranges);
};
};

Example: Nouveau, registers hundreds of pgmap instances that it
uses to recover driver contexts for that specific folio.

This would not scale well.

But most other drivers register between 1-8. That might.

That means this might actually be an effective way to evict pgmap
from struct folio / struct page. (Not making this a requirement or
saying it's reasonable, just an interesting observation).

3. Some existing drivers with 1 pgmap per driver instance instantly get
the folio->lru field back - even if they continue to use ZONE_DEVICE.

At least 3 drivers use page->zone_device_data as a page freelist
rather than actual per-page data. Those drivers could just start
using folio/page->lru instead.

Some store actual per-page zone_device_data that would prevent this,
but from poking around it seems like it might be feasible.

Some use the pgmap as a container_of() argument to get driver
context, may or may not be supportable out of the box, but it seemed
like mild refactoring might get them back the use of folio->lru.

None of this is required, the goal is explicitly not disrupting any
current users of ZONE_DEVICE.

Just some additional food for thought.

As-designed now, this would only apply to NUMA systems, meaning you
can't fully evict pgmap from struct page/folio --- but you could
imagine a world where in non-numa mode we even register a separate
pglist_data specifically for device memory even w/o NUMA.

~Gregory