Re: [PATCH] mm/hmm: skip NUMA balancing on protnone faults

["Thomas Hellström" <thomas.hellstrom@linux.intel.com>]

On Fri, 2026-11-13 at 21:43 +0100, David Hildenbrand (Arm) wrote:
> On 11/13/26 21:34, Thomas Hellström wrote:
> > Hi, David
> > 
> > On Fri, 2026-11-13 at 19:27 +0100, David Hildenbrand (Arm) wrote:
> > > On 11/13/26 17:25, Matthew Auld wrote:
> > > > When automatic NUMA balancing marks a PTE or PMD as protnone
> > > > (PAGE_NONE), hmm_pte_need_fault() requests a fault to restore
> > > > access
> > > > permissions, including _PAGE_RW when write access is needed.
> > > > 
> > > > Calling handle_mm_fault() from hmm_do_fault() currently
> > > > triggers
> > > > NUMA
> > > > balancing in do_numa_page() and do_huge_pmd_numa_page(). When
> > > > hmm_range_fault() runs from a device driver or kworker context
> > > > (such as
> > > > a GPU SVM or userptr rebind worker), folios get migrated toward
> > > > the
> > > > CPU
> > > > node executing the remote fault.
> > > > 
> > > > Also, each folio migration invokes try_to_migrate_one() ->
> > > > mmu_notifier_invalidate_range_start(), which invalidates the
> > > > caller's mmu_interval_notifier while holding mmap_lock. That
> > > > causes
> > > > hmm_range_fault() to abort with -EBUSY after every 2 MiB PMD
> > > > and
> > > > exhaust
> > > > its retry timeout on large buffers, which has been observed
> > > > with
> > > > large
> > > > imported Xe userptr buffers.
> > > > 
> > > > Add FAULT_FLAG_NO_NUMA and set it in hmm_do_fault() so protnone
> > > > entries
> > > > restore PTE/PMD permissions in-place without performing NUMA
> > > > balancing,
> > > > matching GUP's default !FOLL_HONOR_NUMA_FAULT behaviour.
> > > 
> > > Ehm, that's not how it works.
> > > 
> > > GUP will not trigger a "special" fault and will try not
> > > triggering a
> > > fault and
> > > not restore permissions.
> > > 
> > > It will try continuing, leaving the protnone pte as it is.
> > > 
> > > See gup_can_follow_protnone().
> > > 
> > > For read-only access that should always work.
> > > 
> > > For write-access we always trigger an ordinary fault and do NUMA
> > > migration etc.
> > > 
> > > 
> > > So FAULT_FLAG_NO_NUMA is doing something completely different and
> > > I
> > > don't agree
> > > with it.
> > > 
> > 
> > There are two major problems with the current behaviour.
> > 
> > 1) The work item executing hmm_range_fault() might have no affinity
> > at
> > all to the GPU about to access the memory.
> > 
> > 2) If migration happens, then hmm_range_fault() will stop and retry
> > leading to excessive execution time.
> > 
> > While perhaps Matt's commit message doesn't correctly describe the
> > GUP
> > behaviour, do you have any alternative suggestion than to avoid
> > migration in this case? I think it's a reasonable idea given the
> > problems with the current code?
> 
> Adding fault reasons that hack around NUMA hinting such that a actual
> CPU
> accessing it later will not trigger ... NUMA hinting?
> 
> I might be missing the "reasonable" in here :P

Wouldn't the net effect be that content stays put instead of being
migrated? In both cases a later CPU access will not trigger NUMA
hinting except that the proposed change doesn't send content to an
irrelevant node?

> 
> I recall that John (cc) kept saying "GPUs and NUMA balancing is
> inherently
> incompatible, you have to disable NUMA balancing".

Perhaps we could force hmm_range_fault() to run on the node closest to
the GPU. Then at least the migration might be beneficial.

> 
> You could likely implement something similar to GUP, where you would
> allow for
> bypassing protnone for reading.

We could ofc look at that. Might help in some cases.

Thanks,
Thomas


> 
> That wouldn't give you access to the writing side. There might be
> ways to sort
> out some of the writing (similar to FOLL_FORCE handling in GUP), but
> I'd rather
> not go down that path. KVM also doesn't seem to need that.