Re: [PATCH 2/2] Revert "mm, thp: restore node-local hugepage allocations"

David Rientjes
2019-05-31T21:53:40Z

On Fri, 31 May 2019, Michal Hocko wrote:

> > The problem which this patch addresses has apparently gone unreported for
> > 4+ years since
>
> Can we finaly stop considering the time and focus on the what is the
> most reasonable behavior in general case please? Conserving mistakes
> based on an argument that we have them for many years is just not
> productive. It is very well possible that workloads that suffer from
> this simply run on older distribution kernels which are moving towards
> newer kernels very slowly.
>

That's fine, but we also must be mindful of users who have used
MADV_HUGEPAGE over the past four years based on its hard-coded behavior
that would now regress as a result.

> > commit 077fcf116c8c2bd7ee9487b645aa3b50368db7e1
> > Author: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
> > Date: Wed Feb 11 15:27:12 2015 -0800
> >
> > mm/thp: allocate transparent hugepages on local node
>
> Let me quote the commit message to the full lenght
> "
> This make sure that we try to allocate hugepages from local node if
> allowed by mempolicy. If we can't, we fallback to small page allocation
> based on mempolicy. This is based on the observation that allocating
> pages on local node is more beneficial than allocating hugepages on remote
> node.
>
> With this patch applied we may find transparent huge page allocation
> failures if the current node doesn't have enough freee hugepages. Before
> this patch such failures result in us retrying the allocation on other
> nodes in the numa node mask.
> "
>
> I do not see any single numbers backing those claims or any mention of a
> workload that would benefit from the change. Besides that, we have seen
> that THP on a remote (but close) node might be performing better per
> Andrea's numbers. So those claims do not apply in general.
>

I confirm that on every platform I have tested that the access latency to
local pages of the native page size has been less than hugepages on any
remote node. I think it's generally accepted that NUMA-ness is more
important than huge-ness in terms of access latency and this is not the
reason why the revert is being proposed. Certainly if the argument is to
be made that the default behavior should be what is in the best interest
of Linux users in totality, preferring remote hugepages over local pages
of the native page size would not be anywhere close. I agree with
Aneesh's commit message 100%.

> > My goal is to reach a solution that does not cause anybody to incur
> > performance penalties as a result of it.
>
> That is certainly appreciated and I can offer my help there as well. But
> I believe we should start with a code base that cannot generate a
> swapping storm by a trivial code as demonstrated by Mel. A general idea
> on how to approve the situation has been already outlined for a default
> case and a new memory policy has been mentioned as well but we need
> something to start with and neither of the two is compatible with the
> __GFP_THISNODE behavior.
>

Thus far, I haven't seen anybody engage in discussion on how to address
the issue other than proposed reverts that readily acknowledge they cause
other users to regress. If all nodes are fragmented, the swap storms that
are currently reported for the local node would be made worse by the
revert -- if remote hugepages cannot be faulted quickly then it's only
compounded the problem.

The hugepage aware mempolicy idea is one way that could describe what
should be done for these allocations, we could also perhaps consider
heuristics that consider the memory pressure of the local node: just as
I've never seen a platform where remote hugepages have better access
latency than local pages, I've never seen a platform where remote
hugepages aren't a win over remote pages. This, however, is more delicate
on 4 socket and 8 socket platforms but I think a general rule that a
hugepage is better, if readily allocatable, than a page on the same node.
(I've seen cross socket latency for hugepages match the latency for pages,
so not always a win: better to leave the hugepage available remotely for
something running on that node.) If the local node has a workingset that
reaches its capacity, then it makes sense to fault a remote hugepage
instead because otherwise we are thrashing the local node.

That's not a compaction problem, though, it's a reclaim problem. If
compaction fails and it's because we can't allocate target pages, it's
under memory pressure and it's uncertain if reclaim will help: it may fail
after expensive swap, the reclaimed pages could be grabbed by somebody
else and we loop, or the compaction freeing scanner can't find it. Worst
case is we thrash the local node in a swap storm. So the argument I've
made when the removal of __GFP_THISNODE was first proposed is that local
hugepage allocation should be the preference including direct compaction
for users of MADV_HUGEPAGE (or thp enabled=always) but reclaim should
never be done locally. I'd very much like to engage with anybody who
would be willing to discuss fixes that work for everybody rather than only
propose reverts and leave others to deal with new performance regressions.