Re: [RFC PATCH] mm/page_alloc: Add PCP list for THP CMA

[David Hildenbrand <david@redhat.com>]

On 05.08.25 18:57, Juan Yescas wrote:
> On Tue, Aug 5, 2025 at 2:58 AM David Hildenbrand <david@redhat.com> wrote:
>>
>> On 05.08.25 03:22, Juan Yescas wrote:
>>> On Mon, Aug 4, 2025 at 11:50 AM David Hildenbrand <david@redhat.com> wrote:
>>>>
>>>> On 04.08.25 20:20, Juan Yescas wrote:
>>>>> Hi David/Zi,
>>>>>
>>>>> Is there any reason why the MIGRATE_CMA pages are not in the PCP lists?
>>>>>
>>>>> There are many devices that need fast allocation of MIGRATE_CMA pages,
>>>>> and they have to get them from the buddy allocator, which is a bit
>>>>> slower in comparison to the PCP lists.
>>>>>
>>>>> We also have cases where the MIGRATE_CMA memory requirements are big.
>>>>> For example, GPUs need MIGRATE_CMA memory in the ranges of 30MiB to 500MiBs.
>>>>> These cases would benefit if we have THPs for CMAs.
>>>>>
>>>>> Could we add the support for MIGRATE_CMA pages on the PCP and THP lists?
>>>>
>>>> Remember how CMA memory is used:
>>>>
>>>> The owner allocates it through cma_alloc() and friends, where the CMA
>>>> allocator will try allocating *specific physical memory regions* using
>>>> alloc_contig_range(). It doesn't just go ahead and pick a random CMA
>>>> page from the buddy (or PCP) lists. Doesn't work (just imagine having
>>>> different CMA areas etc).
>>>>
>>>> Anybody else is free to use CMA pages for MOVABLE allocations. So we
>>>> treat them as being MOVABLE on the PCP.
>>>>
>>>> Having a separate CMA PCP list doesn't solve or speedup anything, really.
>>>>
>>>
>>> Thanks David for the quick overview.
>>>
>>>> I still have no clue what this patch here tried to solve: it doesn't
>>>> make any sense.
>>>>
>>>
>>> The story started with this out of tree patch that is part of Android.
>>>
>>> https://lore.kernel.org/lkml/cover.1604282969.git.cgoldswo@codeaurora.org/T/#u
>>>
>>> This patch introduced the __GFP_CMA flag that allocates pages from
>>> MIGRATE_MOVABLE
>>> or MIGRATE_CMA. What it happens then, it is that the MIGRATE_MOVABLE
>>> pages in the
>>> PCP lists were consumed pretty fast. To solve this issue, the PCP
>>> MIGRATE_CMA list was added.
>>> This list is initialized by rmqueue_bulk() when it is empty. That's
>>> how we end up with the PCP MIGRATE_CMA list
>>> in Android. In addition to this, the THP list for MIGRATE_MOVABLE was
>>> allowed to contain
>>> MIGRATE_CMA pages. This is causing THP MIGRATE_CMA pages to be used
>>> for THP MIGRATE_MOVABLE
>>> making later allocations from THP MIGRATE_CMA to fail.
>>
>> Okay, so this patch here really is not suitable for the upstream kernel
>> as is. It's purely targeted at the OOT Android patch.
>>
> Right, it is a temporary solution for the pinned MIGRATE_CMA pages.
> 
>>>
>>> These workarounds are mainly because we need to solve this issue upstream:
>>>
>>> - When devices reserve big blocks of MIGRATE_CMA pages, the
>>> underutilized MIGRATE_CMA
>>> can fall back to MIGRATE_MOVABLE and these pages can be pinned, so if
>>> we require MIGRATE_CMA
>>> pages, the allocations might fail.
>>>
>>> I remember that you presented the problem in LPC. Were you able to
>>> make some progress on that?
>>
>> There is the problem of CMA pages getting allocated by someone for a
>> MOVABLE allocation, to then short-term pin it for DMA. Long-term
>> pinnings are disallowed (we just recently fixed a bug where we
>> accidentally allowed it).
>>
> Nice, it is great the issue got caught and fixed upstream :)
> 
>> One concern is that a steady stream of short-term pinnings can turn such
>> pages unmovable. We discussed ideas on how to handle that, but there is
>> no solution upstream yet.
> 
> Are there any plans to continue the discussion? Is it in the priority
> list?

Ohh, it's somewheeeeeere on the todo list :)

Do you (or one of your colleagues) have capacity to work on that? One 
idea was to flag folios as "pending on migration" and disallow any 
further short-term pins until migration is done. IIRC, there were other 
ideas (e.g., isolated pageblock).

> Maybe
> a topic we can discuss in LPC Japan?

Sounds good, feel free to propose this as a topic. I wills end out the 
announcement of the MM MC probably next week.

-- 
Cheers,

David / dhildenb