Re: [PATCH] Revert "perf/aux: Allocate non-contiguous AUX pages by default"

[Dev Jain <dev.jain@arm.com>]

On 04/11/26 2:16 am, Yabin Cui wrote:
> [Resending in plain text for the mailing lists; sorry for the duplicate.]
> 
> Hi Leo, Will, Dev,
> 
> On Sun, Oct 25, 2026 at 04:24:06PM +0000, Leo Yan wrote:
>> The original commit claimed that contiguous AUX pages unnecessarily
>> exacerbate fragmentation for SPE, TRBE and ETR. This is not generally
>> valid: larger allocations can keep pages with the same lifetime together,
>> while order-0 allocations can scatter them across large blocks and prevent
>> free pages from coalescing [1]. The follow-up discussion [2] has not
>> established how forcing order-0 allocations addresses the reported problem.
> ...
>> Restore the previous policy of trying higher-order allocations first.
>> The allocator already uses __GFP_NORETRY and falls back to smaller
>> orders, including order 0, when larger allocations fail.
> 
> Sorry for the late reply, and sorry that the commit message in
> 18049c8cff9c ("perf/aux: Allocate non-contiguous AUX pages by default")
> was imprecise about "exacerbating memory fragmentation". You and Dev are
> right that grouping same-lifetime pages into higher-order allocations
> does not itself worsen fragmentation, and the TLB reduction numbers for
> TRBE and SPE with large vmap() mappings make total sense given TRBE's
> tiny TLB.
> 
> However, a straight revert will reintroduce a severe production issue we
> hit on Android devices, so I'd like to share what actually happened and
> suggest a way to get large mappings for SPE/TRBE without bringing back
> the regression.
> 
> ### What happened in production (even with __GFP_NORETRY)
> 
> On Android Pixel dogfood devices, a background daemon (profcollectd)
> periodically records short (1s) system-wide CoreSight ETM traces every
> ~10 minutes for AutoFDO, using a 4 MiB AUX buffer per CPU with default
> aux_watermark = 0 (so watermark = 2 MiB, max_order = 9).
> 
> Although PERF_AUX_GFP includes __GFP_NORETRY, it is based on GFP_KERNEL,
> which retains __GFP_DIRECT_RECLAIM | __GFP_KSWAPD_RECLAIM. __GFP_NORETRY
> only avoids retry loops and the kernel OOM killer at the end of
> __alloc_pages_slowpath(). On long-running phones where physical memory is
> already fragmented and order 9..4 pages are unavailable, every failed
> high-order call in rb_alloc_aux_page() still enters the slowpath and:
> 
>   1. Wakes kswapd/kcompactd at that high order.
>   2. Runs synchronous direct compaction (__alloc_pages_direct_compact()).
>   3. Runs synchronous direct reclaim (__alloc_pages_direct_reclaim()),
>      evicting pages and raising memory PSI pressure. On Android, PSI
>      reclaim stalls trigger userspace lmkd (Low Memory Killer daemon)
>      even though the kernel OOM killer is never invoked.
> 
> This is amplified by two loops in rb_alloc_aux():
>   - rb_alloc_aux_page() steps down one order at a time (9, 8, ..., 0),
>     hitting direct compaction/reclaim at each unavailable order.
>   - Worse, the outer loop in rb_alloc_aux() never lowers max_order when
>     rb_alloc_aux_page() falls back:
> 
>       order = min(max_order, ilog2(nr_pages - rb->aux_nr_pages));
>       page = rb_alloc_aux_page(node, order);
> 
>     When a 4 MiB buffer (1024 pages) only finds order-0 pages, it retries
>     orders 9..1 on almost every iteration -- thousands of failed
>     high-order slowpath compaction/reclaim attempts for a single
>     perf_mmap() call.
> 
> In field telemetry and local ftrace on Pixel devices (prior to
> 18049c8cff9c, so __GFP_NORETRY was already in place), we observed heavy
> rb_alloc_aux() -> __alloc_pages_direct_compact() stalls alongside
> kcompactd0 pinned on big cores scanning order-9, and this accounted for
>> 20% of all perceptible app kills (via lmkd) across dogfood devices
> until we stopped high-order AUX allocations.
> 
> Initially we worked around this in userspace (simpleperf) by setting
> perf_event_attr::aux_watermark = 4096 so get_order(watermark) == 0. That
> was fine for CoreSight ETR (which uses its own separate scatter-gather
> buffer and only copies into AUX on flush/stop), but when enabling TRBE
> (and SPE) it caused hardware maintenance interrupts every 4 KiB instead
> of every 2 MiB. Also, in overwrite/snapshot mode rb_alloc_aux() ignores
> aux_watermark and uses ilog2(nr_pages). That is why I sent 18049c8cff9c.
> 
> ### Possible solutions
> 
> For PMUs without PERF_PMU_CAP_AUX_NO_SG (such as SPE, TRBE, and modern
> Intel PT), higher-order AUX pages are an opportunistic optimization
> rather than a functional requirement -- very similar to THP allocations
> on fault (GFP_TRANSHUGE_LIGHT) or opportunistic high-order slabs in SLUB
> (allocate_slab()).
> 
> Instead of reverting to GFP_KERNEL reclaim/compaction for all high-order
> attempts, could we do the following?
> 
> 1. Make opportunistic high-order AUX allocations no-reclaim:
>    When !(event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG) and
>    order > 0, clear __GFP_RECLAIM (and add __GFP_NOMEMALLOC, matching
>    SLUB's allocate_slab() and GFP_TRANSHUGE_LIGHT):
> 
>      gfp_t gfp = PERF_AUX_GFP;
> 
>      if (order && !has_no_sg)
>              gfp = (PERF_AUX_GFP | __GFP_NOMEMALLOC) & ~__GFP_RECLAIM;

Thanks for your reply.

This looks like the correct solution. Note commit 0e05a469c294
("mm/page_alloc: avoid direct reclaim and compaction for costly __GFP_NORETRY allocations")
post which the regression you report should reduce drastically. It only
removes GFP_DIRECT_RECLAIM flag, I assume you are seeing latency even
due to reclaim/compaction daemon wakeups.

So instead of enhancing this commit even more by making a page-allocator
change, I would rather make your proposed change on the perf side to
keep it simple. This should completely solve the problem.


> 
>    When order falls back to 0 (or when PERF_PMU_CAP_AUX_NO_SG is set),
>    keep PERF_AUX_GFP so single-page allocations still reclaim normally
>    under memory pressure.
> 
>    That way, when free order-9 / order-4 blocks exist, SPE and TRBE get
>    them immediately for PMD / contiguous-PTE vmap() mappings (and pages
>    stay grouped together); when memory is fragmented on a long-running
>    device, high-order probes fail fast on the freelist without waking
>    kcompactd/kswapd or triggering direct compaction and reclaim.
> 
> 2. Clamp max_order in rb_alloc_aux() after fallback:
>    If rb_alloc_aux_page() requested `order` and had to fall back to a
>    smaller order, update `max_order = page_private(page)` in
>    rb_alloc_aux() so subsequent iterations for the rest of the buffer do
>    not keep re-probing higher orders that just failed.
> 
> (Optionally, if server/benchmark workloads ever want synchronous
> compaction to form huge AUX mappings even on fragmented hosts, that could
> be gated behind an opt-in attr/sysfs knob similar to THP defrag modes,
> with light/no-reclaim as the default.)
> 
> Would this approach work for your SPE/TRBE use cases? Happy to help test
> on our side.
> 
> Thanks,
> Yabin
> 
> 
> On Tue, Nov 3, 2026 at 12:32 PM Yabin Cui <yabinc@google.com> wrote:
>>
>> Hi Leo, Will, Dev,
>>
>> Sorry for the late reply, and sorry that the commit message in
>> 18049c8cff9c ("perf/aux: Allocate non-contiguous AUX pages by default")
>> was imprecise about "exacerbating memory fragmentation". You and Dev are
>> right that grouping same-lifetime pages into higher-order allocations
>> does not itself worsen fragmentation, and the TLB reduction numbers for
>> TRBE and SPE with large vmap() mappings make total sense given TRBE's
>> tiny TLB.
>>
>> However, a straight revert will reintroduce a severe production issue we
>> hit on Android devices, so I'd like to share what actually happened and
>> suggest a way to get large mappings for SPE/TRBE without bringing back
>> the regression.
>>
>> ### What happened in production (even with __GFP_NORETRY)
>>
>> On Android Pixel dogfood devices, a background daemon (profcollectd)
>> periodically records short (1s) system-wide CoreSight ETM traces every
>> ~10 minutes for AutoFDO, using a 4 MiB AUX buffer per CPU with default
>> aux_watermark = 0 (so watermark = 2 MiB, max_order = 9).
>>
>> Although PERF_AUX_GFP includes __GFP_NORETRY, it is based on GFP_KERNEL,
>> which retains __GFP_DIRECT_RECLAIM | __GFP_KSWAPD_RECLAIM. __GFP_NORETRY
>> only avoids retry loops and the kernel OOM killer at the end of
>> __alloc_pages_slowpath(). On long-running phones where physical memory is
>> already fragmented and order 9..4 pages are unavailable, every failed
>> high-order call in rb_alloc_aux_page() still enters the slowpath and:
>>
>>   1. Wakes kswapd/kcompactd at that high order.
>>   2. Runs synchronous direct compaction (__alloc_pages_direct_compact()).
>>   3. Runs synchronous direct reclaim (__alloc_pages_direct_reclaim()),
>>      evicting pages and raising memory PSI pressure. On Android, PSI
>>      reclaim stalls trigger userspace lmkd (Low Memory Killer daemon)
>>      even though the kernel OOM killer is never invoked.
>>
>> This is amplified by two loops in rb_alloc_aux():
>>   - rb_alloc_aux_page() steps down one order at a time (9, 8, ..., 0),
>>     hitting direct compaction/reclaim at each unavailable order.
>>   - Worse, the outer loop in rb_alloc_aux() never lowers max_order when
>>     rb_alloc_aux_page() falls back:
>>
>>       order = min(max_order, ilog2(nr_pages - rb->aux_nr_pages));
>>       page = rb_alloc_aux_page(node, order);
>>
>>     When a 4 MiB buffer (1024 pages) only finds order-0 pages, it retries
>>     orders 9..1 on almost every iteration -- thousands of failed
>>     high-order slowpath compaction/reclaim attempts for a single
>>     perf_mmap() call.
>>
>> In field telemetry and local ftrace on Pixel devices (prior to
>> 18049c8cff9c, so __GFP_NORETRY was already in place), we observed heavy
>> rb_alloc_aux() -> __alloc_pages_direct_compact() stalls alongside
>> kcompactd0 pinned on big cores scanning order-9, and this accounted for
>>> 20% of all perceptible app kills (via lmkd) across dogfood devices
>> until we stopped high-order AUX allocations.
>>
>> Initially we worked around this in userspace (simpleperf) by setting
>> perf_event_attr::aux_watermark = 4096 so get_order(watermark) == 0. That
>> was fine for CoreSight ETR (which uses its own separate scatter-gather
>> buffer and only copies into AUX on flush/stop), but when enabling TRBE
>> (and SPE) it caused hardware maintenance interrupts every 4 KiB instead
>> of every 2 MiB. Also, in overwrite/snapshot mode rb_alloc_aux() ignores
>> aux_watermark and uses ilog2(nr_pages). That is why I sent 18049c8cff9c.
>>
>> ### Possible solutions
>>
>> For PMUs without PERF_PMU_CAP_AUX_NO_SG (such as SPE, TRBE, and modern
>> Intel PT), higher-order AUX pages are an opportunistic optimization
>> rather than a functional requirement -- very similar to THP allocations
>> on fault (GFP_TRANSHUGE_LIGHT) or opportunistic high-order slabs in SLUB
>> (allocate_slab()).
>>
>> Instead of reverting to GFP_KERNEL reclaim/compaction for all high-order
>> attempts, could we do the following?
>>
>> 1. Make opportunistic high-order AUX allocations no-reclaim:
>>    When !(event->pmu->capabilities & PERF_PMU_CAP_AUX_NO_SG) and
>>    order > 0, clear __GFP_RECLAIM (and add __GFP_NOMEMALLOC, matching
>>    SLUB's allocate_slab() and GFP_TRANSHUGE_LIGHT):
>>
>>      gfp_t gfp = PERF_AUX_GFP;
>>
>>      if (order && !has_no_sg)
>>              gfp = (PERF_AUX_GFP | __GFP_NOMEMALLOC) & ~__GFP_RECLAIM;
>>
>>    When order falls back to 0 (or when PERF_PMU_CAP_AUX_NO_SG is set),
>>    keep PERF_AUX_GFP so single-page allocations still reclaim normally
>>    under memory pressure.
>>
>>    That way, when free order-9 / order-4 blocks exist, SPE and TRBE get
>>    them immediately for PMD / contiguous-PTE vmap() mappings (and pages
>>    stay grouped together); when memory is fragmented on a long-running
>>    device, high-order probes fail fast on the freelist without waking
>>    kcompactd/kswapd or triggering direct compaction and reclaim.
>>
>> 2. Clamp max_order in rb_alloc_aux() after fallback:
>>    If rb_alloc_aux_page() requested `order` and had to fall back to a
>>    smaller order, update `max_order = page_private(page)` in
>>    rb_alloc_aux() so subsequent iterations for the rest of the buffer do
>>    not keep re-probing higher orders that just failed.
>>
>> (Optionally, if server/benchmark workloads ever want synchronous
>> compaction to form huge AUX mappings even on fragmented hosts, that could
>> be gated behind an opt-in attr/sysfs knob similar to THP defrag modes,
>> with light/no-reclaim as the default.)
>>
>> Would this approach work for your SPE/TRBE use cases? Happy to help test
>> on our side.
>>
>> Thanks,
>> Yabin
>>
>>
>> On Wed, Oct 28, 2026 at 11:56 PM Dev Jain <dev.jain@arm.com> wrote:
>>>
>>>
>>>
>>> On 25/10/26 9:54 pm, Leo Yan wrote:
>>>> This reverts commit 18049c8cff9cc89daadc4df6975f7d9069638926.
>>>>
>>>> The original commit claimed that contiguous AUX pages unnecessarily
>>>> exacerbate fragmentation for SPE, TRBE and ETR. This is not generally
>>>> valid: larger allocations can keep pages with the same lifetime
>>>> together, while order-0 allocations can scatter them across large
>>>> blocks and prevent free pages from coalescing [1]. The follow-up
>>>> discussion [2] has not established how forcing order-0 allocations
>>>> addresses the reported problem.
>>>>
>>>> TRBE and SPE write to virtually addressed buffers. The TRBE L1 TLB
>>>> has one entry on Neoverse V2 [3] and two on Neoverse V3 [4]. Small
>>>> mappings increase translation pressure, and TTW stalls can contribute
>>>> to FIFO overflow and trace loss on TRBE.
>>>>
>>>> Larger allocations provide contiguous, aligned chunks for PMD or
>>>> contiguous-PTE mappings with the proposed vmap() support [5].
>>>> With the proposed vmap() support already applied, the tests in [2]
>>>> showed that preferring larger AUX allocations reduced L2 data TLB
>>>> refills by 99.68% in the TRBE test and 34.74% in the SPE test.
>>>>
>>>> Restore the previous policy of trying higher-order allocations first.
>>>> The allocator already uses __GFP_NORETRY and falls back to smaller
>>>> orders, including order 0, when larger allocations fail.
>>>>
>>>> Remove PERF_PMU_CAP_AUX_PREFER_LARGE, since the restored default
>>>> provides that preference. Intel PT retains its existing allocation
>>>> behaviour, with no functional change.
>>>>
>>>> [1] https://lore.kernel.org/all/aPjrRkjiIt6HmXmT@casper.infradead.org/
>>>> [2] https://lore.kernel.org/linux-arm-kernel/20260810-perf_aux_trace_large_granule-v1-0-03306c9339e3@arm.com/
>>>> [3] https://documentation-service.arm.com/static/633fe1ec4c59b30b517730b8#page=52
>>>> [4] https://documentation-service.arm.com/static/67361bd2c7fc0d1f211da382#page=54
>>>> [5] https://lore.kernel.org/linux-mm/20260715120813.3609949-1-jiangwen6@xiaomi.com/
>>>>
>>>> Signed-off-by: Leo Yan <leo.yan@arm.com>
>>>> ---
>>>
>>> Reviewed-by: Dev Jain <dev.jain@arm.com>
>>>
>>>
>>>>  arch/x86/events/intel/pt.c  |  2 --
>>>>  include/linux/perf_event.h  |  5 ++---
>>>>  kernel/events/ring_buffer.c | 29 +++++++++--------------------
>>>>  3 files changed, 11 insertions(+), 25 deletions(-)
>>>>
>>>> diff --git a/arch/x86/events/intel/pt.c b/arch/x86/events/intel/pt.c
>>>> index 49349afee6119fb6fad815c847cb3cd9daa97604..3782af40f0832a874a7f0b60961d6652513de241 100644
>>>> --- a/arch/x86/events/intel/pt.c
>>>> +++ b/arch/x86/events/intel/pt.c
>>>> @@ -1896,8 +1896,6 @@ static __init int pt_init(void)
>>>>
>>>>       if (!intel_pt_validate_hw_cap(PT_CAP_topa_multiple_entries))
>>>>               pt_pmu.pmu.capabilities = PERF_PMU_CAP_AUX_NO_SG;
>>>> -     else
>>>> -             pt_pmu.pmu.capabilities = PERF_PMU_CAP_AUX_PREFER_LARGE;
>>>>
>>>>       pt_pmu.pmu.capabilities         |= PERF_PMU_CAP_EXCLUSIVE |
>>>>                                          PERF_PMU_CAP_ITRACE |
>>>> diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
>>>> index eb3f7bab498ca9bee9ce7193744943b24ab2e0c5..b8d8b005f5c8a7c1def7cad02230bf9ffb876bd6 100644
>>>> --- a/include/linux/perf_event.h
>>>> +++ b/include/linux/perf_event.h
>>>> @@ -304,9 +304,8 @@ struct perf_event_pmu_context;
>>>>  #define PERF_PMU_CAP_AUX_OUTPUT              0x0080
>>>>  #define PERF_PMU_CAP_EXTENDED_HW_TYPE        0x0100
>>>>  #define PERF_PMU_CAP_AUX_PAUSE               0x0200
>>>> -#define PERF_PMU_CAP_AUX_PREFER_LARGE        0x0400
>>>> -#define PERF_PMU_CAP_MEDIATED_VPMU   0x0800
>>>> -#define PERF_PMU_CAP_SIMD_REGS               0x1000
>>>> +#define PERF_PMU_CAP_MEDIATED_VPMU   0x0400
>>>> +#define PERF_PMU_CAP_SIMD_REGS               0x0800
>>>>
>>>>  /**
>>>>   * pmu::scope
>>>> diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
>>>> index 64d4dbdc6c071b72b670dd044616f52981206ae8..95f4fbdfd2764de5b72d901d778c46302f525fad 100644
>>>> --- a/kernel/events/ring_buffer.c
>>>> +++ b/kernel/events/ring_buffer.c
>>>> @@ -684,15 +684,7 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event,
>>>>  {
>>>>       bool overwrite = !(flags & RING_BUFFER_WRITABLE);
>>>>       int node = (event->cpu == -1) ? -1 : cpu_to_node(event->cpu);
>>>> -     bool use_contiguous_pages = event->pmu->capabilities & (
>>>> -             PERF_PMU_CAP_AUX_NO_SG | PERF_PMU_CAP_AUX_PREFER_LARGE);
>>>> -     /*
>>>> -      * Initialize max_order to 0 for page allocation. This allocates single
>>>> -      * pages to minimize memory fragmentation. This is overridden if the
>>>> -      * PMU needs or prefers contiguous pages (use_contiguous_pages = true).
>>>> -      */
>>>> -     int max_order = 0;
>>>> -     int ret = -ENOMEM;
>>>> +     int ret = -ENOMEM, max_order;
>>>>
>>>>       if (!has_aux(event))
>>>>               return -EOPNOTSUPP;
>>>> @@ -702,8 +694,8 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event,
>>>>
>>>>       if (!overwrite) {
>>>>               /*
>>>> -              * Watermark defaults to half the buffer, to aid PMU drivers
>>>> -              * in double buffering.
>>>> +              * Watermark defaults to half the buffer, and so does the
>>>> +              * max_order, to aid PMU drivers in double buffering.
>>>>                */
>>>>               if (!watermark)
>>>>                       watermark = min_t(unsigned long,
>>>> @@ -711,19 +703,16 @@ int rb_alloc_aux(struct perf_buffer *rb, struct perf_event *event,
>>>>                                         (unsigned long)nr_pages << (PAGE_SHIFT - 1));
>>>>
>>>>               /*
>>>> -              * If using contiguous pages, use aux_watermark as the basis
>>>> -              * for chunking to help PMU drivers honor the watermark.
>>>> +              * Use aux_watermark as the basis for chunking to
>>>> +              * help PMU drivers honor the watermark.
>>>>                */
>>>> -             if (use_contiguous_pages)
>>>> -                     max_order = get_order(watermark);
>>>> +             max_order = get_order(watermark);
>>>>       } else {
>>>>               /*
>>>> -              * If using contiguous pages, we need to start with the
>>>> -              * max_order that fits in nr_pages, not the other way around,
>>>> -              * hence ilog2() and not get_order.
>>>> +              * We need to start with the max_order that fits in nr_pages,
>>>> +              * not the other way around, hence ilog2() and not get_order.
>>>>                */
>>>> -             if (use_contiguous_pages)
>>>> -                     max_order = ilog2(nr_pages);
>>>> +             max_order = ilog2(nr_pages);
>>>>               watermark = 0;
>>>>       }
>>>>
>>>>
>>>> ---
>>>> base-commit: 21a7f44fd0765fcc5194130b28767786eba1d1e7
>>>> change-id: 20261025-perf_revert_prefer_large_alloc-a0177d6379f6
>>>>
>>>> Best regards,
>>>