Re: [PATCH v2 3/6] memcg: Annotate slow paths in consume_stock() and refill_stock().

[Kuniyuki Iwashima <kuniyu@google.com>]

On Wed, 28 Oct 2026 17:55:31 -0700, Shakeel Butt wrote:
> On Wed, Oct 28, 2026 at 06:50:15PM +0000, Kuniyuki Iwashima wrote:
> > In consume_stock() and refill_stock(), each per-CPU stock slot
> > holds up to MEMCG_STOCK_HIGH (64) pages, so depleting a slot in
> > consume_stock() (!stock_pages) or trimming a slot in refill_stock()
> > (stock_pages > MEMCG_STOCK_HIGH) happens at most once per batch.
> > 
> > Similarly, requests larger than MEMCG_CHARGE_BATCH, local_trylock()
> > failure, and insufficient cached pages (stock_pages < nr_pages)
> > are all slow paths.
> > 
> > Without unlikely() annotations (when building without FDO, or
> > when an FDO profile does not cover a caller), the compiler treats
> > these branches as likely and inlines css_put() in consume_stock()
> > and memcg_uncharge() in refill_stock() right into the middle of
> > the hot path.
> 
> Please state the compiler along with its version here. I didn't see this behavior
> on gcc 14.

Sure, actually I included asm in v1 but thought no one is
interested in it and this might make my patch look AI-sloppy,
but I'm happy that it wasn't the case :)

https://lore.kernel.org/linux-mm/20261016052811.3085635-4-kuniyu@google.com/

This was generated w/ Clang 23, but I was able to see the
same result w/ Clang 21 too.  I'll include them in v3 again.

Before (refill_stock):

  stock_pages = stock->nr_pages[i] + nr_pages;
   7ab2: 43 0f b6 44 26 01       movzbl  0x1(%r14,%r12), %eax
   7ab8: 01 c6                   addl    %eax, %esi
  if (stock_pages > MEMCG_STOCK_HIGH) {
   7aba: 83 fe 41                cmpl    $0x41, %esi
   7abd: 72 1d                   jb      0x7adc <refill_stock+0x21c>
  memcg_uncharge(memcg, stock_pages - MEMCG_STOCK_LOW);   # inlined
   7abf: 41 89 f7                movl    %esi, %r15d
   7ac2: 41 83 c7 e0             addl    $-0x20, %r15d
   7ac6: 48 8d bb 00 01 00 00    leaq    0x100(%rbx), %rdi
   7acd: 4c 89 fe                movq    %r15, %rsi
   7ad0: e8 00 00 00 00          callq   <page_counter_uncharge>
   7ad5: be 20 00 00 00          movl    $0x20, %esi
   7ada: eb 41                   jmp     0x7b1d <refill_stock+0x25d>
  WRITE_ONCE(stock->nr_pages[i], stock_pages);
   7adc: 43 88 74 26 01          movb    %sil, 0x1(%r14,%r12)
  local_unlock(&memcg_stock.lock);
   7ae1: 65 48 8b 05 00 00 00 00 movq    %gs:(%rip), %rax
   7ae9: c6 80 00 00 00 00 00    movb    $0x0, (%rax)
   7af0: e9 03 fe ff ff          jmp     0x78f8 <refill_stock+0x38>

After (refill_stock):

  stock_pages = stock->nr_pages[i] + nr_pages;
   7b76: 42 0f b6 44 33 01       movzbl  0x1(%rbx,%r14), %eax
   7b7c: 01 c6                   addl    %eax, %esi
  if (unlikely(stock_pages > MEMCG_STOCK_HIGH)) {
   7b7e: 83 fe 41                cmpl    $0x41, %esi
   7b81: 73 20                   jae     0x7ba3 <refill_stock+0x233>
  WRITE_ONCE(stock->nr_pages[i], stock_pages);
   7b83: 42 88 74 33 01          movb    %sil, 0x1(%rbx,%r14)
  local_unlock(&memcg_stock.lock);
   7b88: 65 48 8b 05 00 00 00 00 movq    %gs:(%rip), %rax
   7b90: c6 80 00 00 00 00 00    movb    $0x0, (%rax)
  }
   7b97: 5b                      popq    %rbx
   7b98: 41 5e                   popq    %r14
   7b9a: 41 5f                   popq    %r15
   7b9c: 5d                      popq    %rbp
   7b9d: 2e e9 00 00 00 00       jmp     <__x86_return_thunk>
  memcg_uncharge(memcg, stock_pages - MEMCG_STOCK_LOW);
   7ba3: 83 c6 e0                addl    $-0x20, %esi
   7ba6: e8 15 f2 ff ff          callq   0x6dc0 <memcg_uncharge>
   7bab: be 20 00 00 00          movl    $0x20, %esi
   7bb0: eb d1                   jmp     0x7b83 <refill_stock+0x213>