Re: [PATCH v9 00/12] AMD broadcast TLB invalidation

Oleksandr Natalenko
2025-02-06T10:23:10Z

[-- Attachment #1: Type: text/plain, Size: 5121 bytes --]

Hello.

On čtvrtek 6. února 2025 5:43:19, středoevropský standardní čas Rik van Riel wrote:
> Add support for broadcast TLB invalidation using AMD's INVLPGB instruction.
>
> This allows the kernel to invalidate TLB entries on remote CPUs without
> needing to send IPIs, without having to wait for remote CPUs to handle
> those interrupts, and with less interruption to what was running on
> those CPUs.
>
> Because x86 PCID space is limited, and there are some very large
> systems out there, broadcast TLB invalidation is only used for
> processes that are active on 3 or more CPUs, with the threshold
> being gradually increased the more the PCID space gets exhausted.
>
> Combined with the removal of unnecessary lru_add_drain calls
> (see https://lkml.org/lkml/2024/12/19/1388) this results in a
> nice performance boost for the will-it-scale tlb_flush2_threads
> test on an AMD Milan system with 36 cores:
>
> - vanilla kernel: 527k loops/second
> - lru_add_drain removal: 731k loops/second
> - only INVLPGB: 527k loops/second
> - lru_add_drain + INVLPGB: 1157k loops/second
>
> Profiling with only the INVLPGB changes showed while
> TLB invalidation went down from 40% of the total CPU
> time to only around 4% of CPU time, the contention
> simply moved to the LRU lock.
>
> Fixing both at the same time about doubles the
> number of iterations per second from this case.
>
> Some numbers closer to real world performance
> can be found at Phoronix, thanks to Michael:
>
> https://www.phoronix.com/news/AMD-INVLPGB-Linux-Benefits
>
> My current plan is to implement support for Intel's RAR
> (Remote Action Request) TLB flushing in a follow-up series,
> after this thing has been merged into -tip. Making things
> any larger would just be unwieldy for reviewers.
>
> v9:
> - print warning when start or end address was rounded (Peter)

OK, I've just hit one:

TLB flush not stride 200000 aligned. Start 7fffc0000000, end 7fffffe01000
WARNING: CPU: 31 PID: 411 at arch/x86/mm/tlb.c:1342 flush_tlb_mm_range+0x57b/0x600
Modules linked in:
CPU: 31 UID: 0 PID: 411 Comm: modprobe Not tainted 6.13.0-pf3 #1 1366679ca06f46d05d1e9d9c537b0c6b4c922b82
Hardware name: ASUS System Product Name/Pro WS X570-ACE, BIOS 4902 08/29/2024
RIP: 0010:flush_tlb_mm_range+0x57b/0x600
Code: 5f e9 39 b3 3f 00 e8 24 57 f5 ff e9 e9 fc ff ff 48 8b 0c 24 4c 89 e2 48 c7 c7 78 59 27 b0 c6 05 3d 1a 31 02 01 e8 85 e4 01 00 <0f> 0b e9 35 fb ff ff fa 0f 1f 44 00 00 48 89 df e8 a0 f4 ff ff fb
RSP: 0018:ffffc137c11e7a38 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ffff9e6eaf1b5d80 RCX: 00000000ffffdfff
RDX: 0000000000000000 RSI: 00000000ffffffea RDI: 0000000000000001
RBP: ffff9e500244d800 R08: 00000000ffffdfff R09: ffff9e6eae1fffa8
R10: 00000000ffffdfff R11: 0000000000000003 R12: 00007fffc0000000
R13: 000000000000001f R14: 0000000000000015 R15: ffff9e6eaf180000
FS: 0000000000000000(0000) GS:ffff9e6eaf180000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000109966000 CR4: 0000000000f50ef0
PKRU: 55555554
Call Trace:
<TASK>
tlb_flush_mmu+0x125/0x1a0
tlb_finish_mmu+0x41/0x80
relocate_vma_down+0x183/0x200
setup_arg_pages+0x201/0x390
load_elf_binary+0x3a7/0x17d0
bprm_execve+0x244/0x630
kernel_execve+0x180/0x1f0
call_usermodehelper_exec_async+0xd0/0x190
ret_from_fork+0x34/0x50
ret_from_fork_asm+0x1a/0x30
</TASK>

What do I do with it?

Thank you.

> - in the reclaim code, tlbsync at context switch time (Peter)
> - fix !CONFIG_CPU_SUP_AMD compile error in arch_tlbbatch_add_pending (Jan)
> v8:
> - round start & end to handle non-page-aligned callers (Steven & Jan)
> - fix up changelog & add tested-by tags (Manali)
> v7:
> - a few small code cleanups (Nadav)
> - fix spurious VM_WARN_ON_ONCE in mm_global_asid
> - code simplifications & better barriers (Peter & Dave)
> v6:
> - fix info->end check in flush_tlb_kernel_range (Michael)
> - disable broadcast TLB flushing on 32 bit x86
> v5:
> - use byte assembly for compatibility with older toolchains (Borislav, Michael)
> - ensure a panic on an invalid number of extra pages (Dave, Tom)
> - add cant_migrate() assertion to tlbsync (Jann)
> - a bunch more cleanups (Nadav)
> - key TCE enabling off X86_FEATURE_TCE (Andrew)
> - fix a race between reclaim and ASID transition (Jann)
> v4:
> - Use only bitmaps to track free global ASIDs (Nadav)
> - Improved AMD initialization (Borislav & Tom)
> - Various naming and documentation improvements (Peter, Nadav, Tom, Dave)
> - Fixes for subtle race conditions (Jann)
> v3:
> - Remove paravirt tlb_remove_table call (thank you Qi Zheng)
> - More suggested cleanups and changelog fixes by Peter and Nadav
> v2:
> - Apply suggestions by Peter and Borislav (thank you!)
> - Fix bug in arch_tlbbatch_flush, where we need to do both
> the TLBSYNC, and flush the CPUs that are in the cpumask.
> - Some updates to comments and changelogs based on questions.

--
Oleksandr Natalenko, MSE

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]