[RFC 0/8] ARM64 TLB logic revision for more control and enhanced diagnostics

[Christoph Lameter (Ampere) <cl@linux.com>]

WARNING: First draft of the patchset, tested with kernel compiles, may
corrupt your memory.

This patchset intends to aid in help debugging and scaling TLB operations
on ARM64. This is in particular desirable for ARM architectures with
large numbers of cores. What is often seen is that the mesh is flooded with
snoop traffic that is related to TLBI broadcasts.

The patchset adds the following features:

- Allow diagnostics via /proc/vmstat like already possible on X86 with the
	CONFIG_DEBUG_TLB  option.

  Some sample output:

  cat /proc/vmstat
	...
	nr_tlb_remote_flush 554104		Flushes converted to IPIs with local flushing
	nr_tlb_remote_flush_received 1312243	IPIs received to perform local flushing
	nr_tlb_local_flush_all 141837		Local flush alls
	nr_tlb_local_flush_range 571784		Local flush range
	nr_tlb_local_flush_one 8011880		Local individual page flushes
	nr_tlb_flush_all 28239			Flush alls through the mesh
	nr_tlb_flush_range 13003		Flush range through the mesh
	nr_tlb_flush_one 54764			Flush one through the mesh
	nr_tlb_skipped 0			Suppressed flush

- Tracks the cores that have used an address space. With that we can
	compute the weight of cpus that have used this address space
	to decide on how to optimally do the flushing when such an action
	is required.

- Control the TLB flushing behavior via the kernel command line and also on a running system.

    New Kernel parameter  tlb_mode=<tlb_mode>
    New sysfs setting     /sys/kernel/debug/tlb_mode

	tlb_mode is comprised of a set of flags starting at bit 10. Bit 0-9
	are used to set a boundary as to what cpu weight will lead to a mesh
	flush. If the cpu weight is lower then IPIs are send avoiding the mesh.

	Feature flags:

	Bit 10 = If the current cpu is the only one that has ever used an
			address space then perform local invalidation.
			This catches the majority of flushes on boot and
			activities of typical single threaded Unixy processes.

	Bit 11 = Enable TLB range. Various hardware has problems with TLB range.
			This allows the kernel to recognize that TLB range
			should not be used and an alternate method is to be
			used to do the flushing.

	Bit 12 = Suppress TLB flushes if the address space is unused.
			If this bit is set and a flush is requested in an
			unused address space then no flush will be performed
			since there cannot any TLB entries. If this is
			not set then perform mesh flush (just to be sure).

- Autotunes the feature flags on bootup if the user has not specified tlb_mode.
	Calculates an optimal balance between IPIs and mesh flushing
	based on the number of cpus in the system. Enables local validation
	always and tlb range flushing if the processor features indicate
	that the processor supports it.


We need a more detailed description but I hope this is enough to get started.

These issues have been discussed before in an patchset that contains a similar feature 
in 2019:

https://lore.kernel.org/linux-arm-kernel/20190617143255.10462-1-indou.takao@jp.fujitsu.com/


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel