[RFC PATCH 0/8] arm64: Simplify VA space configurations

[RFC PATCH 0/8] arm64: Simplify VA space configurations

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

For historical reasons, we currently support many combinations of page
size and virtual address space size:
- the original arm64 port used only 3 levels of paging (4k/39 bits), and
  the 4 level configuration was only added later
- LVA support on 64k pages could not be disabled at boot, and so it was
  added as a completely separate configuration, even though the number
  of translation levels is the same
- 16k pages gains only 1 bit of VA space when going from 3 to 4 levels
  of translation, and so both are supported
- 16k/36 bits is supported under CONFIG_EXPERT, but 36 bits is a bit on
  the small side for the kernel.

Let's simplify this, by
- removing the configurations that are also the fallbacks on LPA/LPA2
  capable hardware,
- converting 36, 42 and 39 bit configurations into reduced definitions
  of TASK_SIZE, while keeping the larger VA space on the kernel side,
- dropping 16k/48 bits altogether.

The remaining configurations always support up to 52-bit virtual
addressing on the kernel side, and implement the reduced userland VA
space sizes by skipping levels when programming TTBR0_EL1. This is a
quick and dirty hack, but sufficient for the purposes of this RFC.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Kees Cook <keescook@chromium.org>

Ard Biesheuvel (8):
  arm64/Kconfig: force ARM64_PAN=y when enabling TTBR0 sw PAN
  arm64/Kconfig: fix ARCH_MMAP_RND_BITS_MAX for 52-bit virtual
    addressing
  arm64/Kconfig: eliminate 64k/48-bit VA combination
  arm64/Kconfig: eliminate 4k/48-bit VA combination
  arm64/Kconfig: Drop support for 47-bit virtual addressing
  arm64/Kconfig: Drop support for 48-bit virtual addressing
  arm64/mm: Use reduced VA sizes (36/39/42 bits) only for user space
  arm64/mm: Account for reduced VA sizes in T0SZ and skip the levels

 arch/arm64/Kconfig                     | 89 ++++++++------------
 arch/arm64/include/asm/assembler.h     |  2 +-
 arch/arm64/include/asm/memory.h        |  4 -
 arch/arm64/include/asm/mmu_context.h   |  9 +-
 arch/arm64/include/asm/pgtable-hwdef.h |  2 -
 arch/arm64/include/asm/processor.h     |  6 +-
 arch/arm64/kernel/cpufeature.c         |  2 -
 arch/arm64/kernel/head.S               |  4 -
 arch/arm64/mm/init.c                   |  4 +-
 arch/arm64/mm/pgd.c                    |  9 +-
 arch/arm64/mm/proc.S                   |  2 -
 11 files changed, 52 insertions(+), 81 deletions(-)

-- 
2.47.0.163.g1226f6d8fa-goog

[RFC PATCH 1/8] arm64: Kconfig: force ARM64_PAN=y when enabling TTBR0 sw PAN

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

We have a couple of instances of Kconfig constraints where PAN must be
enabled too if TTBR0 sw PAN is enabled, primarily to avoid dealing with
the modified TTBR0_EL1 sysreg format that is used when 52-bit physical
addressing and/or CnP are enabled (support for either implies support
for hardware PAN as well, which will supersede PAN emulation if both are
available)

Let's simplify this, and always enable ARM64_PAN when enabling TTBR0 sw
PAN. This decouples the PAN configuration from the VA size selection,
permitting us to simplify the latter in subsequent patches. (Note that
PAN and TTBR0 sw PAN can still be disabled after this patch, but not
independently)

To avoid a convoluted circular Kconfig dependency involving KCSAN, make
ARM64_MTE select ARM64_PAN too, instead of depending on it.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/Kconfig | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 232dcade2783..ec218ef22f2b 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -1368,7 +1368,6 @@ config ARM64_VA_BITS_48
 
 config ARM64_VA_BITS_52
 	bool "52-bit"
-	depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN
 	help
 	  Enable 52-bit virtual addressing for userspace when explicitly
 	  requested via a hint to mmap(). The kernel will also use 52-bit
@@ -1420,7 +1419,6 @@ config ARM64_PA_BITS_48
 config ARM64_PA_BITS_52
 	bool "52-bit"
 	depends on ARM64_64K_PAGES || ARM64_VA_BITS_52
-	depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN
 	help
 	  Enable support for a 52-bit physical address space, introduced as
 	  part of the ARMv8.2-LPA extension.
@@ -1667,6 +1665,7 @@ config RODATA_FULL_DEFAULT_ENABLED
 config ARM64_SW_TTBR0_PAN
 	bool "Emulate Privileged Access Never using TTBR0_EL1 switching"
 	depends on !KCSAN
+	select ARM64_PAN
 	help
 	  Enabling this option prevents the kernel from accessing
 	  user-space memory directly by pointing TTBR0_EL1 to a reserved
@@ -1923,7 +1922,6 @@ config ARM64_RAS_EXTN
 config ARM64_CNP
 	bool "Enable support for Common Not Private (CNP) translations"
 	default y
-	depends on ARM64_PAN || !ARM64_SW_TTBR0_PAN
 	help
 	  Common Not Private (CNP) allows translation table entries to
 	  be shared between different PEs in the same inner shareable
@@ -2118,7 +2116,7 @@ config ARM64_MTE
 	depends on AS_HAS_ARMV8_5
 	depends on AS_HAS_LSE_ATOMICS
 	# Required for tag checking in the uaccess routines
-	depends on ARM64_PAN
+	select ARM64_PAN
 	select ARCH_HAS_SUBPAGE_FAULTS
 	select ARCH_USES_HIGH_VMA_FLAGS
 	select ARCH_USES_PG_ARCH_2
-- 
2.47.0.163.g1226f6d8fa-goog

[RFC PATCH 2/8] arm64: Kconfig: fix ARCH_MMAP_RND_BITS_MAX for 52-bit virtual addressing

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

The default values for ARCH_MMAP_RND_BITS_MAX fail to take into account
that we now have support for 52-bit virtual addressing. As these configs
are compatible with hardware that is incapable of 52-bit virtual
addressing, use the same value we use for 47/48-bit virtual addressing.
(For the sake of simplicity, use the same value for 47 and 48 bit
virtual addressing for 16k pages)

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/Kconfig | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index ec218ef22f2b..ac8e7550430b 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -324,13 +324,9 @@ config ARCH_MMAP_RND_BITS_MAX
 	default 19 if ARM64_VA_BITS=36
 	default 24 if ARM64_VA_BITS=39
 	default 27 if ARM64_VA_BITS=42
-	default 30 if ARM64_VA_BITS=47
-	default 29 if ARM64_VA_BITS=48 && ARM64_64K_PAGES
-	default 31 if ARM64_VA_BITS=48 && ARM64_16K_PAGES
-	default 33 if ARM64_VA_BITS=48
-	default 14 if ARM64_64K_PAGES
-	default 16 if ARM64_16K_PAGES
-	default 18
+	default 29 if ARM64_64K_PAGES
+	default 30 if ARM64_16K_PAGES
+	default 33
 
 config ARCH_MMAP_RND_COMPAT_BITS_MIN
 	default 7 if ARM64_64K_PAGES
-- 
2.47.0.163.g1226f6d8fa-goog

[RFC PATCH 3/8] arm64: Kconfig: eliminate 64k/48-bit VA combination

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

Now that the vmemmap region is sized dynamically based on the actual
size of the kernel VA space, there are no longer any material
differences between supporting 48-bit and 52-bit VA space sizes for 64k
pages, which use the same number of translation levels. And if needed,
52-bit virtual addressing can be disabled at boot on systems that do
support it but where 48-bit virtual addressing is preferred.

The only remaining difference is the size of a root level user page
table, which grows from 512 bytes to 8k when 52-bit virtual addressing
is enabled, but given that both are less than the size of a page, this
is easily fixed in the pgd_alloc init code. (In all other possible cases
where vabits_actual < VABITS holds, the effective PGD_SIZE equals the
page size, and so this change has no effect.)

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/Kconfig  | 5 +++--
 arch/arm64/mm/pgd.c | 9 +++++----
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index ac8e7550430b..6a73fd61b4aa 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -379,11 +379,11 @@ config PGTABLE_LEVELS
 	int
 	default 2 if ARM64_16K_PAGES && ARM64_VA_BITS_36
 	default 2 if ARM64_64K_PAGES && ARM64_VA_BITS_42
-	default 3 if ARM64_64K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52)
 	default 3 if ARM64_4K_PAGES && ARM64_VA_BITS_39
 	default 3 if ARM64_16K_PAGES && ARM64_VA_BITS_47
+	default 3 if ARM64_64K_PAGES
 	default 4 if ARM64_16K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52)
-	default 4 if !ARM64_64K_PAGES && ARM64_VA_BITS_48
+	default 4 if ARM64_VA_BITS_48
 	default 5 if ARM64_4K_PAGES && ARM64_VA_BITS_52
 
 config ARCH_SUPPORTS_UPROBES
@@ -1361,6 +1361,7 @@ config ARM64_VA_BITS_47
 
 config ARM64_VA_BITS_48
 	bool "48-bit"
+	depends on !PAGE_SIZE_64KB
 
 config ARM64_VA_BITS_52
 	bool "52-bit"
diff --git a/arch/arm64/mm/pgd.c b/arch/arm64/mm/pgd.c
index 0c501cabc238..ecc4b1ec235c 100644
--- a/arch/arm64/mm/pgd.c
+++ b/arch/arm64/mm/pgd.c
@@ -48,20 +48,21 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd)
 
 void __init pgtable_cache_init(void)
 {
+	unsigned int size = PGD_SIZE >> (VA_BITS - vabits_actual);
+
 	if (pgdir_is_page_size())
 		return;
 
-#ifdef CONFIG_ARM64_PA_BITS_52
 	/*
 	 * With 52-bit physical addresses, the architecture requires the
 	 * top-level table to be aligned to at least 64 bytes.
 	 */
-	BUILD_BUG_ON(PGD_SIZE < 64);
-#endif
+	if (IS_ENABLED(CONFIG_ARM64_PA_BITS_52))
+		size = max(size, 64);
 
 	/*
 	 * Naturally aligned pgds required by the architecture.
 	 */
-	pgd_cache = kmem_cache_create("pgd_cache", PGD_SIZE, PGD_SIZE,
+	pgd_cache = kmem_cache_create("pgd_cache", size, size,
 				      SLAB_PANIC, NULL);
 }
-- 
2.47.0.163.g1226f6d8fa-goog

[RFC PATCH 4/8] arm64: Kconfig: eliminate 4k/48-bit VA combination

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

Now that LPA2 is supported seamlessly, using alternatives patching where
appropriate to fall back to 48-bit virtual addressing when running with
4k page size on hardware that lacks LPA support, there is no longer a
need to keep the separate 48-bit VA size configurations. Note that LPA2
support can be overridden at boot time by passing arm64.nolva on the
command line, and given that no LPA2 hardware exists yet in the field,
now is a good time to make this change.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/Kconfig | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 6a73fd61b4aa..099b1a825b9f 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -382,9 +382,8 @@ config PGTABLE_LEVELS
 	default 3 if ARM64_4K_PAGES && ARM64_VA_BITS_39
 	default 3 if ARM64_16K_PAGES && ARM64_VA_BITS_47
 	default 3 if ARM64_64K_PAGES
-	default 4 if ARM64_16K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52)
-	default 4 if ARM64_VA_BITS_48
-	default 5 if ARM64_4K_PAGES && ARM64_VA_BITS_52
+	default 4 if ARM64_16K_PAGES
+	default 5 if ARM64_4K_PAGES
 
 config ARCH_SUPPORTS_UPROBES
 	def_bool y
@@ -1361,7 +1360,7 @@ config ARM64_VA_BITS_47
 
 config ARM64_VA_BITS_48
 	bool "48-bit"
-	depends on !PAGE_SIZE_64KB
+	depends on PAGE_SIZE_16KB
 
 config ARM64_VA_BITS_52
 	bool "52-bit"
-- 
2.47.0.163.g1226f6d8fa-goog

[RFC PATCH 5/8] arm64/Kconfig: Drop support for 47-bit virtual addressing

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

Drop the separate 47-bit virtual address space configuration, which is
identical in practice to 52-bit VA configuration on all 16k pages
capable hardware currently in the field. For future hardware that does
implement support for 52-bit virtual addressing, this mode can be chosen
at boot by passing 'arm64.nolva' on the kernel command line.

This reduces the number of configurations that need to be supported and
validated.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/Kconfig | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 099b1a825b9f..7df7d24c767d 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -380,7 +380,6 @@ config PGTABLE_LEVELS
 	default 2 if ARM64_16K_PAGES && ARM64_VA_BITS_36
 	default 2 if ARM64_64K_PAGES && ARM64_VA_BITS_42
 	default 3 if ARM64_4K_PAGES && ARM64_VA_BITS_39
-	default 3 if ARM64_16K_PAGES && ARM64_VA_BITS_47
 	default 3 if ARM64_64K_PAGES
 	default 4 if ARM64_16K_PAGES
 	default 5 if ARM64_4K_PAGES
@@ -412,12 +411,12 @@ config KASAN_SHADOW_OFFSET
 	hex
 	depends on KASAN_GENERIC || KASAN_SW_TAGS
 	default 0xdfff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && !KASAN_SW_TAGS
-	default 0xdfffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && !KASAN_SW_TAGS
+	default 0xdfffc00000000000 if ARM64_VA_BITS_52 && ARM64_16K_PAGES && !KASAN_SW_TAGS
 	default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS
 	default 0xdfffffc000000000 if ARM64_VA_BITS_39 && !KASAN_SW_TAGS
 	default 0xdffffff800000000 if ARM64_VA_BITS_36 && !KASAN_SW_TAGS
 	default 0xefff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS
-	default 0xefffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS
+	default 0xefffc00000000000 if ARM64_VA_BITS_52 && ARM64_16K_PAGES && KASAN_SW_TAGS
 	default 0xeffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS
 	default 0xefffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS
 	default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS
@@ -1354,10 +1353,6 @@ config ARM64_VA_BITS_42
 	bool "42-bit"
 	depends on PAGE_SIZE_64KB
 
-config ARM64_VA_BITS_47
-	bool "47-bit"
-	depends on PAGE_SIZE_16KB
-
 config ARM64_VA_BITS_48
 	bool "48-bit"
 	depends on PAGE_SIZE_16KB
@@ -1397,7 +1392,6 @@ config ARM64_VA_BITS
 	default 36 if ARM64_VA_BITS_36
 	default 39 if ARM64_VA_BITS_39
 	default 42 if ARM64_VA_BITS_42
-	default 47 if ARM64_VA_BITS_47
 	default 48 if ARM64_VA_BITS_48
 	default 52 if ARM64_VA_BITS_52
 
-- 
2.47.0.163.g1226f6d8fa-goog

[RFC PATCH 6/8] arm64/Kconfig: Drop support for 48-bit virtual addressing

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

Drop support for 48-bit virtual addressing on 16k pages, which is the
only remaining config that supports this explicitly.

On 16k pages, 48-bit virtual addressing uses 4 levels of translation,
where the top level has only 2 entries. This is very inefficient in
terms of TLB utilization, and so 47-bit virtual addressing is usually a
better choice.

Note that x86 supports only 47-bit virtual addressing for user space
with 4 translation levels, due to the way its page tables are
constructed (a single combined root table for both kernel and user
space), and so removing this configuration is unlikely to create
portability concerns.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/Kconfig | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 7df7d24c767d..39d0d2eb5b7c 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -410,12 +410,12 @@ config BUILTIN_RETURN_ADDRESS_STRIPS_PAC
 config KASAN_SHADOW_OFFSET
 	hex
 	depends on KASAN_GENERIC || KASAN_SW_TAGS
-	default 0xdfff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && !KASAN_SW_TAGS
+	default 0xdfff800000000000 if ARM64_VA_BITS_52 && !ARM64_16K_PAGES && !KASAN_SW_TAGS
 	default 0xdfffc00000000000 if ARM64_VA_BITS_52 && ARM64_16K_PAGES && !KASAN_SW_TAGS
 	default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS
 	default 0xdfffffc000000000 if ARM64_VA_BITS_39 && !KASAN_SW_TAGS
 	default 0xdffffff800000000 if ARM64_VA_BITS_36 && !KASAN_SW_TAGS
-	default 0xefff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS
+	default 0xefff800000000000 if ARM64_VA_BITS_52 && !ARM64_16K_PAGES && KASAN_SW_TAGS
 	default 0xefffc00000000000 if ARM64_VA_BITS_52 && ARM64_16K_PAGES && KASAN_SW_TAGS
 	default 0xeffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS
 	default 0xefffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS
@@ -1353,10 +1353,6 @@ config ARM64_VA_BITS_42
 	bool "42-bit"
 	depends on PAGE_SIZE_64KB
 
-config ARM64_VA_BITS_48
-	bool "48-bit"
-	depends on PAGE_SIZE_16KB
-
 config ARM64_VA_BITS_52
 	bool "52-bit"
 	help
@@ -1392,7 +1388,6 @@ config ARM64_VA_BITS
 	default 36 if ARM64_VA_BITS_36
 	default 39 if ARM64_VA_BITS_39
 	default 42 if ARM64_VA_BITS_42
-	default 48 if ARM64_VA_BITS_48
 	default 52 if ARM64_VA_BITS_52
 
 choice
-- 
2.47.0.163.g1226f6d8fa-goog

[RFC PATCH 7/8] arm64/mm: Use reduced VA sizes (36/39/42 bits) only for user space

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

The advantage of a reduced virtual address space size is its impact on
the number of translation levels, which affects TLB pressure. The
working set of translations covering the kernel side is negligible
compared to user space, where each process has its own set of page
tables, and so most of the same benefit can be obtained by reducing the
VA size only for user space.

As a preparatory step towards implementing this, drop all the reduced VA
space sizes in Kconfig, and replace it with a configurable userland VA
space size that is reflected in TASK_SIZE. This will be taken advantage
of in a subsequent patch to actually reduce the number of translations
used by the MMU for translating user space virtual addresses.

As a bonus, this will allow 36-bit virtual addressing for user space to
be used on SoCs whose physical address space layout is based on the
misguided 'Principles of ARM memory maps' document, which struggle with
this currently, given that a 35-bit linear map can only fit 2 GiB of RAM
(and on these designs, the rest of RAM comes after a 30 GiB hole in the
physical address space)

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/Kconfig                     | 56 +++++++++-----------
 arch/arm64/include/asm/assembler.h     |  2 +-
 arch/arm64/include/asm/memory.h        |  4 --
 arch/arm64/include/asm/pgtable-hwdef.h |  2 -
 arch/arm64/include/asm/processor.h     |  6 +--
 arch/arm64/kernel/cpufeature.c         |  2 -
 arch/arm64/kernel/head.S               |  4 --
 arch/arm64/mm/init.c                   |  4 +-
 arch/arm64/mm/proc.S                   |  2 -
 9 files changed, 31 insertions(+), 51 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 39d0d2eb5b7c..78e82383f8e2 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -111,7 +111,7 @@ config ARM64
 	select ARCH_WANT_DEFAULT_BPF_JIT
 	select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT
 	select ARCH_WANT_FRAME_POINTERS
-	select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36)
+	select ARCH_WANT_HUGE_PMD_SHARE if !ARM64_64K_PAGES
 	select ARCH_WANT_LD_ORPHAN_WARN
 	select ARCH_WANTS_EXECMEM_LATE if EXECMEM
 	select ARCH_WANTS_NO_INSTR
@@ -321,9 +321,9 @@ config ARCH_MMAP_RND_BITS_MIN
 # max bits determined by the following formula:
 #  VA_BITS - PAGE_SHIFT - 3
 config ARCH_MMAP_RND_BITS_MAX
-	default 19 if ARM64_VA_BITS=36
-	default 24 if ARM64_VA_BITS=39
-	default 27 if ARM64_VA_BITS=42
+	default 19 if ARM64_USER_VA_BITS_36
+	default 24 if ARM64_USER_VA_BITS_39
+	default 27 if ARM64_USER_VA_BITS_42
 	default 29 if ARM64_64K_PAGES
 	default 30 if ARM64_16K_PAGES
 	default 33
@@ -377,9 +377,6 @@ config FIX_EARLYCON_MEM
 
 config PGTABLE_LEVELS
 	int
-	default 2 if ARM64_16K_PAGES && ARM64_VA_BITS_36
-	default 2 if ARM64_64K_PAGES && ARM64_VA_BITS_42
-	default 3 if ARM64_4K_PAGES && ARM64_VA_BITS_39
 	default 3 if ARM64_64K_PAGES
 	default 4 if ARM64_16K_PAGES
 	default 5 if ARM64_4K_PAGES
@@ -410,16 +407,10 @@ config BUILTIN_RETURN_ADDRESS_STRIPS_PAC
 config KASAN_SHADOW_OFFSET
 	hex
 	depends on KASAN_GENERIC || KASAN_SW_TAGS
-	default 0xdfff800000000000 if ARM64_VA_BITS_52 && !ARM64_16K_PAGES && !KASAN_SW_TAGS
-	default 0xdfffc00000000000 if ARM64_VA_BITS_52 && ARM64_16K_PAGES && !KASAN_SW_TAGS
-	default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS
-	default 0xdfffffc000000000 if ARM64_VA_BITS_39 && !KASAN_SW_TAGS
-	default 0xdffffff800000000 if ARM64_VA_BITS_36 && !KASAN_SW_TAGS
-	default 0xefff800000000000 if ARM64_VA_BITS_52 && !ARM64_16K_PAGES && KASAN_SW_TAGS
-	default 0xefffc00000000000 if ARM64_VA_BITS_52 && ARM64_16K_PAGES && KASAN_SW_TAGS
-	default 0xeffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS
-	default 0xefffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS
-	default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS
+	default 0xdfff800000000000 if !ARM64_16K_PAGES && !KASAN_SW_TAGS
+	default 0xdfffc00000000000 if ARM64_16K_PAGES && !KASAN_SW_TAGS
+	default 0xefff800000000000 if !ARM64_16K_PAGES && KASAN_SW_TAGS
+	default 0xefffc00000000000 if ARM64_16K_PAGES && KASAN_SW_TAGS
 	default 0xffffffffffffffff
 
 config UNWIND_TABLES
@@ -1334,26 +1325,26 @@ config ARM64_64K_PAGES
 endchoice
 
 choice
-	prompt "Virtual address space size"
-	default ARM64_VA_BITS_52
+	prompt "Virtual address space size for user space"
+	default ARM64_USER_VA_BITS_52
 	help
 	  Allows choosing one of multiple possible virtual address
 	  space sizes. The level of translation table is determined by
 	  a combination of page size and virtual address space size.
 
-config ARM64_VA_BITS_36
-	bool "36-bit" if EXPERT
+config ARM64_USER_VA_BITS_36
+	bool "36-bit"
 	depends on PAGE_SIZE_16KB
 
-config ARM64_VA_BITS_39
+config ARM64_USER_VA_BITS_39
 	bool "39-bit"
 	depends on PAGE_SIZE_4KB
 
-config ARM64_VA_BITS_42
+config ARM64_USER_VA_BITS_42
 	bool "42-bit"
 	depends on PAGE_SIZE_64KB
 
-config ARM64_VA_BITS_52
+config ARM64_USER_VA_BITS_52
 	bool "52-bit"
 	help
 	  Enable 52-bit virtual addressing for userspace when explicitly
@@ -1372,7 +1363,7 @@ endchoice
 
 config ARM64_FORCE_52BIT
 	bool "Force 52-bit virtual addresses for userspace"
-	depends on ARM64_VA_BITS_52 && EXPERT
+	depends on ARM64_USER_VA_BITS_52 && EXPERT
 	help
 	  For systems with 52-bit userspace VAs enabled, the kernel will attempt
 	  to maintain compatibility with older software by providing 48-bit VAs
@@ -1385,10 +1376,14 @@ config ARM64_FORCE_52BIT
 
 config ARM64_VA_BITS
 	int
-	default 36 if ARM64_VA_BITS_36
-	default 39 if ARM64_VA_BITS_39
-	default 42 if ARM64_VA_BITS_42
-	default 52 if ARM64_VA_BITS_52
+	default 52
+
+config TASK_SIZE_BITS
+	int
+	default 36 if ARM64_USER_VA_BITS_36
+	default 39 if ARM64_USER_VA_BITS_39
+	default 42 if ARM64_USER_VA_BITS_42
+	default 52
 
 choice
 	prompt "Physical address space size"
@@ -1399,11 +1394,10 @@ choice
 
 config ARM64_PA_BITS_48
 	bool "48-bit"
-	depends on ARM64_64K_PAGES || !ARM64_VA_BITS_52
+	depends on ARM64_64K_PAGES
 
 config ARM64_PA_BITS_52
 	bool "52-bit"
-	depends on ARM64_64K_PAGES || ARM64_VA_BITS_52
 	help
 	  Enable support for a 52-bit physical address space, introduced as
 	  part of the ARMv8.2-LPA extension.
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 3d8d534a7a77..337ba98d0272 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -577,7 +577,7 @@ alternative_endif
  * 	ttbr: Value of ttbr to set, modified.
  */
 	.macro	offset_ttbr1, ttbr, tmp
-#if defined(CONFIG_ARM64_VA_BITS_52) && !defined(CONFIG_ARM64_LPA2)
+#ifndef CONFIG_ARM64_LPA2
 	mrs	\tmp, tcr_el1
 	and	\tmp, \tmp, #TCR_T1SZ_MASK
 	cmp	\tmp, #TCR_T1SZ(VA_BITS_MIN)
diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index 0480c61dbb4f..e138deb78bee 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -53,15 +53,11 @@
 #define PCI_IO_END		(PCI_IO_START + PCI_IO_SIZE)
 #define FIXADDR_TOP		(-UL(SZ_8M))
 
-#if VA_BITS > 48
 #ifdef CONFIG_ARM64_16K_PAGES
 #define VA_BITS_MIN		(47)
 #else
 #define VA_BITS_MIN		(48)
 #endif
-#else
-#define VA_BITS_MIN		(VA_BITS)
-#endif
 
 #define _PAGE_END(va)		(-(UL(1) << ((va) - 1)))
 
diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index fd330c1db289..a7e0c0f3c6c8 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -336,10 +336,8 @@
 #define TTBR_BADDR_MASK_52	GENMASK_ULL(47, 2)
 #endif
 
-#ifdef CONFIG_ARM64_VA_BITS_52
 /* Must be at least 64-byte aligned to prevent corruption of the TTBR */
 #define TTBR1_BADDR_4852_OFFSET	(((UL(1) << (52 - PGDIR_SHIFT)) - \
 				 (UL(1) << (48 - PGDIR_SHIFT))) * 8)
-#endif
 
 #endif
diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
index 1bf1a3b16e88..62f30d8f9738 100644
--- a/arch/arm64/include/asm/processor.h
+++ b/arch/arm64/include/asm/processor.h
@@ -50,9 +50,9 @@
  * TASK_UNMAPPED_BASE - the lower boundary of the mmap VM area.
  */
 
-#define DEFAULT_MAP_WINDOW_64	(UL(1) << VA_BITS_MIN)
-#define TASK_SIZE_64		(UL(1) << vabits_actual)
-#define TASK_SIZE_MAX		(UL(1) << VA_BITS)
+#define DEFAULT_MAP_WINDOW_64	(UL(1) << MIN(VA_BITS_MIN, CONFIG_TASK_SIZE_BITS))
+#define TASK_SIZE_64		(UL(1) << MIN(vabits_actual, CONFIG_TASK_SIZE_BITS))
+#define TASK_SIZE_MAX		(UL(1) << CONFIG_TASK_SIZE_BITS)
 
 #ifdef CONFIG_COMPAT
 #if defined(CONFIG_ARM64_64K_PAGES) && defined(CONFIG_KUSER_HELPERS)
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 37e4c02e0272..c36c66562c0a 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -2866,7 +2866,6 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.cpu_enable = cpu_enable_fpmr,
 		ARM64_CPUID_FIELDS(ID_AA64PFR2_EL1, FPMR, IMP)
 	},
-#ifdef CONFIG_ARM64_VA_BITS_52
 	{
 		.capability = ARM64_HAS_VA52,
 		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
@@ -2883,7 +2882,6 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 #endif
 #endif
 	},
-#endif
 	{
 		.desc = "NV1",
 		.capability = ARM64_HAS_HCR_NV1,
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 5ab1970ee543..15e9c1c5547d 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -380,11 +380,9 @@ SYM_FUNC_START_LOCAL(secondary_startup)
 	 */
 	mov	x20, x0				// preserve boot mode
 
-#ifdef CONFIG_ARM64_VA_BITS_52
 alternative_if ARM64_HAS_VA52
 	bl	__cpu_secondary_check52bitva
 alternative_else_nop_endif
-#endif
 
 	bl	__cpu_setup			// initialise processor
 	adrp	x1, swapper_pg_dir
@@ -488,7 +486,6 @@ SYM_FUNC_START(__enable_mmu)
 	ret
 SYM_FUNC_END(__enable_mmu)
 
-#ifdef CONFIG_ARM64_VA_BITS_52
 SYM_FUNC_START(__cpu_secondary_check52bitva)
 #ifndef CONFIG_ARM64_LPA2
 	mrs_s	x0, SYS_ID_AA64MMFR2_EL1
@@ -509,7 +506,6 @@ SYM_FUNC_START(__cpu_secondary_check52bitva)
 
 2:	ret
 SYM_FUNC_END(__cpu_secondary_check52bitva)
-#endif
 
 SYM_FUNC_START_LOCAL(__no_granule_support)
 	/* Indicate that this CPU can't boot and is stuck in the kernel */
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index d21f67d67cf5..c4349852846f 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -236,8 +236,8 @@ void __init arm64_memblock_init(void)
 	 * we have to move it upward. Since memstart_addr represents the
 	 * physical address of PAGE_OFFSET, we have to *subtract* from it.
 	 */
-	if (IS_ENABLED(CONFIG_ARM64_VA_BITS_52) && (vabits_actual != 52))
-		memstart_addr -= _PAGE_OFFSET(vabits_actual) - _PAGE_OFFSET(52);
+	if (vabits_actual != VA_BITS)
+		memstart_addr -= _PAGE_OFFSET(vabits_actual) - _PAGE_OFFSET(VA_BITS);
 
 	/*
 	 * Apply the memory limit if it was set. Since the kernel may be loaded
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 8abdc7fed321..779f589f947c 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -472,7 +472,6 @@ SYM_FUNC_START(__cpu_setup)
 
 	tcr_clear_errata_bits tcr, x9, x5
 
-#ifdef CONFIG_ARM64_VA_BITS_52
 	mov		x9, #64 - VA_BITS
 alternative_if ARM64_HAS_VA52
 	tcr_set_t1sz	tcr, x9
@@ -480,7 +479,6 @@ alternative_if ARM64_HAS_VA52
 	orr		tcr, tcr, #TCR_DS
 #endif
 alternative_else_nop_endif
-#endif
 
 	/*
 	 * Set the IPS bits in TCR_EL1.
-- 
2.47.0.163.g1226f6d8fa-goog

[RFC PATCH 8/8] arm64/mm: Account for reduced VA sizes in T0SZ and skip the levels

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

Now that a smaller value for TASK_SIZE is used when running with a
reduced virtual address space for userland, it is guaranteed that only
the first entry of each root level page table is populated. This means
that we can reduce the number of levels of translation performed by the
MMU by programming this entry into TTBR0_EL1 directly, and updating T0SZ
accordingly.

This is a quick and dirty hack, but should reap all the benefits in
terms of MMU performance and reduced TLB pressure, at the cost of one
wasted page per process (or 2 on 52-bit VA capable hardware).

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/include/asm/mmu_context.h | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index 48b3d9553b67..99777da39228 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -57,7 +57,13 @@ void cpu_do_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm);
 
 static inline void cpu_switch_mm(pgd_t *pgd, struct mm_struct *mm)
 {
+	int advance = (vabits_actual - CONFIG_TASK_SIZE_BITS) / (PAGE_SHIFT - 3);
+
 	BUG_ON(pgd == swapper_pg_dir);
+
+	while (advance-- > 0)
+		pgd = __va(__pgd_to_phys(*pgd));
+
 	cpu_do_switch_mm(virt_to_phys(pgd),mm);
 }
 
@@ -82,7 +88,8 @@ static inline void __cpu_set_tcr_t0sz(unsigned long t0sz)
 	isb();
 }
 
-#define cpu_set_default_tcr_t0sz()	__cpu_set_tcr_t0sz(TCR_T0SZ(vabits_actual))
+#define cpu_set_default_tcr_t0sz()	__cpu_set_tcr_t0sz(TCR_T0SZ(MIN(vabits_actual, \
+									CONFIG_TASK_SIZE_BITS)))
 #define cpu_set_idmap_tcr_t0sz()	__cpu_set_tcr_t0sz(idmap_t0sz)
 
 /*
-- 
2.47.0.163.g1226f6d8fa-goog

Re: [RFC PATCH 7/8] arm64/mm: Use reduced VA sizes (36/39/42 bits) only for user space

[Marc Zyngier]

On Wed, 30 Oct 2024 10:18:11 +0000,
Ard Biesheuvel <ardb+git@google.com> wrote:
> 
> From: Ard Biesheuvel <ardb@kernel.org>
> 
> The advantage of a reduced virtual address space size is its impact on
> the number of translation levels, which affects TLB pressure. The
> working set of translations covering the kernel side is negligible
> compared to user space, where each process has its own set of page
> tables, and so most of the same benefit can be obtained by reducing the
> VA size only for user space.
> 
> As a preparatory step towards implementing this, drop all the reduced VA
> space sizes in Kconfig, and replace it with a configurable userland VA
> space size that is reflected in TASK_SIZE. This will be taken advantage
> of in a subsequent patch to actually reduce the number of translations
> used by the MMU for translating user space virtual addresses.

I think this may have an impact on KVM's walking of the userspace page
tables to determine whether we are trying to install a block mapping,
which assumes that the start level and the number of VA bits are the
same as the kernel (see get_user_mapping_size()).

Probably nothing too complicated, but something to look into.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [RFC PATCH 7/8] arm64/mm: Use reduced VA sizes (36/39/42 bits) only for user space

[Ard Biesheuvel]

On Wed, 30 Oct 2024 at 13:44, Marc Zyngier <maz@kernel.org> wrote:
>
> On Wed, 30 Oct 2024 10:18:11 +0000,
> Ard Biesheuvel <ardb+git@google.com> wrote:
> >
> > From: Ard Biesheuvel <ardb@kernel.org>
> >
> > The advantage of a reduced virtual address space size is its impact on
> > the number of translation levels, which affects TLB pressure. The
> > working set of translations covering the kernel side is negligible
> > compared to user space, where each process has its own set of page
> > tables, and so most of the same benefit can be obtained by reducing the
> > VA size only for user space.
> >
> > As a preparatory step towards implementing this, drop all the reduced VA
> > space sizes in Kconfig, and replace it with a configurable userland VA
> > space size that is reflected in TASK_SIZE. This will be taken advantage
> > of in a subsequent patch to actually reduce the number of translations
> > used by the MMU for translating user space virtual addresses.
>
> I think this may have an impact on KVM's walking of the userspace page
> tables to determine whether we are trying to install a block mapping,
> which assumes that the start level and the number of VA bits are the
> same as the kernel (see get_user_mapping_size()).
>
> Probably nothing too complicated, but something to look into.
>

With the crude hack I used, things should just work, unless KVM reads
back the value of TTBR0_EL1.

But once that bit is implemented properly, it is definitely something
to be aware of. Thanks.