[PATCH 0/6] arm64: alternatives: switch most used alternatives to callbacks

[PATCH 0/6] arm64: alternatives: switch most used alternatives to callbacks

[Ada Couprie Diaz]

Hello,

This series looks at some size reductions for the kernel by replacing
the most used instruction-based alternatives with callbacks, implementing
ideas of my previous RFC on the insn framework[0] and taking into account
Marc's comments.
The goal here is to create specialized callbacks rather than overhauling
the whole insn framework, which lead to using bitwise operations to modify
the original instructions in most cases.

Using the alternatives dump produced by Mark's patch[0] on v7.3-rc4 defconfig,
the most common instruction-based alternatives are :
	1. 10k entries, `ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE`,
	2. 10k entries, `ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027`,
	3. 3k entries, `ARM64_HAS_VIRT_HOST_EXTN`,
representing a potential maximum 92kB space saved on alternative instructions.

Sadly this is not realized in practice, but I wanted to share the results
below and see what the mailing list thinks.

All in all, it feels like a fair bit of changes for 1% gains on the Image size
and an increase in size for vmlinux.
The gains on the I/O workarounds are more clear cut and could be taken
by themselves, but remain quite slim.

I would be curious if other configurations benefit more, and the impact
when tested on high core-count machines.

=== Structure

- Patches 1 and 2 are preparatory commits for patch 3 to make
  the insn functions used safe to be called when patching alternatives ;
- Patch 5 is an optional clean-up of hard-coded values, to be re-used
  in patch 6 ;
- Patches 3, 4 and 6 are the actual alternatives changes
  - Patch 3 : replaces the NVIDIA workaround to device I/O reads
  - Patch 4 : replaces the Cortex-A57 workaround to device I/O reads
  - Patch 6 : replaces the TPIDR_EL1 to TPIDR_EL2 switch for per-cpu

=== Results

All numbers are on v7.3-rc4 building defconfig with GCC 13.3.0.
Only the patches mentioned are applied on each line.

|   Patches    |  Size (B) |	|    Patch    | Size (B) |
| Base vmlinux | 172826400 |	|  Base Image | 52374016 |
|          1-3 |    -71672 |	|         1-3 |   -65536 |
|            4 |    -71704 |    |           4 |   -65536 |
|          1-4 |    -64712 |	|         1-4 |   -65536 |
|          5-6 |    +68008 |	|         5-6 |       -0 |
|  All patches |    + 3240 |    | All patches |   -65536 |

The impact on alternatives is two-fold :
1. As expected, all alternatives for the two I/O workarounds and
   97% of the `ARM64_HAS_VIRT_HOST_EXTN` alternatives are converted to
   callbacks, saving 85848 bytes (20 pages).
2. The raw number of alternatives *increases* by about 3% (250 new entries)

Almost all alternatives that are patched later, e.g. drivers loading,
are now using callbacks as well.

From my limited testing on a Morello board, boot time seems unaffected.

=== Analysis

I was a bit perplexed by the results, not expecting either the new entries
nor the increase in size due to the TPIDR patch.

Using `readelf`, it appears that most of the increase in size comes from
debug sections massively increasing with the TPIDR patch,
specifically `.debug_line` and `.debug_info`.
Most of the new alternative entries also come from the TPIDR patch,
which is probably due to the compiler making different optimization decisions.

Interestingly, as most of the size increase is in the debug sections,
`CONFIG_DEBUG_INFO_COMPRESSED_` options have a stronger impact. Despite the
increase in size with all patches, compressed debug info shrinks vmlinux
by 57856 bytes compared to v7.3-rc4 with the same debug info compression.

I experimented with adding `volatile` to the asm block in
`__kern_my_cpu_offset()` to see how that would impact both the size and
the amount of alternatives.
Without the patches, it does add 200 new alternatie entries but shrinks
`vmlinux` by 100 kB. The patches add 260 more alternatives and shrinks
`vmlinux` by another 2 kB.
So it looks like compiler optimization would be a large cause of the change.
I have not tested the effects on performance of this change, but given
the comment it would probably be detrimentary so I elected not to include it.

Previous to this series, I also looked into creating a callback to handle
reading and writing to system registers with exception level variants,
mostly used in KVM, but with no benefit. I will share the findings below
in case they can be useful to anyone.



I am curious about what people think, thank you in advance !
Kind regards,
Ada

Based on v7.3-rc4.

[0]: https://lore.kernel.org/r/20250923174903.76283-1-ada.coupriediaz@arm.com 
[1]: https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/commit/?id=ee8dd3d5de66e92a98896d6f316a71ec815cd51e

Ada Couprie Diaz (6):
  arm64: insn: remove deprecated memory barrier types
  arm64: insn: make `aarch64_insn_gen_d{m,s}b()` alternative-safe
  arm64: io: replace NVIDIA Olympus erratum alternative with callback
  arm64: io: replace ARM erratum 832075 alternative with callback
  arm64: insn: operate on MSR/MRS sysreg field via defines
  arm64: use alternatie callback to patch TPIDR_EL1 accesses

 arch/arm64/include/asm/alternative.h |  3 ++
 arch/arm64/include/asm/assembler.h   | 12 ++---
 arch/arm64/include/asm/insn.h        | 65 +++++++++++++++++++++++----
 arch/arm64/include/asm/io.h          | 51 ++++++++++++---------
 arch/arm64/include/asm/percpu.h      | 12 ++---
 arch/arm64/kernel/alternative.c      | 25 +++++++++++
 arch/arm64/kernel/image-vars.h       |  3 ++
 arch/arm64/kernel/io.c               | 49 +++++++++++++++++++++
 arch/arm64/lib/insn.c                | 66 ++--------------------------
 9 files changed, 182 insertions(+), 104 deletions(-)


base-commit: 93f51579e7df248780214094418f205253383cc5
-- 
2.43.0


=== Abandoned : system registers with _ELx variants

KVM uses alternative instructions to access the correct registers
for the desired exception level when in non-VHE mode.
This is used in a good number of places in KVM, so I wondered if
a callback could save some space.

Going through the system register encodings, the following pattern
emerges for registers with multiple variants depending on EL :
- CRn, CRm and op2 are identical
- op1 encodes the exception level of the register access :
  - EL0  : 0b011
  - EL1  : 0b000
  - EL2  : 0b100
  - EL3  : 0b110
  - EL02 : 0b101
  - EL12 : 0b101

So it looked like it would be possible to bit-mask the op1 field per-EL.

However, among the registers with ELx variants there were exceptions :
- TPDIR_EL1 and TPDIR_EL2, which are used for per-cpu functions,
  do not share the same op2 field
- BRBCR_EL1's op1 does not match above pattern
- MAIR2_EL{1,12} and MAIR2_EL{2,3} pairs have different CRm fields
- SP_EL{0,1}'s op1 do not match the above pattern

And, in general :
- Arm ARM DDI 0487 C5.1.1[2] mentions that op1 has different values for EL1
- Indeed, many registers outside of those with multiple ELx variants do not fit
  the above pattern
- There is no hard guarantee that the above logic will be maintained long term


Considering the above, I reimplemented `{read,write}_sysreg_elx` with a
compile time check switching between the current alternative and
the bitmasking callback for the instructions that do match the pattern.

This did convert about 200 alternative entries to callbacks, but because of
the increase in size of debug sections and symbol tables, this only made
`vmlinux` bigger by about 3 kB with no change for the Image.

Given the negative impact on size, limited applicability and benefits,
I did not move further with this.
However, it might be useful to simplify some of the awk script generating
sysreg defines.

[2]: https://support.arm.com/documentation/ddi0487/mc (version M.c)

[PATCH 1/6] arm64: insn: remove deprecated memory barrier types

[Ada Couprie Diaz]

Arm ARM DDI 0487 version M.c[0] deprecated a number of types
used as options for the DMB and DSB instructions, with no change
in behaviour.
The deprecated types are not used in the kernel, so remove them
from our instruction generation framework and add the missing
`AARCH64_INSN_MB_OSH` case in `__get_barrier_crm_val()`.

This simplifies the switch-case and now covers all the enum types,
which allows for making the DSB and DMB generation non instrumentable
in a follow-up commit.

[0]: https://support.arm.com/documentation/ddi0487/mc/
     (See C6.2.150, C6.2.152)

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
---
 arch/arm64/include/asm/insn.h |  6 ------
 arch/arm64/lib/insn.c         | 10 ++--------
 2 files changed, 2 insertions(+), 14 deletions(-)

diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 1ce75a4426387..8ec9d86cc575c 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -302,14 +302,8 @@ enum aarch64_insn_mb_type {
 	AARCH64_INSN_MB_ST,
 	AARCH64_INSN_MB_LD,
 	AARCH64_INSN_MB_ISH,
-	AARCH64_INSN_MB_ISHST,
-	AARCH64_INSN_MB_ISHLD,
 	AARCH64_INSN_MB_NSH,
-	AARCH64_INSN_MB_NSHST,
-	AARCH64_INSN_MB_NSHLD,
 	AARCH64_INSN_MB_OSH,
-	AARCH64_INSN_MB_OSHST,
-	AARCH64_INSN_MB_OSHLD,
 };
 
 #define	__AARCH64_INSN_FUNCS(abbr, mask, val)				\
diff --git a/arch/arm64/lib/insn.c b/arch/arm64/lib/insn.c
index e70ac02385153..c70820d65805f 100644
--- a/arch/arm64/lib/insn.c
+++ b/arch/arm64/lib/insn.c
@@ -1572,16 +1572,10 @@ static u32 __get_barrier_crm_val(enum aarch64_insn_mb_type type)
 		return 0xd;
 	case AARCH64_INSN_MB_ISH:
 		return 0xb;
-	case AARCH64_INSN_MB_ISHST:
-		return 0xa;
-	case AARCH64_INSN_MB_ISHLD:
-		return 0x9;
 	case AARCH64_INSN_MB_NSH:
 		return 0x7;
-	case AARCH64_INSN_MB_NSHST:
-		return 0x6;
-	case AARCH64_INSN_MB_NSHLD:
-		return 0x5;
+	case AARCH64_INSN_MB_OSH:
+		return 0x3;
 	default:
 		pr_err("%s: unknown barrier type %d\n", __func__, type);
 		return AARCH64_BREAK_FAULT;
-- 
2.43.0

Re: [PATCH 1/6] arm64: insn: remove deprecated memory barrier types

[Vladimir Murzin]

Hi Ada,

On 9/28/26 14:30, Ada Couprie Diaz wrote:
> Arm ARM DDI 0487 version M.c[0] deprecated a number of types
> used as options for the DMB and DSB instructions, with no change
> in behaviour.
> The deprecated types are not used in the kernel, so remove them
> from our instruction generation framework and add the missing
> `AARCH64_INSN_MB_OSH` case in `__get_barrier_crm_val()`.
> 

Arm ARM says that OSH has the same behaviour as SY, so I'd assume
any user of OSH should be fine to upgrade to SY, no?

> This simplifies the switch-case and now covers all the enum types,
> which allows for making the DSB and DMB generation non instrumentable
> in a follow-up commit.
> 
> [0]: https://support.arm.com/documentation/ddi0487/mc/
>      (See C6.2.150, C6.2.152)
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> ---
>  arch/arm64/include/asm/insn.h |  6 ------
>  arch/arm64/lib/insn.c         | 10 ++--------
>  2 files changed, 2 insertions(+), 14 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
> index 1ce75a4426387..8ec9d86cc575c 100644
> --- a/arch/arm64/include/asm/insn.h
> +++ b/arch/arm64/include/asm/insn.h
> @@ -302,14 +302,8 @@ enum aarch64_insn_mb_type {
>  	AARCH64_INSN_MB_ST,
>  	AARCH64_INSN_MB_LD,
>  	AARCH64_INSN_MB_ISH,
> -	AARCH64_INSN_MB_ISHST,
> -	AARCH64_INSN_MB_ISHLD,
>  	AARCH64_INSN_MB_NSH,
> -	AARCH64_INSN_MB_NSHST,
> -	AARCH64_INSN_MB_NSHLD,
>  	AARCH64_INSN_MB_OSH,
> -	AARCH64_INSN_MB_OSHST,
> -	AARCH64_INSN_MB_OSHLD,
>  };
>  
>  #define	__AARCH64_INSN_FUNCS(abbr, mask, val)				\
> diff --git a/arch/arm64/lib/insn.c b/arch/arm64/lib/insn.c
> index e70ac02385153..c70820d65805f 100644
> --- a/arch/arm64/lib/insn.c
> +++ b/arch/arm64/lib/insn.c
> @@ -1572,16 +1572,10 @@ static u32 __get_barrier_crm_val(enum aarch64_insn_mb_type type)
>  		return 0xd;
>  	case AARCH64_INSN_MB_ISH:
>  		return 0xb;
> -	case AARCH64_INSN_MB_ISHST:
> -		return 0xa;
> -	case AARCH64_INSN_MB_ISHLD:
> -		return 0x9;
>  	case AARCH64_INSN_MB_NSH:
>  		return 0x7;
> -	case AARCH64_INSN_MB_NSHST:
> -		return 0x6;
> -	case AARCH64_INSN_MB_NSHLD:
> -		return 0x5;
> +	case AARCH64_INSN_MB_OSH:
> +		return 0x3;
>  	default:
>  		pr_err("%s: unknown barrier type %d\n", __func__, type);
>  		return AARCH64_BREAK_FAULT;
> -- 2.43.0
> 

It looks like NSH, ISH and OSH applies to DSB and deprecated in case of DMB
which is slightly confusing. Given that we drop bunch of encodings perhaps we
can inline logic in aarch64_insn_gen_d{m,s}b()?

Cheers
Vladimir

Re: [PATCH 1/6] arm64: insn: remove deprecated memory barrier types

[Ada Couprie Diaz]

Hi Vladimir,

On 07/10/2026 07:21, Vladimir Murzin wrote:
> Hi Ada,
>
> On 9/28/26 14:30, Ada Couprie Diaz wrote:
>> Arm ARM DDI 0487 version M.c[0] deprecated a number of types
>> used as options for the DMB and DSB instructions, with no change
>> in behaviour.
>> The deprecated types are not used in the kernel, so remove them
>> from our instruction generation framework and add the missing
>> `AARCH64_INSN_MB_OSH` case in `__get_barrier_crm_val()`.
>>
> Arm ARM says that OSH has the same behaviour as SY, so I'd assume
> any user of OSH should be fine to upgrade to SY, no?

As you mention below, OSH is _technically_ not deprecated for DSB, so I think
it would make sense to add it for completion.
Though, given that it is indeed equivalent I can see the argument for leaving it.

I initially wanted to silently convert deprecated encodings to SY and co.,
Mark was against the idea but I guess it could be OK just not providing them.

>> This simplifies the switch-case and now covers all the enum types,
>> which allows for making the DSB and DMB generation non instrumentable
>> in a follow-up commit.
>>
>> [0]: https://support.arm.com/documentation/ddi0487/mc/
>>       (See C6.2.150, C6.2.152)
>>
>> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>> ---
>>   arch/arm64/include/asm/insn.h |  6 ------
>>   arch/arm64/lib/insn.c         | 10 ++--------
>>   2 files changed, 2 insertions(+), 14 deletions(-)
>>
>> diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
>> index 1ce75a4426387..8ec9d86cc575c 100644
>> --- a/arch/arm64/include/asm/insn.h
>> +++ b/arch/arm64/include/asm/insn.h
>> @@ -302,14 +302,8 @@ enum aarch64_insn_mb_type {
>>   	AARCH64_INSN_MB_ST,
>>   	AARCH64_INSN_MB_LD,
>>   	AARCH64_INSN_MB_ISH,
>> -	AARCH64_INSN_MB_ISHST,
>> -	AARCH64_INSN_MB_ISHLD,
>>   	AARCH64_INSN_MB_NSH,
>> -	AARCH64_INSN_MB_NSHST,
>> -	AARCH64_INSN_MB_NSHLD,
>>   	AARCH64_INSN_MB_OSH,
>> -	AARCH64_INSN_MB_OSHST,
>> -	AARCH64_INSN_MB_OSHLD,
>>   };
>>   
>>   #define	__AARCH64_INSN_FUNCS(abbr, mask, val)				\
>> diff --git a/arch/arm64/lib/insn.c b/arch/arm64/lib/insn.c
>> index e70ac02385153..c70820d65805f 100644
>> --- a/arch/arm64/lib/insn.c
>> +++ b/arch/arm64/lib/insn.c
>> @@ -1572,16 +1572,10 @@ static u32 __get_barrier_crm_val(enum aarch64_insn_mb_type type)
>>   		return 0xd;
>>   	case AARCH64_INSN_MB_ISH:
>>   		return 0xb;
>> -	case AARCH64_INSN_MB_ISHST:
>> -		return 0xa;
>> -	case AARCH64_INSN_MB_ISHLD:
>> -		return 0x9;
>>   	case AARCH64_INSN_MB_NSH:
>>   		return 0x7;
>> -	case AARCH64_INSN_MB_NSHST:
>> -		return 0x6;
>> -	case AARCH64_INSN_MB_NSHLD:
>> -		return 0x5;
>> +	case AARCH64_INSN_MB_OSH:
>> +		return 0x3;
>>   	default:
>>   		pr_err("%s: unknown barrier type %d\n", __func__, type);
>>   		return AARCH64_BREAK_FAULT;
>> -- 2.43.0
>>
> It looks like NSH, ISH and OSH applies to DSB and deprecated in case of DMB
> which is slightly confusing. Given that we drop bunch of encodings perhaps we
> can inline logic in aarch64_insn_gen_d{m,s}b()?

Yeah I can see an argument for that, I'm not against it and it probably makes
the error handling in patch 2 a bit cleaner...

> Cheers
> Vladimir
Thanks for the comments !
Kind regards,
Ada

Re: [PATCH 1/6] arm64: insn: remove deprecated memory barrier types

[Vladimir Murzin]

Hi Ada,

On 10/7/26 18:05, Ada Couprie Diaz wrote:
> Hi Vladimir,
> 
> On 07/10/2026 07:21, Vladimir Murzin wrote:
>> Hi Ada,
>>
>> On 9/28/26 14:30, Ada Couprie Diaz wrote:
>>> Arm ARM DDI 0487 version M.c[0] deprecated a number of types
>>> used as options for the DMB and DSB instructions, with no change
>>> in behaviour.
>>> The deprecated types are not used in the kernel, so remove them
>>> from our instruction generation framework and add the missing
>>> `AARCH64_INSN_MB_OSH` case in `__get_barrier_crm_val()`.
>>>
>> Arm ARM says that OSH has the same behaviour as SY, so I'd assume
>> any user of OSH should be fine to upgrade to SY, no?
> 
> As you mention below, OSH is _technically_ not deprecated for DSB, so I think
> it would make sense to add it for completion.
> Though, given that it is indeed equivalent I can see the argument for leaving it.
> 

Ahh, missed that. I think if we go for inlining barrier encodings into 
aarch64_insn_gen_d{m,s}b() then we can keep OSH for completion.

> I initially wanted to silently convert deprecated encodings to SY and co.,
> Mark was against the idea but I guess it could be OK just not providing them.
> 
>>> This simplifies the switch-case and now covers all the enum types,
>>> which allows for making the DSB and DMB generation non instrumentable
>>> in a follow-up commit.
>>>
>>> [0]: https://support.arm.com/documentation/ddi0487/mc/
>>>       (See C6.2.150, C6.2.152)
>>>
>>> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>>> ---
>>>   arch/arm64/include/asm/insn.h |  6 ------
>>>   arch/arm64/lib/insn.c         | 10 ++--------
>>>   2 files changed, 2 insertions(+), 14 deletions(-)
>>>
>>> diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
>>> index 1ce75a4426387..8ec9d86cc575c 100644
>>> --- a/arch/arm64/include/asm/insn.h
>>> +++ b/arch/arm64/include/asm/insn.h
>>> @@ -302,14 +302,8 @@ enum aarch64_insn_mb_type {
>>>       AARCH64_INSN_MB_ST,
>>>       AARCH64_INSN_MB_LD,
>>>       AARCH64_INSN_MB_ISH,
>>> -    AARCH64_INSN_MB_ISHST,
>>> -    AARCH64_INSN_MB_ISHLD,
>>>       AARCH64_INSN_MB_NSH,
>>> -    AARCH64_INSN_MB_NSHST,
>>> -    AARCH64_INSN_MB_NSHLD,
>>>       AARCH64_INSN_MB_OSH,
>>> -    AARCH64_INSN_MB_OSHST,
>>> -    AARCH64_INSN_MB_OSHLD,
>>>   };
>>>     #define    __AARCH64_INSN_FUNCS(abbr, mask, val)                \
>>> diff --git a/arch/arm64/lib/insn.c b/arch/arm64/lib/insn.c
>>> index e70ac02385153..c70820d65805f 100644
>>> --- a/arch/arm64/lib/insn.c
>>> +++ b/arch/arm64/lib/insn.c
>>> @@ -1572,16 +1572,10 @@ static u32 __get_barrier_crm_val(enum aarch64_insn_mb_type type)
>>>           return 0xd;
>>>       case AARCH64_INSN_MB_ISH:
>>>           return 0xb;
>>> -    case AARCH64_INSN_MB_ISHST:
>>> -        return 0xa;
>>> -    case AARCH64_INSN_MB_ISHLD:
>>> -        return 0x9;
>>>       case AARCH64_INSN_MB_NSH:
>>>           return 0x7;
>>> -    case AARCH64_INSN_MB_NSHST:
>>> -        return 0x6;
>>> -    case AARCH64_INSN_MB_NSHLD:
>>> -        return 0x5;
>>> +    case AARCH64_INSN_MB_OSH:
>>> +        return 0x3;
>>>       default:
>>>           pr_err("%s: unknown barrier type %d\n", __func__, type);
>>>           return AARCH64_BREAK_FAULT;
>>> -- 2.43.0
>>>
>> It looks like NSH, ISH and OSH applies to DSB and deprecated in case of DMB
>> which is slightly confusing. Given that we drop bunch of encodings perhaps we
>> can inline logic in aarch64_insn_gen_d{m,s}b()?
> 
> Yeah I can see an argument for that, I'm not against it and it probably makes
> the error handling in patch 2 a bit cleaner...
> 

Happy to take a look!

Thanks!

>> Cheers
>> Vladimir
> Thanks for the comments !
> Kind regards,
> Ada

[PATCH 2/6] arm64: insn: make `aarch64_insn_gen_d{m,s}b()` alternative-safe

[Ada Couprie Diaz]

Move `aarch64_insn_gen_d{m,s}b()` and `__get_barrier_crm_val()`
to `insn.h` and mark them `__always_inline`.
Replace the error print by a build-time check now that the compiler can
verify all call sites.

This is possible thanks to all enum types being covered in the previous
commit, and the only current caller being BPF with static arguments.

As all functions are now inlined and do not call any instrumentable
function, they are now safe to call in alternative callbacks.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
---
 arch/arm64/include/asm/insn.h | 56 +++++++++++++++++++++++++++++++++--
 arch/arm64/lib/insn.c         | 52 --------------------------------
 2 files changed, 54 insertions(+), 54 deletions(-)

diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 8ec9d86cc575c..40f13d28a5fd7 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -7,6 +7,7 @@
  */
 #ifndef	__ASM_INSN_H
 #define	__ASM_INSN_H
+#include <linux/bits.h>
 #include <linux/build_bug.h>
 #include <linux/types.h>
 
@@ -699,8 +700,59 @@ u32 aarch64_insn_gen_cas(enum aarch64_insn_register result,
 			 enum aarch64_insn_register value,
 			 enum aarch64_insn_size_type size,
 			 enum aarch64_insn_mem_order_type order);
-u32 aarch64_insn_gen_dmb(enum aarch64_insn_mb_type type);
-u32 aarch64_insn_gen_dsb(enum aarch64_insn_mb_type type);
+
+static __always_inline u32 __get_barrier_crm_val(enum aarch64_insn_mb_type type)
+{
+	switch (type) {
+	case AARCH64_INSN_MB_SY:
+		return 0xf;
+	case AARCH64_INSN_MB_ST:
+		return 0xe;
+	case AARCH64_INSN_MB_LD:
+		return 0xd;
+	case AARCH64_INSN_MB_ISH:
+		return 0xb;
+	case AARCH64_INSN_MB_NSH:
+		return 0x7;
+	case AARCH64_INSN_MB_OSH:
+		return 0x3;
+	default:
+		BUILD_BUG_ON_MSG(1, "unknown barrier type");
+		return AARCH64_BREAK_FAULT;
+	}
+}
+
+static __always_inline u32 aarch64_insn_gen_dmb(enum aarch64_insn_mb_type type)
+{
+	u32 opt;
+	u32 insn;
+
+	opt = __get_barrier_crm_val(type);
+	if (opt == AARCH64_BREAK_FAULT)
+		return AARCH64_BREAK_FAULT;
+
+	insn = aarch64_insn_get_dmb_value();
+	insn &= ~GENMASK(11, 8);
+	insn |= (opt << 8);
+
+	return insn;
+}
+
+static __always_inline u32 aarch64_insn_gen_dsb(enum aarch64_insn_mb_type type)
+{
+	u32 opt, insn;
+
+	opt = __get_barrier_crm_val(type);
+	if (opt == AARCH64_BREAK_FAULT)
+		return AARCH64_BREAK_FAULT;
+
+	insn = aarch64_insn_get_dsb_base_value();
+	insn &= ~GENMASK(11, 8);
+	insn |= (opt << 8);
+
+	return insn;
+}
+
 u32 aarch64_insn_gen_mrs(enum aarch64_insn_register result,
 			 enum aarch64_insn_system_register sysreg);
 
diff --git a/arch/arm64/lib/insn.c b/arch/arm64/lib/insn.c
index c70820d65805f..5c37f3a4c402b 100644
--- a/arch/arm64/lib/insn.c
+++ b/arch/arm64/lib/insn.c
@@ -1561,58 +1561,6 @@ u32 aarch64_insn_gen_extr(enum aarch64_insn_variant variant,
 	return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn, Rm);
 }
 
-static u32 __get_barrier_crm_val(enum aarch64_insn_mb_type type)
-{
-	switch (type) {
-	case AARCH64_INSN_MB_SY:
-		return 0xf;
-	case AARCH64_INSN_MB_ST:
-		return 0xe;
-	case AARCH64_INSN_MB_LD:
-		return 0xd;
-	case AARCH64_INSN_MB_ISH:
-		return 0xb;
-	case AARCH64_INSN_MB_NSH:
-		return 0x7;
-	case AARCH64_INSN_MB_OSH:
-		return 0x3;
-	default:
-		pr_err("%s: unknown barrier type %d\n", __func__, type);
-		return AARCH64_BREAK_FAULT;
-	}
-}
-
-u32 aarch64_insn_gen_dmb(enum aarch64_insn_mb_type type)
-{
-	u32 opt;
-	u32 insn;
-
-	opt = __get_barrier_crm_val(type);
-	if (opt == AARCH64_BREAK_FAULT)
-		return AARCH64_BREAK_FAULT;
-
-	insn = aarch64_insn_get_dmb_value();
-	insn &= ~GENMASK(11, 8);
-	insn |= (opt << 8);
-
-	return insn;
-}
-
-u32 aarch64_insn_gen_dsb(enum aarch64_insn_mb_type type)
-{
-	u32 opt, insn;
-
-	opt = __get_barrier_crm_val(type);
-	if (opt == AARCH64_BREAK_FAULT)
-		return AARCH64_BREAK_FAULT;
-
-	insn = aarch64_insn_get_dsb_base_value();
-	insn &= ~GENMASK(11, 8);
-	insn |= (opt << 8);
-
-	return insn;
-}
-
 u32 aarch64_insn_gen_mrs(enum aarch64_insn_register result,
 			 enum aarch64_insn_system_register sysreg)
 {
-- 
2.43.0

[PATCH 3/6] arm64: io: replace NVIDIA Olympus erratum alternative with callback

[Ada Couprie Diaz]

`__raw_read{b,w,l,q}()` represent about 10k call sites
that need to be patched.
Commit 12aab25ca56e ("arm64: errata: work around NVIDIA Olympus
device store/load ordering") implements its workaround with alternative
instructions, adding 10k extra instructions growing the size of the image.

Implement and use `__io_nvidia_olympus_patch_dmb()` as
a callback alternative instead, saving close to 40kB of image size
with a defconfig.

As the alternative fix replaced a NOP with a DMB in all cases,
make use of `aarch64_insn_gen_dmb()` which was made alternative-safe
in previous commits.

The callback needs to be added to the KVM NVHE namespace as `readl()`
is used in the vgic-v2 driver.

Cc: Vikram Sethi <vsethi@nvidia.com>
Cc: Shanker Donthineni <sdonthineni@nvidia.com>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
---
 arch/arm64/include/asm/io.h    | 24 ++++++++++++++++--------
 arch/arm64/kernel/image-vars.h |  1 +
 arch/arm64/kernel/io.c         | 16 ++++++++++++++++
 3 files changed, 33 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h
index 49a7002661a92..0f1ca651c5c79 100644
--- a/arch/arm64/include/asm/io.h
+++ b/arch/arm64/include/asm/io.h
@@ -19,6 +19,10 @@
 #include <asm/cpufeature.h>
 #include <asm/rsi.h>
 
+/* IO-specific callbacks for alternative patching. */
+void __io_nvidia_olympus_patch_dmb(struct alt_instr *alt, __le32 *origptr,
+				   __le32 *updptr, int nr_inst);
+
 /*
  * Generic IO read/write.  These perform native-endian accesses.
  */
@@ -54,8 +58,9 @@ static __always_inline void __raw_writeq(u64 val, volatile void __iomem *addr)
 static __always_inline u8 __raw_readb(const volatile void __iomem *addr)
 {
 	u8 val;
-	asm volatile(ALTERNATIVE("nop", "dmb osh",
-				 ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
+	asm volatile(ALTERNATIVE_CB("nop",
+				    ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027,
+				    __io_nvidia_olympus_patch_dmb)
 		     ALTERNATIVE("ldrb %w0, [%1]",
 				 "ldarb %w0, [%1]",
 				 ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
@@ -68,8 +73,9 @@ static __always_inline u16 __raw_readw(const volatile void __iomem *addr)
 {
 	u16 val;
 
-	asm volatile(ALTERNATIVE("nop", "dmb osh",
-				 ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
+	asm volatile(ALTERNATIVE_CB("nop",
+				    ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027,
+				    __io_nvidia_olympus_patch_dmb)
 		     ALTERNATIVE("ldrh %w0, [%1]",
 				 "ldarh %w0, [%1]",
 				 ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
@@ -81,8 +87,9 @@ static __always_inline u16 __raw_readw(const volatile void __iomem *addr)
 static __always_inline u32 __raw_readl(const volatile void __iomem *addr)
 {
 	u32 val;
-	asm volatile(ALTERNATIVE("nop", "dmb osh",
-				 ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
+	asm volatile(ALTERNATIVE_CB("nop",
+				    ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027,
+				    __io_nvidia_olympus_patch_dmb)
 		     ALTERNATIVE("ldr %w0, [%1]",
 				 "ldar %w0, [%1]",
 				 ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
@@ -94,8 +101,9 @@ static __always_inline u32 __raw_readl(const volatile void __iomem *addr)
 static __always_inline u64 __raw_readq(const volatile void __iomem *addr)
 {
 	u64 val;
-	asm volatile(ALTERNATIVE("nop", "dmb osh",
-				 ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027)
+	asm volatile(ALTERNATIVE_CB("nop",
+				    ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027,
+				    __io_nvidia_olympus_patch_dmb)
 		     ALTERNATIVE("ldr %0, [%1]",
 				 "ldar %0, [%1]",
 				 ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index 14beb7b9d304c..a22519c5223b4 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -94,6 +94,7 @@ KVM_NVHE_ALIAS(spectre_bhb_patch_clearbhb);
 KVM_NVHE_ALIAS(alt_cb_patch_nops);
 KVM_NVHE_ALIAS(kvm_compute_ich_hcr_trap_bits);
 KVM_NVHE_ALIAS(kvm_patch_ich_vtr_el2);
+KVM_NVHE_ALIAS(__io_nvidia_olympus_patch_dmb);
 
 /* Global kernel state accessed by nVHE hyp code. */
 KVM_NVHE_ALIAS(kvm_vgic_global_state);
diff --git a/arch/arm64/kernel/io.c b/arch/arm64/kernel/io.c
index fe86ada23c7d4..8ad3cd1773d20 100644
--- a/arch/arm64/kernel/io.c
+++ b/arch/arm64/kernel/io.c
@@ -9,6 +9,8 @@
 #include <linux/types.h>
 #include <linux/io.h>
 
+#include <asm/insn.h>
+
 /*
  * This generates a memcpy that works on a from/to address which is aligned to
  * bits. Count is in terms of the number of bits sized quantities to copy. It
@@ -50,3 +52,17 @@ void __iowrite32_copy_full(void __iomem *to, const void *from, size_t count)
 	dgh();
 }
 EXPORT_SYMBOL(__iowrite32_copy_full);
+
+noinstr void __io_nvidia_olympus_patch_dmb(struct alt_instr *alt, __le32 *origptr,
+					   __le32 *updptr, int nr_inst)
+{
+	u32 orinst;
+
+	BUG_ON(nr_inst != 1);
+
+	orinst = le32_to_cpu(origptr[0]);
+	BUG_ON(!aarch64_insn_is_nop(orinst));
+
+	updptr[0] = cpu_to_le32(aarch64_insn_gen_dmb(AARCH64_INSN_MB_OSH));
+}
+EXPORT_SYMBOL(__io_nvidia_olympus_patch_dmb);
-- 
2.43.0

[PATCH 4/6] arm64: io: replace ARM erratum 832075 alternative with callback

[Ada Couprie Diaz]

`__raw_read{b,w,l,q}()` represent about 10k call sites
that need to be patched.
Commit 5afaa1fc1b32 ("arm64: add Cortex-A57 erratum 832075 workaround")
implements its workaround with alternative instructions,
adding 10k extra instructions growing the size of the image.

Implement and use `__io_arm_a57_patch_ladr()` as a callback alternative
instead, saving close to 40kB of image size with a defconfig.

Unsigned offset loads and ordered loads encode the size of the load and
the operand registers identically, but ordered loads cannot offset
the base address.
As the alternatives in device loads do not have offsets, we can safely
bitmask the loads and convert them to ordered load as per the erratum fix.

The callback needs to be added to the KVM NVHE namespace as `readl()`
is used in the vgic-v2 driver.

Cc: Andre Przywara <andre.przywara@arm.com>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
---
 arch/arm64/include/asm/io.h    | 27 +++++++++++++++------------
 arch/arm64/kernel/image-vars.h |  1 +
 arch/arm64/kernel/io.c         | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 49 insertions(+), 12 deletions(-)

diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h
index 0f1ca651c5c79..1791081d97dcb 100644
--- a/arch/arm64/include/asm/io.h
+++ b/arch/arm64/include/asm/io.h
@@ -22,6 +22,9 @@
 /* IO-specific callbacks for alternative patching. */
 void __io_nvidia_olympus_patch_dmb(struct alt_instr *alt, __le32 *origptr,
 				   __le32 *updptr, int nr_inst);
+void __io_arm_a57_patch_ladr(struct alt_instr *alt, __le32 *origptr,
+			     __le32 *updptr, int nr_inst);
+
 
 /*
  * Generic IO read/write.  These perform native-endian accesses.
@@ -61,9 +64,9 @@ static __always_inline u8 __raw_readb(const volatile void __iomem *addr)
 	asm volatile(ALTERNATIVE_CB("nop",
 				    ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027,
 				    __io_nvidia_olympus_patch_dmb)
-		     ALTERNATIVE("ldrb %w0, [%1]",
-				 "ldarb %w0, [%1]",
-				 ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
+		     ALTERNATIVE_CB("ldrb %w0, [%1]",
+				    ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE,
+				    __io_arm_a57_patch_ladr)
 		     : "=r" (val) : "r" (addr));
 	return val;
 }
@@ -76,9 +79,9 @@ static __always_inline u16 __raw_readw(const volatile void __iomem *addr)
 	asm volatile(ALTERNATIVE_CB("nop",
 				    ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027,
 				    __io_nvidia_olympus_patch_dmb)
-		     ALTERNATIVE("ldrh %w0, [%1]",
-				 "ldarh %w0, [%1]",
-				 ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
+		     ALTERNATIVE_CB("ldrh %w0, [%1]",
+				    ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE,
+				    __io_arm_a57_patch_ladr)
 		     : "=r" (val) : "r" (addr));
 	return val;
 }
@@ -90,9 +93,9 @@ static __always_inline u32 __raw_readl(const volatile void __iomem *addr)
 	asm volatile(ALTERNATIVE_CB("nop",
 				    ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027,
 				    __io_nvidia_olympus_patch_dmb)
-		     ALTERNATIVE("ldr %w0, [%1]",
-				 "ldar %w0, [%1]",
-				 ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
+		     ALTERNATIVE_CB("ldr %w0, [%1]",
+				    ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE,
+				    __io_arm_a57_patch_ladr)
 		     : "=r" (val) : "r" (addr));
 	return val;
 }
@@ -104,9 +107,9 @@ static __always_inline u64 __raw_readq(const volatile void __iomem *addr)
 	asm volatile(ALTERNATIVE_CB("nop",
 				    ARM64_WORKAROUND_NVIDIA_OLYMPUS_1027,
 				    __io_nvidia_olympus_patch_dmb)
-		     ALTERNATIVE("ldr %0, [%1]",
-				 "ldar %0, [%1]",
-				 ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE)
+		     ALTERNATIVE_CB("ldr %0, [%1]",
+				    ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE,
+				    __io_arm_a57_patch_ladr)
 		     : "=r" (val) : "r" (addr));
 	return val;
 }
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index a22519c5223b4..615b77ef403fe 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -95,6 +95,7 @@ KVM_NVHE_ALIAS(alt_cb_patch_nops);
 KVM_NVHE_ALIAS(kvm_compute_ich_hcr_trap_bits);
 KVM_NVHE_ALIAS(kvm_patch_ich_vtr_el2);
 KVM_NVHE_ALIAS(__io_nvidia_olympus_patch_dmb);
+KVM_NVHE_ALIAS(__io_arm_a57_patch_ladr);
 
 /* Global kernel state accessed by nVHE hyp code. */
 KVM_NVHE_ALIAS(kvm_vgic_global_state);
diff --git a/arch/arm64/kernel/io.c b/arch/arm64/kernel/io.c
index 8ad3cd1773d20..f56bdf0a70822 100644
--- a/arch/arm64/kernel/io.c
+++ b/arch/arm64/kernel/io.c
@@ -66,3 +66,36 @@ noinstr void __io_nvidia_olympus_patch_dmb(struct alt_instr *alt, __le32 *origpt
 	updptr[0] = cpu_to_le32(aarch64_insn_gen_dmb(AARCH64_INSN_MB_OSH));
 }
 EXPORT_SYMBOL(__io_nvidia_olympus_patch_dmb);
+
+/*
+ * Patch unsigned immediate loads to ordered loads for Arm erratum 832075.
+ * The immediate offset of the load MUST BE 0 for this to make any sense,
+ * as ordered loads do not encode any offset.
+ *
+ * This can patch 8, 16, 32 and 64 bits loads as they share the same encoding,
+ * with the two highest bits encoding size.
+ * See Arm ARM DDI 0487 C4.1 "Load/store register (unsigned immediate)" and
+ * "Load/store ordered" for the complete encodings.
+ */
+noinstr void __io_arm_a57_patch_ladr(struct alt_instr *alt, __le32 *origptr,
+				     __le32 *updptr, int nr_inst)
+{
+	u32 orinst, altinst;
+
+	BUG_ON(nr_inst != 1);
+
+	orinst = le32_to_cpu(origptr[0]);
+	BUG_ON(!aarch64_insn_is_load_imm(orinst));
+	BUG_ON((orinst & GENMASK(21, 10)) != 0);
+
+	/*
+	 * Preserve the size (31, 30) and registers (9,0) of the immediate load,
+	 * as they are encoded identically for ordered loads.
+	 */
+	altinst = orinst & ~GENMASK(29, 10);
+	/* The value defined in insn.h includes the RES1 bits and o0. */
+	altinst |= aarch64_insn_get_load_acq_value();
+
+	updptr[0] = cpu_to_le32(altinst);
+}
+EXPORT_SYMBOL(__io_arm_a57_patch_ladr);
-- 
2.43.0

[PATCH 5/6] arm64: insn: operate on MSR/MRS sysreg field via defines

[Ada Couprie Diaz]

Replace the few instances of hard-coded system register offset and masks
used to operate on MSR/MRS instructions with defines.
This will allow re-use in future commits while making the connection
between those values more explicit.

While we are here, mark `aarch64_insn_extract_system_reg()` `noinstr`
so it can be safe to use in alternative pacthing callbacks.

Changing the mask used in `aarch64_insn_gen_mrs()` to exclude
the lower bits does not change behaviour,
as `aarch64_insn_encode_register()` already clears the bits
used to encode the target register.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
---
 arch/arm64/include/asm/insn.h | 3 +++
 arch/arm64/lib/insn.c         | 8 ++++----
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 40f13d28a5fd7..03620f67a3e47 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -753,6 +753,9 @@ static __always_inline u32 aarch64_insn_gen_dsb(enum aarch64_insn_mb_type type)
 	return insn;
 }
 
+#define AARCH64_INSN_SYSREG_OFFSET	5
+#define AARCH64_INSN_SYSREG_MASK	GENMASK(19, 5)
+
 u32 aarch64_insn_gen_mrs(enum aarch64_insn_register result,
 			 enum aarch64_insn_system_register sysreg);
 
diff --git a/arch/arm64/lib/insn.c b/arch/arm64/lib/insn.c
index 5c37f3a4c402b..2edc5a9c0a9c8 100644
--- a/arch/arm64/lib/insn.c
+++ b/arch/arm64/lib/insn.c
@@ -1362,9 +1362,9 @@ u32 aarch64_insn_adrp_set_offset(u32 insn, s32 offset)
 /*
  * Extract the Op/CR data from a msr/mrs instruction.
  */
-u32 aarch64_insn_extract_system_reg(u32 insn)
+noinstr u32 aarch64_insn_extract_system_reg(u32 insn)
 {
-	return (insn & 0x1FFFE0) >> 5;
+	return (insn & AARCH64_INSN_SYSREG_MASK) >> AARCH64_INSN_SYSREG_OFFSET;
 }
 
 bool aarch32_insn_is_wide(u32 insn)
@@ -1566,8 +1566,8 @@ u32 aarch64_insn_gen_mrs(enum aarch64_insn_register result,
 {
 	u32 insn = aarch64_insn_get_mrs_value();
 
-	insn &= ~GENMASK(19, 0);
-	insn |= sysreg << 5;
+	insn &= ~AARCH64_INSN_SYSREG_MASK;
+	insn |= sysreg << AARCH64_INSN_SYSREG_OFFSET;
 	return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RT,
 					    insn, result);
 }
-- 
2.43.0

Re: [PATCH 5/6] arm64: insn: operate on MSR/MRS sysreg field via defines

[Vladimir Murzin]

Hi Ada,

On 9/28/26 14:30, Ada Couprie Diaz wrote:
> Replace the few instances of hard-coded system register offset and masks
> used to operate on MSR/MRS instructions with defines.
> This will allow re-use in future commits while making the connection
> between those values more explicit.
> 
> While we are here, mark `aarch64_insn_extract_system_reg()` `noinstr`
> so it can be safe to use in alternative pacthing callbacks.
> 
> Changing the mask used in `aarch64_insn_gen_mrs()` to exclude
> the lower bits does not change behaviour,
> as `aarch64_insn_encode_register()` already clears the bits
> used to encode the target register.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> ---
>  arch/arm64/include/asm/insn.h | 3 +++
>  arch/arm64/lib/insn.c         | 8 ++++----
>  2 files changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
> index 40f13d28a5fd7..03620f67a3e47 100644
> --- a/arch/arm64/include/asm/insn.h
> +++ b/arch/arm64/include/asm/insn.h
> @@ -753,6 +753,9 @@ static __always_inline u32 aarch64_insn_gen_dsb(enum aarch64_insn_mb_type type)
>  	return insn;
>  }
>  
> +#define AARCH64_INSN_SYSREG_OFFSET	5
> +#define AARCH64_INSN_SYSREG_MASK	GENMASK(19, 5)
> +

Sashiko has raised comment [1]

| Does this mask unintentionally exclude bit 20?
| The previous hardcoded mask was 0x1FFFE0, which is equivalent to
| GENMASK(20, 5) and includes 16 bits. Defining it as GENMASK(19, 5) yields
| a 15-bit mask.

[1] https://sashiko.dev/#/patchset/20260928133034.243541-1-ada.coupriediaz%40arm.com

Cheers
Vladimir

Re: [PATCH 5/6] arm64: insn: operate on MSR/MRS sysreg field via defines

[Ada Couprie Diaz]

Hi Vladimir,

On 07/10/2026 07:53, Vladimir Murzin wrote:
> Hi Ada,
>
> On 9/28/26 14:30, Ada Couprie Diaz wrote:
>> Replace the few instances of hard-coded system register offset and masks
>> used to operate on MSR/MRS instructions with defines.
>> This will allow re-use in future commits while making the connection
>> between those values more explicit.
>>
>> While we are here, mark `aarch64_insn_extract_system_reg()` `noinstr`
>> so it can be safe to use in alternative pacthing callbacks.
>>
>> Changing the mask used in `aarch64_insn_gen_mrs()` to exclude
>> the lower bits does not change behaviour,
>> as `aarch64_insn_encode_register()` already clears the bits
>> used to encode the target register.
>>
>> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>> ---
>>   arch/arm64/include/asm/insn.h | 3 +++
>>   arch/arm64/lib/insn.c         | 8 ++++----
>>   2 files changed, 7 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
>> index 40f13d28a5fd7..03620f67a3e47 100644
>> --- a/arch/arm64/include/asm/insn.h
>> +++ b/arch/arm64/include/asm/insn.h
>> @@ -753,6 +753,9 @@ static __always_inline u32 aarch64_insn_gen_dsb(enum aarch64_insn_mb_type type)
>>   	return insn;
>>   }
>>   
>> +#define AARCH64_INSN_SYSREG_OFFSET	5
>> +#define AARCH64_INSN_SYSREG_MASK	GENMASK(19, 5)
>> +
> Sashiko has raised comment [1]
>
> | Does this mask unintentionally exclude bit 20?
> | The previous hardcoded mask was 0x1FFFE0, which is equivalent to
> | GENMASK(20, 5) and includes 16 bits. Defining it as GENMASK(19, 5) yields
> | a 15-bit mask.
>
> [1] https://sashiko.dev/#/patchset/20260928133034.243541-1-ada.coupriediaz%40arm.com
>
> Cheers
> Vladimir

Thanks for bringing the report up, it does raise an interesting issue.

This does change the mask used and does indeed change the returned value,
potentially breaking comparisons to the return value of `aarch64_insn_extract_system_reg()`. However, the existing mask is 
incorrect as far as I can tell : the function is supposed to extract the 
Op and CR corresponding to the system registers of the MSR/MRS 
instructions. Those fields are encoded in bits 19-5, which corresponds 
to the GENMASK I used. I think it would make more sense to fix the `enum 
aarch64_insn_special_register` values to represent the registers 
properly and drop bit 20, which is 1 in all cases anyway... I would be 
happy to send a quick v2 with this change ! Thanks for bringing it up, 
Kind regards Ada

Re: [PATCH 5/6] arm64: insn: operate on MSR/MRS sysreg field via defines

[Vladimir Murzin]

Hi Ada,

On 10/7/26 17:31, Ada Couprie Diaz wrote:
> Hi Vladimir,
> 
> On 07/10/2026 07:53, Vladimir Murzin wrote:
>> Hi Ada,
>>
>> On 9/28/26 14:30, Ada Couprie Diaz wrote:
>>> Replace the few instances of hard-coded system register offset and masks
>>> used to operate on MSR/MRS instructions with defines.
>>> This will allow re-use in future commits while making the connection
>>> between those values more explicit.
>>>
>>> While we are here, mark `aarch64_insn_extract_system_reg()` `noinstr`
>>> so it can be safe to use in alternative pacthing callbacks.
>>>
>>> Changing the mask used in `aarch64_insn_gen_mrs()` to exclude
>>> the lower bits does not change behaviour,
>>> as `aarch64_insn_encode_register()` already clears the bits
>>> used to encode the target register.
>>>
>>> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>>> ---
>>>   arch/arm64/include/asm/insn.h | 3 +++
>>>   arch/arm64/lib/insn.c         | 8 ++++----
>>>   2 files changed, 7 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
>>> index 40f13d28a5fd7..03620f67a3e47 100644
>>> --- a/arch/arm64/include/asm/insn.h
>>> +++ b/arch/arm64/include/asm/insn.h
>>> @@ -753,6 +753,9 @@ static __always_inline u32 aarch64_insn_gen_dsb(enum aarch64_insn_mb_type type)
>>>       return insn;
>>>   }
>>>   +#define AARCH64_INSN_SYSREG_OFFSET    5
>>> +#define AARCH64_INSN_SYSREG_MASK    GENMASK(19, 5)
>>> +
>> Sashiko has raised comment [1]
>>
>> | Does this mask unintentionally exclude bit 20?
>> | The previous hardcoded mask was 0x1FFFE0, which is equivalent to
>> | GENMASK(20, 5) and includes 16 bits. Defining it as GENMASK(19, 5) yields
>> | a 15-bit mask.
>>
>> [1] https://sashiko.dev/#/patchset/20260928133034.243541-1-ada.coupriediaz%40arm.com
>>
>> Cheers
>> Vladimir
> 
> Thanks for bringing the report up, it does raise an interesting issue.
> 
> This does change the mask used and does indeed change the returned value,
> potentially breaking comparisons to the return value of `aarch64_insn_extract_system_reg()`. However, the existing mask is incorrect as far as I can tell : the function is supposed to extract the Op and CR corresponding to the system registers of the MSR/MRS instructions. Those fields are encoded in bits 19-5, which corresponds to the GENMASK I used. I think it would make more sense to fix the `enum aarch64_insn_special_register` values to represent the registers properly and drop bit 20, which is 1 in all cases anyway... I would be happy to send a quick v2 with this change ! Thanks for bringing it up, Kind regards Ada
> 

Make sense, please send it as separate change. AFAICS, it should not affect bpf
folk, but worth making them aware ;)

Thanks
Vladimir

[PATCH 6/6] arm64: use alternatie callback to patch TPIDR_EL1 accesses

[Ada Couprie Diaz]

ARM64_HAS_VIRT_HOST_EXTN represents about 3k alternatives, most of them
are found in per_cpu calls switching between TPIDR_EL1 and TPIDR_EL2.

Replace the current alternative instructions with a callback that
changes the target system register, saving a potential 10kB.

`aarch64_insn_extract_system_reg()` is safe to use as it was made
`noinstr` in the previous patch.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
---
 arch/arm64/include/asm/alternative.h |  3 +++
 arch/arm64/include/asm/assembler.h   | 12 ++++--------
 arch/arm64/include/asm/percpu.h      | 12 ++++++------
 arch/arm64/kernel/alternative.c      | 25 +++++++++++++++++++++++++
 arch/arm64/kernel/image-vars.h       |  1 +
 5 files changed, 39 insertions(+), 14 deletions(-)

diff --git a/arch/arm64/include/asm/alternative.h b/arch/arm64/include/asm/alternative.h
index 621aa8550174f..e3842a28f88ba 100644
--- a/arch/arm64/include/asm/alternative.h
+++ b/arch/arm64/include/asm/alternative.h
@@ -37,5 +37,8 @@ static inline int apply_alternatives_module(void *start, size_t length)
 void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr,
 		       __le32 *updptr, int nr_inst);
 
+void alt_cb_patch_tpidr_el2(struct alt_instr *alt, __le32 *origptr,
+			    __le32 *updptr, int nr_inst);
+
 #endif /* __ASSEMBLER__ */
 #endif /* __ASM_ALTERNATIVE_H */
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 0b58b550e8dce..6cbdfefe45382 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -210,19 +210,15 @@ lr	.req	x30		// link register
 	.endm
 #else
 	.macro	get_this_cpu_offset, dst
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
+alternative_cb ARM64_HAS_VIRT_HOST_EXTN, alt_cb_patch_tpidr_el2
 	mrs	\dst, tpidr_el1
-alternative_else
-	mrs	\dst, tpidr_el2
-alternative_endif
+alternative_cb_end
 	.endm
 
 	.macro	set_this_cpu_offset, src
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
+alternative_cb ARM64_HAS_VIRT_HOST_EXTN, alt_cb_patch_tpidr_el2
 	msr	tpidr_el1, \src
-alternative_else
-	msr	tpidr_el2, \src
-alternative_endif
+alternative_cb_end
 	.endm
 #endif
 
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 8cf4068ce1b56..6999ae86a7e9f 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -14,9 +14,9 @@
 
 static inline void set_my_cpu_offset(unsigned long off)
 {
-	asm volatile(ALTERNATIVE("msr tpidr_el1, %0",
-				 "msr tpidr_el2, %0",
-				 ARM64_HAS_VIRT_HOST_EXTN)
+	asm volatile(ALTERNATIVE_CB("msr tpidr_el1, %0",
+				    ARM64_HAS_VIRT_HOST_EXTN,
+				    alt_cb_patch_tpidr_el2)
 			:: "r" (off) : "memory");
 }
 
@@ -37,9 +37,9 @@ static inline unsigned long __kern_my_cpu_offset(void)
 	 * We want to allow caching the value, so avoid using volatile and
 	 * instead use a fake stack read to hazard against barrier().
 	 */
-	asm(ALTERNATIVE("mrs %0, tpidr_el1",
-			"mrs %0, tpidr_el2",
-			ARM64_HAS_VIRT_HOST_EXTN)
+	asm(ALTERNATIVE_CB("mrs %0, tpidr_el1",
+			   ARM64_HAS_VIRT_HOST_EXTN,
+			   alt_cb_patch_tpidr_el2)
 		: "=r" (off) :
 		"Q" (*(const unsigned long *)current_stack_pointer));
 
diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c
index f5ec7e7c1d3fd..d853eda04ed23 100644
--- a/arch/arm64/kernel/alternative.c
+++ b/arch/arm64/kernel/alternative.c
@@ -303,3 +303,28 @@ noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr,
 		updptr[i] = cpu_to_le32(aarch64_insn_gen_nop());
 }
 EXPORT_SYMBOL(alt_cb_patch_nops);
+
+/*
+ * Patch MSR and MRS instructions to switch from TPIDR_EL1 to TPIDR_EL2.
+ * MSR (register) and MRS have the same target system register encoding,
+ * so we can handle both here.
+ */
+noinstr void alt_cb_patch_tpidr_el2(struct alt_instr *alt, __le32 *origptr,
+				    __le32 *updptr, int nr_inst)
+{
+	u32 orinst, altinst;
+
+	BUG_ON(nr_inst != 1);
+
+	orinst = le32_to_cpu(origptr[0]);
+	BUG_ON(!aarch64_insn_is_msr_reg(orinst) &&
+	       !aarch64_insn_is_mrs(orinst));
+	BUG_ON(aarch64_insn_extract_system_reg(orinst) !=
+	       AARCH64_INSN_SYSREG_TPIDR_EL1);
+
+	altinst = orinst & ~AARCH64_INSN_SYSREG_MASK;
+	altinst |= AARCH64_INSN_SYSREG_TPIDR_EL2 << AARCH64_INSN_SYSREG_OFFSET;
+
+	updptr[0] = cpu_to_le32(altinst);
+}
+EXPORT_SYMBOL(alt_cb_patch_tpidr_el2);
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index 615b77ef403fe..49d9984d7cd89 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -96,6 +96,7 @@ KVM_NVHE_ALIAS(kvm_compute_ich_hcr_trap_bits);
 KVM_NVHE_ALIAS(kvm_patch_ich_vtr_el2);
 KVM_NVHE_ALIAS(__io_nvidia_olympus_patch_dmb);
 KVM_NVHE_ALIAS(__io_arm_a57_patch_ladr);
+KVM_NVHE_ALIAS(alt_cb_patch_tpidr_el2);
 
 /* Global kernel state accessed by nVHE hyp code. */
 KVM_NVHE_ALIAS(kvm_vgic_global_state);
-- 
2.43.0

Re: [PATCH 0/6] arm64: alternatives: switch most used alternatives to callbacks

[Catalin Marinas]

Hi Ada,

On Mon, Sep 28, 2026 at 02:30:28PM +0100, Ada Couprie Diaz wrote:
> All numbers are on v7.3-rc4 building defconfig with GCC 13.3.0.
> Only the patches mentioned are applied on each line.
> 
> |   Patches    |  Size (B) |	|    Patch    | Size (B) |
> | Base vmlinux | 172826400 |	|  Base Image | 52374016 |
> |          1-3 |    -71672 |	|         1-3 |   -65536 |
> |            4 |    -71704 |    |           4 |   -65536 |
> |          1-4 |    -64712 |	|         1-4 |   -65536 |
> |          5-6 |    +68008 |	|         5-6 |       -0 |
> |  All patches |    + 3240 |    | All patches |   -65536 |
> 
> The impact on alternatives is two-fold :
> 1. As expected, all alternatives for the two I/O workarounds and
>    97% of the `ARM64_HAS_VIRT_HOST_EXTN` alternatives are converted to
>    callbacks, saving 85848 bytes (20 pages).
> 2. The raw number of alternatives *increases* by about 3% (250 new entries)

TBH, I don't think the saving is worth the additional complexity.

I wonder, could we instead move the replacement instructions to a
separate section we can drop after patching? I can see x86 uses a
separate .altinstr_replacement. Not sure what restrictions we have on
arm64, e.g. if it's placed too far.

That said, are some of the other patches worth picking up as cleanups?

-- 
Catalin

Re: [PATCH 0/6] arm64: alternatives: switch most used alternatives to callbacks

[Ada Couprie Diaz]

Hi Catalin,

On 06/10/2026 17:28, Catalin Marinas wrote:
> Hi Ada,
>
> On Mon, Sep 28, 2026 at 02:30:28PM +0100, Ada Couprie Diaz wrote:
>> All numbers are on v7.3-rc4 building defconfig with GCC 13.3.0.
>> Only the patches mentioned are applied on each line.
>>
>> |   Patches    |  Size (B) |	|    Patch    | Size (B) |
>> | Base vmlinux | 172826400 |	|  Base Image | 52374016 |
>> |          1-3 |    -71672 |	|         1-3 |   -65536 |
>> |            4 |    -71704 |    |           4 |   -65536 |
>> |          1-4 |    -64712 |	|         1-4 |   -65536 |
>> |          5-6 |    +68008 |	|         5-6 |       -0 |
>> |  All patches |    + 3240 |    | All patches |   -65536 |
>>
>> The impact on alternatives is two-fold :
>> 1. As expected, all alternatives for the two I/O workarounds and
>>     97% of the `ARM64_HAS_VIRT_HOST_EXTN` alternatives are converted to
>>     callbacks, saving 85848 bytes (20 pages).
>> 2. The raw number of alternatives *increases* by about 3% (250 new entries)
> TBH, I don't think the saving is worth the additional complexity.

Fair, I did expect it a bit but wanted to share if it could be useful.

> I wonder, could we instead move the replacement instructions to a
> separate section we can drop after patching? I can see x86 uses a
> separate .altinstr_replacement. Not sure what restrictions we have on
> arm64, e.g. if it's placed too far.

I think given the amount of alternatives we have now, there was some
distance issue, but I don't have the exact details. I can look into it !

> That said, are some of the other patches worth picking up as cleanups?

Mh, I think patches 1 and 5 are nice small cleanups by themselves,
patch 2 is less obviously useful as-is but is a start on making the
insn framework safer to use for patching (which is not really reasonable
to tackle in one go anyway).

Thanks Catalin,
Kind regards
Ada