* [PATCH 1/2] powerpc/mm: Add a helper to select PAGE_KERNEL_RO or PAGE_READONLY
From: Christophe Leroy @ 2019-08-21 10:20 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
In a couple of places there is a need to select whether read-only
protection of shadow pages is performed with PAGE_KERNEL_RO or with
PAGE_READONLY.
Add a helper to avoid duplicating the choice.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Cc: stable@vger.kernel.org
---
This small series applies on top of patch "powerpc/kasan: fix parallele
loading of modules" (https://patchwork.ozlabs.org/patch/1144757/).
Otherwise a manual merge has to be done (the modifications are
independant but close to each other in the file.)
---
arch/powerpc/mm/kasan/kasan_init_32.c | 21 +++++++++++++--------
1 file changed, 13 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/mm/kasan/kasan_init_32.c b/arch/powerpc/mm/kasan/kasan_init_32.c
index 802387b231ad..e8ab3cc5f6e4 100644
--- a/arch/powerpc/mm/kasan/kasan_init_32.c
+++ b/arch/powerpc/mm/kasan/kasan_init_32.c
@@ -12,6 +12,14 @@
#include <asm/code-patching.h>
#include <mm/mmu_decl.h>
+static pgprot_t kasan_prot_ro(void)
+{
+ if (early_mmu_has_feature(MMU_FTR_HPTE_TABLE))
+ return PAGE_READONLY;
+
+ return PAGE_KERNEL_RO;
+}
+
static void kasan_populate_pte(pte_t *ptep, pgprot_t prot)
{
unsigned long va = (unsigned long)kasan_early_shadow_page;
@@ -26,6 +34,7 @@ static int __ref kasan_init_shadow_page_tables(unsigned long k_start, unsigned l
{
pmd_t *pmd;
unsigned long k_cur, k_next;
+ pgprot_t prot = kasan_prot_ro();
pmd = pmd_offset(pud_offset(pgd_offset_k(k_start), k_start), k_start);
@@ -43,10 +52,7 @@ static int __ref kasan_init_shadow_page_tables(unsigned long k_start, unsigned l
if (!new)
return -ENOMEM;
- if (early_mmu_has_feature(MMU_FTR_HPTE_TABLE))
- kasan_populate_pte(new, PAGE_READONLY);
- else
- kasan_populate_pte(new, PAGE_KERNEL_RO);
+ kasan_populate_pte(new, prot);
smp_wmb(); /* See comment in __pte_alloc */
@@ -103,10 +109,9 @@ static int __ref kasan_init_region(void *start, size_t size)
static void __init kasan_remap_early_shadow_ro(void)
{
- if (early_mmu_has_feature(MMU_FTR_HPTE_TABLE))
- kasan_populate_pte(kasan_early_shadow_pte, PAGE_READONLY);
- else
- kasan_populate_pte(kasan_early_shadow_pte, PAGE_KERNEL_RO);
+ pgprot_t prot = kasan_prot_ro();
+
+ kasan_populate_pte(kasan_early_shadow_pte, prot);
flush_tlb_kernel_range(KASAN_SHADOW_START, KASAN_SHADOW_END);
}
--
2.13.3
^ permalink raw reply related
* [PATCH 2/2] powerpc/mm: Fix an Oops in kasan_mmu_init()
From: Christophe Leroy @ 2019-08-21 10:20 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
In-Reply-To: <9f33f44b9cd741c4a02b3dce7b8ef9438fe2cd2a.1566382750.git.christophe.leroy@c-s.fr>
Uncompressing Kernel Image ... OK
Loading Device Tree to 01ff7000, end 01fff74f ... OK
[ 0.000000] printk: bootconsole [udbg0] enabled
[ 0.000000] BUG: Unable to handle kernel data access at 0xf818c000
[ 0.000000] Faulting instruction address: 0xc0013c7c
[ 0.000000] Thread overran stack, or stack corrupted
[ 0.000000] Oops: Kernel access of bad area, sig: 11 [#1]
[ 0.000000] BE PAGE_SIZE=16K PREEMPT
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.3.0-rc4-s3k-dev-00743-g5abe4a3e8fd3-dirty #2080
[ 0.000000] NIP: c0013c7c LR: c0013310 CTR: 00000000
[ 0.000000] REGS: c0c5ff38 TRAP: 0300 Not tainted (5.3.0-rc4-s3k-dev-00743-g5abe4a3e8fd3-dirty)
[ 0.000000] MSR: 00001032 <ME,IR,DR,RI> CR: 99033955 XER: 80002100
[ 0.000000] DAR: f818c000 DSISR: 82000000
[ 0.000000] GPR00: c0013310 c0c5fff0 c0ad6ac0 c0c600c0 f818c031 82000000 00000000 ffffffff
[ 0.000000] GPR08: 00000000 f1f1f1f1 c0013c2c c0013304 99033955 00400008 00000000 07ff9598
[ 0.000000] GPR16: 00000000 07ffb94c 00000000 00000000 00000000 00000000 00000000 f818cfb2
[ 0.000000] GPR24: 00000000 00000000 00001000 ffffffff 00000000 c07dbf80 00000000 f818c000
[ 0.000000] NIP [c0013c7c] do_page_fault+0x50/0x904
[ 0.000000] LR [c0013310] handle_page_fault+0xc/0x38
[ 0.000000] Call Trace:
[ 0.000000] Instruction dump:
[ 0.000000] be010080 91410014 553fe8fe 3d40c001 3d20f1f1 7d800026 394a3c2c 3fffe000
[ 0.000000] 6129f1f1 900100c4 9181007c 91410018 <913f0000> 3d2001f4 6129f4f4 913f0004
Don't map the early shadow page read-only yet when creating the new
page tables for the real shadow memory, otherwise the memblock
allocations that immediately follows to create the real shadow pages
that are about to replace the early shadow page trigger a page fault
if they fall into the region being worked on at the moment.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Fixes: 2edb16efc899 ("powerpc/32: Add KASAN support")
Cc: stable@vger.kernel.org
---
arch/powerpc/mm/kasan/kasan_init_32.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/mm/kasan/kasan_init_32.c b/arch/powerpc/mm/kasan/kasan_init_32.c
index e8ab3cc5f6e4..0e6ed4413eea 100644
--- a/arch/powerpc/mm/kasan/kasan_init_32.c
+++ b/arch/powerpc/mm/kasan/kasan_init_32.c
@@ -34,7 +34,7 @@ static int __ref kasan_init_shadow_page_tables(unsigned long k_start, unsigned l
{
pmd_t *pmd;
unsigned long k_cur, k_next;
- pgprot_t prot = kasan_prot_ro();
+ pgprot_t prot = slab_is_available() ? kasan_prot_ro() : PAGE_KERNEL;
pmd = pmd_offset(pud_offset(pgd_offset_k(k_start), k_start), k_start);
@@ -110,9 +110,22 @@ static int __ref kasan_init_region(void *start, size_t size)
static void __init kasan_remap_early_shadow_ro(void)
{
pgprot_t prot = kasan_prot_ro();
+ unsigned long k_start = KASAN_SHADOW_START;
+ unsigned long k_end = KASAN_SHADOW_END;
+ unsigned long k_cur;
+ phys_addr_t pa = __pa(kasan_early_shadow_page);
kasan_populate_pte(kasan_early_shadow_pte, prot);
+ for (k_cur = k_start & PAGE_MASK; k_cur < k_end; k_cur += PAGE_SIZE) {
+ pmd_t *pmd = pmd_offset(pud_offset(pgd_offset_k(k_cur), k_cur), k_cur);
+ pte_t *ptep = pte_offset_kernel(pmd, k_cur);
+
+ if ((pte_val(*ptep) & PTE_RPN_MASK) != pa)
+ continue;
+
+ __set_pte_at(&init_mm, k_cur, ptep, pfn_pte(PHYS_PFN(pa), prot), 0);
+ }
flush_tlb_kernel_range(KASAN_SHADOW_START, KASAN_SHADOW_END);
}
--
2.13.3
^ permalink raw reply related
* [PATCH] powerpc/8xx: set STACK_END_MAGIC earlier on the init_stack
From: Christophe Leroy @ 2019-08-21 10:20 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
Today, the STACK_END_MAGIC is set on init_stack in start_kernel().
To avoid a false 'Thread overran stack, or stack corrupted' message
on early Oopses, setup STACK_END_MAGIC as soon as possible.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
arch/powerpc/kernel/head_8xx.S | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/powerpc/kernel/head_8xx.S b/arch/powerpc/kernel/head_8xx.S
index 5ab9178c2347..b8ca5b43e587 100644
--- a/arch/powerpc/kernel/head_8xx.S
+++ b/arch/powerpc/kernel/head_8xx.S
@@ -15,6 +15,7 @@
*/
#include <linux/init.h>
+#include <linux/magic.h>
#include <asm/processor.h>
#include <asm/page.h>
#include <asm/mmu.h>
@@ -741,6 +742,9 @@ start_here:
/* stack */
lis r1,init_thread_union@ha
addi r1,r1,init_thread_union@l
+ lis r0, STACK_END_MAGIC@h
+ ori r0, r0, STACK_END_MAGIC@l
+ stw r0, 0(r1)
li r0,0
stwu r0,THREAD_SIZE-STACK_FRAME_OVERHEAD(r1)
--
2.13.3
^ permalink raw reply related
* Regression fix for bpf in v5.3 (was Re: [RFC PATCH] bpf: handle 32-bit zext during constant blinding)
From: Michael Ellerman @ 2019-08-21 10:25 UTC (permalink / raw)
To: Naveen N. Rao, Alexei Starovoitov, Daniel Borkmann, Jiong Wang
Cc: netdev, bpf, linuxppc-dev, linux-kernel
In-Reply-To: <20190813171018.28221-1-naveen.n.rao@linux.vnet.ibm.com>
"Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes:
> Since BPF constant blinding is performed after the verifier pass, there
> are certain ALU32 instructions inserted which don't have a corresponding
> zext instruction inserted after. This is causing a kernel oops on
> powerpc and can be reproduced by running 'test_cgroup_storage' with
> bpf_jit_harden=2.
>
> Fix this by emitting BPF_ZEXT during constant blinding if
> prog->aux->verifier_zext is set.
>
> Fixes: a4b1d3c1ddf6cb ("bpf: verifier: insert zero extension according to analysis result")
> Reported-by: Michael Ellerman <mpe@ellerman.id.au>
> Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
> ---
> This approach (the location where zext is being introduced below, in
> particular) works for powerpc, but I am not entirely sure if this is
> sufficient for other architectures as well. This is broken on v5.3-rc4.
Any comment on this?
This is a regression in v5.3, which results in a kernel crash, it would
be nice to get it fixed before the release please?
cheers
> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index 8191a7db2777..d84146e6fd9e 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
> @@ -890,7 +890,8 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog,
>
> static int bpf_jit_blind_insn(const struct bpf_insn *from,
> const struct bpf_insn *aux,
> - struct bpf_insn *to_buff)
> + struct bpf_insn *to_buff,
> + bool emit_zext)
> {
> struct bpf_insn *to = to_buff;
> u32 imm_rnd = get_random_int();
> @@ -939,6 +940,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
> *to++ = BPF_ALU32_IMM(BPF_MOV, BPF_REG_AX, imm_rnd ^ from->imm);
> *to++ = BPF_ALU32_IMM(BPF_XOR, BPF_REG_AX, imm_rnd);
> *to++ = BPF_ALU32_REG(from->code, from->dst_reg, BPF_REG_AX);
> + if (emit_zext)
> + *to++ = BPF_ZEXT_REG(from->dst_reg);
> break;
>
> case BPF_ALU64 | BPF_ADD | BPF_K:
> @@ -992,6 +995,10 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
> off -= 2;
> *to++ = BPF_ALU32_IMM(BPF_MOV, BPF_REG_AX, imm_rnd ^ from->imm);
> *to++ = BPF_ALU32_IMM(BPF_XOR, BPF_REG_AX, imm_rnd);
> + if (emit_zext) {
> + *to++ = BPF_ZEXT_REG(BPF_REG_AX);
> + off--;
> + }
> *to++ = BPF_JMP32_REG(from->code, from->dst_reg, BPF_REG_AX,
> off);
> break;
> @@ -1005,6 +1012,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
> case 0: /* Part 2 of BPF_LD | BPF_IMM | BPF_DW. */
> *to++ = BPF_ALU32_IMM(BPF_MOV, BPF_REG_AX, imm_rnd ^ aux[0].imm);
> *to++ = BPF_ALU32_IMM(BPF_XOR, BPF_REG_AX, imm_rnd);
> + if (emit_zext)
> + *to++ = BPF_ZEXT_REG(BPF_REG_AX);
> *to++ = BPF_ALU64_REG(BPF_OR, aux[0].dst_reg, BPF_REG_AX);
> break;
>
> @@ -1088,7 +1097,8 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog)
> insn[1].code == 0)
> memcpy(aux, insn, sizeof(aux));
>
> - rewritten = bpf_jit_blind_insn(insn, aux, insn_buff);
> + rewritten = bpf_jit_blind_insn(insn, aux, insn_buff,
> + clone->aux->verifier_zext);
> if (!rewritten)
> continue;
>
> --
> 2.22.0
^ permalink raw reply
* Re: [PATCH] powerpc/vdso64: inline __get_datapage()
From: Santosh Sivaraj @ 2019-08-21 11:20 UTC (permalink / raw)
To: Christophe Leroy, Benjamin Herrenschmidt; +Cc: linuxppc-dev
In-Reply-To: <3724e1ba-d0f8-7247-73c2-6d83a3dbd040@c-s.fr>
Christophe Leroy <christophe.leroy@c-s.fr> writes:
> Le 21/08/2019 à 11:29, Santosh Sivaraj a écrit :
>> __get_datapage() is only a few instructions to retrieve the
>> address of the page where the kernel stores data to the VDSO.
>>
>> By inlining this function into its users, a bl/blr pair and
>> a mflr/mtlr pair is avoided, plus a few reg moves.
>>
>> clock-gettime-monotonic: syscall: 514 nsec/call 396 nsec/call
>> clock-gettime-monotonic: libc: 25 nsec/call 24 nsec/call
>> clock-gettime-monotonic: vdso: 20 nsec/call 20 nsec/call
>> clock-getres-monotonic: syscall: 347 nsec/call 372 nsec/call
>> clock-getres-monotonic: libc: 19 nsec/call 19 nsec/call
>> clock-getres-monotonic: vdso: 10 nsec/call 10 nsec/call
>> clock-gettime-monotonic-coarse: syscall: 511 nsec/call 396 nsec/call
>> clock-gettime-monotonic-coarse: libc: 23 nsec/call 21 nsec/call
>> clock-gettime-monotonic-coarse: vdso: 15 nsec/call 13 nsec/call
>> clock-gettime-realtime: syscall: 526 nsec/call 405 nsec/call
>> clock-gettime-realtime: libc: 24 nsec/call 23 nsec/call
>> clock-gettime-realtime: vdso: 18 nsec/call 18 nsec/call
>> clock-getres-realtime: syscall: 342 nsec/call 372 nsec/call
>> clock-getres-realtime: libc: 19 nsec/call 19 nsec/call
>> clock-getres-realtime: vdso: 10 nsec/call 10 nsec/call
>> clock-gettime-realtime-coarse: syscall: 515 nsec/call 373 nsec/call
>> clock-gettime-realtime-coarse: libc: 23 nsec/call 22 nsec/call
>> clock-gettime-realtime-coarse: vdso: 14 nsec/call 13 nsec/call
>
> I think you should only put the measurements on vdso calls, and only the
> ones that are impacted by the change. For exemple, getres function
> doesn't use __get_datapage so showing it here is pointless.
>
> gettimeofday should be shown there as it uses __get_datapage()
>
>
>>
>> Based on the patch by Christophe Leroy <christophe.leroy@c-s.fr> for vdso32.
>>
>> Signed-off-by: Santosh Sivaraj <santosh@fossix.org>
>> ---
>>
>> except for a couple of calls (1 or 2 nsec reduction), there are no
>> improvements in the call times. Or is 10 nsec the minimum granularity??
>
> Maybe the ones that show no improvements are the ones that don't use
> __get_datapage() at all ...
Yes makes sense.
>
>>
>> So I don't know if its even worth updating vdso64 except to keep vdso32 and
>> vdso64 equal.
>
> 2ns on a 15ns call is 13% so it is worth it I think.
true. Since datapage.h is the same for both 32 and 64, may be we should put
it in include/asm.
Thanks,
Santosh
>
> Christophe
>
>
>>
>>
>> arch/powerpc/kernel/vdso64/cacheflush.S | 10 ++++----
>> arch/powerpc/kernel/vdso64/datapage.S | 29 ++++-------------------
>> arch/powerpc/kernel/vdso64/datapage.h | 10 ++++++++
>> arch/powerpc/kernel/vdso64/gettimeofday.S | 8 ++++---
>> 4 files changed, 24 insertions(+), 33 deletions(-)
>> create mode 100644 arch/powerpc/kernel/vdso64/datapage.h
>>
>> diff --git a/arch/powerpc/kernel/vdso64/cacheflush.S b/arch/powerpc/kernel/vdso64/cacheflush.S
>> index 3f92561a64c4..30e8b0d29bea 100644
>> --- a/arch/powerpc/kernel/vdso64/cacheflush.S
>> +++ b/arch/powerpc/kernel/vdso64/cacheflush.S
>> @@ -10,6 +10,8 @@
>> #include <asm/vdso.h>
>> #include <asm/asm-offsets.h>
>>
>> +#include "datapage.h"
>> +
>> .text
>>
>> /*
>> @@ -24,14 +26,12 @@ V_FUNCTION_BEGIN(__kernel_sync_dicache)
>> .cfi_startproc
>> mflr r12
>> .cfi_register lr,r12
>> - mr r11,r3
>> - bl V_LOCAL_FUNC(__get_datapage)
>> + get_datapage r11, r0
>> mtlr r12
>> - mr r10,r3
>>
>> lwz r7,CFG_DCACHE_BLOCKSZ(r10)
>> addi r5,r7,-1
>> - andc r6,r11,r5 /* round low to line bdy */
>> + andc r6,r3,r5 /* round low to line bdy */
>> subf r8,r6,r4 /* compute length */
>> add r8,r8,r5 /* ensure we get enough */
>> lwz r9,CFG_DCACHE_LOGBLOCKSZ(r10)
>> @@ -48,7 +48,7 @@ V_FUNCTION_BEGIN(__kernel_sync_dicache)
>>
>> lwz r7,CFG_ICACHE_BLOCKSZ(r10)
>> addi r5,r7,-1
>> - andc r6,r11,r5 /* round low to line bdy */
>> + andc r6,r3,r5 /* round low to line bdy */
>> subf r8,r6,r4 /* compute length */
>> add r8,r8,r5
>> lwz r9,CFG_ICACHE_LOGBLOCKSZ(r10)
>> diff --git a/arch/powerpc/kernel/vdso64/datapage.S b/arch/powerpc/kernel/vdso64/datapage.S
>> index dc84f5ae3802..8712f57c931c 100644
>> --- a/arch/powerpc/kernel/vdso64/datapage.S
>> +++ b/arch/powerpc/kernel/vdso64/datapage.S
>> @@ -11,34 +11,13 @@
>> #include <asm/unistd.h>
>> #include <asm/vdso.h>
>>
>> +#include "datapage.h"
>> +
>> .text
>> .global __kernel_datapage_offset;
>> __kernel_datapage_offset:
>> .long 0
>>
>> -V_FUNCTION_BEGIN(__get_datapage)
>> - .cfi_startproc
>> - /* We don't want that exposed or overridable as we want other objects
>> - * to be able to bl directly to here
>> - */
>> - .protected __get_datapage
>> - .hidden __get_datapage
>> -
>> - mflr r0
>> - .cfi_register lr,r0
>> -
>> - bcl 20,31,data_page_branch
>> -data_page_branch:
>> - mflr r3
>> - mtlr r0
>> - addi r3, r3, __kernel_datapage_offset-data_page_branch
>> - lwz r0,0(r3)
>> - .cfi_restore lr
>> - add r3,r0,r3
>> - blr
>> - .cfi_endproc
>> -V_FUNCTION_END(__get_datapage)
>> -
>> /*
>> * void *__kernel_get_syscall_map(unsigned int *syscall_count) ;
>> *
>> @@ -53,7 +32,7 @@ V_FUNCTION_BEGIN(__kernel_get_syscall_map)
>> mflr r12
>> .cfi_register lr,r12
>> mr r4,r3
>> - bl V_LOCAL_FUNC(__get_datapage)
>> + get_datapage r3, r0
>> mtlr r12
>> addi r3,r3,CFG_SYSCALL_MAP64
>> cmpldi cr0,r4,0
>> @@ -75,7 +54,7 @@ V_FUNCTION_BEGIN(__kernel_get_tbfreq)
>> .cfi_startproc
>> mflr r12
>> .cfi_register lr,r12
>> - bl V_LOCAL_FUNC(__get_datapage)
>> + get_datapage r3, r0
>> ld r3,CFG_TB_TICKS_PER_SEC(r3)
>> mtlr r12
>> crclr cr0*4+so
>> diff --git a/arch/powerpc/kernel/vdso64/datapage.h b/arch/powerpc/kernel/vdso64/datapage.h
>> new file mode 100644
>> index 000000000000..f2f0da0f65f3
>> --- /dev/null
>> +++ b/arch/powerpc/kernel/vdso64/datapage.h
>> @@ -0,0 +1,10 @@
>> +/* SPDX-License-Identifier: GPL-2.0-or-later */
>> +
>> +.macro get_datapage ptr, tmp
>> + bcl 20,31,888f
>> +888:
>> + mflr \ptr
>> + addi \ptr, \ptr, __kernel_datapage_offset - 888b
>> + lwz \tmp, 0(\ptr)
>> + add \ptr, \tmp, \ptr
>> +.endm
>> diff --git a/arch/powerpc/kernel/vdso64/gettimeofday.S b/arch/powerpc/kernel/vdso64/gettimeofday.S
>> index 07bfe33fe874..7bcc879392cc 100644
>> --- a/arch/powerpc/kernel/vdso64/gettimeofday.S
>> +++ b/arch/powerpc/kernel/vdso64/gettimeofday.S
>> @@ -12,6 +12,8 @@
>> #include <asm/asm-offsets.h>
>> #include <asm/unistd.h>
>>
>> +#include "datapage.h"
>> +
>> .text
>> /*
>> * Exact prototype of gettimeofday
>> @@ -26,7 +28,7 @@ V_FUNCTION_BEGIN(__kernel_gettimeofday)
>>
>> mr r11,r3 /* r11 holds tv */
>> mr r10,r4 /* r10 holds tz */
>> - bl V_LOCAL_FUNC(__get_datapage) /* get data page */
>> + get_datapage r3, r0
>> cmpldi r11,0 /* check if tv is NULL */
>> beq 2f
>> lis r7,1000000@ha /* load up USEC_PER_SEC */
>> @@ -71,7 +73,7 @@ V_FUNCTION_BEGIN(__kernel_clock_gettime)
>> mflr r12 /* r12 saves lr */
>> .cfi_register lr,r12
>> mr r11,r4 /* r11 saves tp */
>> - bl V_LOCAL_FUNC(__get_datapage) /* get data page */
>> + get_datapage r3, r0 /* get data page */
>> lis r7,NSEC_PER_SEC@h /* want nanoseconds */
>> ori r7,r7,NSEC_PER_SEC@l
>> beq cr5,70f
>> @@ -218,7 +220,7 @@ V_FUNCTION_BEGIN(__kernel_time)
>> .cfi_register lr,r12
>>
>> mr r11,r3 /* r11 holds t */
>> - bl V_LOCAL_FUNC(__get_datapage)
>> + get_datapage r3, r0
>>
>> ld r4,STAMP_XTIME+TSPC64_TV_SEC(r3)
>>
>>
^ permalink raw reply
* Re: [PATCH] powerpc/vdso64: inline __get_datapage()
From: Segher Boessenkool @ 2019-08-21 11:44 UTC (permalink / raw)
To: Santosh Sivaraj; +Cc: linuxppc-dev
In-Reply-To: <20190821092959.16066-1-santosh@fossix.org>
Hi!
On Wed, Aug 21, 2019 at 02:59:59PM +0530, Santosh Sivaraj wrote:
> except for a couple of calls (1 or 2 nsec reduction), there are no
> improvements in the call times. Or is 10 nsec the minimum granularity??
>
> So I don't know if its even worth updating vdso64 except to keep vdso32 and
> vdso64 equal.
Calls are cheap, in principle... It is the LR stuff that can make it
slower on some cores, and a lot of calling sequence stuff may have
considerable overhead of course.
> +.macro get_datapage ptr, tmp
> + bcl 20,31,888f
> +888:
> + mflr \ptr
> + addi \ptr, \ptr, __kernel_datapage_offset - 888b
> + lwz \tmp, 0(\ptr)
> + add \ptr, \tmp, \ptr
> +.endm
(You can just write that as
bcl 20,31,$+4
mflr \ptr
etc. Useless labels are useless :-) )
One thing you might want to do to improve performance is to do this without
the bcl etc., because you cannot really hide the LR latency of that. But
that isn't very many ns either... Superscalar helps, OoO helps, but it is
mostly just that >100MHz helps ;-)
Segher
^ permalink raw reply
* Re: [PATCH] powerpc/vdso64: inline __get_datapage()
From: Christophe Leroy @ 2019-08-21 11:50 UTC (permalink / raw)
To: Segher Boessenkool, Santosh Sivaraj; +Cc: linuxppc-dev
In-Reply-To: <20190821114423.GC31406@gate.crashing.org>
Le 21/08/2019 à 13:44, Segher Boessenkool a écrit :
> Hi!
>
> On Wed, Aug 21, 2019 at 02:59:59PM +0530, Santosh Sivaraj wrote:
>> except for a couple of calls (1 or 2 nsec reduction), there are no
>> improvements in the call times. Or is 10 nsec the minimum granularity??
>>
>> So I don't know if its even worth updating vdso64 except to keep vdso32 and
>> vdso64 equal.
>
> Calls are cheap, in principle... It is the LR stuff that can make it
> slower on some cores, and a lot of calling sequence stuff may have
> considerable overhead of course.
On an 8xx, a taken branch is 2 cycles and a non taken branch in 1 cycle
(+ the refetch if that was not the anticipate branch).
>
>> +.macro get_datapage ptr, tmp
>> + bcl 20,31,888f
>> +888:
>> + mflr \ptr
>> + addi \ptr, \ptr, __kernel_datapage_offset - 888b
>> + lwz \tmp, 0(\ptr)
>> + add \ptr, \tmp, \ptr
>> +.endm
>
> (You can just write that as
> bcl 20,31,$+4
> mflr \ptr
> etc. Useless labels are useless :-) )
Nice trick. Will use that.
>
> One thing you might want to do to improve performance is to do this without
> the bcl etc., because you cannot really hide the LR latency of that. But
> that isn't very many ns either... Superscalar helps, OoO helps, but it is
> mostly just that >100MHz helps ;-)
Good idea. Did you have a look at my vdso32 similar patch ?
https://patchwork.ozlabs.org/patch/1148274/
Do you have any idea on how to avoid that bcl/mflr stuff ?
Christophe
^ permalink raw reply
* Re: [PATCH] powerpc/vdso64: inline __get_datapage()
From: Segher Boessenkool @ 2019-08-21 12:15 UTC (permalink / raw)
To: Christophe Leroy; +Cc: Santosh Sivaraj, linuxppc-dev
In-Reply-To: <b7cc5d77-dfca-1cf7-f316-636a4f603b04@c-s.fr>
On Wed, Aug 21, 2019 at 01:50:52PM +0200, Christophe Leroy wrote:
> Le 21/08/2019 à 13:44, Segher Boessenkool a écrit :
> >Calls are cheap, in principle... It is the LR stuff that can make it
> >slower on some cores, and a lot of calling sequence stuff may have
> >considerable overhead of course.
>
> On an 8xx, a taken branch is 2 cycles and a non taken branch in 1 cycle
> (+ the refetch if that was not the anticipate branch).
Yup. And on the big cores they are all 0 cycles, if correctly predicted.
(Taken branches end your fetch group, of course, there are small
inefficiencies everywhere, but that's about the gist of it).
> >>+.macro get_datapage ptr, tmp
> >>+ bcl 20,31,888f
> >>+888:
> >>+ mflr \ptr
> >>+ addi \ptr, \ptr, __kernel_datapage_offset - 888b
> >>+ lwz \tmp, 0(\ptr)
> >>+ add \ptr, \tmp, \ptr
> >>+.endm
> >
> >(You can just write that as
> > bcl 20,31,$+4
> > mflr \ptr
> >etc. Useless labels are useless :-) )
>
> Nice trick. Will use that.
Or .+4 if you like that syntax better... It's all the same thing.
> >One thing you might want to do to improve performance is to do this without
> >the bcl etc., because you cannot really hide the LR latency of that. But
> >that isn't very many ns either... Superscalar helps, OoO helps, but it is
> >mostly just that >100MHz helps ;-)
>
> Good idea. Did you have a look at my vdso32 similar patch ?
> https://patchwork.ozlabs.org/patch/1148274/
Yes, I saw it.
> Do you have any idea on how to avoid that bcl/mflr stuff ?
Do a load from some fixed address? Maybe an absolute address, even?
lwz r3,-12344(0) or similar (that address is in kernel space...)
There aren't many options, and certainly not many *good* options!
Segher
^ permalink raw reply
* Re: [PATCH v2 21/44] powerpc/64s/exception: remove 0xb00 handler
From: Michael Ellerman @ 2019-08-21 12:18 UTC (permalink / raw)
To: Nicholas Piggin, linuxppc-dev; +Cc: Nicholas Piggin
In-Reply-To: <20190802105709.27696-22-npiggin@gmail.com>
Nicholas Piggin <npiggin@gmail.com> writes:
> This vector is not used by any supported processor, and has been
> implemented as an unknown exception going back to 2.6. There is
> nothing special about 0xb00, so remove it like other unused
> vectors.
Actually it goes back to the original ppc64 submission.
See (takes a while to load):
https://github.com/mpe/linux-fullhistory/commit/c3aa9878533e724f639852c3d951e6a169e04081#diff-c7b0adae374819e9003279ff5f69226fR340
That commit had handlers for all the vectors from 0x100 through 0xf00,
with stubs for 0xa00, 0xb00 and 0xe00. But it's not at all clear why it
needed the stubs, possibly it was just being verbose.
0xa00 eventually became doorbell_super and 0xe00 became h_data_storage.
Leaving just 0xb00 as the lone relic.
</irrelevant-history>
Patch looks good.
cheers
> diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
> index 723c37f3da17..9c407392774c 100644
> --- a/arch/powerpc/kernel/exceptions-64s.S
> +++ b/arch/powerpc/kernel/exceptions-64s.S
> @@ -1563,10 +1563,8 @@ EXC_COMMON_ASYNC(doorbell_super_common, 0xa00, unknown_exception)
> #endif
>
>
> -EXC_REAL(trap_0b, 0xb00, 0x100)
> -EXC_VIRT(trap_0b, 0x4b00, 0x100, 0xb00)
> -TRAMP_KVM(PACA_EXGEN, 0xb00)
> -EXC_COMMON(trap_0b_common, 0xb00, unknown_exception)
> +EXC_REAL_NONE(0xb00, 0x100)
> +EXC_VIRT_NONE(0x4b00, 0x100)
>
> /*
> * system call / hypercall (0xc00, 0x4c00)
> --
> 2.22.0
^ permalink raw reply
* [PATCH 01/10] powerpc/64s/exception: Add ISIDE option
From: Nicholas Piggin @ 2019-08-21 12:23 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin
In-Reply-To: <20190821122315.9535-1-npiggin@gmail.com>
Rather than using DAR=2 to select the i-side registers, add an
explicit option.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/exceptions-64s.S | 23 ++++++++++++++++-------
1 file changed, 16 insertions(+), 7 deletions(-)
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 5d9fdf9dce55..2d2c6f19eec1 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -199,6 +199,7 @@ END_FTR_SECTION_NESTED(ftr,ftr,943)
#define IVEC .L_IVEC_\name\()
#define IHSRR .L_IHSRR_\name\()
#define IAREA .L_IAREA_\name\()
+#define IISIDE .L_IISIDE_\name\()
#define IDAR .L_IDAR_\name\()
#define IDSISR .L_IDSISR_\name\()
#define ISET_RI .L_ISET_RI_\name\()
@@ -231,6 +232,9 @@ do_define_int n
.ifndef IAREA
.error "IAREA not defined"
.endif
+ .ifndef IISIDE
+ IISIDE=0
+ .endif
.ifndef IDAR
IDAR=0
.endif
@@ -546,7 +550,7 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
*/
GET_SCRATCH0(r10)
std r10,IAREA+EX_R13(r13)
- .if IDAR == 1
+ .if IDAR && !IISIDE
.if IHSRR
mfspr r10,SPRN_HDAR
.else
@@ -554,7 +558,7 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
.endif
std r10,IAREA+EX_DAR(r13)
.endif
- .if IDSISR == 1
+ .if IDSISR && !IISIDE
.if IHSRR
mfspr r10,SPRN_HDSISR
.else
@@ -629,16 +633,18 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
std r9,GPR11(r1)
std r10,GPR12(r1)
std r11,GPR13(r1)
+
.if IDAR
- .if IDAR == 2
+ .if IISIDE
ld r10,_NIP(r1)
.else
ld r10,IAREA+EX_DAR(r13)
.endif
std r10,_DAR(r1)
.endif
+
.if IDSISR
- .if IDSISR == 2
+ .if IISIDE
ld r10,_MSR(r1)
lis r11,DSISR_SRR1_MATCH_64S@h
and r10,r10,r11
@@ -647,6 +653,7 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
.endif
std r10,_DSISR(r1)
.endif
+
BEGIN_FTR_SECTION_NESTED(66)
ld r10,IAREA+EX_CFAR(r13)
std r10,ORIG_GPR3(r1)
@@ -1324,8 +1331,9 @@ INT_DEFINE_BEGIN(instruction_access)
IVEC=0x400
IHSRR=EXC_STD
IAREA=PACA_EXGEN
- IDAR=2
- IDSISR=2
+ IISIDE=1
+ IDAR=1
+ IDSISR=1
IKVM_REAL=1
INT_DEFINE_END(instruction_access)
@@ -1355,7 +1363,8 @@ INT_DEFINE_BEGIN(instruction_access_slb)
IHSRR=EXC_STD
IAREA=PACA_EXSLB
IRECONCILE=0
- IDAR=2
+ IISIDE=1
+ IDAR=1
IKVM_REAL=1
INT_DEFINE_END(instruction_access_slb)
--
2.22.0
^ permalink raw reply related
* [PATCH 00/10] powerpc/64s/exception: initial reworking of
From: Nicholas Piggin @ 2019-08-21 12:23 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin
This applies on top of the previous exception series.
Until now the focus was on making the code generation macros sane,
ironing out irregularities and needless differences in handlers,
and sharing code. Some change of generated code was required, but
minimised as much as possible.
This series begins to change generated code in a bigger way. This is
an assortment of changes, one of the big goals goals is really moving
code and especially branches and logic and conditional compilation
out of "unrelocated" interrupt entry points, with the hope of making
KASLR more effective when it is implemented.
After this is a more significant rework of the instruction sequences
with the aim of improving performance, but I will stop here for the
next merge window.
Thanks,
Nick
Nicholas Piggin (10):
powerpc/64s/exception: Add ISIDE option
powerpc/64s/exception: move real->virt switch into the common handler
powerpc/64s/exception: move soft-mask test to common code
powerpc/64s/exception: move KVM test to common code
powerpc/64s/exception: remove confusing IEARLY option
powerpc/64s/exception: remove the SPR saving patch code macros
powerpc/64s/exception: trim unused arguments from KVMTEST macro
powerpc/64s/exception: hdecrementer avoid touching the stack
powerpc/64s/exception: re-inline some handlers
powerpc/64s/exception: add more comments for interrupt handlers
arch/powerpc/include/asm/exception-64s.h | 4 -
arch/powerpc/include/asm/time.h | 1 -
arch/powerpc/kernel/exceptions-64s.S | 1151 ++++++++++++++--------
arch/powerpc/kernel/time.c | 9 -
arch/powerpc/kvm/book3s_hv_rmhandlers.S | 11 -
arch/powerpc/kvm/book3s_segment.S | 7 -
6 files changed, 716 insertions(+), 467 deletions(-)
--
2.22.0
^ permalink raw reply
* [PATCH 02/10] powerpc/64s/exception: move real->virt switch into the common handler
From: Nicholas Piggin @ 2019-08-21 12:23 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin
In-Reply-To: <20190821122315.9535-1-npiggin@gmail.com>
The real mode interrupt entry points currently use rfid to branch to
the common handler in virtual mode. This is a significant amount of
code, and forces other code (notably the KVM test) to live in the
real mode handler.
In the interest of minimising the amount of code that runs unrelocated
move the switch to virt mode into the common code, and do it with
mtmsrd, which avoids clobbering SRRs (although the post-KVMTEST
performance of real-mode interrupt handlers is not a big concern these
days).
This requires CTR to always be saved (real-mode needs to reach 0xc...)
but that's not a huge impact these days. It could be optimized away in
future.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/exception-64s.h | 4 -
arch/powerpc/kernel/exceptions-64s.S | 248 ++++++++++-------------
2 files changed, 106 insertions(+), 146 deletions(-)
diff --git a/arch/powerpc/include/asm/exception-64s.h b/arch/powerpc/include/asm/exception-64s.h
index 33f4f72eb035..47bd4ea0837d 100644
--- a/arch/powerpc/include/asm/exception-64s.h
+++ b/arch/powerpc/include/asm/exception-64s.h
@@ -33,11 +33,7 @@
#include <asm/feature-fixups.h>
/* PACA save area size in u64 units (exgen, exmc, etc) */
-#if defined(CONFIG_RELOCATABLE)
#define EX_SIZE 10
-#else
-#define EX_SIZE 9
-#endif
/*
* maximum recursive depth of MCE exceptions
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 2d2c6f19eec1..6ea6176de00a 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -32,16 +32,10 @@
#define EX_CCR 52
#define EX_CFAR 56
#define EX_PPR 64
-#if defined(CONFIG_RELOCATABLE)
#define EX_CTR 72
.if EX_SIZE != 10
.error "EX_SIZE is wrong"
.endif
-#else
-.if EX_SIZE != 9
- .error "EX_SIZE is wrong"
-.endif
-#endif
/*
* Following are fixed section helper macros.
@@ -124,22 +118,6 @@ name:
#define EXC_HV 1
#define EXC_STD 0
-#if defined(CONFIG_RELOCATABLE)
-/*
- * If we support interrupts with relocation on AND we're a relocatable kernel,
- * we need to use CTR to get to the 2nd level handler. So, save/restore it
- * when required.
- */
-#define SAVE_CTR(reg, area) mfctr reg ; std reg,area+EX_CTR(r13)
-#define GET_CTR(reg, area) ld reg,area+EX_CTR(r13)
-#define RESTORE_CTR(reg, area) ld reg,area+EX_CTR(r13) ; mtctr reg
-#else
-/* ...else CTR is unused and in register. */
-#define SAVE_CTR(reg, area)
-#define GET_CTR(reg, area) mfctr reg
-#define RESTORE_CTR(reg, area)
-#endif
-
/*
* PPR save/restore macros used in exceptions-64s.S
* Used for P7 or later processors
@@ -199,6 +177,7 @@ END_FTR_SECTION_NESTED(ftr,ftr,943)
#define IVEC .L_IVEC_\name\()
#define IHSRR .L_IHSRR_\name\()
#define IAREA .L_IAREA_\name\()
+#define IVIRT .L_IVIRT_\name\()
#define IISIDE .L_IISIDE_\name\()
#define IDAR .L_IDAR_\name\()
#define IDSISR .L_IDSISR_\name\()
@@ -232,6 +211,9 @@ do_define_int n
.ifndef IAREA
.error "IAREA not defined"
.endif
+ .ifndef IVIRT
+ IVIRT=1
+ .endif
.ifndef IISIDE
IISIDE=0
.endif
@@ -325,7 +307,7 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
* outside the head section. CONFIG_RELOCATABLE KVM expects CTR
* to be saved in HSTATE_SCRATCH1.
*/
- mfctr r9
+ ld r9,IAREA+EX_CTR(r13)
std r9,HSTATE_SCRATCH1(r13)
__LOAD_FAR_HANDLER(r9, kvmppc_interrupt)
mtctr r9
@@ -362,101 +344,6 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
.endm
#endif
-.macro INT_SAVE_SRR_AND_JUMP label, hsrr, set_ri
- ld r10,PACAKMSR(r13) /* get MSR value for kernel */
- .if ! \set_ri
- xori r10,r10,MSR_RI /* Clear MSR_RI */
- .endif
- .if \hsrr == EXC_HV_OR_STD
- BEGIN_FTR_SECTION
- mfspr r11,SPRN_HSRR0 /* save HSRR0 */
- mfspr r12,SPRN_HSRR1 /* and HSRR1 */
- mtspr SPRN_HSRR1,r10
- FTR_SECTION_ELSE
- mfspr r11,SPRN_SRR0 /* save SRR0 */
- mfspr r12,SPRN_SRR1 /* and SRR1 */
- mtspr SPRN_SRR1,r10
- ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206)
- .elseif \hsrr
- mfspr r11,SPRN_HSRR0 /* save HSRR0 */
- mfspr r12,SPRN_HSRR1 /* and HSRR1 */
- mtspr SPRN_HSRR1,r10
- .else
- mfspr r11,SPRN_SRR0 /* save SRR0 */
- mfspr r12,SPRN_SRR1 /* and SRR1 */
- mtspr SPRN_SRR1,r10
- .endif
- LOAD_HANDLER(r10, \label\())
- .if \hsrr == EXC_HV_OR_STD
- BEGIN_FTR_SECTION
- mtspr SPRN_HSRR0,r10
- HRFI_TO_KERNEL
- FTR_SECTION_ELSE
- mtspr SPRN_SRR0,r10
- RFI_TO_KERNEL
- ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206)
- .elseif \hsrr
- mtspr SPRN_HSRR0,r10
- HRFI_TO_KERNEL
- .else
- mtspr SPRN_SRR0,r10
- RFI_TO_KERNEL
- .endif
- b . /* prevent speculative execution */
-.endm
-
-/* INT_SAVE_SRR_AND_JUMP works for real or virt, this is faster but virt only */
-.macro INT_VIRT_SAVE_SRR_AND_JUMP label, hsrr
-#ifdef CONFIG_RELOCATABLE
- .if \hsrr == EXC_HV_OR_STD
- BEGIN_FTR_SECTION
- mfspr r11,SPRN_HSRR0 /* save HSRR0 */
- FTR_SECTION_ELSE
- mfspr r11,SPRN_SRR0 /* save SRR0 */
- ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206)
- .elseif \hsrr
- mfspr r11,SPRN_HSRR0 /* save HSRR0 */
- .else
- mfspr r11,SPRN_SRR0 /* save SRR0 */
- .endif
- LOAD_HANDLER(r12, \label\())
- mtctr r12
- .if \hsrr == EXC_HV_OR_STD
- BEGIN_FTR_SECTION
- mfspr r12,SPRN_HSRR1 /* and HSRR1 */
- FTR_SECTION_ELSE
- mfspr r12,SPRN_SRR1 /* and HSRR1 */
- ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206)
- .elseif \hsrr
- mfspr r12,SPRN_HSRR1 /* and HSRR1 */
- .else
- mfspr r12,SPRN_SRR1 /* and HSRR1 */
- .endif
- li r10,MSR_RI
- mtmsrd r10,1 /* Set RI (EE=0) */
- bctr
-#else
- .if \hsrr == EXC_HV_OR_STD
- BEGIN_FTR_SECTION
- mfspr r11,SPRN_HSRR0 /* save HSRR0 */
- mfspr r12,SPRN_HSRR1 /* and HSRR1 */
- FTR_SECTION_ELSE
- mfspr r11,SPRN_SRR0 /* save SRR0 */
- mfspr r12,SPRN_SRR1 /* and SRR1 */
- ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206)
- .elseif \hsrr
- mfspr r11,SPRN_HSRR0 /* save HSRR0 */
- mfspr r12,SPRN_HSRR1 /* and HSRR1 */
- .else
- mfspr r11,SPRN_SRR0 /* save SRR0 */
- mfspr r12,SPRN_SRR1 /* and SRR1 */
- .endif
- li r10,MSR_RI
- mtmsrd r10,1 /* Set RI (EE=0) */
- b \label
-#endif
-.endm
-
/*
* This is the BOOK3S interrupt entry code macro.
*
@@ -481,6 +368,22 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
#define GEN_VIRT 1
#define GEN_OOL 1
+.macro GEN_BRANCH_TO_COMMON name virt
+ .if \virt
+#ifndef CONFIG_RELOCATABLE
+ b \name\()_common_virt
+#else
+ LOAD_HANDLER(r10, \name\()_common_virt)
+ mtctr r10
+ bctr
+#endif
+ .else
+ LOAD_HANDLER(r10, \name\()_common_real)
+ mtctr r10
+ bctr
+ .endif
+.endm
+
.macro GEN_INT_ENTRY name virt ool=0
SET_SCRATCH0(r13) /* save r13 */
GET_PACA(r13)
@@ -504,8 +407,10 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
OPT_SAVE_REG_TO_PACA(IAREA+EX_PPR, r9, CPU_FTR_HAS_PPR)
OPT_SAVE_REG_TO_PACA(IAREA+EX_CFAR, r10, CPU_FTR_CFAR)
INTERRUPT_TO_KERNEL
- SAVE_CTR(r10, IAREA)
+ mfctr r10
+ std r10,IAREA+EX_CTR(r13)
mfcr r9
+
.if (!\virt && IKVM_REAL) || (\virt && IKVM_VIRT)
KVMTEST \name IHSRR IVEC
.endif
@@ -570,27 +475,59 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
.if IEARLY == 2
/* nothing more */
.elseif IEARLY
- mfctr r10 /* save ctr, even for !RELOCATABLE */
BRANCH_TO_C000(r11, \name\()_common)
- .elseif !\virt
- INT_SAVE_SRR_AND_JUMP \name\()_common, IHSRR, ISET_RI
.else
- INT_VIRT_SAVE_SRR_AND_JUMP \name\()_common, IHSRR
+ .if IHSRR == EXC_HV_OR_STD
+ BEGIN_FTR_SECTION
+ mfspr r11,SPRN_HSRR0 /* save HSRR0 */
+ mfspr r12,SPRN_HSRR1 /* and HSRR1 */
+ FTR_SECTION_ELSE
+ mfspr r11,SPRN_SRR0 /* save SRR0 */
+ mfspr r12,SPRN_SRR1 /* and SRR1 */
+ ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206)
+ .elseif IHSRR
+ mfspr r11,SPRN_HSRR0 /* save HSRR0 */
+ mfspr r12,SPRN_HSRR1 /* and HSRR1 */
+ .else
+ mfspr r11,SPRN_SRR0 /* save SRR0 */
+ mfspr r12,SPRN_SRR1 /* and SRR1 */
.endif
+ GEN_BRANCH_TO_COMMON \name \virt
+ .endif
+
.if \ool
.popsection
.endif
.endm
/*
- * On entry r13 points to the paca, r9-r13 are saved in the paca,
- * r9 contains the saved CR, r11 and r12 contain the saved SRR0 and
- * SRR1, and relocation is on.
- *
- * If stack=0, then the stack is already set in r1, and r1 is saved in r10.
- * PPR save and CPU accounting is not done for the !stack case (XXX why not?)
+ * __GEN_COMMON_ENTRY is required to receive the branch from interrupt
+ * entry, except in the case of the IEARLY handlers.
+ * This switches to virtual mode and sets MSR[RI].
*/
-.macro GEN_COMMON name
+.macro __GEN_COMMON_ENTRY name
+DEFINE_FIXED_SYMBOL(\name\()_common_real)
+\name\()_common_real:
+ ld r10,PACAKMSR(r13) /* get MSR value for kernel */
+ .if ! ISET_RI
+ xori r10,r10,MSR_RI /* Clear MSR_RI */
+ .endif
+ mtmsrd r10
+
+ .if IVIRT
+ b 1f /* real entry skips the virt entry code */
+ .balign IFETCH_ALIGN_BYTES
+DEFINE_FIXED_SYMBOL(\name\()_common_virt)
+\name\()_common_virt:
+ .if ISET_RI
+ li r10,MSR_RI
+ mtmsrd r10,1 /* Set MSR_RI */
+ .endif
+1:
+ .endif /* IVIRT */
+.endm
+
+.macro __GEN_COMMON_BODY name
.if ISTACK
andi. r10,r12,MSR_PR /* See if coming from user */
mr r10,r1 /* Save r1 */
@@ -658,7 +595,7 @@ BEGIN_FTR_SECTION_NESTED(66)
ld r10,IAREA+EX_CFAR(r13)
std r10,ORIG_GPR3(r1)
END_FTR_SECTION_NESTED(CPU_FTR_CFAR, CPU_FTR_CFAR, 66)
- GET_CTR(r10, IAREA)
+ ld r10,IAREA+EX_CTR(r13)
std r10,_CTR(r1)
std r2,GPR2(r1) /* save r2 in stackframe */
SAVE_4GPRS(3, r1) /* save r3 - r6 in stackframe */
@@ -686,6 +623,19 @@ END_FTR_SECTION_NESTED(CPU_FTR_CFAR, CPU_FTR_CFAR, 66)
.endif
.endm
+/*
+ * On entry r13 points to the paca, r9-r13 are saved in the paca,
+ * r9 contains the saved CR, r11 and r12 contain the saved SRR0 and
+ * SRR1, and relocation is on.
+ *
+ * If stack=0, then the stack is already set in r1, and r1 is saved in r10.
+ * PPR save and CPU accounting is not done for the !stack case (XXX why not?)
+ */
+.macro GEN_COMMON name
+ __GEN_COMMON_ENTRY \name
+ __GEN_COMMON_BODY \name
+.endm
+
/*
* Restore all registers including H/SRR0/1 saved in a stack frame of a
* standard exception.
@@ -839,6 +789,7 @@ INT_DEFINE_BEGIN(system_reset)
IVEC=0x100
IHSRR=EXC_STD
IAREA=PACA_EXNMI
+ IVIRT=0 /* no virt entry point */
/*
* MSR_RI is not enabled, because PACA_EXNMI and nmi stack is
* being used, so a nested NMI exception would corrupt it.
@@ -918,6 +869,7 @@ TRAMP_REAL_BEGIN(system_reset_fwnmi)
#endif /* CONFIG_PPC_PSERIES */
EXC_COMMON_BEGIN(system_reset_common)
+ __GEN_COMMON_ENTRY system_reset
/*
* Increment paca->in_nmi then enable MSR_RI. SLB or MCE will be able
* to recover, but nested NMI will notice in_nmi and not recover
@@ -933,7 +885,7 @@ EXC_COMMON_BEGIN(system_reset_common)
mr r10,r1
ld r1,PACA_NMI_EMERG_SP(r13)
subi r1,r1,INT_FRAME_SIZE
- GEN_COMMON system_reset
+ __GEN_COMMON_BODY system_reset
bl save_nvgprs
/*
* Set IRQS_ALL_DISABLED unconditionally so arch_irqs_disabled does
@@ -979,6 +931,7 @@ INT_DEFINE_BEGIN(machine_check_early)
IVEC=0x200
IHSRR=EXC_STD
IAREA=PACA_EXMC
+ IVIRT=0 /* no virt entry point */
/*
* MSR_RI is not enabled, because PACA_EXMC is being used, so a
* nested machine check corrupts it. machine_check_common enables
@@ -997,6 +950,7 @@ INT_DEFINE_BEGIN(machine_check)
IVEC=0x200
IHSRR=EXC_STD
IAREA=PACA_EXMC
+ IVIRT=0 /* no virt entry point */
ISET_RI=0
IDAR=1
IDSISR=1
@@ -1029,7 +983,6 @@ TRAMP_KVM_BEGIN(machine_check_kvm)
EXCEPTION_RESTORE_REGS EXC_STD
EXC_COMMON_BEGIN(machine_check_early_common)
- mtctr r10 /* Restore ctr */
mfspr r11,SPRN_SRR0
mfspr r12,SPRN_SRR1
@@ -1068,7 +1021,7 @@ EXC_COMMON_BEGIN(machine_check_early_common)
bgt cr1,unrecoverable_mce /* Check if we hit limit of 4 */
subi r1,r1,INT_FRAME_SIZE /* alloc stack frame */
- GEN_COMMON machine_check_early
+ __GEN_COMMON_BODY machine_check_early
BEGIN_FTR_SECTION
bl enable_machine_check
@@ -1467,6 +1420,8 @@ EXC_VIRT_END(program_check, 0x4700, 0x100)
TRAMP_KVM_BEGIN(program_check_kvm)
GEN_KVM program_check
EXC_COMMON_BEGIN(program_check_common)
+ __GEN_COMMON_ENTRY program_check
+
/*
* It's possible to receive a TM Bad Thing type program check with
* userspace register values (in particular r1), but with SRR1 reporting
@@ -1492,11 +1447,11 @@ EXC_COMMON_BEGIN(program_check_common)
ld r1,PACAEMERGSP(r13) /* Use emergency stack */
subi r1,r1,INT_FRAME_SIZE /* alloc stack frame */
__ISTACK(program_check)=0
- GEN_COMMON program_check
+ __GEN_COMMON_BODY program_check
b 3f
2:
__ISTACK(program_check)=1
- GEN_COMMON program_check
+ __GEN_COMMON_BODY program_check
3:
bl save_nvgprs
addi r3,r1,STACK_FRAME_OVERHEAD
@@ -1885,6 +1840,7 @@ INT_DEFINE_BEGIN(hmi_exception_early)
IVEC=0xe60
IHSRR=EXC_HV
IAREA=PACA_EXGEN
+ IVIRT=0 /* no virt entry point */
IEARLY=1
ISTACK=0
IRECONCILE=0
@@ -1896,6 +1852,7 @@ INT_DEFINE_BEGIN(hmi_exception)
IVEC=0xe60
IHSRR=EXC_HV
IAREA=PACA_EXGEN
+ IVIRT=0 /* no virt entry point */
IMASK=IRQS_DISABLED
IKVM_REAL=1
INT_DEFINE_END(hmi_exception)
@@ -1910,14 +1867,13 @@ TRAMP_KVM_BEGIN(hmi_exception_kvm)
GEN_KVM hmi_exception
EXC_COMMON_BEGIN(hmi_exception_early_common)
- mtctr r10 /* Restore ctr */
mfspr r11,SPRN_HSRR0 /* Save HSRR0 */
mfspr r12,SPRN_HSRR1 /* Save HSRR1 */
mr r10,r1 /* Save r1 */
ld r1,PACAEMERGSP(r13) /* Use emergency stack for realmode */
subi r1,r1,INT_FRAME_SIZE /* alloc stack frame */
- GEN_COMMON hmi_exception_early
+ __GEN_COMMON_BODY hmi_exception_early
addi r3,r1,STACK_FRAME_OVERHEAD
bl hmi_exception_realmode
@@ -2259,7 +2215,9 @@ EXC_REAL_BEGIN(denorm_exception, 0x1500, 0x100)
bne+ denorm_assist
#endif
KVMTEST denorm_exception, EXC_HV, 0x1500
- INT_SAVE_SRR_AND_JUMP denorm_exception_common, EXC_HV, 1
+ mfspr r11,SPRN_HSRR0
+ mfspr r12,SPRN_HSRR1
+ GEN_BRANCH_TO_COMMON denorm_exception, GEN_REAL
EXC_REAL_END(denorm_exception, 0x1500, 0x100)
#ifdef CONFIG_PPC_DENORMALISATION
EXC_VIRT_BEGIN(denorm_exception, 0x5500, 0x100)
@@ -2267,7 +2225,9 @@ EXC_VIRT_BEGIN(denorm_exception, 0x5500, 0x100)
mfspr r10,SPRN_HSRR1
andis. r10,r10,(HSRR1_DENORM)@h /* denorm? */
bne+ denorm_assist
- INT_VIRT_SAVE_SRR_AND_JUMP denorm_exception_common, EXC_HV
+ mfspr r11,SPRN_HSRR0
+ mfspr r12,SPRN_HSRR1
+ GEN_BRANCH_TO_COMMON denorm_exception, GEN_VIRT
EXC_VIRT_END(denorm_exception, 0x5500, 0x100)
#else
EXC_VIRT_NONE(0x5500, 0x100)
@@ -2442,7 +2402,11 @@ EXC_VIRT_NONE(0x5800, 0x100)
std r12,PACA_EXGEN+EX_R12(r13); \
GET_SCRATCH0(r10); \
std r10,PACA_EXGEN+EX_R13(r13); \
- INT_SAVE_SRR_AND_JUMP soft_nmi_common, _H, 1
+ mfspr r11,SPRN_SRR0; /* save SRR0 */ \
+ mfspr r12,SPRN_SRR1; /* and SRR1 */ \
+ LOAD_HANDLER(r10, soft_nmi_common); \
+ mtctr r10; \
+ bctr
/*
* Branch to soft_nmi_interrupt using the emergency stack. The emergency
@@ -2458,7 +2422,7 @@ EXC_COMMON_BEGIN(soft_nmi_common)
ld r1,PACAEMERGSP(r13)
subi r1,r1,INT_FRAME_SIZE
__ISTACK(decrementer)=0
- GEN_COMMON decrementer
+ __GEN_COMMON_BODY decrementer
bl save_nvgprs
addi r3,r1,STACK_FRAME_OVERHEAD
bl soft_nmi_interrupt
--
2.22.0
^ permalink raw reply related
* [PATCH 03/10] powerpc/64s/exception: move soft-mask test to common code
From: Nicholas Piggin @ 2019-08-21 12:23 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin
In-Reply-To: <20190821122315.9535-1-npiggin@gmail.com>
As well as moving code out of the unrelocated vectors, this allows the
masked handlers to be moved to common code, and allows the soft_nmi
handler to be generated more like a regular handler.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/exceptions-64s.S | 108 +++++++++++++--------------
1 file changed, 51 insertions(+), 57 deletions(-)
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 6ea6176de00a..b78167ce6355 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -414,36 +414,6 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
.if (!\virt && IKVM_REAL) || (\virt && IKVM_VIRT)
KVMTEST \name IHSRR IVEC
.endif
- .if IMASK
- lbz r10,PACAIRQSOFTMASK(r13)
- andi. r10,r10,IMASK
- /* Associate vector numbers with bits in paca->irq_happened */
- .if IVEC == 0x500 || IVEC == 0xea0
- li r10,PACA_IRQ_EE
- .elseif IVEC == 0x900
- li r10,PACA_IRQ_DEC
- .elseif IVEC == 0xa00 || IVEC == 0xe80
- li r10,PACA_IRQ_DBELL
- .elseif IVEC == 0xe60
- li r10,PACA_IRQ_HMI
- .elseif IVEC == 0xf00
- li r10,PACA_IRQ_PMI
- .else
- .abort "Bad maskable vector"
- .endif
-
- .if IHSRR == EXC_HV_OR_STD
- BEGIN_FTR_SECTION
- bne masked_Hinterrupt
- FTR_SECTION_ELSE
- bne masked_interrupt
- ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206)
- .elseif IHSRR
- bne masked_Hinterrupt
- .else
- bne masked_interrupt
- .endif
- .endif
std r11,IAREA+EX_R11(r13)
std r12,IAREA+EX_R12(r13)
@@ -528,6 +498,37 @@ DEFINE_FIXED_SYMBOL(\name\()_common_virt)
.endm
.macro __GEN_COMMON_BODY name
+ .if IMASK
+ lbz r10,PACAIRQSOFTMASK(r13)
+ andi. r10,r10,IMASK
+ /* Associate vector numbers with bits in paca->irq_happened */
+ .if IVEC == 0x500 || IVEC == 0xea0
+ li r10,PACA_IRQ_EE
+ .elseif IVEC == 0x900
+ li r10,PACA_IRQ_DEC
+ .elseif IVEC == 0xa00 || IVEC == 0xe80
+ li r10,PACA_IRQ_DBELL
+ .elseif IVEC == 0xe60
+ li r10,PACA_IRQ_HMI
+ .elseif IVEC == 0xf00
+ li r10,PACA_IRQ_PMI
+ .else
+ .abort "Bad maskable vector"
+ .endif
+
+ .if IHSRR == EXC_HV_OR_STD
+ BEGIN_FTR_SECTION
+ bne masked_Hinterrupt
+ FTR_SECTION_ELSE
+ bne masked_interrupt
+ ALT_FTR_SECTION_END_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206)
+ .elseif IHSRR
+ bne masked_Hinterrupt
+ .else
+ bne masked_interrupt
+ .endif
+ .endif
+
.if ISTACK
andi. r10,r12,MSR_PR /* See if coming from user */
mr r10,r1 /* Save r1 */
@@ -2395,18 +2396,12 @@ EXC_VIRT_NONE(0x5800, 0x100)
#ifdef CONFIG_PPC_WATCHDOG
-#define MASKED_DEC_HANDLER_LABEL 3f
-
-#define MASKED_DEC_HANDLER(_H) \
-3: /* soft-nmi */ \
- std r12,PACA_EXGEN+EX_R12(r13); \
- GET_SCRATCH0(r10); \
- std r10,PACA_EXGEN+EX_R13(r13); \
- mfspr r11,SPRN_SRR0; /* save SRR0 */ \
- mfspr r12,SPRN_SRR1; /* and SRR1 */ \
- LOAD_HANDLER(r10, soft_nmi_common); \
- mtctr r10; \
- bctr
+INT_DEFINE_BEGIN(soft_nmi)
+ IVEC=0x900
+ IHSRR=EXC_STD
+ IAREA=PACA_EXGEN
+ ISTACK=0
+INT_DEFINE_END(soft_nmi)
/*
* Branch to soft_nmi_interrupt using the emergency stack. The emergency
@@ -2418,19 +2413,16 @@ EXC_VIRT_NONE(0x5800, 0x100)
* and run it entirely with interrupts hard disabled.
*/
EXC_COMMON_BEGIN(soft_nmi_common)
+ mfspr r11,SPRN_SRR0
mr r10,r1
ld r1,PACAEMERGSP(r13)
subi r1,r1,INT_FRAME_SIZE
- __ISTACK(decrementer)=0
- __GEN_COMMON_BODY decrementer
+ __GEN_COMMON_BODY soft_nmi
bl save_nvgprs
addi r3,r1,STACK_FRAME_OVERHEAD
bl soft_nmi_interrupt
b ret_from_except
-#else /* CONFIG_PPC_WATCHDOG */
-#define MASKED_DEC_HANDLER_LABEL 2f /* normal return */
-#define MASKED_DEC_HANDLER(_H)
#endif /* CONFIG_PPC_WATCHDOG */
/*
@@ -2449,7 +2441,6 @@ masked_Hinterrupt:
.else
masked_interrupt:
.endif
- std r11,PACA_EXGEN+EX_R11(r13)
lbz r11,PACAIRQHAPPENED(r13)
or r11,r11,r10
stb r11,PACAIRQHAPPENED(r13)
@@ -2458,26 +2449,30 @@ masked_interrupt:
lis r10,0x7fff
ori r10,r10,0xffff
mtspr SPRN_DEC,r10
- b MASKED_DEC_HANDLER_LABEL
+#ifdef CONFIG_PPC_WATCHDOG
+ b soft_nmi_common
+#else
+ b 2f
+#endif
1: andi. r10,r10,PACA_IRQ_MUST_HARD_MASK
beq 2f
+ xori r12,r12,MSR_EE /* clear MSR_EE */
.if \hsrr
- mfspr r10,SPRN_HSRR1
- xori r10,r10,MSR_EE /* clear MSR_EE */
- mtspr SPRN_HSRR1,r10
+ mtspr SPRN_HSRR1,r12
.else
- mfspr r10,SPRN_SRR1
- xori r10,r10,MSR_EE /* clear MSR_EE */
- mtspr SPRN_SRR1,r10
+ mtspr SPRN_SRR1,r12
.endif
ori r11,r11,PACA_IRQ_HARD_DIS
stb r11,PACAIRQHAPPENED(r13)
2: /* done */
+ ld r10,PACA_EXGEN+EX_CTR(r13)
+ mtctr r10
mtcrf 0x80,r9
std r1,PACAR1(r13)
ld r9,PACA_EXGEN+EX_R9(r13)
ld r10,PACA_EXGEN+EX_R10(r13)
ld r11,PACA_EXGEN+EX_R11(r13)
+ ld r12,PACA_EXGEN+EX_R12(r13)
/* returns to kernel where r13 must be set up, so don't restore it */
.if \hsrr
HRFI_TO_KERNEL
@@ -2485,7 +2480,6 @@ masked_interrupt:
RFI_TO_KERNEL
.endif
b .
- MASKED_DEC_HANDLER(\hsrr\())
.endm
TRAMP_REAL_BEGIN(stf_barrier_fallback)
@@ -2592,7 +2586,7 @@ TRAMP_REAL_BEGIN(hrfi_flush_fallback)
* instruction code patches (which end up in the common .text area)
* cannot reach these if they are put there.
*/
-USE_FIXED_SECTION(virt_trampolines)
+USE_TEXT_SECTION()
MASKED_INTERRUPT EXC_STD
MASKED_INTERRUPT EXC_HV
--
2.22.0
^ permalink raw reply related
* [PATCH 04/10] powerpc/64s/exception: move KVM test to common code
From: Nicholas Piggin @ 2019-08-21 12:23 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin
In-Reply-To: <20190821122315.9535-1-npiggin@gmail.com>
This allows more code to be moved out of unrelocated regions. The system
call KVMTEST is changed to be open-coded and remain in the tramp area to
avoid having to move it to entry_64.S. The custom nature of the system
call entry code means the hcall case can be made more streamlined than
regular interrupt handlers.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/exceptions-64s.S | 235 ++++++++++++------------
arch/powerpc/kvm/book3s_hv_rmhandlers.S | 11 --
arch/powerpc/kvm/book3s_segment.S | 7 -
3 files changed, 115 insertions(+), 138 deletions(-)
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index b78167ce6355..e5d5d81d6107 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -44,7 +44,6 @@
* EXC_VIRT_BEGIN/END - virt (AIL), unrelocated exception vectors
* TRAMP_REAL_BEGIN - real, unrelocated helpers (virt may call these)
* TRAMP_VIRT_BEGIN - virt, unreloc helpers (in practice, real can use)
- * TRAMP_KVM_BEGIN - KVM handlers, these are put into real, unrelocated
* EXC_COMMON - After switching to virtual, relocated mode.
*/
@@ -74,13 +73,6 @@ name:
#define TRAMP_VIRT_BEGIN(name) \
FIXED_SECTION_ENTRY_BEGIN(virt_trampolines, name)
-#ifdef CONFIG_KVM_BOOK3S_64_HANDLER
-#define TRAMP_KVM_BEGIN(name) \
- TRAMP_VIRT_BEGIN(name)
-#else
-#define TRAMP_KVM_BEGIN(name)
-#endif
-
#define EXC_REAL_NONE(start, size) \
FIXED_SECTION_ENTRY_BEGIN_LOCATION(real_vectors, exc_real_##start##_##unused, start, size); \
FIXED_SECTION_ENTRY_END_LOCATION(real_vectors, exc_real_##start##_##unused, start, size)
@@ -271,6 +263,9 @@ do_define_int n
.endm
.macro GEN_KVM name
+ .balign IFETCH_ALIGN_BYTES
+\name\()_kvm:
+
.if IKVM_SKIP
cmpwi r10,KVM_GUEST_MODE_SKIP
beq 89f
@@ -281,13 +276,18 @@ BEGIN_FTR_SECTION_NESTED(947)
END_FTR_SECTION_NESTED(CPU_FTR_CFAR,CPU_FTR_CFAR,947)
.endif
+ ld r10,PACA_EXGEN+EX_CTR(r13)
+ mtctr r10
BEGIN_FTR_SECTION_NESTED(948)
ld r10,IAREA+EX_PPR(r13)
std r10,HSTATE_PPR(r13)
END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
- ld r10,IAREA+EX_R10(r13)
+ ld r11,IAREA+EX_R11(r13)
+ ld r12,IAREA+EX_R12(r13)
std r12,HSTATE_SCRATCH0(r13)
sldi r12,r9,32
+ ld r9,IAREA+EX_R9(r13)
+ ld r10,IAREA+EX_R10(r13)
/* HSRR variants have the 0x2 bit added to their trap number */
.if IHSRR == EXC_HV_OR_STD
BEGIN_FTR_SECTION
@@ -300,29 +300,16 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
.else
ori r12,r12,(IVEC)
.endif
-
-#ifdef CONFIG_RELOCATABLE
- /*
- * KVM requires __LOAD_FAR_HANDLER beause kvmppc_interrupt lives
- * outside the head section. CONFIG_RELOCATABLE KVM expects CTR
- * to be saved in HSTATE_SCRATCH1.
- */
- ld r9,IAREA+EX_CTR(r13)
- std r9,HSTATE_SCRATCH1(r13)
- __LOAD_FAR_HANDLER(r9, kvmppc_interrupt)
- mtctr r9
- ld r9,IAREA+EX_R9(r13)
- bctr
-#else
- ld r9,IAREA+EX_R9(r13)
b kvmppc_interrupt
-#endif
-
.if IKVM_SKIP
89: mtocrf 0x80,r9
+ ld r10,PACA_EXGEN+EX_CTR(r13)
+ mtctr r10
ld r9,IAREA+EX_R9(r13)
ld r10,IAREA+EX_R10(r13)
+ ld r11,IAREA+EX_R11(r13)
+ ld r12,IAREA+EX_R12(r13)
.if IHSRR == EXC_HV_OR_STD
BEGIN_FTR_SECTION
b kvmppc_skip_Hinterrupt
@@ -410,11 +397,6 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
mfctr r10
std r10,IAREA+EX_CTR(r13)
mfcr r9
-
- .if (!\virt && IKVM_REAL) || (\virt && IKVM_VIRT)
- KVMTEST \name IHSRR IVEC
- .endif
-
std r11,IAREA+EX_R11(r13)
std r12,IAREA+EX_R12(r13)
@@ -478,6 +460,10 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
.macro __GEN_COMMON_ENTRY name
DEFINE_FIXED_SYMBOL(\name\()_common_real)
\name\()_common_real:
+ .if IKVM_REAL
+ KVMTEST \name IHSRR IVEC
+ .endif
+
ld r10,PACAKMSR(r13) /* get MSR value for kernel */
.if ! ISET_RI
xori r10,r10,MSR_RI /* Clear MSR_RI */
@@ -489,6 +475,10 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real)
.balign IFETCH_ALIGN_BYTES
DEFINE_FIXED_SYMBOL(\name\()_common_virt)
\name\()_common_virt:
+ .if IKVM_VIRT
+ KVMTEST \name IHSRR IVEC
+ .endif
+
.if ISET_RI
li r10,MSR_RI
mtmsrd r10,1 /* Set MSR_RI */
@@ -848,8 +838,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_HVMODE | CPU_FTR_ARCH_206)
*/
EXC_REAL_END(system_reset, 0x100, 0x100)
EXC_VIRT_NONE(0x4100, 0x100)
-TRAMP_KVM_BEGIN(system_reset_kvm)
- GEN_KVM system_reset
#ifdef CONFIG_PPC_P7_NAP
TRAMP_REAL_BEGIN(system_reset_idle_wake)
@@ -927,6 +915,8 @@ EXC_COMMON_BEGIN(system_reset_common)
EXCEPTION_RESTORE_REGS EXC_STD
RFI_TO_USER_OR_KERNEL
+ GEN_KVM system_reset
+
INT_DEFINE_BEGIN(machine_check_early)
IVEC=0x200
@@ -970,9 +960,6 @@ TRAMP_REAL_BEGIN(machine_check_fwnmi)
GEN_INT_ENTRY machine_check_early, GEN_REAL
#endif
-TRAMP_KVM_BEGIN(machine_check_kvm)
- GEN_KVM machine_check
-
#define MACHINE_CHECK_HANDLER_WINDUP \
/* Clear MSR_RI before setting SRR0 and SRR1. */\
li r9,0; \
@@ -1129,6 +1116,9 @@ EXC_COMMON_BEGIN(machine_check_common)
bl machine_check_exception
b ret_from_except
+ GEN_KVM machine_check
+
+
#ifdef CONFIG_PPC_P7_NAP
/*
* This is an idle wakeup. Low level machine check has already been
@@ -1223,8 +1213,6 @@ EXC_REAL_END(data_access, 0x300, 0x80)
EXC_VIRT_BEGIN(data_access, 0x4300, 0x80)
GEN_INT_ENTRY data_access, GEN_VIRT
EXC_VIRT_END(data_access, 0x4300, 0x80)
-TRAMP_KVM_BEGIN(data_access_kvm)
- GEN_KVM data_access
EXC_COMMON_BEGIN(data_access_common)
GEN_COMMON data_access
ld r4,_DAR(r1)
@@ -1237,6 +1225,8 @@ MMU_FTR_SECTION_ELSE
b handle_page_fault
ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
+ GEN_KVM data_access
+
INT_DEFINE_BEGIN(data_access_slb)
IVEC=0x380
@@ -1254,8 +1244,6 @@ EXC_REAL_END(data_access_slb, 0x380, 0x80)
EXC_VIRT_BEGIN(data_access_slb, 0x4380, 0x80)
GEN_INT_ENTRY data_access_slb, GEN_VIRT
EXC_VIRT_END(data_access_slb, 0x4380, 0x80)
-TRAMP_KVM_BEGIN(data_access_slb_kvm)
- GEN_KVM data_access_slb
EXC_COMMON_BEGIN(data_access_slb_common)
GEN_COMMON data_access_slb
ld r4,_DAR(r1)
@@ -1280,6 +1268,8 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
bl do_bad_slb_fault
b ret_from_except
+ GEN_KVM data_access_slb
+
INT_DEFINE_BEGIN(instruction_access)
IVEC=0x400
@@ -1297,8 +1287,6 @@ EXC_REAL_END(instruction_access, 0x400, 0x80)
EXC_VIRT_BEGIN(instruction_access, 0x4400, 0x80)
GEN_INT_ENTRY instruction_access, GEN_VIRT
EXC_VIRT_END(instruction_access, 0x4400, 0x80)
-TRAMP_KVM_BEGIN(instruction_access_kvm)
- GEN_KVM instruction_access
EXC_COMMON_BEGIN(instruction_access_common)
GEN_COMMON instruction_access
ld r4,_DAR(r1)
@@ -1311,6 +1299,8 @@ MMU_FTR_SECTION_ELSE
b handle_page_fault
ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
+ GEN_KVM instruction_access
+
INT_DEFINE_BEGIN(instruction_access_slb)
IVEC=0x480
@@ -1328,8 +1318,6 @@ EXC_REAL_END(instruction_access_slb, 0x480, 0x80)
EXC_VIRT_BEGIN(instruction_access_slb, 0x4480, 0x80)
GEN_INT_ENTRY instruction_access_slb, GEN_VIRT
EXC_VIRT_END(instruction_access_slb, 0x4480, 0x80)
-TRAMP_KVM_BEGIN(instruction_access_slb_kvm)
- GEN_KVM instruction_access_slb
EXC_COMMON_BEGIN(instruction_access_slb_common)
GEN_COMMON instruction_access_slb
ld r4,_DAR(r1)
@@ -1354,6 +1342,9 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
bl do_bad_slb_fault
b ret_from_except
+ GEN_KVM instruction_access_slb
+
+
INT_DEFINE_BEGIN(hardware_interrupt)
IVEC=0x500
IHSRR=EXC_HV_OR_STD
@@ -1369,8 +1360,6 @@ EXC_REAL_END(hardware_interrupt, 0x500, 0x100)
EXC_VIRT_BEGIN(hardware_interrupt, 0x4500, 0x100)
GEN_INT_ENTRY hardware_interrupt, GEN_VIRT
EXC_VIRT_END(hardware_interrupt, 0x4500, 0x100)
-TRAMP_KVM_BEGIN(hardware_interrupt_kvm)
- GEN_KVM hardware_interrupt
EXC_COMMON_BEGIN(hardware_interrupt_common)
GEN_COMMON hardware_interrupt
FINISH_NAP
@@ -1379,6 +1368,8 @@ EXC_COMMON_BEGIN(hardware_interrupt_common)
bl do_IRQ
b ret_from_except_lite
+ GEN_KVM hardware_interrupt
+
INT_DEFINE_BEGIN(alignment)
IVEC=0x600
@@ -1395,8 +1386,6 @@ EXC_REAL_END(alignment, 0x600, 0x100)
EXC_VIRT_BEGIN(alignment, 0x4600, 0x100)
GEN_INT_ENTRY alignment, GEN_VIRT
EXC_VIRT_END(alignment, 0x4600, 0x100)
-TRAMP_KVM_BEGIN(alignment_kvm)
- GEN_KVM alignment
EXC_COMMON_BEGIN(alignment_common)
GEN_COMMON alignment
bl save_nvgprs
@@ -1404,6 +1393,8 @@ EXC_COMMON_BEGIN(alignment_common)
bl alignment_exception
b ret_from_except
+ GEN_KVM alignment
+
INT_DEFINE_BEGIN(program_check)
IVEC=0x700
@@ -1418,8 +1409,6 @@ EXC_REAL_END(program_check, 0x700, 0x100)
EXC_VIRT_BEGIN(program_check, 0x4700, 0x100)
GEN_INT_ENTRY program_check, GEN_VIRT
EXC_VIRT_END(program_check, 0x4700, 0x100)
-TRAMP_KVM_BEGIN(program_check_kvm)
- GEN_KVM program_check
EXC_COMMON_BEGIN(program_check_common)
__GEN_COMMON_ENTRY program_check
@@ -1459,6 +1448,8 @@ EXC_COMMON_BEGIN(program_check_common)
bl program_check_exception
b ret_from_except
+ GEN_KVM program_check
+
INT_DEFINE_BEGIN(fp_unavailable)
IVEC=0x800
@@ -1474,8 +1465,6 @@ EXC_REAL_END(fp_unavailable, 0x800, 0x100)
EXC_VIRT_BEGIN(fp_unavailable, 0x4800, 0x100)
GEN_INT_ENTRY fp_unavailable, GEN_VIRT
EXC_VIRT_END(fp_unavailable, 0x4800, 0x100)
-TRAMP_KVM_BEGIN(fp_unavailable_kvm)
- GEN_KVM fp_unavailable
EXC_COMMON_BEGIN(fp_unavailable_common)
GEN_COMMON fp_unavailable
bne 1f /* if from user, just load it up */
@@ -1505,6 +1494,8 @@ END_FTR_SECTION_IFSET(CPU_FTR_TM)
b ret_from_except
#endif
+ GEN_KVM fp_unavailable
+
INT_DEFINE_BEGIN(decrementer)
IVEC=0x900
@@ -1520,8 +1511,6 @@ EXC_REAL_END(decrementer, 0x900, 0x80)
EXC_VIRT_BEGIN(decrementer, 0x4900, 0x80)
GEN_INT_ENTRY decrementer, GEN_VIRT
EXC_VIRT_END(decrementer, 0x4900, 0x80)
-TRAMP_KVM_BEGIN(decrementer_kvm)
- GEN_KVM decrementer
EXC_COMMON_BEGIN(decrementer_common)
GEN_COMMON decrementer
FINISH_NAP
@@ -1530,6 +1519,8 @@ EXC_COMMON_BEGIN(decrementer_common)
bl timer_interrupt
b ret_from_except_lite
+ GEN_KVM decrementer
+
INT_DEFINE_BEGIN(hdecrementer)
IVEC=0x980
@@ -1545,8 +1536,6 @@ EXC_REAL_END(hdecrementer, 0x980, 0x80)
EXC_VIRT_BEGIN(hdecrementer, 0x4980, 0x80)
GEN_INT_ENTRY hdecrementer, GEN_VIRT
EXC_VIRT_END(hdecrementer, 0x4980, 0x80)
-TRAMP_KVM_BEGIN(hdecrementer_kvm)
- GEN_KVM hdecrementer
EXC_COMMON_BEGIN(hdecrementer_common)
GEN_COMMON hdecrementer
bl save_nvgprs
@@ -1554,6 +1543,8 @@ EXC_COMMON_BEGIN(hdecrementer_common)
bl hdec_interrupt
b ret_from_except
+ GEN_KVM hdecrementer
+
INT_DEFINE_BEGIN(doorbell_super)
IVEC=0xa00
@@ -1569,8 +1560,6 @@ EXC_REAL_END(doorbell_super, 0xa00, 0x100)
EXC_VIRT_BEGIN(doorbell_super, 0x4a00, 0x100)
GEN_INT_ENTRY doorbell_super, GEN_VIRT
EXC_VIRT_END(doorbell_super, 0x4a00, 0x100)
-TRAMP_KVM_BEGIN(doorbell_super_kvm)
- GEN_KVM doorbell_super
EXC_COMMON_BEGIN(doorbell_super_common)
GEN_COMMON doorbell_super
FINISH_NAP
@@ -1583,6 +1572,8 @@ EXC_COMMON_BEGIN(doorbell_super_common)
#endif
b ret_from_except_lite
+ GEN_KVM doorbell_super
+
EXC_REAL_NONE(0xb00, 0x100)
EXC_VIRT_NONE(0x4b00, 0x100)
@@ -1704,6 +1695,7 @@ EXC_VIRT_BEGIN(system_call, 0x4c00, 0x100)
EXC_VIRT_END(system_call, 0x4c00, 0x100)
#ifdef CONFIG_KVM_BOOK3S_64_HANDLER
+TRAMP_REAL_BEGIN(system_call_kvm)
/*
* This is a hcall, so register convention is as above, with these
* differences:
@@ -1711,20 +1703,35 @@ EXC_VIRT_END(system_call, 0x4c00, 0x100)
* ctr = orig r13
* orig r10 saved in PACA
*/
-TRAMP_KVM_BEGIN(system_call_kvm)
/*
* Save the PPR (on systems that support it) before changing to
* HMT_MEDIUM. That allows the KVM code to save that value into the
* guest state (it is the guest's PPR value).
*/
- OPT_GET_SPR(r10, SPRN_PPR, CPU_FTR_HAS_PPR)
+BEGIN_FTR_SECTION_NESTED(948)
+ mfspr r10,SPRN_PPR
+ std r10,HSTATE_PPR(r13)
+END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
HMT_MEDIUM
- OPT_SAVE_REG_TO_PACA(PACA_EXGEN+EX_PPR, r10, CPU_FTR_HAS_PPR)
mfctr r10
SET_SCRATCH0(r10)
- std r9,PACA_EXGEN+EX_R9(r13)
- mfcr r9
- GEN_KVM system_call
+ mfcr r10
+ std r12,HSTATE_SCRATCH0(r13)
+ sldi r12,r10,32
+ ori r12,r12,0xc00
+#ifdef CONFIG_RELOCATABLE
+ /*
+ * Requires __LOAD_FAR_HANDLER beause kvmppc_interrupt lives
+ * outside the head section.
+ */
+ __LOAD_FAR_HANDLER(r10, kvmppc_interrupt)
+ mtctr r10
+ ld r10,PACA_EXGEN+EX_R10(r13)
+ bctr
+#else
+ ld r10,PACA_EXGEN+EX_R10(r13)
+ b kvmppc_interrupt
+#endif
#endif
@@ -1741,8 +1748,6 @@ EXC_REAL_END(single_step, 0xd00, 0x100)
EXC_VIRT_BEGIN(single_step, 0x4d00, 0x100)
GEN_INT_ENTRY single_step, GEN_VIRT
EXC_VIRT_END(single_step, 0x4d00, 0x100)
-TRAMP_KVM_BEGIN(single_step_kvm)
- GEN_KVM single_step
EXC_COMMON_BEGIN(single_step_common)
GEN_COMMON single_step
bl save_nvgprs
@@ -1750,6 +1755,8 @@ EXC_COMMON_BEGIN(single_step_common)
bl single_step_exception
b ret_from_except
+ GEN_KVM single_step
+
INT_DEFINE_BEGIN(h_data_storage)
IVEC=0xe00
@@ -1768,8 +1775,6 @@ EXC_REAL_END(h_data_storage, 0xe00, 0x20)
EXC_VIRT_BEGIN(h_data_storage, 0x4e00, 0x20)
GEN_INT_ENTRY h_data_storage, GEN_VIRT, GEN_OOL
EXC_VIRT_END(h_data_storage, 0x4e00, 0x20)
-TRAMP_KVM_BEGIN(h_data_storage_kvm)
- GEN_KVM h_data_storage
EXC_COMMON_BEGIN(h_data_storage_common)
GEN_COMMON h_data_storage
bl save_nvgprs
@@ -1783,6 +1788,8 @@ MMU_FTR_SECTION_ELSE
ALT_MMU_FTR_SECTION_END_IFSET(MMU_FTR_TYPE_RADIX)
b ret_from_except
+ GEN_KVM h_data_storage
+
INT_DEFINE_BEGIN(h_instr_storage)
IVEC=0xe20
@@ -1798,8 +1805,6 @@ EXC_REAL_END(h_instr_storage, 0xe20, 0x20)
EXC_VIRT_BEGIN(h_instr_storage, 0x4e20, 0x20)
GEN_INT_ENTRY h_instr_storage, GEN_VIRT, GEN_OOL
EXC_VIRT_END(h_instr_storage, 0x4e20, 0x20)
-TRAMP_KVM_BEGIN(h_instr_storage_kvm)
- GEN_KVM h_instr_storage
EXC_COMMON_BEGIN(h_instr_storage_common)
GEN_COMMON h_instr_storage
bl save_nvgprs
@@ -1807,6 +1812,8 @@ EXC_COMMON_BEGIN(h_instr_storage_common)
bl unknown_exception
b ret_from_except
+ GEN_KVM h_instr_storage
+
INT_DEFINE_BEGIN(emulation_assist)
IVEC=0xe40
@@ -1822,8 +1829,6 @@ EXC_REAL_END(emulation_assist, 0xe40, 0x20)
EXC_VIRT_BEGIN(emulation_assist, 0x4e40, 0x20)
GEN_INT_ENTRY emulation_assist, GEN_VIRT, GEN_OOL
EXC_VIRT_END(emulation_assist, 0x4e40, 0x20)
-TRAMP_KVM_BEGIN(emulation_assist_kvm)
- GEN_KVM emulation_assist
EXC_COMMON_BEGIN(emulation_assist_common)
GEN_COMMON emulation_assist
bl save_nvgprs
@@ -1831,6 +1836,8 @@ EXC_COMMON_BEGIN(emulation_assist_common)
bl emulation_assist_interrupt
b ret_from_except
+ GEN_KVM emulation_assist
+
/*
* hmi_exception trampoline is a special case. It jumps to hmi_exception_early
@@ -1862,10 +1869,6 @@ EXC_REAL_BEGIN(hmi_exception, 0xe60, 0x20)
GEN_INT_ENTRY hmi_exception_early, GEN_REAL, GEN_OOL
EXC_REAL_END(hmi_exception, 0xe60, 0x20)
EXC_VIRT_NONE(0x4e60, 0x20)
-TRAMP_KVM_BEGIN(hmi_exception_early_kvm)
- GEN_KVM hmi_exception_early
-TRAMP_KVM_BEGIN(hmi_exception_kvm)
- GEN_KVM hmi_exception
EXC_COMMON_BEGIN(hmi_exception_early_common)
mfspr r11,SPRN_HSRR0 /* Save HSRR0 */
@@ -1892,6 +1895,8 @@ EXC_COMMON_BEGIN(hmi_exception_early_common)
EXCEPTION_RESTORE_REGS EXC_HV
GEN_INT_ENTRY hmi_exception, GEN_REAL
+ GEN_KVM hmi_exception_early
+
EXC_COMMON_BEGIN(hmi_exception_common)
GEN_COMMON hmi_exception
FINISH_NAP
@@ -1901,6 +1906,8 @@ EXC_COMMON_BEGIN(hmi_exception_common)
bl handle_hmi_exception
b ret_from_except
+ GEN_KVM hmi_exception
+
INT_DEFINE_BEGIN(h_doorbell)
IVEC=0xe80
@@ -1917,8 +1924,6 @@ EXC_REAL_END(h_doorbell, 0xe80, 0x20)
EXC_VIRT_BEGIN(h_doorbell, 0x4e80, 0x20)
GEN_INT_ENTRY h_doorbell, GEN_VIRT, GEN_OOL
EXC_VIRT_END(h_doorbell, 0x4e80, 0x20)
-TRAMP_KVM_BEGIN(h_doorbell_kvm)
- GEN_KVM h_doorbell
EXC_COMMON_BEGIN(h_doorbell_common)
GEN_COMMON h_doorbell
FINISH_NAP
@@ -1931,6 +1936,8 @@ EXC_COMMON_BEGIN(h_doorbell_common)
#endif
b ret_from_except_lite
+ GEN_KVM h_doorbell
+
INT_DEFINE_BEGIN(h_virt_irq)
IVEC=0xea0
@@ -1947,8 +1954,6 @@ EXC_REAL_END(h_virt_irq, 0xea0, 0x20)
EXC_VIRT_BEGIN(h_virt_irq, 0x4ea0, 0x20)
GEN_INT_ENTRY h_virt_irq, GEN_VIRT, GEN_OOL
EXC_VIRT_END(h_virt_irq, 0x4ea0, 0x20)
-TRAMP_KVM_BEGIN(h_virt_irq_kvm)
- GEN_KVM h_virt_irq
EXC_COMMON_BEGIN(h_virt_irq_common)
GEN_COMMON h_virt_irq
FINISH_NAP
@@ -1957,6 +1962,8 @@ EXC_COMMON_BEGIN(h_virt_irq_common)
bl do_IRQ
b ret_from_except_lite
+ GEN_KVM h_virt_irq
+
EXC_REAL_NONE(0xec0, 0x20)
EXC_VIRT_NONE(0x4ec0, 0x20)
@@ -1978,8 +1985,6 @@ EXC_REAL_END(performance_monitor, 0xf00, 0x20)
EXC_VIRT_BEGIN(performance_monitor, 0x4f00, 0x20)
GEN_INT_ENTRY performance_monitor, GEN_VIRT, GEN_OOL
EXC_VIRT_END(performance_monitor, 0x4f00, 0x20)
-TRAMP_KVM_BEGIN(performance_monitor_kvm)
- GEN_KVM performance_monitor
EXC_COMMON_BEGIN(performance_monitor_common)
GEN_COMMON performance_monitor
FINISH_NAP
@@ -1988,6 +1993,8 @@ EXC_COMMON_BEGIN(performance_monitor_common)
bl performance_monitor_exception
b ret_from_except_lite
+ GEN_KVM performance_monitor
+
INT_DEFINE_BEGIN(altivec_unavailable)
IVEC=0xf20
@@ -2003,8 +2010,6 @@ EXC_REAL_END(altivec_unavailable, 0xf20, 0x20)
EXC_VIRT_BEGIN(altivec_unavailable, 0x4f20, 0x20)
GEN_INT_ENTRY altivec_unavailable, GEN_VIRT, GEN_OOL
EXC_VIRT_END(altivec_unavailable, 0x4f20, 0x20)
-TRAMP_KVM_BEGIN(altivec_unavailable_kvm)
- GEN_KVM altivec_unavailable
EXC_COMMON_BEGIN(altivec_unavailable_common)
GEN_COMMON altivec_unavailable
#ifdef CONFIG_ALTIVEC
@@ -2038,6 +2043,8 @@ END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC)
bl altivec_unavailable_exception
b ret_from_except
+ GEN_KVM altivec_unavailable
+
INT_DEFINE_BEGIN(vsx_unavailable)
IVEC=0xf40
@@ -2053,8 +2060,6 @@ EXC_REAL_END(vsx_unavailable, 0xf40, 0x20)
EXC_VIRT_BEGIN(vsx_unavailable, 0x4f40, 0x20)
GEN_INT_ENTRY vsx_unavailable, GEN_VIRT, GEN_OOL
EXC_VIRT_END(vsx_unavailable, 0x4f40, 0x20)
-TRAMP_KVM_BEGIN(vsx_unavailable_kvm)
- GEN_KVM vsx_unavailable
EXC_COMMON_BEGIN(vsx_unavailable_common)
GEN_COMMON vsx_unavailable
#ifdef CONFIG_VSX
@@ -2087,6 +2092,8 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX)
bl vsx_unavailable_exception
b ret_from_except
+ GEN_KVM vsx_unavailable
+
INT_DEFINE_BEGIN(facility_unavailable)
IVEC=0xf60
@@ -2101,8 +2108,6 @@ EXC_REAL_END(facility_unavailable, 0xf60, 0x20)
EXC_VIRT_BEGIN(facility_unavailable, 0x4f60, 0x20)
GEN_INT_ENTRY facility_unavailable, GEN_VIRT, GEN_OOL
EXC_VIRT_END(facility_unavailable, 0x4f60, 0x20)
-TRAMP_KVM_BEGIN(facility_unavailable_kvm)
- GEN_KVM facility_unavailable
EXC_COMMON_BEGIN(facility_unavailable_common)
GEN_COMMON facility_unavailable
bl save_nvgprs
@@ -2110,6 +2115,8 @@ EXC_COMMON_BEGIN(facility_unavailable_common)
bl facility_unavailable_exception
b ret_from_except
+ GEN_KVM facility_unavailable
+
INT_DEFINE_BEGIN(h_facility_unavailable)
IVEC=0xf80
@@ -2125,8 +2132,6 @@ EXC_REAL_END(h_facility_unavailable, 0xf80, 0x20)
EXC_VIRT_BEGIN(h_facility_unavailable, 0x4f80, 0x20)
GEN_INT_ENTRY h_facility_unavailable, GEN_VIRT, GEN_OOL
EXC_VIRT_END(h_facility_unavailable, 0x4f80, 0x20)
-TRAMP_KVM_BEGIN(h_facility_unavailable_kvm)
- GEN_KVM h_facility_unavailable
EXC_COMMON_BEGIN(h_facility_unavailable_common)
GEN_COMMON h_facility_unavailable
bl save_nvgprs
@@ -2134,6 +2139,8 @@ EXC_COMMON_BEGIN(h_facility_unavailable_common)
bl facility_unavailable_exception
b ret_from_except
+ GEN_KVM h_facility_unavailable
+
EXC_REAL_NONE(0xfa0, 0x20)
EXC_VIRT_NONE(0x4fa0, 0x20)
@@ -2160,14 +2167,15 @@ EXC_REAL_BEGIN(cbe_system_error, 0x1200, 0x100)
GEN_INT_ENTRY cbe_system_error, GEN_REAL
EXC_REAL_END(cbe_system_error, 0x1200, 0x100)
EXC_VIRT_NONE(0x5200, 0x100)
-TRAMP_KVM_BEGIN(cbe_system_error_kvm)
- GEN_KVM cbe_system_error
EXC_COMMON_BEGIN(cbe_system_error_common)
GEN_COMMON cbe_system_error
bl save_nvgprs
addi r3,r1,STACK_FRAME_OVERHEAD
bl cbe_system_error_exception
b ret_from_except
+
+ GEN_KVM cbe_system_error
+
#else /* CONFIG_CBE_RAS */
EXC_REAL_NONE(0x1200, 0x100)
EXC_VIRT_NONE(0x5200, 0x100)
@@ -2188,8 +2196,6 @@ EXC_REAL_END(instruction_breakpoint, 0x1300, 0x100)
EXC_VIRT_BEGIN(instruction_breakpoint, 0x5300, 0x100)
GEN_INT_ENTRY instruction_breakpoint, GEN_VIRT
EXC_VIRT_END(instruction_breakpoint, 0x5300, 0x100)
-TRAMP_KVM_BEGIN(instruction_breakpoint_kvm)
- GEN_KVM instruction_breakpoint
EXC_COMMON_BEGIN(instruction_breakpoint_common)
GEN_COMMON instruction_breakpoint
bl save_nvgprs
@@ -2197,6 +2203,8 @@ EXC_COMMON_BEGIN(instruction_breakpoint_common)
bl instruction_breakpoint_exception
b ret_from_except
+ GEN_KVM instruction_breakpoint
+
EXC_REAL_NONE(0x1400, 0x100)
EXC_VIRT_NONE(0x5400, 0x100)
@@ -2206,6 +2214,7 @@ INT_DEFINE_BEGIN(denorm_exception)
IHSRR=EXC_HV
IAREA=PACA_EXGEN
IEARLY=2
+ IKVM_REAL=1
INT_DEFINE_END(denorm_exception)
EXC_REAL_BEGIN(denorm_exception, 0x1500, 0x100)
@@ -2215,7 +2224,6 @@ EXC_REAL_BEGIN(denorm_exception, 0x1500, 0x100)
andis. r10,r10,(HSRR1_DENORM)@h /* denorm? */
bne+ denorm_assist
#endif
- KVMTEST denorm_exception, EXC_HV, 0x1500
mfspr r11,SPRN_HSRR0
mfspr r12,SPRN_HSRR1
GEN_BRANCH_TO_COMMON denorm_exception, GEN_REAL
@@ -2233,8 +2241,6 @@ EXC_VIRT_END(denorm_exception, 0x5500, 0x100)
#else
EXC_VIRT_NONE(0x5500, 0x100)
#endif
-TRAMP_KVM_BEGIN(denorm_exception_kvm)
- GEN_KVM denorm_exception
#ifdef CONFIG_PPC_DENORMALISATION
TRAMP_REAL_BEGIN(denorm_assist)
@@ -2312,6 +2318,8 @@ EXC_COMMON_BEGIN(denorm_exception_common)
bl unknown_exception
b ret_from_except
+ GEN_KVM denorm_exception
+
#ifdef CONFIG_CBE_RAS
INT_DEFINE_BEGIN(cbe_maintenance)
@@ -2326,14 +2334,15 @@ EXC_REAL_BEGIN(cbe_maintenance, 0x1600, 0x100)
GEN_INT_ENTRY cbe_maintenance, GEN_REAL
EXC_REAL_END(cbe_maintenance, 0x1600, 0x100)
EXC_VIRT_NONE(0x5600, 0x100)
-TRAMP_KVM_BEGIN(cbe_maintenance_kvm)
- GEN_KVM cbe_maintenance
EXC_COMMON_BEGIN(cbe_maintenance_common)
GEN_COMMON cbe_maintenance
bl save_nvgprs
addi r3,r1,STACK_FRAME_OVERHEAD
bl cbe_maintenance_exception
b ret_from_except
+
+ GEN_KVM cbe_maintenance
+
#else /* CONFIG_CBE_RAS */
EXC_REAL_NONE(0x1600, 0x100)
EXC_VIRT_NONE(0x5600, 0x100)
@@ -2353,8 +2362,6 @@ EXC_REAL_END(altivec_assist, 0x1700, 0x100)
EXC_VIRT_BEGIN(altivec_assist, 0x5700, 0x100)
GEN_INT_ENTRY altivec_assist, GEN_VIRT
EXC_VIRT_END(altivec_assist, 0x5700, 0x100)
-TRAMP_KVM_BEGIN(altivec_assist_kvm)
- GEN_KVM altivec_assist
EXC_COMMON_BEGIN(altivec_assist_common)
GEN_COMMON altivec_assist
bl save_nvgprs
@@ -2366,6 +2373,8 @@ EXC_COMMON_BEGIN(altivec_assist_common)
#endif
b ret_from_except
+ GEN_KVM altivec_assist
+
#ifdef CONFIG_CBE_RAS
INT_DEFINE_BEGIN(cbe_thermal)
@@ -2380,14 +2389,15 @@ EXC_REAL_BEGIN(cbe_thermal, 0x1800, 0x100)
GEN_INT_ENTRY cbe_thermal, GEN_REAL
EXC_REAL_END(cbe_thermal, 0x1800, 0x100)
EXC_VIRT_NONE(0x5800, 0x100)
-TRAMP_KVM_BEGIN(cbe_thermal_kvm)
- GEN_KVM cbe_thermal
EXC_COMMON_BEGIN(cbe_thermal_common)
GEN_COMMON cbe_thermal
bl save_nvgprs
addi r3,r1,STACK_FRAME_OVERHEAD
bl cbe_thermal_exception
b ret_from_except
+
+ GEN_KVM cbe_thermal
+
#else /* CONFIG_CBE_RAS */
EXC_REAL_NONE(0x1800, 0x100)
EXC_VIRT_NONE(0x5800, 0x100)
@@ -2581,17 +2591,12 @@ TRAMP_REAL_BEGIN(hrfi_flush_fallback)
GET_SCRATCH0(r13);
hrfid
-/*
- * Real mode exceptions actually use this too, but alternate
- * instruction code patches (which end up in the common .text area)
- * cannot reach these if they are put there.
- */
USE_TEXT_SECTION()
MASKED_INTERRUPT EXC_STD
MASKED_INTERRUPT EXC_HV
#ifdef CONFIG_KVM_BOOK3S_64_HANDLER
-TRAMP_REAL_BEGIN(kvmppc_skip_interrupt)
+kvmppc_skip_interrupt:
/*
* Here all GPRs are unchanged from when the interrupt happened
* except for r13, which is saved in SPRG_SCRATCH0.
@@ -2603,7 +2608,7 @@ TRAMP_REAL_BEGIN(kvmppc_skip_interrupt)
RFI_TO_KERNEL
b .
-TRAMP_REAL_BEGIN(kvmppc_skip_Hinterrupt)
+kvmppc_skip_Hinterrupt:
/*
* Here all GPRs are unchanged from when the interrupt happened
* except for r13, which is saved in SPRG_SCRATCH0.
@@ -2616,16 +2621,6 @@ TRAMP_REAL_BEGIN(kvmppc_skip_Hinterrupt)
b .
#endif
-/*
- * Ensure that any handlers that get invoked from the exception prologs
- * above are below the first 64KB (0x10000) of the kernel image because
- * the prologs assemble the addresses of these handlers using the
- * LOAD_HANDLER macro, which uses an ori instruction.
- */
-
-/*** Common interrupt handlers ***/
-
-
/*
* Relocation-on interrupts: A subset of the interrupts can be delivered
* with IR=1/DR=1, if AIL==2 and MSR.HV won't be changed by delivering
diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
index 337e64468d78..4d5c11ff256b 100644
--- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S
+++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
@@ -1234,7 +1234,6 @@ kvmppc_interrupt_hv:
* R12 = (guest CR << 32) | interrupt vector
* R13 = PACA
* guest R12 saved in shadow VCPU SCRATCH0
- * guest CTR saved in shadow VCPU SCRATCH1 if RELOCATABLE
* guest R13 saved in SPRN_SCRATCH0
*/
std r9, HSTATE_SCRATCH2(r13)
@@ -1335,12 +1334,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
11: stw r3,VCPU_HEIR(r9)
/* these are volatile across C function calls */
-#ifdef CONFIG_RELOCATABLE
- ld r3, HSTATE_SCRATCH1(r13)
- mtctr r3
-#else
mfctr r3
-#endif
mfxer r4
std r3, VCPU_CTR(r9)
std r4, VCPU_XER(r9)
@@ -3184,7 +3178,6 @@ END_FTR_SECTION_IFCLR(CPU_FTR_P9_TM_HV_ASSIST)
* r12 is (CR << 32) | vector
* r13 points to our PACA
* r12 is saved in HSTATE_SCRATCH0(r13)
- * ctr is saved in HSTATE_SCRATCH1(r13) if RELOCATABLE
* r9 is saved in HSTATE_SCRATCH2(r13)
* r13 is saved in HSPRG1
* cfar is saved in HSTATE_CFAR(r13)
@@ -3233,11 +3226,7 @@ kvmppc_bad_host_intr:
ld r5, HSTATE_CFAR(r13)
std r5, ORIG_GPR3(r1)
mflr r3
-#ifdef CONFIG_RELOCATABLE
- ld r4, HSTATE_SCRATCH1(r13)
-#else
mfctr r4
-#endif
mfxer r5
lbz r6, PACAIRQSOFTMASK(r13)
std r3, _LINK(r1)
diff --git a/arch/powerpc/kvm/book3s_segment.S b/arch/powerpc/kvm/book3s_segment.S
index 0169bab544dd..1f492aa4c8d6 100644
--- a/arch/powerpc/kvm/book3s_segment.S
+++ b/arch/powerpc/kvm/book3s_segment.S
@@ -167,16 +167,9 @@ kvmppc_interrupt_pr:
* R12 = (guest CR << 32) | exit handler id
* R13 = PACA
* HSTATE.SCRATCH0 = guest R12
- * HSTATE.SCRATCH1 = guest CTR if RELOCATABLE
*/
#ifdef CONFIG_PPC64
/* Match 32-bit entry */
-#ifdef CONFIG_RELOCATABLE
- std r9, HSTATE_SCRATCH2(r13)
- ld r9, HSTATE_SCRATCH1(r13)
- mtctr r9
- ld r9, HSTATE_SCRATCH2(r13)
-#endif
rotldi r12, r12, 32 /* Flip R12 halves for stw */
stw r12, HSTATE_SCRATCH1(r13) /* CR is now in the low half */
srdi r12, r12, 32 /* shift trap into low half */
--
2.22.0
^ permalink raw reply related
* [PATCH 05/10] powerpc/64s/exception: remove confusing IEARLY option
From: Nicholas Piggin @ 2019-08-21 12:23 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin
In-Reply-To: <20190821122315.9535-1-npiggin@gmail.com>
Replace IEARLY=1 and IEARLY=2 with IBRANCH_COMMON, which controls if
the entry code branches to a common handler; and IREALMODE_COMMON,
which controls whether the common handler should remain in real mode.
These special cases no longer avoid loading the SRR registers, there
is no point as most of them load the registers immediately anyway.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/exceptions-64s.S | 48 ++++++++++++++--------------
1 file changed, 24 insertions(+), 24 deletions(-)
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index e5d5d81d6107..705cca95e03c 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -174,7 +174,8 @@ END_FTR_SECTION_NESTED(ftr,ftr,943)
#define IDAR .L_IDAR_\name\()
#define IDSISR .L_IDSISR_\name\()
#define ISET_RI .L_ISET_RI_\name\()
-#define IEARLY .L_IEARLY_\name\()
+#define IBRANCH_COMMON .L_IBRANCH_COMMON_\name\()
+#define IREALMODE_COMMON .L_IREALMODE_COMMON_\name\()
#define IMASK .L_IMASK_\name\()
#define IKVM_SKIP .L_IKVM_SKIP_\name\()
#define IKVM_REAL .L_IKVM_REAL_\name\()
@@ -218,8 +219,15 @@ do_define_int n
.ifndef ISET_RI
ISET_RI=1
.endif
- .ifndef IEARLY
- IEARLY=0
+ .ifndef IBRANCH_COMMON
+ IBRANCH_COMMON=1
+ .endif
+ .ifndef IREALMODE_COMMON
+ IREALMODE_COMMON=0
+ .else
+ .if ! IBRANCH_COMMON
+ .error "IREALMODE_COMMON=1 but IBRANCH_COMMON=0"
+ .endif
.endif
.ifndef IMASK
IMASK=0
@@ -356,6 +364,11 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
#define GEN_OOL 1
.macro GEN_BRANCH_TO_COMMON name virt
+ .if IREALMODE_COMMON
+ LOAD_HANDLER(r10, \name\()_common)
+ mtctr r10
+ bctr
+ .else
.if \virt
#ifndef CONFIG_RELOCATABLE
b \name\()_common_virt
@@ -369,6 +382,7 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
mtctr r10
bctr
.endif
+ .endif
.endm
.macro GEN_INT_ENTRY name virt ool=0
@@ -424,11 +438,6 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
stw r10,IAREA+EX_DSISR(r13)
.endif
- .if IEARLY == 2
- /* nothing more */
- .elseif IEARLY
- BRANCH_TO_C000(r11, \name\()_common)
- .else
.if IHSRR == EXC_HV_OR_STD
BEGIN_FTR_SECTION
mfspr r11,SPRN_HSRR0 /* save HSRR0 */
@@ -444,6 +453,8 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
mfspr r11,SPRN_SRR0 /* save SRR0 */
mfspr r12,SPRN_SRR1 /* and SRR1 */
.endif
+
+ .if IBRANCH_COMMON
GEN_BRANCH_TO_COMMON \name \virt
.endif
@@ -923,6 +934,7 @@ INT_DEFINE_BEGIN(machine_check_early)
IHSRR=EXC_STD
IAREA=PACA_EXMC
IVIRT=0 /* no virt entry point */
+ IREALMODE_COMMON=1
/*
* MSR_RI is not enabled, because PACA_EXMC is being used, so a
* nested machine check corrupts it. machine_check_common enables
@@ -930,7 +942,6 @@ INT_DEFINE_BEGIN(machine_check_early)
*/
ISET_RI=0
ISTACK=0
- IEARLY=1
IDAR=1
IDSISR=1
IRECONCILE=0
@@ -971,9 +982,6 @@ TRAMP_REAL_BEGIN(machine_check_fwnmi)
EXCEPTION_RESTORE_REGS EXC_STD
EXC_COMMON_BEGIN(machine_check_early_common)
- mfspr r11,SPRN_SRR0
- mfspr r12,SPRN_SRR1
-
/*
* Switch to mc_emergency stack and handle re-entrancy (we limit
* the nested MCE upto level 4 to avoid stack overflow).
@@ -1849,7 +1857,7 @@ INT_DEFINE_BEGIN(hmi_exception_early)
IHSRR=EXC_HV
IAREA=PACA_EXGEN
IVIRT=0 /* no virt entry point */
- IEARLY=1
+ IREALMODE_COMMON=1
ISTACK=0
IRECONCILE=0
IKUAP=0 /* We don't touch AMR here, we never go to virtual mode */
@@ -1871,8 +1879,6 @@ EXC_REAL_END(hmi_exception, 0xe60, 0x20)
EXC_VIRT_NONE(0x4e60, 0x20)
EXC_COMMON_BEGIN(hmi_exception_early_common)
- mfspr r11,SPRN_HSRR0 /* Save HSRR0 */
- mfspr r12,SPRN_HSRR1 /* Save HSRR1 */
mr r10,r1 /* Save r1 */
ld r1,PACAEMERGSP(r13) /* Use emergency stack for realmode */
subi r1,r1,INT_FRAME_SIZE /* alloc stack frame */
@@ -2213,29 +2219,23 @@ INT_DEFINE_BEGIN(denorm_exception)
IVEC=0x1500
IHSRR=EXC_HV
IAREA=PACA_EXGEN
- IEARLY=2
+ IBRANCH_COMMON=0
IKVM_REAL=1
INT_DEFINE_END(denorm_exception)
EXC_REAL_BEGIN(denorm_exception, 0x1500, 0x100)
GEN_INT_ENTRY denorm_exception, GEN_REAL
#ifdef CONFIG_PPC_DENORMALISATION
- mfspr r10,SPRN_HSRR1
- andis. r10,r10,(HSRR1_DENORM)@h /* denorm? */
+ andis. r10,r12,(HSRR1_DENORM)@h /* denorm? */
bne+ denorm_assist
#endif
- mfspr r11,SPRN_HSRR0
- mfspr r12,SPRN_HSRR1
GEN_BRANCH_TO_COMMON denorm_exception, GEN_REAL
EXC_REAL_END(denorm_exception, 0x1500, 0x100)
#ifdef CONFIG_PPC_DENORMALISATION
EXC_VIRT_BEGIN(denorm_exception, 0x5500, 0x100)
GEN_INT_ENTRY denorm_exception, GEN_VIRT
- mfspr r10,SPRN_HSRR1
- andis. r10,r10,(HSRR1_DENORM)@h /* denorm? */
+ andis. r10,r12,(HSRR1_DENORM)@h /* denorm? */
bne+ denorm_assist
- mfspr r11,SPRN_HSRR0
- mfspr r12,SPRN_HSRR1
GEN_BRANCH_TO_COMMON denorm_exception, GEN_VIRT
EXC_VIRT_END(denorm_exception, 0x5500, 0x100)
#else
--
2.22.0
^ permalink raw reply related
* [PATCH 06/10] powerpc/64s/exception: remove the SPR saving patch code macros
From: Nicholas Piggin @ 2019-08-21 12:23 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin
In-Reply-To: <20190821122315.9535-1-npiggin@gmail.com>
These are used infrequently enough they don't provide much help, so
inline them.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/exceptions-64s.S | 82 ++++++++++------------------
1 file changed, 28 insertions(+), 54 deletions(-)
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 705cca95e03c..14351e1b27df 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -110,46 +110,6 @@ name:
#define EXC_HV 1
#define EXC_STD 0
-/*
- * PPR save/restore macros used in exceptions-64s.S
- * Used for P7 or later processors
- */
-#define SAVE_PPR(area, ra) \
-BEGIN_FTR_SECTION_NESTED(940) \
- ld ra,area+EX_PPR(r13); /* Read PPR from paca */ \
- std ra,_PPR(r1); \
-END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,940)
-
-#define RESTORE_PPR_PACA(area, ra) \
-BEGIN_FTR_SECTION_NESTED(941) \
- ld ra,area+EX_PPR(r13); \
- mtspr SPRN_PPR,ra; \
-END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,941)
-
-/*
- * Get an SPR into a register if the CPU has the given feature
- */
-#define OPT_GET_SPR(ra, spr, ftr) \
-BEGIN_FTR_SECTION_NESTED(943) \
- mfspr ra,spr; \
-END_FTR_SECTION_NESTED(ftr,ftr,943)
-
-/*
- * Set an SPR from a register if the CPU has the given feature
- */
-#define OPT_SET_SPR(ra, spr, ftr) \
-BEGIN_FTR_SECTION_NESTED(943) \
- mtspr spr,ra; \
-END_FTR_SECTION_NESTED(ftr,ftr,943)
-
-/*
- * Save a register to the PACA if the CPU has the given feature
- */
-#define OPT_SAVE_REG_TO_PACA(offset, ra, ftr) \
-BEGIN_FTR_SECTION_NESTED(943) \
- std ra,offset(r13); \
-END_FTR_SECTION_NESTED(ftr,ftr,943)
-
/*
* Branch to label using its 0xC000 address. This results in instruction
* address suitable for MSR[IR]=0 or 1, which allows relocation to be turned
@@ -278,18 +238,18 @@ do_define_int n
cmpwi r10,KVM_GUEST_MODE_SKIP
beq 89f
.else
-BEGIN_FTR_SECTION_NESTED(947)
+BEGIN_FTR_SECTION
ld r10,IAREA+EX_CFAR(r13)
std r10,HSTATE_CFAR(r13)
-END_FTR_SECTION_NESTED(CPU_FTR_CFAR,CPU_FTR_CFAR,947)
+END_FTR_SECTION_IFSET(CPU_FTR_CFAR)
.endif
ld r10,PACA_EXGEN+EX_CTR(r13)
mtctr r10
-BEGIN_FTR_SECTION_NESTED(948)
+BEGIN_FTR_SECTION
ld r10,IAREA+EX_PPR(r13)
std r10,HSTATE_PPR(r13)
-END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
+END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
ld r11,IAREA+EX_R11(r13)
ld r12,IAREA+EX_R12(r13)
std r12,HSTATE_SCRATCH0(r13)
@@ -389,10 +349,14 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
SET_SCRATCH0(r13) /* save r13 */
GET_PACA(r13)
std r9,IAREA+EX_R9(r13) /* save r9 */
- OPT_GET_SPR(r9, SPRN_PPR, CPU_FTR_HAS_PPR)
+BEGIN_FTR_SECTION
+ mfspr r9,SPRN_PPR
+END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
HMT_MEDIUM
std r10,IAREA+EX_R10(r13) /* save r10 - r12 */
- OPT_GET_SPR(r10, SPRN_CFAR, CPU_FTR_CFAR)
+BEGIN_FTR_SECTION
+ mfspr r10,SPRN_CFAR
+END_FTR_SECTION_IFSET(CPU_FTR_CFAR)
.if \ool
.if !\virt
b tramp_real_\name
@@ -405,8 +369,12 @@ END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
.endif
.endif
- OPT_SAVE_REG_TO_PACA(IAREA+EX_PPR, r9, CPU_FTR_HAS_PPR)
- OPT_SAVE_REG_TO_PACA(IAREA+EX_CFAR, r10, CPU_FTR_CFAR)
+BEGIN_FTR_SECTION
+ std r9,IAREA+EX_PPR(r13)
+END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
+BEGIN_FTR_SECTION
+ std r10,IAREA+EX_CFAR(r13)
+END_FTR_SECTION_IFSET(CPU_FTR_CFAR)
INTERRUPT_TO_KERNEL
mfctr r10
std r10,IAREA+EX_CTR(r13)
@@ -553,7 +521,10 @@ DEFINE_FIXED_SYMBOL(\name\()_common_virt)
.endif
beq 101f /* if from kernel mode */
ACCOUNT_CPU_USER_ENTRY(r13, r9, r10)
- SAVE_PPR(IAREA, r9)
+BEGIN_FTR_SECTION
+ ld r9,IAREA+EX_PPR(r13) /* Read PPR from paca */
+ std r9,_PPR(r1)
+END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
101:
.else
.if IKUAP
@@ -593,10 +564,10 @@ DEFINE_FIXED_SYMBOL(\name\()_common_virt)
std r10,_DSISR(r1)
.endif
-BEGIN_FTR_SECTION_NESTED(66)
+BEGIN_FTR_SECTION
ld r10,IAREA+EX_CFAR(r13)
std r10,ORIG_GPR3(r1)
-END_FTR_SECTION_NESTED(CPU_FTR_CFAR, CPU_FTR_CFAR, 66)
+END_FTR_SECTION_IFSET(CPU_FTR_CFAR)
ld r10,IAREA+EX_CTR(r13)
std r10,_CTR(r1)
std r2,GPR2(r1) /* save r2 in stackframe */
@@ -1716,10 +1687,10 @@ TRAMP_REAL_BEGIN(system_call_kvm)
* HMT_MEDIUM. That allows the KVM code to save that value into the
* guest state (it is the guest's PPR value).
*/
-BEGIN_FTR_SECTION_NESTED(948)
+BEGIN_FTR_SECTION
mfspr r10,SPRN_PPR
std r10,HSTATE_PPR(r13)
-END_FTR_SECTION_NESTED(CPU_FTR_HAS_PPR,CPU_FTR_HAS_PPR,948)
+END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
HMT_MEDIUM
mfctr r10
SET_SCRATCH0(r10)
@@ -2298,7 +2269,10 @@ denorm_done:
mtspr SPRN_HSRR0,r11
mtcrf 0x80,r9
ld r9,PACA_EXGEN+EX_R9(r13)
- RESTORE_PPR_PACA(PACA_EXGEN, r10)
+BEGIN_FTR_SECTION
+ ld r10,PACA_EXGEN+EX_PPR(r13)
+ mtspr SPRN_PPR,r10
+END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
BEGIN_FTR_SECTION
ld r10,PACA_EXGEN+EX_CFAR(r13)
mtspr SPRN_CFAR,r10
--
2.22.0
^ permalink raw reply related
* [PATCH 07/10] powerpc/64s/exception: trim unused arguments from KVMTEST macro
From: Nicholas Piggin @ 2019-08-21 12:23 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin
In-Reply-To: <20190821122315.9535-1-npiggin@gmail.com>
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/exceptions-64s.S | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 14351e1b27df..a4ceb88c53c4 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -224,7 +224,7 @@ do_define_int n
#define kvmppc_interrupt kvmppc_interrupt_pr
#endif
-.macro KVMTEST name, hsrr, n
+.macro KVMTEST name
lbz r10,HSTATE_IN_GUEST(r13)
cmpwi r10,0
bne \name\()_kvm
@@ -293,7 +293,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
.endm
#else
-.macro KVMTEST name, hsrr, n
+.macro KVMTEST name
.endm
.macro GEN_KVM name
.endm
@@ -440,7 +440,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_CFAR)
DEFINE_FIXED_SYMBOL(\name\()_common_real)
\name\()_common_real:
.if IKVM_REAL
- KVMTEST \name IHSRR IVEC
+ KVMTEST \name
.endif
ld r10,PACAKMSR(r13) /* get MSR value for kernel */
@@ -455,7 +455,7 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real)
DEFINE_FIXED_SYMBOL(\name\()_common_virt)
\name\()_common_virt:
.if IKVM_VIRT
- KVMTEST \name IHSRR IVEC
+ KVMTEST \name
.endif
.if ISET_RI
@@ -1615,7 +1615,7 @@ INT_DEFINE_END(system_call)
GET_PACA(r13)
std r10,PACA_EXGEN+EX_R10(r13)
INTERRUPT_TO_KERNEL
- KVMTEST system_call EXC_STD 0xc00 /* uses r10, branch to system_call_kvm */
+ KVMTEST system_call /* uses r10, branch to system_call_kvm */
mfctr r9
#else
mr r9,r13
--
2.22.0
^ permalink raw reply related
* [PATCH 08/10] powerpc/64s/exception: hdecrementer avoid touching the stack
From: Nicholas Piggin @ 2019-08-21 12:23 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin
In-Reply-To: <20190821122315.9535-1-npiggin@gmail.com>
The hdec interrupt handler is reported to sometimes fire in Linux if
KVM leaves it pending after a guest exists. This is harmless, so there
is a no-op handler for it.
The interrupt handler currently uses the regular kernel stack. Change
this to avoid touching the stack entirely.
This should be the last place where the regular Linux stack can be
accessed with asynchronous interrupts (including PMI) soft-masked.
It might be possible to take advantage of this invariant, e.g., to
context switch the kernel stack SLB entry without clearing MSR[EE].
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/include/asm/time.h | 1 -
arch/powerpc/kernel/exceptions-64s.S | 25 ++++++++++++++++++++-----
arch/powerpc/kernel/time.c | 9 ---------
3 files changed, 20 insertions(+), 15 deletions(-)
diff --git a/arch/powerpc/include/asm/time.h b/arch/powerpc/include/asm/time.h
index 54f4ec1f9fab..0e17c41c4f46 100644
--- a/arch/powerpc/include/asm/time.h
+++ b/arch/powerpc/include/asm/time.h
@@ -24,7 +24,6 @@ extern struct clock_event_device decrementer_clockevent;
extern void generic_calibrate_decr(void);
-extern void hdec_interrupt(struct pt_regs *regs);
/* Some sane defaults: 125 MHz timebase, 1GHz processor */
extern unsigned long ppc_proc_freq;
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index a4ceb88c53c4..3f4b7dfa800b 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -1505,6 +1505,8 @@ INT_DEFINE_BEGIN(hdecrementer)
IVEC=0x980
IHSRR=EXC_HV
IAREA=PACA_EXGEN
+ ISTACK=0
+ IRECONCILE=0
IKVM_REAL=1
IKVM_VIRT=1
INT_DEFINE_END(hdecrementer)
@@ -1516,11 +1518,24 @@ EXC_VIRT_BEGIN(hdecrementer, 0x4980, 0x80)
GEN_INT_ENTRY hdecrementer, GEN_VIRT
EXC_VIRT_END(hdecrementer, 0x4980, 0x80)
EXC_COMMON_BEGIN(hdecrementer_common)
- GEN_COMMON hdecrementer
- bl save_nvgprs
- addi r3,r1,STACK_FRAME_OVERHEAD
- bl hdec_interrupt
- b ret_from_except
+ __GEN_COMMON_ENTRY hdecrementer
+ /*
+ * Hypervisor decrementer interrupts not caught by the KVM test
+ * shouldn't occur but are sometimes left pending on exit from a KVM
+ * guest. We don't need to do anything to clear them, as they are
+ * edge-triggered.
+ *
+ * Be careful to avoid touching the kernel stack.
+ */
+ ld r10,PACA_EXGEN+EX_CTR(r13)
+ mtctr r10
+ mtcrf 0x80,r9
+ ld r9,PACA_EXGEN+EX_R9(r13)
+ ld r10,PACA_EXGEN+EX_R10(r13)
+ ld r11,PACA_EXGEN+EX_R11(r13)
+ ld r12,PACA_EXGEN+EX_R12(r13)
+ ld r13,PACA_EXGEN+EX_R13(r13)
+ HRFI_TO_KERNEL
GEN_KVM hdecrementer
diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c
index 694522308cd5..bebc8c440289 100644
--- a/arch/powerpc/kernel/time.c
+++ b/arch/powerpc/kernel/time.c
@@ -663,15 +663,6 @@ void timer_broadcast_interrupt(void)
}
#endif
-/*
- * Hypervisor decrementer interrupts shouldn't occur but are sometimes
- * left pending on exit from a KVM guest. We don't need to do anything
- * to clear them, as they are edge-triggered.
- */
-void hdec_interrupt(struct pt_regs *regs)
-{
-}
-
#ifdef CONFIG_SUSPEND
static void generic_suspend_disable_irqs(void)
{
--
2.22.0
^ permalink raw reply related
* [PATCH 09/10] powerpc/64s/exception: re-inline some handlers
From: Nicholas Piggin @ 2019-08-21 12:23 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin
In-Reply-To: <20190821122315.9535-1-npiggin@gmail.com>
The reduction in interrupt entry size allows some handlers to be
re-inlined.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/exceptions-64s.S | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 3f4b7dfa800b..87bce1fd7e11 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -1187,7 +1187,7 @@ INT_DEFINE_BEGIN(data_access)
INT_DEFINE_END(data_access)
EXC_REAL_BEGIN(data_access, 0x300, 0x80)
- GEN_INT_ENTRY data_access, GEN_REAL, GEN_OOL
+ GEN_INT_ENTRY data_access, GEN_REAL
EXC_REAL_END(data_access, 0x300, 0x80)
EXC_VIRT_BEGIN(data_access, 0x4300, 0x80)
GEN_INT_ENTRY data_access, GEN_VIRT
@@ -1218,7 +1218,7 @@ INT_DEFINE_BEGIN(data_access_slb)
INT_DEFINE_END(data_access_slb)
EXC_REAL_BEGIN(data_access_slb, 0x380, 0x80)
- GEN_INT_ENTRY data_access_slb, GEN_REAL, GEN_OOL
+ GEN_INT_ENTRY data_access_slb, GEN_REAL
EXC_REAL_END(data_access_slb, 0x380, 0x80)
EXC_VIRT_BEGIN(data_access_slb, 0x4380, 0x80)
GEN_INT_ENTRY data_access_slb, GEN_VIRT
@@ -1485,7 +1485,7 @@ INT_DEFINE_BEGIN(decrementer)
INT_DEFINE_END(decrementer)
EXC_REAL_BEGIN(decrementer, 0x900, 0x80)
- GEN_INT_ENTRY decrementer, GEN_REAL, GEN_OOL
+ GEN_INT_ENTRY decrementer, GEN_REAL
EXC_REAL_END(decrementer, 0x900, 0x80)
EXC_VIRT_BEGIN(decrementer, 0x4900, 0x80)
GEN_INT_ENTRY decrementer, GEN_VIRT
--
2.22.0
^ permalink raw reply related
* [PATCH 10/10] powerpc/64s/exception: add more comments for interrupt handlers
From: Nicholas Piggin @ 2019-08-21 12:23 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Nicholas Piggin
In-Reply-To: <20190821122315.9535-1-npiggin@gmail.com>
A few of the non-standard handlers are left uncommented. Some more
description could be added to some.
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/exceptions-64s.S | 408 ++++++++++++++++++++++++---
1 file changed, 369 insertions(+), 39 deletions(-)
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 87bce1fd7e11..2963b46f9580 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -126,25 +126,25 @@ name:
/*
* Interrupt code generation macros
*/
-#define IVEC .L_IVEC_\name\()
-#define IHSRR .L_IHSRR_\name\()
-#define IAREA .L_IAREA_\name\()
-#define IVIRT .L_IVIRT_\name\()
-#define IISIDE .L_IISIDE_\name\()
-#define IDAR .L_IDAR_\name\()
-#define IDSISR .L_IDSISR_\name\()
-#define ISET_RI .L_ISET_RI_\name\()
-#define IBRANCH_COMMON .L_IBRANCH_COMMON_\name\()
-#define IREALMODE_COMMON .L_IREALMODE_COMMON_\name\()
-#define IMASK .L_IMASK_\name\()
-#define IKVM_SKIP .L_IKVM_SKIP_\name\()
-#define IKVM_REAL .L_IKVM_REAL_\name\()
+#define IVEC .L_IVEC_\name\() /* Interrupt vector address */
+#define IHSRR .L_IHSRR_\name\() /* Sets SRR or HSRR registers */
+#define IAREA .L_IAREA_\name\() /* PACA save area */
+#define IVIRT .L_IVIRT_\name\() /* Has virt mode entry point */
+#define IISIDE .L_IISIDE_\name\() /* Uses SRR0/1 not DAR/DSISR */
+#define IDAR .L_IDAR_\name\() /* Uses DAR (or SRR0) */
+#define IDSISR .L_IDSISR_\name\() /* Uses DSISR (or SRR1) */
+#define ISET_RI .L_ISET_RI_\name\() /* Run common code w/ MSR[RI]=1 */
+#define IBRANCH_COMMON .L_IBRANCH_COMMON_\name\() /* ENTRY branch to common */
+#define IREALMODE_COMMON .L_IREALMODE_COMMON_\name\() /* Common runs in realmode */
+#define IMASK .L_IMASK_\name\() /* IRQ soft-mask bit */
+#define IKVM_SKIP .L_IKVM_SKIP_\name\() /* Generate KVM skip handler */
+#define IKVM_REAL .L_IKVM_REAL_\name\() /* Real entry tests KVM */
#define __IKVM_REAL(name) .L_IKVM_REAL_ ## name
-#define IKVM_VIRT .L_IKVM_VIRT_\name\()
-#define ISTACK .L_ISTACK_\name\()
+#define IKVM_VIRT .L_IKVM_VIRT_\name\() /* Virt entry tests KVM */
+#define ISTACK .L_ISTACK_\name\() /* Set regular kernel stack */
#define __ISTACK(name) .L_ISTACK_ ## name
-#define IRECONCILE .L_IRECONCILE_\name\()
-#define IKUAP .L_IKUAP_\name\()
+#define IRECONCILE .L_IRECONCILE_\name\() /* Do RECONCILE_IRQ_STATE */
+#define IKUAP .L_IKUAP_\name\() /* Do KUAP lock */
#define INT_DEFINE_BEGIN(n) \
.macro int_define_ ## n name
@@ -215,6 +215,20 @@ do_define_int n
#ifdef CONFIG_KVM_BOOK3S_64_HANDLER
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
/*
+ * All interrupts which set HSRR registers (EXC_HV) as well as SRESET and
+ * MCE and syscall when invoked with "sc 1" switch to the hypervisor to
+ * be taken, so all generally need to test whether they were taken in guest
+ * context.
+ *
+ * SRESET and MCE may also be sent to the guest by the hypervisor.
+ *
+ * Interrupts which set SRR registers except SRESET and MCE and sc 1 are not
+ * elevated to the hypervisor, though many can be taken when running in
+ * hypervisor mode (e.g., bare metal kernel and userspace). These generally
+ * need to test whether taken in guest context for PR KVM guests. PR KVM
+ * does not enable AIL interrupts, so always takes them in real mode, which
+ * is why these generally only need test the real-mode case.
+ *
* If hv is possible, interrupts come into to the hv version
* of the kvmppc_interrupt code, which then jumps to the PR handler,
* kvmppc_interrupt_pr, if the guest is a PR guest.
@@ -758,6 +772,39 @@ __start_interrupts:
EXC_VIRT_NONE(0x4000, 0x100)
+/**
+ * Interrupt 0x100 - System Reset Interrupt (SRESET aka NMI).
+ * This is a non-maskable, asynchronous interrupt always taken in real-mode.
+ * It is caused by:
+ * - Wake from power-saving state, on powernv.
+ * - An NMI from another CPU, triggered by firmware or hypercall.
+ * - As crash/debug signal injected from BMC, firmware or hypervisor.
+ *
+ * Handling:
+ * Power-save wakeup is the only performance critical path, so this is
+ * determined quickly as possible first. In this case volatile registers
+ * can be discarded and SPRs like CFAR don't need to be read.
+ *
+ * If not a powersave wakeup, then it's run as a regular interrupt, however
+ * it uses its own stack and PACA save area to preserve the regular kernel
+ * environment for debugging.
+ *
+ * This interrupt is not maskable, so triggering it when MSR[RI] is clear,
+ * or SCRATCH0 is in use, etc. may cause a crash. It's also not entirely
+ * correct to switch to virtual mode to run the regular interrupt handler
+ * because it might be interrupted when the MMU is in a bad state (e.g., SLB
+ * is clear).
+ *
+ * FWNMI:
+ * PAPR specifies a "fwnmi" facility which sends the sreset to a different
+ * entry point with a different register set up. Some hypervisors will
+ * send the sreset to 0x100 in the guest if it is not fwnmi capable.
+ *
+ * KVM:
+ * Unlike most EXC_STD interrupts, this may be taken by the host while
+ * executing in a guest, so a KVM test is required. KVM will pull the CPU out
+ * of guest mode and then raise the sreset.
+ */
INT_DEFINE_BEGIN(system_reset)
IVEC=0x100
IHSRR=EXC_STD
@@ -834,6 +881,7 @@ TRAMP_REAL_BEGIN(system_reset_idle_wake)
* Vectors for the FWNMI option. Share common code.
*/
TRAMP_REAL_BEGIN(system_reset_fwnmi)
+ /* XXX: fwnmi guest could run a nested/PR guest, so why no test? */
__IKVM_REAL(system_reset)=0
GEN_INT_ENTRY system_reset, GEN_REAL
@@ -900,6 +948,44 @@ EXC_COMMON_BEGIN(system_reset_common)
GEN_KVM system_reset
+/**
+ * Interrupt 0x200 - Machine Check Interrupt (MCE).
+ * This is a non-maskable interrupt always taken in real-mode. It can be
+ * synchronous or asynchronous, caused by hardware or software, and it may be
+ * taken in a power-saving state.
+ *
+ * Handling:
+ * Similarly to system reset, this uses its own stack and PACA save area,
+ * the difference is re-entrancy is allowed on the machine check stack.
+ *
+ * machine_check_early is run in real mode, and carefully decodes the
+ * machine check and tries to handle it (e.g., flush the SLB if there was an
+ * error detected there), determines if it was recoverable and logs the
+ * event.
+ *
+ * Then, depending on the execution context when the interrupt is taken, there
+ * are 3 main actions:
+ * - Executing in kernel mode. The event is queued with irq_work, which means
+ * it is handled when it is next safe to do so (i.e., the kernel has enabled
+ * interrupts), which could be immediately when the interrupt returns. This
+ * avoids nasty issues like switching to virtual mode when the MMU is in a
+ * bad state, or when executing OPAL code. (SRESET is exposed to such issues,
+ * but it has different priorities). Check to see if the CPU was in power
+ * save, and return via the wake up code if it was.
+ *
+ * - Executing in user mode. machine_check_exception is run like a normal
+ * interrupt handler, which processes the data generated by the early handler.
+ *
+ * - Executing in guest mode. The interrupt is run with its KVM test, and
+ * branches to KVM to deal with. KVM may queue the event for the host
+ * to report later.
+ *
+ * This interrupt is not maskable, so if it triggers when MSR[RI] is clear,
+ * or SCRATCH0 is in use, it may cause a crash.
+ *
+ * KVM:
+ * See SRESET.
+ */
INT_DEFINE_BEGIN(machine_check_early)
IVEC=0x200
IHSRR=EXC_STD
@@ -1162,19 +1248,28 @@ END_FTR_SECTION_IFSET(CPU_FTR_HVMODE)
/**
- * 0x300 - Data Storage Interrupt (DSI)
- * This interrupt is generated due to a data access which does not have a valid
- * page table entry with permissions to allow the data access to be performed.
- * DAWR matches also fault here, as do RC updates, and minor misc errors e.g.,
- * copy/paste, AMO, certain invalid CI accesses, etc.
+ * Interrupt 0x300 - Data Storage Interrupt (DSI).
+ * This is a synchronous interrupt generated due to a data access exception,
+ * e.g., a load orstore which does not have a valid page table entry with
+ * permissions. DAWR matches also fault here, as do RC updates, and minor misc
+ * errors e.g., copy/paste, AMO, certain invalid CI accesses, etc.
+ *
+ * Handling:
+ * - Hash MMU
+ * Go to do_hash_page first to see if the HPT can be filled from an entry in
+ * the Linux page table. Hash faults can hit in kernel mode in a fairly
+ * arbitrary state (e.g., interrupts disabled, locks held) when accessing
+ * "non-bolted" regions, e.g., vmalloc space. However these should always be
+ * backed by Linux page tables.
*
- * This interrupt is delivered to the guest (HV bit unchanged).
+ * If none is found, do a Linux page fault. Linux page faults can happen in
+ * kernel mode due to user copy operations of course.
*
- * Linux HPT responds by first attempting to refill the hash table from the
- * Linux page table, then going to a full page fault if the Linux page table
- * entry was insufficient. RPT goes straight to full page fault.
+ * - Radix MMU
+ * The hardware loads from the Linux page table directly, so a fault goes
+ * immediately to Linux page fault.
*
- * PR KVM ...?
+ * Conditions like DAWR match are handled on the way in to Linux page fault.
*/
INT_DEFINE_BEGIN(data_access)
IVEC=0x300
@@ -1207,6 +1302,24 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
GEN_KVM data_access
+/**
+ * Interrupt 0x380 - Data Segment Interrupt (DSLB).
+ * This is a synchronous interrupt in response to an MMU fault missing SLB
+ * entry for HPT, or an address outside RPT translation range.
+ *
+ * Handling:
+ * - HPT:
+ * This refills the SLB, or reports an access fault similarly to a bad page
+ * fault. When coming from user-mode, the SLB handler may access any kernel
+ * data, though it may itself take a DSLB. When coming from kernel mode,
+ * recursive faults must be avoided so access is restricted to the kernel
+ * image text/data, kernel stack, and any data allocated below
+ * ppc64_bolted_size (first segment). The kernel handler must avoid stomping
+ * on user-handler data structures.
+ *
+ * A dedicated save area EXSLB is used (XXX: but it actually need not be
+ * these days, we could use EXGEN).
+ */
INT_DEFINE_BEGIN(data_access_slb)
IVEC=0x380
IHSRR=EXC_STD
@@ -1250,6 +1363,15 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
GEN_KVM data_access_slb
+/**
+ * Interrupt 0x400 - Instruction Storage Interrupt (ISI).
+ * This is a synchronous interrupt in response to an MMU fault due to an
+ * instruction fetch.
+ *
+ * Handling:
+ * Similar to DSI, though in response to fetch. The faulting address is found
+ * in SRR0 (rather than DAR), and status in SRR1 (rather than DSISR).
+ */
INT_DEFINE_BEGIN(instruction_access)
IVEC=0x400
IHSRR=EXC_STD
@@ -1281,6 +1403,15 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
GEN_KVM instruction_access
+/**
+ * Interrupt 0x480 - Instruction Segment Interrupt (ISLB).
+ * This is a synchronous interrupt in response to an MMU fault due to an
+ * instruction fetch.
+ *
+ * Handling:
+ * Similar to DSLB, though in response to fetch. The faulting address is found
+ * in SRR0 (rather than DAR).
+ */
INT_DEFINE_BEGIN(instruction_access_slb)
IVEC=0x480
IHSRR=EXC_STD
@@ -1324,6 +1455,29 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX)
GEN_KVM instruction_access_slb
+/**
+ * Interrupt 0x500 - External Interrupt.
+ * This is an asynchronous maskable interrupt in response to an "external
+ * exception" from the interrupt controller or hypervisor (e.g., device
+ * interrupt). It is maskable in hardware by clearing MSR[EE], and
+ * soft-maskable with IRQS_DISABLED mask (i.e., local_irq_disable()).
+ *
+ * When running in HV mode, Linux sets up the LPCR[LPES] bit such that
+ * interrupts are delivered with HSRR registers, guests use SRRs, which
+ * reqiures EXC_HV_OR_STD.
+ *
+ * On bare metal POWER9 and later, Linux sets the LPCR[HVICE] bit such that
+ * external interrupts are delivered as Hypervisor Virtualization Interrupts
+ * rather than External Interrupts.
+ *
+ * Handling:
+ * This calls into Linux IRQ handler. NVGPRs are not saved to reduce overhead,
+ * because registers at the time of the interrupt are not so important as it is
+ * asynchronous.
+ *
+ * If soft masked, the masked handler will note the pending interrupt for
+ * replay, and clear MSR[EE] in the interrupted context.
+ */
INT_DEFINE_BEGIN(hardware_interrupt)
IVEC=0x500
IHSRR=EXC_HV_OR_STD
@@ -1350,6 +1504,10 @@ EXC_COMMON_BEGIN(hardware_interrupt_common)
GEN_KVM hardware_interrupt
+/**
+ * Interrupt 0x600 - Alignment Interrupt
+ * This is a synchronous interrupt in response to data alignment fault.
+ */
INT_DEFINE_BEGIN(alignment)
IVEC=0x600
IHSRR=EXC_STD
@@ -1375,6 +1533,15 @@ EXC_COMMON_BEGIN(alignment_common)
GEN_KVM alignment
+/**
+ * Interrupt 0x700 - Program Interrupt (program check).
+ * This is a synchronous interrupt in response to various instruction faults:
+ * traps, privilege errors, TM errors, floating point exceptions.
+ *
+ * Handling:
+ * This interrupt may use the "emergency stack" in some cases when being taken
+ * from kernel context, which complicates handling.
+ */
INT_DEFINE_BEGIN(program_check)
IVEC=0x700
IHSRR=EXC_STD
@@ -1430,6 +1597,15 @@ EXC_COMMON_BEGIN(program_check_common)
GEN_KVM program_check
+/*
+ * Interrupt 0x800 - Floating-Point Unavailable Interrupt.
+ * This is a synchronous interrupt in response to executing an fp instruction
+ * with MSR[FP]=0.
+ *
+ * Handling:
+ * This will load FP registers and enable the FP bit if coming from userspace,
+ * otherwise report a bad kernel use of FP.
+ */
INT_DEFINE_BEGIN(fp_unavailable)
IVEC=0x800
IHSRR=EXC_STD
@@ -1476,6 +1652,23 @@ END_FTR_SECTION_IFSET(CPU_FTR_TM)
GEN_KVM fp_unavailable
+/**
+ * Interrupt 0x900 - Decrementer Interrupt.
+ * This is an asynchronous interrupt in response to a decrementer exception
+ * (e.g., DEC has wrapped below zero). It is maskable in hardware by clearing
+ * MSR[EE], and soft-maskable with IRQS_DISABLED mask (i.e.,
+ * local_irq_disable()).
+ *
+ * Handling:
+ * This calls into Linux timer handler. NVGPRs are not saved (see 0x500).
+ *
+ * If soft masked, the masked handler will note the pending interrupt for
+ * replay, and bump the decrementer to a high value, leaving MSR[EE] enabled
+ * in the interrupted context.
+ * If PPC_WATCHDOG is configured, the soft masked handler will actually set
+ * things back up to run soft_nmi_interrupt as a regular interrupt handler
+ * on the emergency stack.
+ */
INT_DEFINE_BEGIN(decrementer)
IVEC=0x900
IHSRR=EXC_STD
@@ -1501,6 +1694,16 @@ EXC_COMMON_BEGIN(decrementer_common)
GEN_KVM decrementer
+/**
+ * Interrupt 0x980 - Hypervisor Decrementer Interrupt.
+ * This is an asynchronous interrupt, similar to 0x900 but for the HDEC
+ * register.
+ *
+ * Handling:
+ * Linux does not use this outside KVM where it's used to keep a host timer
+ * while the guest is given control of DEC. It should normally be caught by
+ * the KVM test and routed there.
+ */
INT_DEFINE_BEGIN(hdecrementer)
IVEC=0x980
IHSRR=EXC_HV
@@ -1540,6 +1743,20 @@ EXC_COMMON_BEGIN(hdecrementer_common)
GEN_KVM hdecrementer
+/**
+ * Interrupt 0xa00 - Directed Privileged Doorbell Interrupt.
+ * This is an asynchronous interrupt in response to a msgsndp doorbell.
+ * It is maskable in hardware by clearing MSR[EE], and soft-maskable with
+ * IRQS_DISABLED mask (i.e., local_irq_disable()).
+ *
+ * Handling:
+ * Guests may use this for IPIs between threads in a core if the
+ * hypervisor supports it. NVGPRS are not saved (see 0x500).
+ *
+ * If soft masked, the masked handler will note the pending interrupt for
+ * replay, leaving MSR[EE] enabled in the interrupted context because the
+ * doorbells are edge triggered.
+ */
INT_DEFINE_BEGIN(doorbell_super)
IVEC=0xa00
IHSRR=EXC_STD
@@ -1572,18 +1789,23 @@ EXC_COMMON_BEGIN(doorbell_super_common)
EXC_REAL_NONE(0xb00, 0x100)
EXC_VIRT_NONE(0x4b00, 0x100)
-/*
- * system call / hypercall (0xc00, 0x4c00)
- *
- * The system call exception is invoked with "sc 0" and does not alter HV bit.
- * There is support for kernel code to invoke system calls but there are no
- * in-tree users.
- *
- * The hypercall is invoked with "sc 1" and sets HV=1.
+/**
+ * Interrupt 0xc00 - System Call Interrupt (syscall, hcall).
+ * This is a synchronous interrupt invoked with the "sc" instruction. The
+ * sc 0 does not alter the HV mode so it goes to the currently running OS
+ * and is used for system calls. sc 1 elevates to hypervisor and is used for
+ * hypercalls from guest.
*
* In HPT, sc 1 always goes to 0xc00 real mode. In RADIX, sc 1 can go to
* 0x4c00 virtual mode.
*
+ * Handling:
+ * If the KVM test fires then it was due to a hypercall and is accordingly
+ * routed to KVM. Otherwise this executes a normal Linux system call.
+ *
+ * There is support for kernel code to invoke system calls but there are no
+ * in-tree users.
+ *
* Call convention:
*
* syscall register convention is in Documentation/powerpc/syscall64-abi.rst
@@ -1729,6 +1951,11 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
#endif
+/**
+ * Interrupt 0xd00 - Trace Interrupt.
+ * This is a synchronous interrupt in response to instruction step or
+ * breakpoint faults.
+ */
INT_DEFINE_BEGIN(single_step)
IVEC=0xd00
IHSRR=EXC_STD
@@ -1752,6 +1979,18 @@ EXC_COMMON_BEGIN(single_step_common)
GEN_KVM single_step
+/**
+ * Interrupt 0xe00 - Hypervisor Data Storage Interrupt (HDSI).
+ * This is a synchronous interrupt in response to an MMU fault caused by a
+ * guest data access.
+ *
+ * Handling:
+ * This should always get routed to KVM. In radix MMU mode, this is caused
+ * by a guest nested radix access that can't be performed due to the
+ * partition scope page table. In hash mode, this can be caused by guests
+ * running with translation disabled (virtual real mode) or with VPM enabled.
+ * KVM will update the page table structures or disallow the access.
+ */
INT_DEFINE_BEGIN(h_data_storage)
IVEC=0xe00
IHSRR=EXC_HV
@@ -1785,6 +2024,11 @@ ALT_MMU_FTR_SECTION_END_IFSET(MMU_FTR_TYPE_RADIX)
GEN_KVM h_data_storage
+/**
+ * Interrupt 0xe20 - Hypervisor Instruction Storage Interrupt (HISI).
+ * This is a synchronous interrupt in response to an MMU fault caused by a
+ * guest instruction fetch, similar to HDSI.
+ */
INT_DEFINE_BEGIN(h_instr_storage)
IVEC=0xe20
IHSRR=EXC_HV
@@ -1809,6 +2053,9 @@ EXC_COMMON_BEGIN(h_instr_storage_common)
GEN_KVM h_instr_storage
+/**
+ * Interrupt 0xe40 - Hypervisor Emulation Assistance Interrupt.
+ */
INT_DEFINE_BEGIN(emulation_assist)
IVEC=0xe40
IHSRR=EXC_HV
@@ -1833,10 +2080,29 @@ EXC_COMMON_BEGIN(emulation_assist_common)
GEN_KVM emulation_assist
-/*
- * hmi_exception trampoline is a special case. It jumps to hmi_exception_early
- * first, and then eventaully from there to the trampoline to get into virtual
- * mode.
+/**
+ * Interrupt 0xe60 - Hypervisor Maintenance Interrupt (HMI).
+ * This is an asynchronous interrupt caused by a Hypervisor Maintenance
+ * Exception. It is always taken in real mode but uses HSRR registers
+ * unlike SRESET and MCE.
+ *
+ * It is maskable in hardware by clearing MSR[EE], and partially soft-maskable
+ * with IRQS_DISABLED mask (i.e., local_irq_disable()).
+ *
+ * Handling:
+ * This is a special case, this is handled similarly to machine checks, with an
+ * initial real mode handler that is not soft-masked, which attempts to fix the
+ * problem. Then a regular handler which is soft-maskable and reports the
+ * problem.
+ *
+ * The emergency stack is used for the early real mode handler.
+ *
+ * XXX: unclear why MCE and HMI schemes could not be made common, e.g.,
+ * either use soft-masking for the MCE, or use irq_work for the HMI.
+ *
+ * KVM:
+ * Unlike MCE, this calls into KVM without calling the real mode handler
+ * first.
*/
INT_DEFINE_BEGIN(hmi_exception_early)
IVEC=0xe60
@@ -1901,6 +2167,11 @@ EXC_COMMON_BEGIN(hmi_exception_common)
GEN_KVM hmi_exception
+/**
+ * Interrupt 0xe80 - Directed Hypervisor Doorbell Interrupt.
+ * This is an asynchronous interrupt in response to a msgsnd doorbell.
+ * Similar to the 0xa00 doorbell but for host rather than guest.
+ */
INT_DEFINE_BEGIN(h_doorbell)
IVEC=0xe80
IHSRR=EXC_HV
@@ -1931,6 +2202,11 @@ EXC_COMMON_BEGIN(h_doorbell_common)
GEN_KVM h_doorbell
+/**
+ * Interrupt 0xea0 - Hypervisor Virtualization Interrupt.
+ * This is an asynchronous interrupt in response to an "external exception".
+ * Similar to 0x500 but for host only.
+ */
INT_DEFINE_BEGIN(h_virt_irq)
IVEC=0xea0
IHSRR=EXC_HV
@@ -1963,6 +2239,22 @@ EXC_REAL_NONE(0xee0, 0x20)
EXC_VIRT_NONE(0x4ee0, 0x20)
+/*
+ * Interrupt 0xf00 - Performance Monitor Interrupt (PMI, PMU).
+ * This is an asynchronous interrupt in response to a PMU exception.
+ * It is maskable in hardware by clearing MSR[EE], and soft-maskable with
+ * IRQS_PMI_DISABLED mask (NOTE: NOT local_irq_disable()).
+ *
+ * Handling:
+ * This calls into the perf subsystem.
+ *
+ * Like the watchdog soft-nmi, it appears an NMI interrupt to Linux, in that it
+ * runs under local_irq_disable. However it may be soft-masked in
+ * powerpc-specific code.
+ *
+ * If soft masked, the masked handler will note the pending interrupt for
+ * replay, and clear MSR[EE] in the interrupted context.
+ */
INT_DEFINE_BEGIN(performance_monitor)
IVEC=0xf00
IHSRR=EXC_STD
@@ -1988,6 +2280,12 @@ EXC_COMMON_BEGIN(performance_monitor_common)
GEN_KVM performance_monitor
+/**
+ * Interrupt 0xf20 - Vector Unavailable Interrupt.
+ * This is a synchronous interrupt in response to
+ * executing a vector (or altivec) instruction with MSR[VEC]=0.
+ * Similar to FP unavailable.
+ */
INT_DEFINE_BEGIN(altivec_unavailable)
IVEC=0xf20
IHSRR=EXC_STD
@@ -2038,6 +2336,12 @@ END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC)
GEN_KVM altivec_unavailable
+/**
+ * Interrupt 0xf40 - VSX Unavailable Interrupt.
+ * This is a synchronous interrupt in response to
+ * executing a VSX instruction with MSR[VSX]=0.
+ * Similar to FP unavailable.
+ */
INT_DEFINE_BEGIN(vsx_unavailable)
IVEC=0xf40
IHSRR=EXC_STD
@@ -2087,6 +2391,13 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX)
GEN_KVM vsx_unavailable
+/**
+ * Interrupt 0xf60 - Facility Unavailable Interrupt.
+ * This is a synchronous interrupt in response to
+ * executing an instruction without access to the facility that can be
+ * resolved by the OS (e.g., FSCR, MSR).
+ * Similar to FP unavailable.
+ */
INT_DEFINE_BEGIN(facility_unavailable)
IVEC=0xf60
IHSRR=EXC_STD
@@ -2110,6 +2421,13 @@ EXC_COMMON_BEGIN(facility_unavailable_common)
GEN_KVM facility_unavailable
+/**
+ * Interrupt 0xf60 - Hypervisor Facility Unavailable Interrupt.
+ * This is a synchronous interrupt in response to
+ * executing an instruction without access to the facility that can only
+ * be resolved in HV mode (e.g., HFSCR).
+ * Similar to FP unavailable.
+ */
INT_DEFINE_BEGIN(h_facility_unavailable)
IVEC=0xf80
IHSRR=EXC_HV
@@ -2201,6 +2519,18 @@ EXC_COMMON_BEGIN(instruction_breakpoint_common)
EXC_REAL_NONE(0x1400, 0x100)
EXC_VIRT_NONE(0x5400, 0x100)
+/**
+ * Interrupt 0x1500 - Soft Patch Interrupt
+ *
+ * Handling:
+ * This is an implementation specific interrupt which can be used for a
+ * range of exceptions.
+ *
+ * This interrupt handler is unique in that it runs the denormal assist
+ * code even for guests (and even in guest context) without going to KVM,
+ * for speed. POWER9 does not raise denorm exceptions, so this special case
+ * could be phased out in future to reduce special cases.
+ */
INT_DEFINE_BEGIN(denorm_exception)
IVEC=0x1500
IHSRR=EXC_HV
--
2.22.0
^ permalink raw reply related
* Re: [RFC PATCH] bpf: handle 32-bit zext during constant blinding
From: Jiong Wang @ 2019-08-21 9:05 UTC (permalink / raw)
To: Naveen N. Rao
Cc: Daniel Borkmann, Jiong Wang, linux-kernel, netdev, bpf,
linuxppc-dev, Alexei Starovoitov
In-Reply-To: <1566376025.68ldwx3wc7.naveen@linux.ibm.com>
Naveen N. Rao writes:
> Naveen N. Rao wrote:
>> Since BPF constant blinding is performed after the verifier pass, there
>> are certain ALU32 instructions inserted which don't have a corresponding
>> zext instruction inserted after. This is causing a kernel oops on
>> powerpc and can be reproduced by running 'test_cgroup_storage' with
>> bpf_jit_harden=2.
>>
>> Fix this by emitting BPF_ZEXT during constant blinding if
>> prog->aux->verifier_zext is set.
>>
>> Fixes: a4b1d3c1ddf6cb ("bpf: verifier: insert zero extension according to analysis result")
>> Reported-by: Michael Ellerman <mpe@ellerman.id.au>
>> Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
>> ---
>> This approach (the location where zext is being introduced below, in
>> particular) works for powerpc, but I am not entirely sure if this is
>> sufficient for other architectures as well. This is broken on v5.3-rc4.
>
> Alexie, Daniel, Jiong,
> Any feedback on this?
The fix on BPF_LD | BPF_IMM | BPF_DW looks correct to me, but the two other
places looks to me is unnecessary, as those destinations are exposed to
external and if they are used as 64-bit then there will be zext inserted
for them.
Have you verified removing those two fixes will still cause the bug?
Regards,
Jiong
>
> - Naveen
^ permalink raw reply
* Re: Regression fix for bpf in v5.3 (was Re: [RFC PATCH] bpf: handle 32-bit zext during constant blinding)
From: Jiong Wang @ 2019-08-21 10:55 UTC (permalink / raw)
To: Michael Ellerman
Cc: Daniel Borkmann, Jiong Wang, netdev, linux-kernel,
Alexei Starovoitov, Naveen N. Rao, linuxppc-dev, bpf
In-Reply-To: <87d0gy6cj6.fsf@concordia.ellerman.id.au>
Michael Ellerman writes:
> "Naveen N. Rao" <naveen.n.rao@linux.vnet.ibm.com> writes:
>> Since BPF constant blinding is performed after the verifier pass, there
>> are certain ALU32 instructions inserted which don't have a corresponding
>> zext instruction inserted after. This is causing a kernel oops on
>> powerpc and can be reproduced by running 'test_cgroup_storage' with
>> bpf_jit_harden=2.
>>
>> Fix this by emitting BPF_ZEXT during constant blinding if
>> prog->aux->verifier_zext is set.
>>
>> Fixes: a4b1d3c1ddf6cb ("bpf: verifier: insert zero extension according to analysis result")
>> Reported-by: Michael Ellerman <mpe@ellerman.id.au>
>> Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
>> ---
>> This approach (the location where zext is being introduced below, in
>> particular) works for powerpc, but I am not entirely sure if this is
>> sufficient for other architectures as well. This is broken on v5.3-rc4.
>
> Any comment on this?
Have commented on https://marc.info/?l=linux-netdev&m=156637836024743&w=2
The fix looks correct to me on "BPF_LD | BPF_IMM | BPF_DW", but looks
unnecessary on two other places. It would be great if you or Naveen could
confirm it.
Thanks.
Regards,
Jiong
> This is a regression in v5.3, which results in a kernel crash, it would
> be nice to get it fixed before the release please?
>
> cheers
>
>> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
>> index 8191a7db2777..d84146e6fd9e 100644
>> --- a/kernel/bpf/core.c
>> +++ b/kernel/bpf/core.c
>> @@ -890,7 +890,8 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog,
>>
>> static int bpf_jit_blind_insn(const struct bpf_insn *from,
>> const struct bpf_insn *aux,
>> - struct bpf_insn *to_buff)
>> + struct bpf_insn *to_buff,
>> + bool emit_zext)
>> {
>> struct bpf_insn *to = to_buff;
>> u32 imm_rnd = get_random_int();
>> @@ -939,6 +940,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
>> *to++ = BPF_ALU32_IMM(BPF_MOV, BPF_REG_AX, imm_rnd ^ from->imm);
>> *to++ = BPF_ALU32_IMM(BPF_XOR, BPF_REG_AX, imm_rnd);
>> *to++ = BPF_ALU32_REG(from->code, from->dst_reg, BPF_REG_AX);
>> + if (emit_zext)
>> + *to++ = BPF_ZEXT_REG(from->dst_reg);
>> break;
>>
>> case BPF_ALU64 | BPF_ADD | BPF_K:
>> @@ -992,6 +995,10 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
>> off -= 2;
>> *to++ = BPF_ALU32_IMM(BPF_MOV, BPF_REG_AX, imm_rnd ^ from->imm);
>> *to++ = BPF_ALU32_IMM(BPF_XOR, BPF_REG_AX, imm_rnd);
>> + if (emit_zext) {
>> + *to++ = BPF_ZEXT_REG(BPF_REG_AX);
>> + off--;
>> + }
>> *to++ = BPF_JMP32_REG(from->code, from->dst_reg, BPF_REG_AX,
>> off);
>> break;
>> @@ -1005,6 +1012,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
>> case 0: /* Part 2 of BPF_LD | BPF_IMM | BPF_DW. */
>> *to++ = BPF_ALU32_IMM(BPF_MOV, BPF_REG_AX, imm_rnd ^ aux[0].imm);
>> *to++ = BPF_ALU32_IMM(BPF_XOR, BPF_REG_AX, imm_rnd);
>> + if (emit_zext)
>> + *to++ = BPF_ZEXT_REG(BPF_REG_AX);
>> *to++ = BPF_ALU64_REG(BPF_OR, aux[0].dst_reg, BPF_REG_AX);
>> break;
>>
>> @@ -1088,7 +1097,8 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog)
>> insn[1].code == 0)
>> memcpy(aux, insn, sizeof(aux));
>>
>> - rewritten = bpf_jit_blind_insn(insn, aux, insn_buff);
>> + rewritten = bpf_jit_blind_insn(insn, aux, insn_buff,
>> + clone->aux->verifier_zext);
>> if (!rewritten)
>> continue;
>>
>> --
>> 2.22.0
^ permalink raw reply
* Re: [PATCH] powerpc: Add barrier_nospec to raw_copy_in_user()
From: Michal Suchánek @ 2019-08-21 13:36 UTC (permalink / raw)
To: Suraj Jitindar Singh; +Cc: linuxppc-dev, Al Viro
In-Reply-To: <20190306011038.16449-1-sjitindarsingh@gmail.com>
On Wed, 6 Mar 2019 12:10:38 +1100
Suraj Jitindar Singh <sjitindarsingh@gmail.com> wrote:
> Commit ddf35cf3764b ("powerpc: Use barrier_nospec in copy_from_user()")
> Added barrier_nospec before loading from user-controller pointers.
> The intention was to order the load from the potentially user-controlled
> pointer vs a previous branch based on an access_ok() check or similar.
>
> In order to achieve the same result, add a barrier_nospec to the
> raw_copy_in_user() function before loading from such a user-controlled
> pointer.
>
> Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
> ---
> arch/powerpc/include/asm/uaccess.h | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/arch/powerpc/include/asm/uaccess.h b/arch/powerpc/include/asm/uaccess.h
> index e3a731793ea2..bb615592d5bb 100644
> --- a/arch/powerpc/include/asm/uaccess.h
> +++ b/arch/powerpc/include/asm/uaccess.h
> @@ -306,6 +306,7 @@ extern unsigned long __copy_tofrom_user(void __user *to,
> static inline unsigned long
> raw_copy_in_user(void __user *to, const void __user *from, unsigned long n)
> {
> + barrier_nospec();
> return __copy_tofrom_user(to, from, n);
> }
> #endif /* __powerpc64__ */
Hello,
AFAICT this is not needed. The data cannot be used in the kernel without
subsequent copy_from_user which already has the nospec barrier.
Thanks
Michal
^ permalink raw reply
* [PATCH v2] btrfs: fix allocation of bitmap pages.
From: Christophe Leroy @ 2019-08-21 15:05 UTC (permalink / raw)
To: erhard_f, Chris Mason, Josef Bacik, David Sterba, Andrew Morton
Cc: linux-mm, linux-btrfs, linuxppc-dev, linux-kernel, stable
Various notifications of type "BUG kmalloc-4096 () : Redzone
overwritten" have been observed recently in various parts of
the kernel. After some time, it has been made a relation with
the use of BTRFS filesystem.
[ 22.809700] BUG kmalloc-4096 (Tainted: G W ): Redzone overwritten
[ 22.809971] -----------------------------------------------------------------------------
[ 22.810286] INFO: 0xbe1a5921-0xfbfc06cd. First byte 0x0 instead of 0xcc
[ 22.810866] INFO: Allocated in __load_free_space_cache+0x588/0x780 [btrfs] age=22 cpu=0 pid=224
[ 22.811193] __slab_alloc.constprop.26+0x44/0x70
[ 22.811345] kmem_cache_alloc_trace+0xf0/0x2ec
[ 22.811588] __load_free_space_cache+0x588/0x780 [btrfs]
[ 22.811848] load_free_space_cache+0xf4/0x1b0 [btrfs]
[ 22.812090] cache_block_group+0x1d0/0x3d0 [btrfs]
[ 22.812321] find_free_extent+0x680/0x12a4 [btrfs]
[ 22.812549] btrfs_reserve_extent+0xec/0x220 [btrfs]
[ 22.812785] btrfs_alloc_tree_block+0x178/0x5f4 [btrfs]
[ 22.813032] __btrfs_cow_block+0x150/0x5d4 [btrfs]
[ 22.813262] btrfs_cow_block+0x194/0x298 [btrfs]
[ 22.813484] commit_cowonly_roots+0x44/0x294 [btrfs]
[ 22.813718] btrfs_commit_transaction+0x63c/0xc0c [btrfs]
[ 22.813973] close_ctree+0xf8/0x2a4 [btrfs]
[ 22.814107] generic_shutdown_super+0x80/0x110
[ 22.814250] kill_anon_super+0x18/0x30
[ 22.814437] btrfs_kill_super+0x18/0x90 [btrfs]
[ 22.814590] INFO: Freed in proc_cgroup_show+0xc0/0x248 age=41 cpu=0 pid=83
[ 22.814841] proc_cgroup_show+0xc0/0x248
[ 22.814967] proc_single_show+0x54/0x98
[ 22.815086] seq_read+0x278/0x45c
[ 22.815190] __vfs_read+0x28/0x17c
[ 22.815289] vfs_read+0xa8/0x14c
[ 22.815381] ksys_read+0x50/0x94
[ 22.815475] ret_from_syscall+0x0/0x38
Commit 69d2480456d1 ("btrfs: use copy_page for copying pages instead
of memcpy") changed the way bitmap blocks are copied. But allthough
bitmaps have the size of a page, they were allocated with kzalloc().
Most of the time, kzalloc() allocates aligned blocks of memory, so
copy_page() can be used. But when some debug options like SLAB_DEBUG
are activated, kzalloc() may return unaligned pointer.
On powerpc, memcpy(), copy_page() and other copying functions use
'dcbz' instruction which provides an entire zeroed cacheline to avoid
memory read when the intention is to overwrite a full line. Functions
like memcpy() are writen to care about partial cachelines at the start
and end of the destination, but copy_page() assumes it gets pages. As
pages are naturally cache aligned, copy_page() doesn't care about
partial lines. This means that when copy_page() is called with a
misaligned pointer, a few leading bytes are zeroed.
To fix it, allocate bitmaps through kmem_cache instead of using kzalloc()
The cache pool is created with PAGE_SIZE alignment constraint.
Reported-by: Erhard F. <erhard_f@mailbox.org>
Link: https://bugzilla.kernel.org/show_bug.cgi?id=204371
Fixes: 69d2480456d1 ("btrfs: use copy_page for copying pages instead of memcpy")
Cc: stable@vger.kernel.org
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
v2: Using kmem_cache instead of get_zeroed_page() in order to benefit from SLAB debugging features like redzone.
---
fs/btrfs/ctree.h | 1 +
fs/btrfs/free-space-cache.c | 17 ++++++++++-------
fs/btrfs/inode.c | 7 +++++++
3 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 299e11e6c554..26abb95becc9 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -43,6 +43,7 @@ extern struct kmem_cache *btrfs_trans_handle_cachep;
extern struct kmem_cache *btrfs_bit_radix_cachep;
extern struct kmem_cache *btrfs_path_cachep;
extern struct kmem_cache *btrfs_free_space_cachep;
+extern struct kmem_cache *btrfs_bitmap_cachep;
struct btrfs_ordered_sum;
struct btrfs_ref;
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 062be9dde4c6..9a708e7920a0 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -764,7 +764,8 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
} else {
ASSERT(num_bitmaps);
num_bitmaps--;
- e->bitmap = kzalloc(PAGE_SIZE, GFP_NOFS);
+ e->bitmap = kmem_cache_zalloc(btrfs_bitmap_cachep,
+ GFP_NOFS);
if (!e->bitmap) {
kmem_cache_free(
btrfs_free_space_cachep, e);
@@ -1881,7 +1882,7 @@ static void free_bitmap(struct btrfs_free_space_ctl *ctl,
struct btrfs_free_space *bitmap_info)
{
unlink_free_space(ctl, bitmap_info);
- kfree(bitmap_info->bitmap);
+ kmem_cache_free(btrfs_bitmap_cachep, bitmap_info->bitmap);
kmem_cache_free(btrfs_free_space_cachep, bitmap_info);
ctl->total_bitmaps--;
ctl->op->recalc_thresholds(ctl);
@@ -2135,7 +2136,7 @@ static int insert_into_bitmap(struct btrfs_free_space_ctl *ctl,
}
/* allocate the bitmap */
- info->bitmap = kzalloc(PAGE_SIZE, GFP_NOFS);
+ info->bitmap = kmem_cache_zalloc(btrfs_bitmap_cachep, GFP_NOFS);
spin_lock(&ctl->tree_lock);
if (!info->bitmap) {
ret = -ENOMEM;
@@ -2146,7 +2147,8 @@ static int insert_into_bitmap(struct btrfs_free_space_ctl *ctl,
out:
if (info) {
- kfree(info->bitmap);
+ if (info->bitmap)
+ kmem_cache_free(btrfs_bitmap_cachep, info->bitmap);
kmem_cache_free(btrfs_free_space_cachep, info);
}
@@ -2802,7 +2804,7 @@ u64 btrfs_alloc_from_cluster(struct btrfs_block_group_cache *block_group,
if (entry->bytes == 0) {
ctl->free_extents--;
if (entry->bitmap) {
- kfree(entry->bitmap);
+ kmem_cache_free(btrfs_bitmap_cachep, entry->bitmap);
ctl->total_bitmaps--;
ctl->op->recalc_thresholds(ctl);
}
@@ -3606,7 +3608,7 @@ int test_add_free_space_entry(struct btrfs_block_group_cache *cache,
}
if (!map) {
- map = kzalloc(PAGE_SIZE, GFP_NOFS);
+ map = kmem_cache_zalloc(btrfs_bitmap_cachep, GFP_NOFS);
if (!map) {
kmem_cache_free(btrfs_free_space_cachep, info);
return -ENOMEM;
@@ -3635,7 +3637,8 @@ int test_add_free_space_entry(struct btrfs_block_group_cache *cache,
if (info)
kmem_cache_free(btrfs_free_space_cachep, info);
- kfree(map);
+ if (map)
+ kmem_cache_free(btrfs_bitmap_cachep, map);
return 0;
}
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index ee582a36653d..da470af9d328 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -74,6 +74,7 @@ static struct kmem_cache *btrfs_inode_cachep;
struct kmem_cache *btrfs_trans_handle_cachep;
struct kmem_cache *btrfs_path_cachep;
struct kmem_cache *btrfs_free_space_cachep;
+struct kmem_cache *btrfs_bitmap_cachep;
static int btrfs_setsize(struct inode *inode, struct iattr *attr);
static int btrfs_truncate(struct inode *inode, bool skip_writeback);
@@ -9380,6 +9381,7 @@ void __cold btrfs_destroy_cachep(void)
kmem_cache_destroy(btrfs_trans_handle_cachep);
kmem_cache_destroy(btrfs_path_cachep);
kmem_cache_destroy(btrfs_free_space_cachep);
+ kmem_cache_destroy(btrfs_bitmap_cachep);
}
int __init btrfs_init_cachep(void)
@@ -9409,6 +9411,11 @@ int __init btrfs_init_cachep(void)
if (!btrfs_free_space_cachep)
goto fail;
+ btrfs_bitmap_cachep = kmem_cache_create("btrfs_bitmap", PAGE_SIZE,
+ PAGE_SIZE, SLAB_RED_ZONE, NULL);
+ if (!btrfs_bitmap_cachep)
+ goto fail;
+
return 0;
fail:
btrfs_destroy_cachep();
--
2.13.3
^ permalink raw reply related
* [PATCH v2 0/4] powerpc: expose secure variables to the kernel and userspace
From: Nayna Jain @ 2019-08-21 15:08 UTC (permalink / raw)
To: linuxppc-dev, linux-efi, linux-integrity
Cc: Ard Biesheuvel, Eric Ricther, Nayna Jain, linux-kernel,
Mimi Zohar, Claudio Carvalho, Matthew Garret, Greg Kroah-Hartman,
Paul Mackerras, Jeremy Kerr, Elaine Palmer, Oliver O'Halloran,
George Wilson
In order to verify the OS kernel on PowerNV systems, secure boot requires
X.509 certificates trusted by the platform. These are stored in secure
variables controlled by OPAL, called OPAL secure variables. In order to
enable users to manage the keys, the secure variables need to be exposed
to userspace.
OPAL provides the runtime services for the kernel to be able to access the
secure variables[1]. This patchset defines the kernel interface for the
OPAL APIs. These APIs are used by the hooks, which load these variables
to the keyring and expose them to the userspace for reading/writing.
The previous version[2] of the patchset added support only for the sysfs
interface. This patch adds two more patches that involves loading of
the firmware trusted keys to the kernel keyring. This patchset is
dependent on the base CONFIG PPC_SECURE_BOOT added by ima arch specific
patches for POWER[3]
Overall, this patchset adds the following support:
* expose secure variables to the kernel via OPAL Runtime API interface
* expose secure variables to the userspace via kernel sysfs interface
* load kernel verification and revocation keys to .platform and
.blacklist keyring respectively.
The secure variables can be read/written using simple linux utilities
cat/hexdump.
For example:
Path to the secure variables is:
/sys/firmware/secvar/vars
Each secure variable is listed as directory.
$ ls -l
total 0
drwxr-xr-x. 2 root root 0 Aug 20 21:20 db
drwxr-xr-x. 2 root root 0 Aug 20 21:20 KEK
drwxr-xr-x. 2 root root 0 Aug 20 21:20 PK
The attributes of each of the secure variables are(for example: PK):
[PK]$ ls -l
total 0
-r--r--r--. 1 root root 32000 Aug 21 08:28 data
-r--r--r--. 1 root root 65536 Aug 21 08:28 name
-r--r--r--. 1 root root 65536 Aug 21 08:28 size
--w-------. 1 root root 32000 Aug 21 08:28 update
The "data" is used to read the existing variable value using hexdump. The
data is stored in ESL format.
The "update" is used to write a new value using cat. The update is
to be submitted as AUTH file.
[1] Depends on skiboot OPAL API changes which removes metadata from
the API. The new version with the changes are going to be posted soon.
[2] https://lkml.org/lkml/2019/6/13/1644
[3] https://lkml.org/lkml/2019/8/19/402
Changelog:
v2:
* removes complete efi-sms from the sysfs implementation and is simplified
* includes Greg's and Oliver's feedbacks:
* adds sysfs documentation
* moves sysfs code to arch/powerpc
* other code related feedbacks.
* adds two new patches to load keys to .platform and .blacklist keyring.
These patches are added to this series as they are also dependent on
OPAL APIs.
Nayna Jain (4):
powerpc/powernv: Add OPAL API interface to access secure variable
powerpc: expose secure variables to userspace via sysfs
x86/efi: move common keyring handler functions to new file
powerpc: load firmware trusted keys into kernel keyring
Documentation/ABI/testing/sysfs-secvar | 27 +++
arch/powerpc/Kconfig | 9 +
arch/powerpc/include/asm/opal-api.h | 5 +-
arch/powerpc/include/asm/opal.h | 6 +
arch/powerpc/include/asm/secvar.h | 55 +++++
arch/powerpc/kernel/Makefile | 3 +-
arch/powerpc/kernel/secvar-ops.c | 25 +++
arch/powerpc/kernel/secvar-sysfs.c | 210 ++++++++++++++++++
arch/powerpc/platforms/powernv/Kconfig | 6 +
arch/powerpc/platforms/powernv/Makefile | 1 +
arch/powerpc/platforms/powernv/opal-call.c | 3 +
arch/powerpc/platforms/powernv/opal-secvar.c | 102 +++++++++
arch/powerpc/platforms/powernv/opal.c | 5 +
security/integrity/Kconfig | 9 +
security/integrity/Makefile | 6 +-
.../platform_certs/keyring_handler.c | 80 +++++++
.../platform_certs/keyring_handler.h | 35 +++
.../integrity/platform_certs/load_powerpc.c | 94 ++++++++
security/integrity/platform_certs/load_uefi.c | 67 +-----
19 files changed, 679 insertions(+), 69 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-secvar
create mode 100644 arch/powerpc/include/asm/secvar.h
create mode 100644 arch/powerpc/kernel/secvar-ops.c
create mode 100644 arch/powerpc/kernel/secvar-sysfs.c
create mode 100644 arch/powerpc/platforms/powernv/opal-secvar.c
create mode 100644 security/integrity/platform_certs/keyring_handler.c
create mode 100644 security/integrity/platform_certs/keyring_handler.h
create mode 100644 security/integrity/platform_certs/load_powerpc.c
--
2.20.1
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox