* Re: [PATCH v4 44/46] KVM: PPC: Book3S HV P9: implement hash guest support
From: Nicholas Piggin @ 2021-03-24 1:34 UTC (permalink / raw)
To: Fabiano Rosas, kvm-ppc; +Cc: linuxppc-dev
In-Reply-To: <87tup1kgtb.fsf@linux.ibm.com>
Excerpts from Fabiano Rosas's message of March 24, 2021 1:53 am:
> Nicholas Piggin <npiggin@gmail.com> writes:
>
>> Guest entry/exit has to restore and save/clear the SLB, plus several
>> other bits to accommodate hash guests in the P9 path.
>>
>> Radix host, hash guest support is removed from the P7/8 path.
>>
>> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>> ---
>
> <snip>
>
>> diff --git a/arch/powerpc/kvm/book3s_hv_interrupt.c b/arch/powerpc/kvm/book3s_hv_interrupt.c
>> index cd84d2c37632..03fbfef708a8 100644
>> --- a/arch/powerpc/kvm/book3s_hv_interrupt.c
>> +++ b/arch/powerpc/kvm/book3s_hv_interrupt.c
>> @@ -55,6 +55,50 @@ static void __accumulate_time(struct kvm_vcpu *vcpu, struct kvmhv_tb_accumulator
>> #define accumulate_time(vcpu, next) do {} while (0)
>> #endif
>>
>> +static inline void mfslb(unsigned int idx, u64 *slbee, u64 *slbev)
>> +{
>> + asm volatile("slbmfev %0,%1" : "=r" (*slbev) : "r" (idx));
>> + asm volatile("slbmfee %0,%1" : "=r" (*slbee) : "r" (idx));
>> +}
>> +
>> +static inline void __mtslb(u64 slbee, u64 slbev)
>> +{
>> + asm volatile("slbmte %0,%1" :: "r" (slbev), "r" (slbee));
>> +}
>> +
>> +static inline void mtslb(unsigned int idx, u64 slbee, u64 slbev)
>> +{
>> + BUG_ON((slbee & 0xfff) != idx);
>> +
>> + __mtslb(slbee, slbev);
>> +}
>> +
>> +static inline void slb_invalidate(unsigned int ih)
>> +{
>> + asm volatile("slbia %0" :: "i"(ih));
>> +}
>
> Fyi, in my environment the assembler complains:
>
> {standard input}: Assembler messages:
> {standard input}:1293: Error: junk at end of line: `6'
> {standard input}:2138: Error: junk at end of line: `6'
> make[3]: *** [../scripts/Makefile.build:271:
> arch/powerpc/kvm/book3s_hv_interrupt.o] Error 1
>
> This works:
>
> - asm volatile("slbia %0" :: "i"(ih));
> + asm volatile(PPC_SLBIA(%0) :: "i"(ih));
>
> But I don't know what is going on.
Ah yes, we still need to use PPC_SLBIA. IH parameter to slbia was only
added in binutils 2.27 and we support down to 2.23.
Thanks for the fix I'll add it.
Thanks,
Nick
^ permalink raw reply
* Re: [PATCH v4 22/46] KVM: PPC: Book3S HV P9: Stop handling hcalls in real-mode in the P9 path
From: Nicholas Piggin @ 2021-03-24 1:27 UTC (permalink / raw)
To: Fabiano Rosas, kvm-ppc; +Cc: linuxppc-dev
In-Reply-To: <87y2ed5vi0.fsf@linux.ibm.com>
Excerpts from Fabiano Rosas's message of March 24, 2021 8:57 am:
> Nicholas Piggin <npiggin@gmail.com> writes:
>
>> In the interest of minimising the amount of code that is run in
>> "real-mode", don't handle hcalls in real mode in the P9 path.
>>
>> POWER8 and earlier are much more expensive to exit from HV real mode
>> and switch to host mode, because on those processors HV interrupts get
>> to the hypervisor with the MMU off, and the other threads in the core
>> need to be pulled out of the guest, and SLBs all need to be saved,
>> ERATs invalidated, and host SLB reloaded before the MMU is re-enabled
>> in host mode. Hash guests also require a lot of hcalls to run. The
>> XICS interrupt controller requires hcalls to run.
>>
>> By contrast, POWER9 has independent thread switching, and in radix mode
>> the hypervisor is already in a host virtual memory mode when the HV
>> interrupt is taken. Radix + xive guests don't need hcalls to handle
>> interrupts or manage translations.
>>
>> So it's much less important to handle hcalls in real mode in P9.
>>
>> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>> ---
>
> <snip>
>
>> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
>> index fa7614c37e08..17739aaee3d8 100644
>> --- a/arch/powerpc/kvm/book3s_hv.c
>> +++ b/arch/powerpc/kvm/book3s_hv.c
>> @@ -1142,12 +1142,13 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)
>> }
>>
>> /*
>> - * Handle H_CEDE in the nested virtualization case where we haven't
>> - * called the real-mode hcall handlers in book3s_hv_rmhandlers.S.
>> + * Handle H_CEDE in the P9 path where we don't call the real-mode hcall
>> + * handlers in book3s_hv_rmhandlers.S.
>> + *
>> * This has to be done early, not in kvmppc_pseries_do_hcall(), so
>> * that the cede logic in kvmppc_run_single_vcpu() works properly.
>> */
>> -static void kvmppc_nested_cede(struct kvm_vcpu *vcpu)
>> +static void kvmppc_cede(struct kvm_vcpu *vcpu)
>> {
>> vcpu->arch.shregs.msr |= MSR_EE;
>> vcpu->arch.ceded = 1;
>> @@ -1403,9 +1404,15 @@ static int kvmppc_handle_exit_hv(struct kvm_vcpu *vcpu,
>> /* hcall - punt to userspace */
>> int i;
>>
>> - /* hypercall with MSR_PR has already been handled in rmode,
>> - * and never reaches here.
>> - */
>> + if (unlikely(vcpu->arch.shregs.msr & MSR_PR)) {
>> + /*
>> + * Guest userspace executed sc 1, reflect it back as a
>> + * privileged program check interrupt.
>> + */
>> + kvmppc_core_queue_program(vcpu, SRR1_PROGPRIV);
>> + r = RESUME_GUEST;
>> + break;
>> + }
>
> This patch bypasses sc_1_fast_return so it breaks KVM-PR. L1 loops with
> the following output:
>
> [ 9.503929][ T3443] Couldn't emulate instruction 0x4e800020 (op 19 xop 16)
> [ 9.503990][ T3443] kvmppc_exit_pr_progint: emulation at 48f4 failed (4e800020)
> [ 9.504080][ T3443] Couldn't emulate instruction 0x4e800020 (op 19 xop 16)
> [ 9.504170][ T3443] kvmppc_exit_pr_progint: emulation at 48f4 failed (4e800020)
>
> 0x4e800020 is a blr after a sc 1 in SLOF.
>
> For KVM-PR we need to inject a 0xc00 at some point, either here or
> before branching to no_try_real in book3s_hv_rmhandlers.S.
Ah, I didn't know about that PR KVM (I suppose I should test it but I
haven't been able to get it running in the past).
Should be able to deal with that. This patch probably shouldn't change
the syscall behaviour like this anyway.
Thanks,
Nick
^ permalink raw reply
* Re: [PATCH v4 22/46] KVM: PPC: Book3S HV P9: Stop handling hcalls in real-mode in the P9 path
From: Nicholas Piggin @ 2021-03-24 1:22 UTC (permalink / raw)
To: Fabiano Rosas, kvm-ppc; +Cc: linuxppc-dev
In-Reply-To: <871rc5g32x.fsf@linux.ibm.com>
Excerpts from Fabiano Rosas's message of March 24, 2021 4:03 am:
> Nicholas Piggin <npiggin@gmail.com> writes:
>
>> In the interest of minimising the amount of code that is run in
>> "real-mode", don't handle hcalls in real mode in the P9 path.
>>
>> POWER8 and earlier are much more expensive to exit from HV real mode
>> and switch to host mode, because on those processors HV interrupts get
>> to the hypervisor with the MMU off, and the other threads in the core
>> need to be pulled out of the guest, and SLBs all need to be saved,
>> ERATs invalidated, and host SLB reloaded before the MMU is re-enabled
>> in host mode. Hash guests also require a lot of hcalls to run. The
>> XICS interrupt controller requires hcalls to run.
>>
>> By contrast, POWER9 has independent thread switching, and in radix mode
>> the hypervisor is already in a host virtual memory mode when the HV
>> interrupt is taken. Radix + xive guests don't need hcalls to handle
>> interrupts or manage translations.
>>
>> So it's much less important to handle hcalls in real mode in P9.
>>
>> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>
> I tried this again in the L2 with xive=off and it works as expected now.
>
> Tested-by: Fabiano Rosas <farosas@linux.ibm.com>
Oh good, thanks for spotting the problem and re testing.
Thanks,
Nick
^ permalink raw reply
* Re: [PATCH v4 22/46] KVM: PPC: Book3S HV P9: Stop handling hcalls in real-mode in the P9 path
From: Nicholas Piggin @ 2021-03-24 1:21 UTC (permalink / raw)
To: Cédric Le Goater, kvm-ppc; +Cc: linuxppc-dev
In-Reply-To: <5b1361d9-723e-1868-efb0-ab0abf217d92@kaod.org>
Excerpts from Cédric Le Goater's message of March 23, 2021 11:23 pm:
> On 3/23/21 2:02 AM, Nicholas Piggin wrote:
>> In the interest of minimising the amount of code that is run in
>> "real-mode", don't handle hcalls in real mode in the P9 path.
>>
>> POWER8 and earlier are much more expensive to exit from HV real mode
>> and switch to host mode, because on those processors HV interrupts get
>> to the hypervisor with the MMU off, and the other threads in the core
>> need to be pulled out of the guest, and SLBs all need to be saved,
>> ERATs invalidated, and host SLB reloaded before the MMU is re-enabled
>> in host mode. Hash guests also require a lot of hcalls to run. The
>> XICS interrupt controller requires hcalls to run.
>>
>> By contrast, POWER9 has independent thread switching, and in radix mode
>> the hypervisor is already in a host virtual memory mode when the HV
>> interrupt is taken. Radix + xive guests don't need hcalls to handle
>> interrupts or manage translations.
>>
>> So it's much less important to handle hcalls in real mode in P9.
>>
>> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>> ---
>> arch/powerpc/include/asm/kvm_ppc.h | 5 ++
>> arch/powerpc/kvm/book3s_hv.c | 57 ++++++++++++++++----
>> arch/powerpc/kvm/book3s_hv_rmhandlers.S | 5 ++
>> arch/powerpc/kvm/book3s_xive.c | 70 +++++++++++++++++++++++++
>> 4 files changed, 127 insertions(+), 10 deletions(-)
>>
>> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
>> index 73b1ca5a6471..db6646c2ade2 100644
>> --- a/arch/powerpc/include/asm/kvm_ppc.h
>> +++ b/arch/powerpc/include/asm/kvm_ppc.h
>> @@ -607,6 +607,7 @@ extern void kvmppc_free_pimap(struct kvm *kvm);
>> extern int kvmppc_xics_rm_complete(struct kvm_vcpu *vcpu, u32 hcall);
>> extern void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu);
>> extern int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 cmd);
>> +extern int kvmppc_xive_xics_hcall(struct kvm_vcpu *vcpu, u32 req);
>> extern u64 kvmppc_xics_get_icp(struct kvm_vcpu *vcpu);
>> extern int kvmppc_xics_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
>> extern int kvmppc_xics_connect_vcpu(struct kvm_device *dev,
>> @@ -639,6 +640,8 @@ static inline int kvmppc_xics_enabled(struct kvm_vcpu *vcpu)
>> static inline void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu) { }
>> static inline int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 cmd)
>> { return 0; }
>> +static inline int kvmppc_xive_xics_hcall(struct kvm_vcpu *vcpu, u32 req)
>> + { return 0; }
>> #endif
>>
>> #ifdef CONFIG_KVM_XIVE
>> @@ -673,6 +676,7 @@ extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
>> int level, bool line_status);
>> extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
>> extern void kvmppc_xive_pull_vcpu(struct kvm_vcpu *vcpu);
>> +extern void kvmppc_xive_cede_vcpu(struct kvm_vcpu *vcpu);
>>
>> static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
>> {
>> @@ -714,6 +718,7 @@ static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 ir
>> int level, bool line_status) { return -ENODEV; }
>> static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
>> static inline void kvmppc_xive_pull_vcpu(struct kvm_vcpu *vcpu) { }
>> +static inline void kvmppc_xive_cede_vcpu(struct kvm_vcpu *vcpu) { }
>>
>> static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
>> { return 0; }
>> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
>> index fa7614c37e08..17739aaee3d8 100644
>> --- a/arch/powerpc/kvm/book3s_hv.c
>> +++ b/arch/powerpc/kvm/book3s_hv.c
>> @@ -1142,12 +1142,13 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)
>> }
>>
>> /*
>> - * Handle H_CEDE in the nested virtualization case where we haven't
>> - * called the real-mode hcall handlers in book3s_hv_rmhandlers.S.
>> + * Handle H_CEDE in the P9 path where we don't call the real-mode hcall
>> + * handlers in book3s_hv_rmhandlers.S.
>> + *
>> * This has to be done early, not in kvmppc_pseries_do_hcall(), so
>> * that the cede logic in kvmppc_run_single_vcpu() works properly.
>> */
>> -static void kvmppc_nested_cede(struct kvm_vcpu *vcpu)
>> +static void kvmppc_cede(struct kvm_vcpu *vcpu)
>> {
>> vcpu->arch.shregs.msr |= MSR_EE;
>> vcpu->arch.ceded = 1;
>> @@ -1403,9 +1404,15 @@ static int kvmppc_handle_exit_hv(struct kvm_vcpu *vcpu,
>> /* hcall - punt to userspace */
>> int i;
>>
>> - /* hypercall with MSR_PR has already been handled in rmode,
>> - * and never reaches here.
>> - */
>> + if (unlikely(vcpu->arch.shregs.msr & MSR_PR)) {
>> + /*
>> + * Guest userspace executed sc 1, reflect it back as a
>> + * privileged program check interrupt.
>> + */
>> + kvmppc_core_queue_program(vcpu, SRR1_PROGPRIV);
>> + r = RESUME_GUEST;
>> + break;
>> + }
>>
>> run->papr_hcall.nr = kvmppc_get_gpr(vcpu, 3);
>> for (i = 0; i < 9; ++i)
>> @@ -3663,6 +3670,12 @@ static int kvmhv_load_hv_regs_and_go(struct kvm_vcpu *vcpu, u64 time_limit,
>> return trap;
>> }
>>
>> +static inline bool hcall_is_xics(unsigned long req)
>> +{
>> + return (req == H_EOI || req == H_CPPR || req == H_IPI ||
>> + req == H_IPOLL || req == H_XIRR || req == H_XIRR_X);
>> +}
>> +
>> /*
>> * Virtual-mode guest entry for POWER9 and later when the host and
>> * guest are both using the radix MMU. The LPIDR has already been set.
>> @@ -3774,15 +3787,36 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,
>> /* H_CEDE has to be handled now, not later */
>> if (trap == BOOK3S_INTERRUPT_SYSCALL && !vcpu->arch.nested &&
>> kvmppc_get_gpr(vcpu, 3) == H_CEDE) {
>> - kvmppc_nested_cede(vcpu);
>> + kvmppc_cede(vcpu);
>> kvmppc_set_gpr(vcpu, 3, 0);
>> trap = 0;
>> }
>> } else {
>> kvmppc_xive_push_vcpu(vcpu);
>> trap = kvmhv_load_hv_regs_and_go(vcpu, time_limit, lpcr);
>> + if (trap == BOOK3S_INTERRUPT_SYSCALL && !vcpu->arch.nested &&
>> + !(vcpu->arch.shregs.msr & MSR_PR)) {
>> + unsigned long req = kvmppc_get_gpr(vcpu, 3);
>> +
>> + /* H_CEDE has to be handled now, not later */
>> + if (req == H_CEDE) {
>> + kvmppc_cede(vcpu);
>> + kvmppc_xive_cede_vcpu(vcpu); /* may un-cede */> + kvmppc_set_gpr(vcpu, 3, 0);
>> + trap = 0;
>> +
>> + /* XICS hcalls must be handled before xive is pulled */
>> + } else if (hcall_is_xics(req)) {
>> + int ret;
>> +
>> + ret = kvmppc_xive_xics_hcall(vcpu, req);
>> + if (ret != H_TOO_HARD) {
>> + kvmppc_set_gpr(vcpu, 3, ret);
>> + trap = 0;
>> + }
>> + }
>> + }
>> kvmppc_xive_pull_vcpu(vcpu);
>> -
>> }
>>
>> vcpu->arch.slb_max = 0;
>> @@ -4442,8 +4476,11 @@ static int kvmppc_vcpu_run_hv(struct kvm_vcpu *vcpu)
>> else
>> r = kvmppc_run_vcpu(vcpu);
>>
>> - if (run->exit_reason == KVM_EXIT_PAPR_HCALL &&
>> - !(vcpu->arch.shregs.msr & MSR_PR)) {
>> + if (run->exit_reason == KVM_EXIT_PAPR_HCALL) {
>> + if (WARN_ON_ONCE(vcpu->arch.shregs.msr & MSR_PR)) {
>> + r = RESUME_GUEST;
>> + continue;
>> + }
>> trace_kvm_hcall_enter(vcpu);
>> r = kvmppc_pseries_do_hcall(vcpu);
>> trace_kvm_hcall_exit(vcpu, r);
>> diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
>> index c11597f815e4..2d0d14ed1d92 100644
>> --- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S
>> +++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
>> @@ -1397,9 +1397,14 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
>> mr r4,r9
>> bge fast_guest_return
>> 2:
>> + /* If we came in through the P9 short path, no real mode hcalls */
>> + lwz r0, STACK_SLOT_SHORT_PATH(r1)
>> + cmpwi r0, 0
>> + bne no_try_real
>> /* See if this is an hcall we can handle in real mode */
>> cmpwi r12,BOOK3S_INTERRUPT_SYSCALL
>> beq hcall_try_real_mode
>> +no_try_real:
>>
>> /* Hypervisor doorbell - exit only if host IPI flag set */
>> cmpwi r12, BOOK3S_INTERRUPT_H_DOORBELL
>> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
>> index 741bf1f4387a..dcc07ceaf5ca 100644
>> --- a/arch/powerpc/kvm/book3s_xive.c
>> +++ b/arch/powerpc/kvm/book3s_xive.c
>> @@ -158,6 +158,40 @@ void kvmppc_xive_pull_vcpu(struct kvm_vcpu *vcpu)
>> }
>> EXPORT_SYMBOL_GPL(kvmppc_xive_pull_vcpu);
>>
>> +void kvmppc_xive_cede_vcpu(struct kvm_vcpu *vcpu)
>
> kvmppc_xive_rearm_escalation() may be ? It has more meaning to me.
Sure whatever you prefer.
>
>> +{
>> + void __iomem *esc_vaddr = (void __iomem *)vcpu->arch.xive_esc_vaddr;
>> +
>> + if (!esc_vaddr)
>> + return;
>> +
>> + /* we are using XIVE with single escalation */
>> +
>> + if (vcpu->arch.xive_esc_on) {
>> + /*
>> + * If we still have a pending escalation, abort the cede,
>> + * and we must set PQ to 10 rather than 00 so that we don't
>> + * potentially end up with two entries for the escalation
>> + * interrupt in the XIVE interrupt queue. In that case
>> + * we also don't want to set xive_esc_on to 1 here in
>> + * case we race with xive_esc_irq().
>> + */
>> + vcpu->arch.ceded = 0;
>> + /*
>> + * The escalation interrupts are special as we don't EOI them.
>> + * There is no need to use the load-after-store ordering offset
>> + * to set PQ to 10 as we won't use StoreEOI.
>> + */
>> + __raw_readq(esc_vaddr + XIVE_ESB_SET_PQ_10);
>> + } else {
>> + vcpu->arch.xive_esc_on = true;> + mb();
>> + __raw_readq(esc_vaddr + XIVE_ESB_SET_PQ_00);
>> + }
>> + mb();
>> +}
>> +EXPORT_SYMBOL_GPL(kvmppc_xive_cede_vcpu);
>> +
>> /*
>> * This is a simple trigger for a generic XIVE IRQ. This must
>> * only be called for interrupts that support a trigger page
>> @@ -2106,6 +2140,42 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
>> return 0;
>> }
>>
>> +int kvmppc_xive_xics_hcall(struct kvm_vcpu *vcpu, u32 req)
>> +{
>> + struct kvmppc_vcore *vc = vcpu->arch.vcore;
>> +
>> + /*
>> + * This test covers the case in which a vCPU does XICS hcalls without
>> + * QEMU having connected the vCPU to a XICS ICP. The ICP is the KVM
>> + * XICS device on P8 or XICS-on-XIVE on P9. It catches QEMU errors when
>> + * the interrupt mode is negotiated, we don't want the OS to do XICS
>> + * hcalls after having negotiated the XIVE interrupt mode.
>> + */
>
> I think a comment like the following should be enough.
>
> The VM should have configured XICS mode before doing XICS hcalls.
I'll change it.
> No need to resend for that though.
>
> Reviewed-by: Cédric Le Goater <clg@kaod.org>
Thanks,
Nick
^ permalink raw reply
* [powerpc:next-test] BUILD SUCCESS 8a83feefbd5254ae7f13aff3e4097dd7d8723bce
From: kernel test robot @ 2021-03-24 1:11 UTC (permalink / raw)
To: Michael Ellerman; +Cc: linuxppc-dev
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next-test
branch HEAD: 8a83feefbd5254ae7f13aff3e4097dd7d8723bce cxl: Fix couple of spellings
elapsed time: 725m
configs tested: 109
configs skipped: 2
The following configs have been built successfully.
More configs may be tested in the coming days.
gcc tested configs:
arm defconfig
arm64 allyesconfig
arm64 defconfig
arm allyesconfig
arm allmodconfig
x86_64 allyesconfig
riscv allmodconfig
i386 allyesconfig
riscv allyesconfig
h8300 allyesconfig
arc nsimosci_hs_smp_defconfig
powerpc ppc40x_defconfig
powerpc makalu_defconfig
m68k m5208evb_defconfig
mips cu1000-neo_defconfig
powerpc ksi8560_defconfig
arm mps2_defconfig
powerpc walnut_defconfig
arm rpc_defconfig
mips jmr3927_defconfig
arm am200epdkit_defconfig
powerpc currituck_defconfig
sh sh7710voipgw_defconfig
arc vdk_hs38_defconfig
mips bmips_stb_defconfig
ia64 generic_defconfig
arc nsim_700_defconfig
arm pxa910_defconfig
xtensa nommu_kc705_defconfig
powerpc mpc8272_ads_defconfig
powerpc linkstation_defconfig
powerpc rainier_defconfig
mips maltaup_defconfig
arm pxa168_defconfig
arm collie_defconfig
arm pxa_defconfig
powerpc tqm8555_defconfig
powerpc eiger_defconfig
arm aspeed_g5_defconfig
powerpc pseries_defconfig
arm pxa255-idp_defconfig
arm exynos_defconfig
h8300 alldefconfig
sh se7780_defconfig
ia64 allmodconfig
ia64 defconfig
ia64 allyesconfig
m68k allmodconfig
m68k defconfig
m68k allyesconfig
nios2 defconfig
arc allyesconfig
nds32 allnoconfig
nds32 defconfig
nios2 allyesconfig
csky defconfig
alpha defconfig
alpha allyesconfig
xtensa allyesconfig
arc defconfig
sh allmodconfig
parisc defconfig
s390 allyesconfig
s390 allmodconfig
parisc allyesconfig
s390 defconfig
sparc allyesconfig
sparc defconfig
i386 tinyconfig
i386 defconfig
mips allyesconfig
mips allmodconfig
powerpc allyesconfig
powerpc allmodconfig
powerpc allnoconfig
x86_64 randconfig-a006-20210323
x86_64 randconfig-a004-20210323
x86_64 randconfig-a005-20210323
i386 randconfig-a003-20210323
i386 randconfig-a004-20210323
i386 randconfig-a001-20210323
i386 randconfig-a002-20210323
i386 randconfig-a006-20210323
i386 randconfig-a005-20210323
i386 randconfig-a014-20210323
i386 randconfig-a011-20210323
i386 randconfig-a015-20210323
i386 randconfig-a016-20210323
i386 randconfig-a012-20210323
i386 randconfig-a013-20210323
x86_64 randconfig-a002-20210323
x86_64 randconfig-a003-20210323
x86_64 randconfig-a001-20210323
riscv nommu_k210_defconfig
riscv nommu_virt_defconfig
riscv allnoconfig
riscv defconfig
riscv rv32_defconfig
x86_64 rhel-7.6-kselftests
x86_64 defconfig
x86_64 rhel-8.3
x86_64 rhel-8.3-kbuiltin
x86_64 kexec
clang tested configs:
x86_64 randconfig-a012-20210323
x86_64 randconfig-a015-20210323
x86_64 randconfig-a013-20210323
x86_64 randconfig-a014-20210323
x86_64 randconfig-a011-20210323
x86_64 randconfig-a016-20210323
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
^ permalink raw reply
* [powerpc:merge] BUILD SUCCESS 909b15d4ac3524a89c6df8c60e0cb0b4d5a3c248
From: kernel test robot @ 2021-03-24 1:11 UTC (permalink / raw)
To: Michael Ellerman; +Cc: linuxppc-dev
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git merge
branch HEAD: 909b15d4ac3524a89c6df8c60e0cb0b4d5a3c248 Automatic merge of 'fixes' into merge (2021-03-23 22:53)
elapsed time: 725m
configs tested: 127
configs skipped: 2
The following configs have been built successfully.
More configs may be tested in the coming days.
gcc tested configs:
arm defconfig
arm64 allyesconfig
arm64 defconfig
arm allyesconfig
arm allmodconfig
x86_64 allyesconfig
i386 allyesconfig
powerpc maple_defconfig
powerpc lite5200b_defconfig
sparc alldefconfig
mips maltaup_xpa_defconfig
powerpc mpc836x_rdk_defconfig
arm lart_defconfig
m68k m5208evb_defconfig
mips cu1000-neo_defconfig
powerpc ksi8560_defconfig
arm mps2_defconfig
powerpc tqm8xx_defconfig
m68k alldefconfig
powerpc mgcoge_defconfig
sh se7751_defconfig
mips omega2p_defconfig
powerpc ppc64e_defconfig
powerpc walnut_defconfig
arm rpc_defconfig
mips jmr3927_defconfig
arm am200epdkit_defconfig
powerpc currituck_defconfig
sh sh7710voipgw_defconfig
powerpc socrates_defconfig
nds32 allnoconfig
arm imx_v6_v7_defconfig
arm neponset_defconfig
sh hp6xx_defconfig
arm orion5x_defconfig
mips malta_qemu_32r6_defconfig
mips capcella_defconfig
arm lubbock_defconfig
sh alldefconfig
powerpc ep8248e_defconfig
powerpc tqm8540_defconfig
arm integrator_defconfig
riscv rv32_defconfig
powerpc mpc866_ads_defconfig
arm mainstone_defconfig
sh sh03_defconfig
m68k multi_defconfig
arm pxa_defconfig
powerpc tqm8555_defconfig
powerpc eiger_defconfig
arm pxa168_defconfig
mips cu1830-neo_defconfig
powerpc obs600_defconfig
powerpc64 defconfig
mips ath25_defconfig
arm aspeed_g5_defconfig
powerpc pseries_defconfig
arm pxa255-idp_defconfig
arm exynos_defconfig
h8300 alldefconfig
sh se7780_defconfig
ia64 allmodconfig
ia64 defconfig
ia64 allyesconfig
m68k allmodconfig
m68k defconfig
m68k allyesconfig
nds32 defconfig
nios2 allyesconfig
csky defconfig
alpha defconfig
alpha allyesconfig
xtensa allyesconfig
h8300 allyesconfig
arc defconfig
sh allmodconfig
parisc defconfig
s390 allyesconfig
s390 allmodconfig
parisc allyesconfig
s390 defconfig
sparc allyesconfig
sparc defconfig
i386 tinyconfig
i386 defconfig
nios2 defconfig
arc allyesconfig
mips allyesconfig
mips allmodconfig
powerpc allyesconfig
powerpc allmodconfig
powerpc allnoconfig
x86_64 randconfig-a002-20210323
x86_64 randconfig-a003-20210323
x86_64 randconfig-a006-20210323
x86_64 randconfig-a001-20210323
x86_64 randconfig-a004-20210323
x86_64 randconfig-a005-20210323
i386 randconfig-a003-20210323
i386 randconfig-a004-20210323
i386 randconfig-a001-20210323
i386 randconfig-a002-20210323
i386 randconfig-a006-20210323
i386 randconfig-a005-20210323
i386 randconfig-a016-20210323
i386 randconfig-a014-20210323
i386 randconfig-a011-20210323
i386 randconfig-a015-20210323
i386 randconfig-a012-20210323
i386 randconfig-a013-20210323
riscv nommu_k210_defconfig
riscv allyesconfig
riscv nommu_virt_defconfig
riscv allnoconfig
riscv defconfig
riscv allmodconfig
x86_64 rhel-7.6-kselftests
x86_64 defconfig
x86_64 rhel-8.3
x86_64 rhel-8.3-kbuiltin
x86_64 kexec
clang tested configs:
x86_64 randconfig-a012-20210323
x86_64 randconfig-a015-20210323
x86_64 randconfig-a013-20210323
x86_64 randconfig-a014-20210323
x86_64 randconfig-a011-20210323
x86_64 randconfig-a016-20210323
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
^ permalink raw reply
* [powerpc:fixes-test] BUILD SUCCESS 274cb1ca2e7ce02cab56f5f4c61a74aeb566f931
From: kernel test robot @ 2021-03-24 1:04 UTC (permalink / raw)
To: Michael Ellerman; +Cc: linuxppc-dev
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git fixes-test
branch HEAD: 274cb1ca2e7ce02cab56f5f4c61a74aeb566f931 powerpc/pseries/mobility: handle premature return from H_JOIN
elapsed time: 721m
configs tested: 206
configs skipped: 2
The following configs have been built successfully.
More configs may be tested in the coming days.
gcc tested configs:
arm defconfig
arm64 allyesconfig
arm64 defconfig
arm allyesconfig
arm allmodconfig
x86_64 allyesconfig
riscv allmodconfig
riscv allyesconfig
i386 allyesconfig
powerpc sam440ep_defconfig
powerpc mpc834x_itxgp_defconfig
powerpc ppa8548_defconfig
mips workpad_defconfig
powerpc mpc8313_rdb_defconfig
mips cu1000-neo_defconfig
arm mini2440_defconfig
powerpc redwood_defconfig
mips ar7_defconfig
m68k mac_defconfig
m68k allyesconfig
arm imote2_defconfig
mips ip27_defconfig
sh se7724_defconfig
mips jazz_defconfig
m68k sun3_defconfig
powerpc skiroot_defconfig
mips bmips_stb_defconfig
arm milbeaut_m10v_defconfig
arm mv78xx0_defconfig
arm h5000_defconfig
arm spear3xx_defconfig
powerpc mpc836x_rdk_defconfig
arm lart_defconfig
m68k m5307c3_defconfig
h8300 h8300h-sim_defconfig
sh rts7751r2dplus_defconfig
arm sunxi_defconfig
sh ap325rxa_defconfig
mips bigsur_defconfig
m68k m5407c3_defconfig
arm hisi_defconfig
arm vexpress_defconfig
m68k m5208evb_defconfig
powerpc ksi8560_defconfig
arm mps2_defconfig
arm shannon_defconfig
arm omap2plus_defconfig
arm tct_hammer_defconfig
powerpc cm5200_defconfig
powerpc iss476-smp_defconfig
mips malta_kvm_guest_defconfig
arc haps_hs_smp_defconfig
powerpc lite5200b_defconfig
powerpc sequoia_defconfig
mips rbtx49xx_defconfig
arm pxa255-idp_defconfig
arm exynos_defconfig
powerpc bluestone_defconfig
powerpc adder875_defconfig
arm omap1_defconfig
powerpc tqm8560_defconfig
arm trizeps4_defconfig
xtensa cadence_csp_defconfig
sh magicpanelr2_defconfig
m68k m5475evb_defconfig
mips db1xxx_defconfig
mips maltaaprp_defconfig
mips tb0287_defconfig
sh sdk7780_defconfig
powerpc storcenter_defconfig
powerpc kilauea_defconfig
mips sb1250_swarm_defconfig
arc vdk_hs38_defconfig
ia64 generic_defconfig
arc nsim_700_defconfig
arm pxa910_defconfig
xtensa nommu_kc705_defconfig
powerpc socrates_defconfig
nds32 allnoconfig
arm imx_v6_v7_defconfig
arm neponset_defconfig
sh hp6xx_defconfig
arm orion5x_defconfig
mips tb0219_defconfig
mips jmr3927_defconfig
ia64 gensparse_defconfig
microblaze mmu_defconfig
arm nhk8815_defconfig
mips malta_qemu_32r6_defconfig
mips capcella_defconfig
arm lubbock_defconfig
s390 alldefconfig
mips e55_defconfig
sh sh7710voipgw_defconfig
m68k stmark2_defconfig
powerpc ppc64e_defconfig
powerpc sbc8548_defconfig
mips bcm47xx_defconfig
mips loongson3_defconfig
mips ip28_defconfig
powerpc obs600_defconfig
m68k mvme16x_defconfig
nios2 3c120_defconfig
sh landisk_defconfig
sh secureedge5410_defconfig
arm integrator_defconfig
powerpc mpc836x_mds_defconfig
powerpc mpc8272_ads_defconfig
powerpc linkstation_defconfig
powerpc rainier_defconfig
mips maltaup_defconfig
arm pxa168_defconfig
arm collie_defconfig
arm lpc18xx_defconfig
sh ecovec24-romimage_defconfig
mips rs90_defconfig
sh sh7785lcr_defconfig
sh se7721_defconfig
arm davinci_all_defconfig
powerpc ppc6xx_defconfig
powerpc mpc834x_mds_defconfig
sh rsk7201_defconfig
powerpc tqm8541_defconfig
powerpc mpc834x_itx_defconfig
sh rsk7203_defconfig
mips loongson1b_defconfig
arm pxa_defconfig
powerpc tqm8555_defconfig
powerpc eiger_defconfig
mips cu1830-neo_defconfig
powerpc64 defconfig
mips ath25_defconfig
arm axm55xx_defconfig
arc nsimosci_hs_smp_defconfig
powerpc asp8347_defconfig
arc hsdk_defconfig
ia64 allmodconfig
ia64 defconfig
ia64 allyesconfig
m68k allmodconfig
m68k defconfig
nios2 defconfig
arc allyesconfig
nds32 defconfig
nios2 allyesconfig
csky defconfig
alpha defconfig
alpha allyesconfig
xtensa allyesconfig
h8300 allyesconfig
arc defconfig
sh allmodconfig
parisc defconfig
s390 allyesconfig
s390 allmodconfig
parisc allyesconfig
s390 defconfig
sparc allyesconfig
sparc defconfig
i386 tinyconfig
i386 defconfig
mips allyesconfig
mips allmodconfig
powerpc allyesconfig
powerpc allmodconfig
powerpc allnoconfig
x86_64 randconfig-a002-20210323
x86_64 randconfig-a003-20210323
x86_64 randconfig-a006-20210323
x86_64 randconfig-a001-20210323
x86_64 randconfig-a004-20210323
x86_64 randconfig-a005-20210323
i386 randconfig-a003-20210323
i386 randconfig-a001-20210323
i386 randconfig-a002-20210323
i386 randconfig-a004-20210323
i386 randconfig-a006-20210323
i386 randconfig-a005-20210323
i386 randconfig-a004-20210324
i386 randconfig-a003-20210324
i386 randconfig-a001-20210324
i386 randconfig-a002-20210324
i386 randconfig-a006-20210324
i386 randconfig-a005-20210324
i386 randconfig-a015-20210323
i386 randconfig-a016-20210323
i386 randconfig-a014-20210323
i386 randconfig-a011-20210323
i386 randconfig-a012-20210323
i386 randconfig-a013-20210323
riscv nommu_virt_defconfig
riscv rv32_defconfig
riscv nommu_k210_defconfig
riscv allnoconfig
riscv defconfig
x86_64 rhel-7.6-kselftests
x86_64 defconfig
x86_64 rhel-8.3
x86_64 rhel-8.3-kbuiltin
x86_64 kexec
clang tested configs:
x86_64 randconfig-a012-20210323
x86_64 randconfig-a015-20210323
x86_64 randconfig-a013-20210323
x86_64 randconfig-a014-20210323
x86_64 randconfig-a011-20210323
x86_64 randconfig-a016-20210323
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
^ permalink raw reply
* Re: [PATCH v2 -next] powerpc: kernel/time.c - cleanup warnings
From: Alexandre Belloni @ 2021-03-23 23:05 UTC (permalink / raw)
To: He Ying
Cc: linux-rtc, a.zummo, geert+renesas, peterz, frederic, linux-kernel,
npiggin, paulus, kernelfans, tglx, msuchanek, linuxppc-dev
In-Reply-To: <YFppJkpZRHMJFay0@piout.net>
On 23/03/2021 23:18:17+0100, Alexandre Belloni wrote:
> Hello,
>
> On 23/03/2021 05:12:57-0400, He Ying wrote:
> > We found these warnings in arch/powerpc/kernel/time.c as follows:
> > warning: symbol 'decrementer_max' was not declared. Should it be static?
> > warning: symbol 'rtc_lock' was not declared. Should it be static?
> > warning: symbol 'dtl_consumer' was not declared. Should it be static?
> >
> > Declare 'decrementer_max' and 'rtc_lock' in powerpc asm/time.h.
> > Rename 'rtc_lock' in drviers/rtc/rtc-vr41xx.c to 'vr41xx_rtc_lock' to
> > avoid the conflict with the variable in powerpc asm/time.h.
> > Move 'dtl_consumer' definition behind "include <asm/dtl.h>" because it
> > is declared there.
> >
> > Reported-by: Hulk Robot <hulkci@huawei.com>
> > Signed-off-by: He Ying <heying24@huawei.com>
> > ---
> > v2:
> > - Instead of including linux/mc146818rtc.h in powerpc kernel/time.c, declare
> > rtc_lock in powerpc asm/time.h.
> >
>
> V1 was actually the correct thing to do. rtc_lock is there exactly
> because chrp and maple are using mc146818 compatible RTCs. This is then
> useful because then drivers/char/nvram.c is enabled. The proper fix
> would be to scrap all of that and use rtc-cmos for those platforms as
> this drives the RTC properly and exposes the NVRAM for the mc146818.
>
> Or at least, if there are no users for the char/nvram driver on those
> two platforms, remove the spinlock and stop enabling CONFIG_NVRAM or
> more likely rename the symbol as it seems to be abused by both chrp and
> powermac.
>
Ok so rtc_lock is not even used by the char/nvram.c driver as it is
completely compiled out.
I guess it is fine having it move to the individual platform as looking
very quickly at the Kconfig, it is not possible to select both
simultaneously. Tentative patch:
8<-----
From dfa59b6f44fdfdefafffa7666aec89e62bbd5c80 Mon Sep 17 00:00:00 2001
From: Alexandre Belloni <alexandre.belloni@bootlin.com>
Date: Wed, 24 Mar 2021 00:00:03 +0100
Subject: [PATCH] powerpc: move rtc_lock to specific platforms
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
---
arch/powerpc/kernel/time.c | 3 ---
arch/powerpc/platforms/chrp/time.c | 2 +-
arch/powerpc/platforms/maple/time.c | 2 ++
3 files changed, 3 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c
index 67feb3524460..d3bb189ea7f4 100644
--- a/arch/powerpc/kernel/time.c
+++ b/arch/powerpc/kernel/time.c
@@ -123,9 +123,6 @@ EXPORT_SYMBOL(tb_ticks_per_usec);
unsigned long tb_ticks_per_sec;
EXPORT_SYMBOL(tb_ticks_per_sec); /* for cputime_t conversions */
-DEFINE_SPINLOCK(rtc_lock);
-EXPORT_SYMBOL_GPL(rtc_lock);
-
static u64 tb_to_ns_scale __read_mostly;
static unsigned tb_to_ns_shift __read_mostly;
static u64 boot_tb __read_mostly;
diff --git a/arch/powerpc/platforms/chrp/time.c b/arch/powerpc/platforms/chrp/time.c
index acde7bbe0716..ea90c15f5edd 100644
--- a/arch/powerpc/platforms/chrp/time.c
+++ b/arch/powerpc/platforms/chrp/time.c
@@ -30,7 +30,7 @@
#include <platforms/chrp/chrp.h>
-extern spinlock_t rtc_lock;
+DEFINE_SPINLOCK(rtc_lock);
#define NVRAM_AS0 0x74
#define NVRAM_AS1 0x75
diff --git a/arch/powerpc/platforms/maple/time.c b/arch/powerpc/platforms/maple/time.c
index 78209bb7629c..ddda02010d86 100644
--- a/arch/powerpc/platforms/maple/time.c
+++ b/arch/powerpc/platforms/maple/time.c
@@ -34,6 +34,8 @@
#define DBG(x...)
#endif
+DEFINE_SPINLOCK(rtc_lock);
+
static int maple_rtc_addr;
static int maple_clock_read(int addr)
--
2.25.1
> I'm not completely against the rename in vr41xxx but the fix for the
> warnings can and should be contained in arch/powerpc.
>
> > arch/powerpc/include/asm/time.h | 3 +++
> > arch/powerpc/kernel/time.c | 6 ++----
> > drivers/rtc/rtc-vr41xx.c | 22 +++++++++++-----------
> > 3 files changed, 16 insertions(+), 15 deletions(-)
> >
> > diff --git a/arch/powerpc/include/asm/time.h b/arch/powerpc/include/asm/time.h
> > index 8dd3cdb25338..64a3ef0b4270 100644
> > --- a/arch/powerpc/include/asm/time.h
> > +++ b/arch/powerpc/include/asm/time.h
> > @@ -12,6 +12,7 @@
> > #ifdef __KERNEL__
> > #include <linux/types.h>
> > #include <linux/percpu.h>
> > +#include <linux/spinlock.h>
> >
> > #include <asm/processor.h>
> > #include <asm/cpu_has_feature.h>
> > @@ -22,6 +23,8 @@ extern unsigned long tb_ticks_per_jiffy;
> > extern unsigned long tb_ticks_per_usec;
> > extern unsigned long tb_ticks_per_sec;
> > extern struct clock_event_device decrementer_clockevent;
> > +extern u64 decrementer_max;
> > +extern spinlock_t rtc_lock;
> >
> >
> > extern void generic_calibrate_decr(void);
> > diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c
> > index b67d93a609a2..60b6ac7d3685 100644
> > --- a/arch/powerpc/kernel/time.c
> > +++ b/arch/powerpc/kernel/time.c
> > @@ -150,10 +150,6 @@ bool tb_invalid;
> > u64 __cputime_usec_factor;
> > EXPORT_SYMBOL(__cputime_usec_factor);
> >
> > -#ifdef CONFIG_PPC_SPLPAR
> > -void (*dtl_consumer)(struct dtl_entry *, u64);
> > -#endif
> > -
> > static void calc_cputime_factors(void)
> > {
> > struct div_result res;
> > @@ -179,6 +175,8 @@ static inline unsigned long read_spurr(unsigned long tb)
> >
> > #include <asm/dtl.h>
> >
> > +void (*dtl_consumer)(struct dtl_entry *, u64);
> > +
> > /*
> > * Scan the dispatch trace log and count up the stolen time.
> > * Should be called with interrupts disabled.
> > diff --git a/drivers/rtc/rtc-vr41xx.c b/drivers/rtc/rtc-vr41xx.c
> > index 5a9f9ad86d32..cc31db058197 100644
> > --- a/drivers/rtc/rtc-vr41xx.c
> > +++ b/drivers/rtc/rtc-vr41xx.c
> > @@ -72,7 +72,7 @@ static void __iomem *rtc2_base;
> >
> > static unsigned long epoch = 1970; /* Jan 1 1970 00:00:00 */
> >
> > -static DEFINE_SPINLOCK(rtc_lock);
> > +static DEFINE_SPINLOCK(vr41xx_rtc_lock);
> > static char rtc_name[] = "RTC";
> > static unsigned long periodic_count;
> > static unsigned int alarm_enabled;
> > @@ -101,13 +101,13 @@ static inline time64_t read_elapsed_second(void)
> >
> > static inline void write_elapsed_second(time64_t sec)
> > {
> > - spin_lock_irq(&rtc_lock);
> > + spin_lock_irq(&vr41xx_rtc_lock);
> >
> > rtc1_write(ETIMELREG, (uint16_t)(sec << 15));
> > rtc1_write(ETIMEMREG, (uint16_t)(sec >> 1));
> > rtc1_write(ETIMEHREG, (uint16_t)(sec >> 17));
> >
> > - spin_unlock_irq(&rtc_lock);
> > + spin_unlock_irq(&vr41xx_rtc_lock);
> > }
> >
> > static int vr41xx_rtc_read_time(struct device *dev, struct rtc_time *time)
> > @@ -139,14 +139,14 @@ static int vr41xx_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *wkalrm)
> > unsigned long low, mid, high;
> > struct rtc_time *time = &wkalrm->time;
> >
> > - spin_lock_irq(&rtc_lock);
> > + spin_lock_irq(&vr41xx_rtc_lock);
> >
> > low = rtc1_read(ECMPLREG);
> > mid = rtc1_read(ECMPMREG);
> > high = rtc1_read(ECMPHREG);
> > wkalrm->enabled = alarm_enabled;
> >
> > - spin_unlock_irq(&rtc_lock);
> > + spin_unlock_irq(&vr41xx_rtc_lock);
> >
> > rtc_time64_to_tm((high << 17) | (mid << 1) | (low >> 15), time);
> >
> > @@ -159,7 +159,7 @@ static int vr41xx_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *wkalrm)
> >
> > alarm_sec = rtc_tm_to_time64(&wkalrm->time);
> >
> > - spin_lock_irq(&rtc_lock);
> > + spin_lock_irq(&vr41xx_rtc_lock);
> >
> > if (alarm_enabled)
> > disable_irq(aie_irq);
> > @@ -173,7 +173,7 @@ static int vr41xx_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *wkalrm)
> >
> > alarm_enabled = wkalrm->enabled;
> >
> > - spin_unlock_irq(&rtc_lock);
> > + spin_unlock_irq(&vr41xx_rtc_lock);
> >
> > return 0;
> > }
> > @@ -202,7 +202,7 @@ static int vr41xx_rtc_ioctl(struct device *dev, unsigned int cmd, unsigned long
> >
> > static int vr41xx_rtc_alarm_irq_enable(struct device *dev, unsigned int enabled)
> > {
> > - spin_lock_irq(&rtc_lock);
> > + spin_lock_irq(&vr41xx_rtc_lock);
> > if (enabled) {
> > if (!alarm_enabled) {
> > enable_irq(aie_irq);
> > @@ -214,7 +214,7 @@ static int vr41xx_rtc_alarm_irq_enable(struct device *dev, unsigned int enabled)
> > alarm_enabled = 0;
> > }
> > }
> > - spin_unlock_irq(&rtc_lock);
> > + spin_unlock_irq(&vr41xx_rtc_lock);
> > return 0;
> > }
> >
> > @@ -296,7 +296,7 @@ static int rtc_probe(struct platform_device *pdev)
> > rtc->range_max = (1ULL << 33) - 1;
> > rtc->max_user_freq = MAX_PERIODIC_RATE;
> >
> > - spin_lock_irq(&rtc_lock);
> > + spin_lock_irq(&vr41xx_rtc_lock);
> >
> > rtc1_write(ECMPLREG, 0);
> > rtc1_write(ECMPMREG, 0);
> > @@ -304,7 +304,7 @@ static int rtc_probe(struct platform_device *pdev)
> > rtc1_write(RTCL1LREG, 0);
> > rtc1_write(RTCL1HREG, 0);
> >
> > - spin_unlock_irq(&rtc_lock);
> > + spin_unlock_irq(&vr41xx_rtc_lock);
> >
> > aie_irq = platform_get_irq(pdev, 0);
> > if (aie_irq <= 0) {
> > --
> > 2.17.1
> >
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply related
* Re: [PATCH v4 22/46] KVM: PPC: Book3S HV P9: Stop handling hcalls in real-mode in the P9 path
From: Fabiano Rosas @ 2021-03-23 22:57 UTC (permalink / raw)
To: Nicholas Piggin, kvm-ppc; +Cc: linuxppc-dev, Nicholas Piggin
In-Reply-To: <20210323010305.1045293-23-npiggin@gmail.com>
Nicholas Piggin <npiggin@gmail.com> writes:
> In the interest of minimising the amount of code that is run in
> "real-mode", don't handle hcalls in real mode in the P9 path.
>
> POWER8 and earlier are much more expensive to exit from HV real mode
> and switch to host mode, because on those processors HV interrupts get
> to the hypervisor with the MMU off, and the other threads in the core
> need to be pulled out of the guest, and SLBs all need to be saved,
> ERATs invalidated, and host SLB reloaded before the MMU is re-enabled
> in host mode. Hash guests also require a lot of hcalls to run. The
> XICS interrupt controller requires hcalls to run.
>
> By contrast, POWER9 has independent thread switching, and in radix mode
> the hypervisor is already in a host virtual memory mode when the HV
> interrupt is taken. Radix + xive guests don't need hcalls to handle
> interrupts or manage translations.
>
> So it's much less important to handle hcalls in real mode in P9.
>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---
<snip>
> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
> index fa7614c37e08..17739aaee3d8 100644
> --- a/arch/powerpc/kvm/book3s_hv.c
> +++ b/arch/powerpc/kvm/book3s_hv.c
> @@ -1142,12 +1142,13 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)
> }
>
> /*
> - * Handle H_CEDE in the nested virtualization case where we haven't
> - * called the real-mode hcall handlers in book3s_hv_rmhandlers.S.
> + * Handle H_CEDE in the P9 path where we don't call the real-mode hcall
> + * handlers in book3s_hv_rmhandlers.S.
> + *
> * This has to be done early, not in kvmppc_pseries_do_hcall(), so
> * that the cede logic in kvmppc_run_single_vcpu() works properly.
> */
> -static void kvmppc_nested_cede(struct kvm_vcpu *vcpu)
> +static void kvmppc_cede(struct kvm_vcpu *vcpu)
> {
> vcpu->arch.shregs.msr |= MSR_EE;
> vcpu->arch.ceded = 1;
> @@ -1403,9 +1404,15 @@ static int kvmppc_handle_exit_hv(struct kvm_vcpu *vcpu,
> /* hcall - punt to userspace */
> int i;
>
> - /* hypercall with MSR_PR has already been handled in rmode,
> - * and never reaches here.
> - */
> + if (unlikely(vcpu->arch.shregs.msr & MSR_PR)) {
> + /*
> + * Guest userspace executed sc 1, reflect it back as a
> + * privileged program check interrupt.
> + */
> + kvmppc_core_queue_program(vcpu, SRR1_PROGPRIV);
> + r = RESUME_GUEST;
> + break;
> + }
This patch bypasses sc_1_fast_return so it breaks KVM-PR. L1 loops with
the following output:
[ 9.503929][ T3443] Couldn't emulate instruction 0x4e800020 (op 19 xop 16)
[ 9.503990][ T3443] kvmppc_exit_pr_progint: emulation at 48f4 failed (4e800020)
[ 9.504080][ T3443] Couldn't emulate instruction 0x4e800020 (op 19 xop 16)
[ 9.504170][ T3443] kvmppc_exit_pr_progint: emulation at 48f4 failed (4e800020)
0x4e800020 is a blr after a sc 1 in SLOF.
For KVM-PR we need to inject a 0xc00 at some point, either here or
before branching to no_try_real in book3s_hv_rmhandlers.S.
>
> run->papr_hcall.nr = kvmppc_get_gpr(vcpu, 3);
> for (i = 0; i < 9; ++i)
> @@ -3663,6 +3670,12 @@ static int kvmhv_load_hv_regs_and_go(struct kvm_vcpu *vcpu, u64 time_limit,
> return trap;
> }
>
> +static inline bool hcall_is_xics(unsigned long req)
> +{
> + return (req == H_EOI || req == H_CPPR || req == H_IPI ||
> + req == H_IPOLL || req == H_XIRR || req == H_XIRR_X);
> +}
> +
> /*
> * Virtual-mode guest entry for POWER9 and later when the host and
> * guest are both using the radix MMU. The LPIDR has already been set.
> @@ -3774,15 +3787,36 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,
> /* H_CEDE has to be handled now, not later */
> if (trap == BOOK3S_INTERRUPT_SYSCALL && !vcpu->arch.nested &&
> kvmppc_get_gpr(vcpu, 3) == H_CEDE) {
> - kvmppc_nested_cede(vcpu);
> + kvmppc_cede(vcpu);
> kvmppc_set_gpr(vcpu, 3, 0);
> trap = 0;
> }
> } else {
> kvmppc_xive_push_vcpu(vcpu);
> trap = kvmhv_load_hv_regs_and_go(vcpu, time_limit, lpcr);
> + if (trap == BOOK3S_INTERRUPT_SYSCALL && !vcpu->arch.nested &&
> + !(vcpu->arch.shregs.msr & MSR_PR)) {
> + unsigned long req = kvmppc_get_gpr(vcpu, 3);
> +
> + /* H_CEDE has to be handled now, not later */
> + if (req == H_CEDE) {
> + kvmppc_cede(vcpu);
> + kvmppc_xive_cede_vcpu(vcpu); /* may un-cede */
> + kvmppc_set_gpr(vcpu, 3, 0);
> + trap = 0;
> +
> + /* XICS hcalls must be handled before xive is pulled */
> + } else if (hcall_is_xics(req)) {
> + int ret;
> +
> + ret = kvmppc_xive_xics_hcall(vcpu, req);
> + if (ret != H_TOO_HARD) {
> + kvmppc_set_gpr(vcpu, 3, ret);
> + trap = 0;
> + }
> + }
> + }
> kvmppc_xive_pull_vcpu(vcpu);
> -
> }
>
> vcpu->arch.slb_max = 0;
> @@ -4442,8 +4476,11 @@ static int kvmppc_vcpu_run_hv(struct kvm_vcpu *vcpu)
> else
> r = kvmppc_run_vcpu(vcpu);
>
> - if (run->exit_reason == KVM_EXIT_PAPR_HCALL &&
> - !(vcpu->arch.shregs.msr & MSR_PR)) {
> + if (run->exit_reason == KVM_EXIT_PAPR_HCALL) {
> + if (WARN_ON_ONCE(vcpu->arch.shregs.msr & MSR_PR)) {
> + r = RESUME_GUEST;
> + continue;
> + }
Note that this hunk might need to be dropped.
> trace_kvm_hcall_enter(vcpu);
> r = kvmppc_pseries_do_hcall(vcpu);
> trace_kvm_hcall_exit(vcpu, r);
> diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
> index c11597f815e4..2d0d14ed1d92 100644
> --- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S
> +++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
> @@ -1397,9 +1397,14 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
> mr r4,r9
> bge fast_guest_return
> 2:
> + /* If we came in through the P9 short path, no real mode hcalls */
> + lwz r0, STACK_SLOT_SHORT_PATH(r1)
> + cmpwi r0, 0
> + bne no_try_real
> /* See if this is an hcall we can handle in real mode */
> cmpwi r12,BOOK3S_INTERRUPT_SYSCALL
> beq hcall_try_real_mode
> +no_try_real:
>
> /* Hypervisor doorbell - exit only if host IPI flag set */
> cmpwi r12, BOOK3S_INTERRUPT_H_DOORBELL
^ permalink raw reply
* Re: [PATCH v2 -next] powerpc: kernel/time.c - cleanup warnings
From: Alexandre Belloni @ 2021-03-23 22:18 UTC (permalink / raw)
To: He Ying
Cc: linux-rtc, a.zummo, geert+renesas, peterz, frederic, linux-kernel,
npiggin, paulus, kernelfans, tglx, msuchanek, linuxppc-dev
In-Reply-To: <20210323091257.90054-1-heying24@huawei.com>
Hello,
On 23/03/2021 05:12:57-0400, He Ying wrote:
> We found these warnings in arch/powerpc/kernel/time.c as follows:
> warning: symbol 'decrementer_max' was not declared. Should it be static?
> warning: symbol 'rtc_lock' was not declared. Should it be static?
> warning: symbol 'dtl_consumer' was not declared. Should it be static?
>
> Declare 'decrementer_max' and 'rtc_lock' in powerpc asm/time.h.
> Rename 'rtc_lock' in drviers/rtc/rtc-vr41xx.c to 'vr41xx_rtc_lock' to
> avoid the conflict with the variable in powerpc asm/time.h.
> Move 'dtl_consumer' definition behind "include <asm/dtl.h>" because it
> is declared there.
>
> Reported-by: Hulk Robot <hulkci@huawei.com>
> Signed-off-by: He Ying <heying24@huawei.com>
> ---
> v2:
> - Instead of including linux/mc146818rtc.h in powerpc kernel/time.c, declare
> rtc_lock in powerpc asm/time.h.
>
V1 was actually the correct thing to do. rtc_lock is there exactly
because chrp and maple are using mc146818 compatible RTCs. This is then
useful because then drivers/char/nvram.c is enabled. The proper fix
would be to scrap all of that and use rtc-cmos for those platforms as
this drives the RTC properly and exposes the NVRAM for the mc146818.
Or at least, if there are no users for the char/nvram driver on those
two platforms, remove the spinlock and stop enabling CONFIG_NVRAM or
more likely rename the symbol as it seems to be abused by both chrp and
powermac.
I'm not completely against the rename in vr41xxx but the fix for the
warnings can and should be contained in arch/powerpc.
> arch/powerpc/include/asm/time.h | 3 +++
> arch/powerpc/kernel/time.c | 6 ++----
> drivers/rtc/rtc-vr41xx.c | 22 +++++++++++-----------
> 3 files changed, 16 insertions(+), 15 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/time.h b/arch/powerpc/include/asm/time.h
> index 8dd3cdb25338..64a3ef0b4270 100644
> --- a/arch/powerpc/include/asm/time.h
> +++ b/arch/powerpc/include/asm/time.h
> @@ -12,6 +12,7 @@
> #ifdef __KERNEL__
> #include <linux/types.h>
> #include <linux/percpu.h>
> +#include <linux/spinlock.h>
>
> #include <asm/processor.h>
> #include <asm/cpu_has_feature.h>
> @@ -22,6 +23,8 @@ extern unsigned long tb_ticks_per_jiffy;
> extern unsigned long tb_ticks_per_usec;
> extern unsigned long tb_ticks_per_sec;
> extern struct clock_event_device decrementer_clockevent;
> +extern u64 decrementer_max;
> +extern spinlock_t rtc_lock;
>
>
> extern void generic_calibrate_decr(void);
> diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c
> index b67d93a609a2..60b6ac7d3685 100644
> --- a/arch/powerpc/kernel/time.c
> +++ b/arch/powerpc/kernel/time.c
> @@ -150,10 +150,6 @@ bool tb_invalid;
> u64 __cputime_usec_factor;
> EXPORT_SYMBOL(__cputime_usec_factor);
>
> -#ifdef CONFIG_PPC_SPLPAR
> -void (*dtl_consumer)(struct dtl_entry *, u64);
> -#endif
> -
> static void calc_cputime_factors(void)
> {
> struct div_result res;
> @@ -179,6 +175,8 @@ static inline unsigned long read_spurr(unsigned long tb)
>
> #include <asm/dtl.h>
>
> +void (*dtl_consumer)(struct dtl_entry *, u64);
> +
> /*
> * Scan the dispatch trace log and count up the stolen time.
> * Should be called with interrupts disabled.
> diff --git a/drivers/rtc/rtc-vr41xx.c b/drivers/rtc/rtc-vr41xx.c
> index 5a9f9ad86d32..cc31db058197 100644
> --- a/drivers/rtc/rtc-vr41xx.c
> +++ b/drivers/rtc/rtc-vr41xx.c
> @@ -72,7 +72,7 @@ static void __iomem *rtc2_base;
>
> static unsigned long epoch = 1970; /* Jan 1 1970 00:00:00 */
>
> -static DEFINE_SPINLOCK(rtc_lock);
> +static DEFINE_SPINLOCK(vr41xx_rtc_lock);
> static char rtc_name[] = "RTC";
> static unsigned long periodic_count;
> static unsigned int alarm_enabled;
> @@ -101,13 +101,13 @@ static inline time64_t read_elapsed_second(void)
>
> static inline void write_elapsed_second(time64_t sec)
> {
> - spin_lock_irq(&rtc_lock);
> + spin_lock_irq(&vr41xx_rtc_lock);
>
> rtc1_write(ETIMELREG, (uint16_t)(sec << 15));
> rtc1_write(ETIMEMREG, (uint16_t)(sec >> 1));
> rtc1_write(ETIMEHREG, (uint16_t)(sec >> 17));
>
> - spin_unlock_irq(&rtc_lock);
> + spin_unlock_irq(&vr41xx_rtc_lock);
> }
>
> static int vr41xx_rtc_read_time(struct device *dev, struct rtc_time *time)
> @@ -139,14 +139,14 @@ static int vr41xx_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *wkalrm)
> unsigned long low, mid, high;
> struct rtc_time *time = &wkalrm->time;
>
> - spin_lock_irq(&rtc_lock);
> + spin_lock_irq(&vr41xx_rtc_lock);
>
> low = rtc1_read(ECMPLREG);
> mid = rtc1_read(ECMPMREG);
> high = rtc1_read(ECMPHREG);
> wkalrm->enabled = alarm_enabled;
>
> - spin_unlock_irq(&rtc_lock);
> + spin_unlock_irq(&vr41xx_rtc_lock);
>
> rtc_time64_to_tm((high << 17) | (mid << 1) | (low >> 15), time);
>
> @@ -159,7 +159,7 @@ static int vr41xx_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *wkalrm)
>
> alarm_sec = rtc_tm_to_time64(&wkalrm->time);
>
> - spin_lock_irq(&rtc_lock);
> + spin_lock_irq(&vr41xx_rtc_lock);
>
> if (alarm_enabled)
> disable_irq(aie_irq);
> @@ -173,7 +173,7 @@ static int vr41xx_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *wkalrm)
>
> alarm_enabled = wkalrm->enabled;
>
> - spin_unlock_irq(&rtc_lock);
> + spin_unlock_irq(&vr41xx_rtc_lock);
>
> return 0;
> }
> @@ -202,7 +202,7 @@ static int vr41xx_rtc_ioctl(struct device *dev, unsigned int cmd, unsigned long
>
> static int vr41xx_rtc_alarm_irq_enable(struct device *dev, unsigned int enabled)
> {
> - spin_lock_irq(&rtc_lock);
> + spin_lock_irq(&vr41xx_rtc_lock);
> if (enabled) {
> if (!alarm_enabled) {
> enable_irq(aie_irq);
> @@ -214,7 +214,7 @@ static int vr41xx_rtc_alarm_irq_enable(struct device *dev, unsigned int enabled)
> alarm_enabled = 0;
> }
> }
> - spin_unlock_irq(&rtc_lock);
> + spin_unlock_irq(&vr41xx_rtc_lock);
> return 0;
> }
>
> @@ -296,7 +296,7 @@ static int rtc_probe(struct platform_device *pdev)
> rtc->range_max = (1ULL << 33) - 1;
> rtc->max_user_freq = MAX_PERIODIC_RATE;
>
> - spin_lock_irq(&rtc_lock);
> + spin_lock_irq(&vr41xx_rtc_lock);
>
> rtc1_write(ECMPLREG, 0);
> rtc1_write(ECMPMREG, 0);
> @@ -304,7 +304,7 @@ static int rtc_probe(struct platform_device *pdev)
> rtc1_write(RTCL1LREG, 0);
> rtc1_write(RTCL1HREG, 0);
>
> - spin_unlock_irq(&rtc_lock);
> + spin_unlock_irq(&vr41xx_rtc_lock);
>
> aie_irq = platform_get_irq(pdev, 0);
> if (aie_irq <= 0) {
> --
> 2.17.1
>
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply
* Re: [PATCH] macintosh: A typo fix
From: Randy Dunlap @ 2021-03-23 22:01 UTC (permalink / raw)
To: Bhaskar Chowdhury, benh, linuxppc-dev, linux-kernel
In-Reply-To: <20210323204652.23059-1-unixbhaskar@gmail.com>
On 3/23/21 1:46 PM, Bhaskar Chowdhury wrote:
>
> s/coment/comment/
>
> Signed-off-by: Bhaskar Chowdhury <unixbhaskar@gmail.com>
Acked-by: Randy Dunlap <rdunlap@infradead.org>
> ---
> drivers/macintosh/windfarm_smu_controls.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/macintosh/windfarm_smu_controls.c b/drivers/macintosh/windfarm_smu_controls.c
> index 79cb1ad09bfd..75966052819a 100644
> --- a/drivers/macintosh/windfarm_smu_controls.c
> +++ b/drivers/macintosh/windfarm_smu_controls.c
> @@ -94,7 +94,7 @@ static int smu_set_fan(int pwm, u8 id, u16 value)
> return rc;
> wait_for_completion(&comp);
>
> - /* Handle fallback (see coment above) */
> + /* Handle fallback (see comment above) */
> if (cmd.status != 0 && smu_supports_new_fans_ops) {
> printk(KERN_WARNING "windfarm: SMU failed new fan command "
> "falling back to old method\n");
> --
--
~Randy
^ permalink raw reply
* [PATCH v2 1/1] hotplug-cpu.c: show 'last online CPU' error in dlpar_cpu_offline()
From: Daniel Henrique Barboza @ 2021-03-23 20:50 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Daniel Henrique Barboza
In-Reply-To: <20210323205056.52768-1-danielhb413@gmail.com>
One of the reasons that dlpar_cpu_offline can fail is when attempting to
offline the last online CPU of the kernel. This can be observed in a
pseries QEMU guest that has hotplugged CPUs. If the user offlines all
other CPUs of the guest, and a hotplugged CPU is now the last online
CPU, trying to reclaim it will fail. See [1] for an example.
The current error message in this situation returns rc with -EBUSY and a
generic explanation, e.g.:
pseries-hotplug-cpu: Failed to offline CPU PowerPC,POWER9, rc: -16
EBUSY can be caused by other conditions, such as cpu_hotplug_disable
being true. Throwing a more specific error message for this case,
instead of just "Failed to offline CPU", makes it clearer that the error
is in fact a known error situation instead of other generic/unknown
cause.
This patch adds a 'last online' check in dlpar_cpu_offline() to catch
the 'last online CPU' offline error, eturning a more informative error
message:
pseries-hotplug-cpu: Unable to remove last online CPU PowerPC,POWER9
[1] https://bugzilla.redhat.com/1911414
Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com>
---
arch/powerpc/platforms/pseries/hotplug-cpu.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/arch/powerpc/platforms/pseries/hotplug-cpu.c b/arch/powerpc/platforms/pseries/hotplug-cpu.c
index 12cbffd3c2e3..3ac7e904385c 100644
--- a/arch/powerpc/platforms/pseries/hotplug-cpu.c
+++ b/arch/powerpc/platforms/pseries/hotplug-cpu.c
@@ -271,6 +271,18 @@ static int dlpar_offline_cpu(struct device_node *dn)
if (!cpu_online(cpu))
break;
+ /* device_offline() will return -EBUSY (via cpu_down())
+ * if there is only one CPU left. Check it here to fail
+ * earlier and with a more informative error message,
+ * while also retaining the cpu_add_remove_lock to be sure
+ * that no CPUs are being online/offlined during this
+ * check. */
+ if (num_online_cpus() == 1) {
+ pr_warn("Unable to remove last online CPU %pOFn\n", dn);
+ rc = -EBUSY;
+ goto out_unlock;
+ }
+
cpu_maps_update_done();
rc = device_offline(get_cpu_device(cpu));
if (rc)
@@ -283,6 +295,7 @@ static int dlpar_offline_cpu(struct device_node *dn)
thread);
}
}
+out_unlock:
cpu_maps_update_done();
out:
--
2.30.2
^ permalink raw reply related
* [PATCH v2 0/1] show 'last online CPU' error in dlpar_cpu_offline()
From: Daniel Henrique Barboza @ 2021-03-23 20:50 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Daniel Henrique Barboza
changes in v2 after Michael Ellerman review:
- moved the verification code from dlpar_cpu_remove() to
dlpar_cpu_offline(), while holding cpu_add_remove_lock
- reworded the commit message and code comment
v1 link:
https://patchwork.ozlabs.org/project/linuxppc-dev/patch/20210305173845.451158-1-danielhb413@gmail.com/
Daniel Henrique Barboza (1):
hotplug-cpu.c: show 'last online CPU' error in dlpar_cpu_offline()
arch/powerpc/platforms/pseries/hotplug-cpu.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
--
2.30.2
^ permalink raw reply
* [PATCH] macintosh: A typo fix
From: Bhaskar Chowdhury @ 2021-03-23 20:46 UTC (permalink / raw)
To: benh, unixbhaskar, linuxppc-dev, linux-kernel; +Cc: rdunlap
s/coment/comment/
Signed-off-by: Bhaskar Chowdhury <unixbhaskar@gmail.com>
---
drivers/macintosh/windfarm_smu_controls.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/macintosh/windfarm_smu_controls.c b/drivers/macintosh/windfarm_smu_controls.c
index 79cb1ad09bfd..75966052819a 100644
--- a/drivers/macintosh/windfarm_smu_controls.c
+++ b/drivers/macintosh/windfarm_smu_controls.c
@@ -94,7 +94,7 @@ static int smu_set_fan(int pwm, u8 id, u16 value)
return rc;
wait_for_completion(&comp);
- /* Handle fallback (see coment above) */
+ /* Handle fallback (see comment above) */
if (cmd.status != 0 && smu_supports_new_fans_ops) {
printk(KERN_WARNING "windfarm: SMU failed new fan command "
"falling back to old method\n");
--
2.30.1
^ permalink raw reply related
* Re: [PATCH 02/10] ARM: disable CONFIG_IDE in footbridge_defconfig
From: Russell King - ARM Linux admin @ 2021-03-23 18:43 UTC (permalink / raw)
To: Cye Borg
Cc: Jens Axboe, Thomas Bogendoerfer, linux-doc, linux-alpha,
linux-kernel, Christoph Hellwig, linux-ide, linux-m68k,
Geert Uytterhoeven, John Paul Adrian Glaubitz, Ivan Kokshaysky,
Matt Turner, linux-mips, linuxppc-dev, David S. Miller,
linux-arm-kernel, Richard Henderson
In-Reply-To: <CAD4NMuZWoV0m85OyBDHLt+J8NYCV5wYx7fFZaivBNEgDnrN5xw@mail.gmail.com>
On Mon, Mar 22, 2021 at 06:10:01PM +0100, Cye Borg wrote:
> PWS 500au:
>
> snow / # lspci -vvx -s 7.1
> 00:07.1 IDE interface: Contaq Microsystems 82c693 (prog-if 80 [ISA
> Compatibility mode-only controller, supports bus mastering])
> Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr+ Stepping- SERR- FastB2B- DisINTx-
> Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
> >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0
> Interrupt: pin A routed to IRQ 0
> Region 0: I/O ports at 01f0 [size=8]
> Region 1: I/O ports at 03f4
> Region 4: I/O ports at 9080 [size=16]
> Kernel driver in use: pata_cypress
> Kernel modules: pata_cypress
> 00: 80 10 93 c6 45 00 80 02 00 80 01 01 00 00 80 00
> 10: f1 01 00 00 f5 03 00 00 00 00 00 00 00 00 00 00
> 20: 81 90 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00
>
> snow / # lspci -vvx -s 7.2
> 00:07.2 IDE interface: Contaq Microsystems 82c693 (prog-if 00 [ISA
> Compatibility mode-only controller])
> Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr+ Stepping- SERR- FastB2B- DisINTx-
> Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium
> >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0
> Interrupt: pin B routed to IRQ 0
> Region 0: I/O ports at 0170 [size=8]
> Region 1: I/O ports at 0374
> Region 4: Memory at 0c240000 (32-bit, non-prefetchable)
> [disabled] [size=64K]
> Kernel modules: pata_cypress
> 00: 80 10 93 c6 45 00 80 02 00 00 01 01 00 00 80 00
> 10: 71 01 00 00 75 03 00 00 00 00 00 00 00 00 00 00
> 20: 00 00 24 0c 00 00 00 00 00 00 00 00 00 00 00 00
> 30: 00 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00
Thanks very much.
Could I also ask for the output of:
# lspci -vxxx -s 7.0
as well please - this will dump all 256 bytes for the ISA bridge, which
contains a bunch of configuration registers. Thanks.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
^ permalink raw reply
* Re: [PATCH v4 02/46] KVM: PPC: Book3S HV: Add a function to filter guest LPCR bits
From: Fabiano Rosas @ 2021-03-23 18:17 UTC (permalink / raw)
To: Nicholas Piggin, kvm-ppc; +Cc: linuxppc-dev, Nicholas Piggin
In-Reply-To: <20210323010305.1045293-3-npiggin@gmail.com>
Nicholas Piggin <npiggin@gmail.com> writes:
> Guest LPCR depends on hardware type, and future changes will add
> restrictions based on errata and guest MMU mode. Move this logic
> to a common function and use it for the cases where the guest
> wants to update its LPCR (or the LPCR of a nested guest).
>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Fabiano Rosas <farosas@linux.ibm.com>
> ---
> arch/powerpc/include/asm/kvm_book3s.h | 2 +
> arch/powerpc/kvm/book3s_hv.c | 60 ++++++++++++++++++---------
> arch/powerpc/kvm/book3s_hv_nested.c | 3 +-
> 3 files changed, 45 insertions(+), 20 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/kvm_book3s.h b/arch/powerpc/include/asm/kvm_book3s.h
> index 2f5f919f6cd3..3eec3ef6f083 100644
> --- a/arch/powerpc/include/asm/kvm_book3s.h
> +++ b/arch/powerpc/include/asm/kvm_book3s.h
> @@ -258,6 +258,8 @@ extern long kvmppc_hv_get_dirty_log_hpt(struct kvm *kvm,
> extern void kvmppc_harvest_vpa_dirty(struct kvmppc_vpa *vpa,
> struct kvm_memory_slot *memslot,
> unsigned long *map);
> +extern unsigned long kvmppc_filter_lpcr_hv(struct kvmppc_vcore *vc,
> + unsigned long lpcr);
> extern void kvmppc_update_lpcr(struct kvm *kvm, unsigned long lpcr,
> unsigned long mask);
> extern void kvmppc_set_fscr(struct kvm_vcpu *vcpu, u64 fscr);
> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
> index 13bad6bf4c95..c4539c38c639 100644
> --- a/arch/powerpc/kvm/book3s_hv.c
> +++ b/arch/powerpc/kvm/book3s_hv.c
> @@ -1635,6 +1635,27 @@ static int kvm_arch_vcpu_ioctl_set_sregs_hv(struct kvm_vcpu *vcpu,
> return 0;
> }
>
> +/*
> + * Enforce limits on guest LPCR values based on hardware availability,
> + * guest configuration, and possibly hypervisor support and security
> + * concerns.
> + */
> +unsigned long kvmppc_filter_lpcr_hv(struct kvmppc_vcore *vc, unsigned long lpcr)
> +{
> + /* On POWER8 and above, userspace can modify AIL */
> + if (!cpu_has_feature(CPU_FTR_ARCH_207S))
> + lpcr &= ~LPCR_AIL;
> +
> + /*
> + * On POWER9, allow userspace to enable large decrementer for the
> + * guest, whether or not the host has it enabled.
> + */
> + if (!cpu_has_feature(CPU_FTR_ARCH_300))
> + lpcr &= ~LPCR_LD;
> +
> + return lpcr;
> +}
> +
> static void kvmppc_set_lpcr(struct kvm_vcpu *vcpu, u64 new_lpcr,
> bool preserve_top32)
> {
> @@ -1643,6 +1664,23 @@ static void kvmppc_set_lpcr(struct kvm_vcpu *vcpu, u64 new_lpcr,
> u64 mask;
>
> spin_lock(&vc->lock);
> +
> + /*
> + * Userspace can only modify
> + * DPFD (default prefetch depth), ILE (interrupt little-endian),
> + * TC (translation control), AIL (alternate interrupt location),
> + * LD (large decrementer).
> + * These are subject to restrictions from kvmppc_filter_lcpr_hv().
> + */
> + mask = LPCR_DPFD | LPCR_ILE | LPCR_TC | LPCR_AIL | LPCR_LD;
> +
> + /* Broken 32-bit version of LPCR must not clear top bits */
> + if (preserve_top32)
> + mask &= 0xFFFFFFFF;
> +
> + new_lpcr = kvmppc_filter_lpcr_hv(vc,
> + (vc->lpcr & ~mask) | (new_lpcr & mask));
> +
> /*
> * If ILE (interrupt little-endian) has changed, update the
> * MSR_LE bit in the intr_msr for each vcpu in this vcore.
> @@ -1661,25 +1699,8 @@ static void kvmppc_set_lpcr(struct kvm_vcpu *vcpu, u64 new_lpcr,
> }
> }
>
> - /*
> - * Userspace can only modify DPFD (default prefetch depth),
> - * ILE (interrupt little-endian) and TC (translation control).
> - * On POWER8 and POWER9 userspace can also modify AIL (alt. interrupt loc.).
> - */
> - mask = LPCR_DPFD | LPCR_ILE | LPCR_TC;
> - if (cpu_has_feature(CPU_FTR_ARCH_207S))
> - mask |= LPCR_AIL;
> - /*
> - * On POWER9, allow userspace to enable large decrementer for the
> - * guest, whether or not the host has it enabled.
> - */
> - if (cpu_has_feature(CPU_FTR_ARCH_300))
> - mask |= LPCR_LD;
> + vc->lpcr = new_lpcr;
>
> - /* Broken 32-bit version of LPCR must not clear top bits */
> - if (preserve_top32)
> - mask &= 0xFFFFFFFF;
> - vc->lpcr = (vc->lpcr & ~mask) | (new_lpcr & mask);
> spin_unlock(&vc->lock);
> }
>
> @@ -4641,8 +4662,9 @@ void kvmppc_update_lpcr(struct kvm *kvm, unsigned long lpcr, unsigned long mask)
> struct kvmppc_vcore *vc = kvm->arch.vcores[i];
> if (!vc)
> continue;
> +
> spin_lock(&vc->lock);
> - vc->lpcr = (vc->lpcr & ~mask) | lpcr;
> + vc->lpcr = kvmppc_filter_lpcr_hv(vc, (vc->lpcr & ~mask) | lpcr);
> spin_unlock(&vc->lock);
> if (++cores_done >= kvm->arch.online_vcores)
> break;
> diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_hv_nested.c
> index 2fe1fea4c934..f7b441b3eb17 100644
> --- a/arch/powerpc/kvm/book3s_hv_nested.c
> +++ b/arch/powerpc/kvm/book3s_hv_nested.c
> @@ -142,7 +142,8 @@ static void sanitise_hv_regs(struct kvm_vcpu *vcpu, struct hv_guest_state *hr)
> */
> mask = LPCR_DPFD | LPCR_ILE | LPCR_TC | LPCR_AIL | LPCR_LD |
> LPCR_LPES | LPCR_MER;
> - hr->lpcr = (vc->lpcr & ~mask) | (hr->lpcr & mask);
> + hr->lpcr = kvmppc_filter_lpcr_hv(vc,
> + (vc->lpcr & ~mask) | (hr->lpcr & mask));
>
> /*
> * Don't let L1 enable features for L2 which we've disabled for L1,
^ permalink raw reply
* Re: [PATCH v4 01/46] KVM: PPC: Book3S HV: Nested move LPCR sanitising to sanitise_hv_regs
From: Fabiano Rosas @ 2021-03-23 18:08 UTC (permalink / raw)
To: Nicholas Piggin, kvm-ppc; +Cc: linuxppc-dev, Nicholas Piggin
In-Reply-To: <20210323010305.1045293-2-npiggin@gmail.com>
Nicholas Piggin <npiggin@gmail.com> writes:
> This will get a bit more complicated in future patches. Move it
> into the helper function.
>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Reviewed-by: Fabiano Rosas <farosas@linux.ibm.com>
> ---
> arch/powerpc/kvm/book3s_hv_nested.c | 18 ++++++++++++------
> 1 file changed, 12 insertions(+), 6 deletions(-)
>
> diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_hv_nested.c
> index 0cd0e7aad588..2fe1fea4c934 100644
> --- a/arch/powerpc/kvm/book3s_hv_nested.c
> +++ b/arch/powerpc/kvm/book3s_hv_nested.c
> @@ -134,6 +134,16 @@ static void save_hv_return_state(struct kvm_vcpu *vcpu, int trap,
>
> static void sanitise_hv_regs(struct kvm_vcpu *vcpu, struct hv_guest_state *hr)
> {
> + struct kvmppc_vcore *vc = vcpu->arch.vcore;
> + u64 mask;
> +
> + /*
> + * Don't let L1 change LPCR bits for the L2 except these:
> + */
> + mask = LPCR_DPFD | LPCR_ILE | LPCR_TC | LPCR_AIL | LPCR_LD |
> + LPCR_LPES | LPCR_MER;
> + hr->lpcr = (vc->lpcr & ~mask) | (hr->lpcr & mask);
> +
> /*
> * Don't let L1 enable features for L2 which we've disabled for L1,
> * but preserve the interrupt cause field.
> @@ -271,8 +281,6 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu)
> u64 hv_ptr, regs_ptr;
> u64 hdec_exp;
> s64 delta_purr, delta_spurr, delta_ic, delta_vtb;
> - u64 mask;
> - unsigned long lpcr;
>
> if (vcpu->kvm->arch.l1_ptcr == 0)
> return H_NOT_AVAILABLE;
> @@ -321,9 +329,7 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu)
> vcpu->arch.nested_vcpu_id = l2_hv.vcpu_token;
> vcpu->arch.regs = l2_regs;
> vcpu->arch.shregs.msr = vcpu->arch.regs.msr;
> - mask = LPCR_DPFD | LPCR_ILE | LPCR_TC | LPCR_AIL | LPCR_LD |
> - LPCR_LPES | LPCR_MER;
> - lpcr = (vc->lpcr & ~mask) | (l2_hv.lpcr & mask);
> +
> sanitise_hv_regs(vcpu, &l2_hv);
> restore_hv_regs(vcpu, &l2_hv);
>
> @@ -335,7 +341,7 @@ long kvmhv_enter_nested_guest(struct kvm_vcpu *vcpu)
> r = RESUME_HOST;
> break;
> }
> - r = kvmhv_run_single_vcpu(vcpu, hdec_exp, lpcr);
> + r = kvmhv_run_single_vcpu(vcpu, hdec_exp, l2_hv.lpcr);
> } while (is_kvmppc_resume_guest(r));
>
> /* save L2 state for return */
^ permalink raw reply
* Re: [PATCH v4 22/46] KVM: PPC: Book3S HV P9: Stop handling hcalls in real-mode in the P9 path
From: Fabiano Rosas @ 2021-03-23 18:03 UTC (permalink / raw)
To: Nicholas Piggin, kvm-ppc; +Cc: linuxppc-dev, Nicholas Piggin
In-Reply-To: <20210323010305.1045293-23-npiggin@gmail.com>
Nicholas Piggin <npiggin@gmail.com> writes:
> In the interest of minimising the amount of code that is run in
> "real-mode", don't handle hcalls in real mode in the P9 path.
>
> POWER8 and earlier are much more expensive to exit from HV real mode
> and switch to host mode, because on those processors HV interrupts get
> to the hypervisor with the MMU off, and the other threads in the core
> need to be pulled out of the guest, and SLBs all need to be saved,
> ERATs invalidated, and host SLB reloaded before the MMU is re-enabled
> in host mode. Hash guests also require a lot of hcalls to run. The
> XICS interrupt controller requires hcalls to run.
>
> By contrast, POWER9 has independent thread switching, and in radix mode
> the hypervisor is already in a host virtual memory mode when the HV
> interrupt is taken. Radix + xive guests don't need hcalls to handle
> interrupts or manage translations.
>
> So it's much less important to handle hcalls in real mode in P9.
>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
I tried this again in the L2 with xive=off and it works as expected now.
Tested-by: Fabiano Rosas <farosas@linux.ibm.com>
> ---
> arch/powerpc/include/asm/kvm_ppc.h | 5 ++
> arch/powerpc/kvm/book3s_hv.c | 57 ++++++++++++++++----
> arch/powerpc/kvm/book3s_hv_rmhandlers.S | 5 ++
> arch/powerpc/kvm/book3s_xive.c | 70 +++++++++++++++++++++++++
> 4 files changed, 127 insertions(+), 10 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
> index 73b1ca5a6471..db6646c2ade2 100644
> --- a/arch/powerpc/include/asm/kvm_ppc.h
> +++ b/arch/powerpc/include/asm/kvm_ppc.h
> @@ -607,6 +607,7 @@ extern void kvmppc_free_pimap(struct kvm *kvm);
> extern int kvmppc_xics_rm_complete(struct kvm_vcpu *vcpu, u32 hcall);
> extern void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu);
> extern int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 cmd);
> +extern int kvmppc_xive_xics_hcall(struct kvm_vcpu *vcpu, u32 req);
> extern u64 kvmppc_xics_get_icp(struct kvm_vcpu *vcpu);
> extern int kvmppc_xics_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
> extern int kvmppc_xics_connect_vcpu(struct kvm_device *dev,
> @@ -639,6 +640,8 @@ static inline int kvmppc_xics_enabled(struct kvm_vcpu *vcpu)
> static inline void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu) { }
> static inline int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 cmd)
> { return 0; }
> +static inline int kvmppc_xive_xics_hcall(struct kvm_vcpu *vcpu, u32 req)
> + { return 0; }
> #endif
>
> #ifdef CONFIG_KVM_XIVE
> @@ -673,6 +676,7 @@ extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
> int level, bool line_status);
> extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
> extern void kvmppc_xive_pull_vcpu(struct kvm_vcpu *vcpu);
> +extern void kvmppc_xive_cede_vcpu(struct kvm_vcpu *vcpu);
>
> static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
> {
> @@ -714,6 +718,7 @@ static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 ir
> int level, bool line_status) { return -ENODEV; }
> static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
> static inline void kvmppc_xive_pull_vcpu(struct kvm_vcpu *vcpu) { }
> +static inline void kvmppc_xive_cede_vcpu(struct kvm_vcpu *vcpu) { }
>
> static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
> { return 0; }
> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
> index fa7614c37e08..17739aaee3d8 100644
> --- a/arch/powerpc/kvm/book3s_hv.c
> +++ b/arch/powerpc/kvm/book3s_hv.c
> @@ -1142,12 +1142,13 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)
> }
>
> /*
> - * Handle H_CEDE in the nested virtualization case where we haven't
> - * called the real-mode hcall handlers in book3s_hv_rmhandlers.S.
> + * Handle H_CEDE in the P9 path where we don't call the real-mode hcall
> + * handlers in book3s_hv_rmhandlers.S.
> + *
> * This has to be done early, not in kvmppc_pseries_do_hcall(), so
> * that the cede logic in kvmppc_run_single_vcpu() works properly.
> */
> -static void kvmppc_nested_cede(struct kvm_vcpu *vcpu)
> +static void kvmppc_cede(struct kvm_vcpu *vcpu)
> {
> vcpu->arch.shregs.msr |= MSR_EE;
> vcpu->arch.ceded = 1;
> @@ -1403,9 +1404,15 @@ static int kvmppc_handle_exit_hv(struct kvm_vcpu *vcpu,
> /* hcall - punt to userspace */
> int i;
>
> - /* hypercall with MSR_PR has already been handled in rmode,
> - * and never reaches here.
> - */
> + if (unlikely(vcpu->arch.shregs.msr & MSR_PR)) {
> + /*
> + * Guest userspace executed sc 1, reflect it back as a
> + * privileged program check interrupt.
> + */
> + kvmppc_core_queue_program(vcpu, SRR1_PROGPRIV);
> + r = RESUME_GUEST;
> + break;
> + }
>
> run->papr_hcall.nr = kvmppc_get_gpr(vcpu, 3);
> for (i = 0; i < 9; ++i)
> @@ -3663,6 +3670,12 @@ static int kvmhv_load_hv_regs_and_go(struct kvm_vcpu *vcpu, u64 time_limit,
> return trap;
> }
>
> +static inline bool hcall_is_xics(unsigned long req)
> +{
> + return (req == H_EOI || req == H_CPPR || req == H_IPI ||
> + req == H_IPOLL || req == H_XIRR || req == H_XIRR_X);
> +}
> +
> /*
> * Virtual-mode guest entry for POWER9 and later when the host and
> * guest are both using the radix MMU. The LPIDR has already been set.
> @@ -3774,15 +3787,36 @@ static int kvmhv_p9_guest_entry(struct kvm_vcpu *vcpu, u64 time_limit,
> /* H_CEDE has to be handled now, not later */
> if (trap == BOOK3S_INTERRUPT_SYSCALL && !vcpu->arch.nested &&
> kvmppc_get_gpr(vcpu, 3) == H_CEDE) {
> - kvmppc_nested_cede(vcpu);
> + kvmppc_cede(vcpu);
> kvmppc_set_gpr(vcpu, 3, 0);
> trap = 0;
> }
> } else {
> kvmppc_xive_push_vcpu(vcpu);
> trap = kvmhv_load_hv_regs_and_go(vcpu, time_limit, lpcr);
> + if (trap == BOOK3S_INTERRUPT_SYSCALL && !vcpu->arch.nested &&
> + !(vcpu->arch.shregs.msr & MSR_PR)) {
> + unsigned long req = kvmppc_get_gpr(vcpu, 3);
> +
> + /* H_CEDE has to be handled now, not later */
> + if (req == H_CEDE) {
> + kvmppc_cede(vcpu);
> + kvmppc_xive_cede_vcpu(vcpu); /* may un-cede */
> + kvmppc_set_gpr(vcpu, 3, 0);
> + trap = 0;
> +
> + /* XICS hcalls must be handled before xive is pulled */
> + } else if (hcall_is_xics(req)) {
> + int ret;
> +
> + ret = kvmppc_xive_xics_hcall(vcpu, req);
> + if (ret != H_TOO_HARD) {
> + kvmppc_set_gpr(vcpu, 3, ret);
> + trap = 0;
> + }
> + }
> + }
> kvmppc_xive_pull_vcpu(vcpu);
> -
> }
>
> vcpu->arch.slb_max = 0;
> @@ -4442,8 +4476,11 @@ static int kvmppc_vcpu_run_hv(struct kvm_vcpu *vcpu)
> else
> r = kvmppc_run_vcpu(vcpu);
>
> - if (run->exit_reason == KVM_EXIT_PAPR_HCALL &&
> - !(vcpu->arch.shregs.msr & MSR_PR)) {
> + if (run->exit_reason == KVM_EXIT_PAPR_HCALL) {
> + if (WARN_ON_ONCE(vcpu->arch.shregs.msr & MSR_PR)) {
> + r = RESUME_GUEST;
> + continue;
> + }
> trace_kvm_hcall_enter(vcpu);
> r = kvmppc_pseries_do_hcall(vcpu);
> trace_kvm_hcall_exit(vcpu, r);
> diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
> index c11597f815e4..2d0d14ed1d92 100644
> --- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S
> +++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
> @@ -1397,9 +1397,14 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
> mr r4,r9
> bge fast_guest_return
> 2:
> + /* If we came in through the P9 short path, no real mode hcalls */
> + lwz r0, STACK_SLOT_SHORT_PATH(r1)
> + cmpwi r0, 0
> + bne no_try_real
> /* See if this is an hcall we can handle in real mode */
> cmpwi r12,BOOK3S_INTERRUPT_SYSCALL
> beq hcall_try_real_mode
> +no_try_real:
>
> /* Hypervisor doorbell - exit only if host IPI flag set */
> cmpwi r12, BOOK3S_INTERRUPT_H_DOORBELL
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index 741bf1f4387a..dcc07ceaf5ca 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -158,6 +158,40 @@ void kvmppc_xive_pull_vcpu(struct kvm_vcpu *vcpu)
> }
> EXPORT_SYMBOL_GPL(kvmppc_xive_pull_vcpu);
>
> +void kvmppc_xive_cede_vcpu(struct kvm_vcpu *vcpu)
> +{
> + void __iomem *esc_vaddr = (void __iomem *)vcpu->arch.xive_esc_vaddr;
> +
> + if (!esc_vaddr)
> + return;
> +
> + /* we are using XIVE with single escalation */
> +
> + if (vcpu->arch.xive_esc_on) {
> + /*
> + * If we still have a pending escalation, abort the cede,
> + * and we must set PQ to 10 rather than 00 so that we don't
> + * potentially end up with two entries for the escalation
> + * interrupt in the XIVE interrupt queue. In that case
> + * we also don't want to set xive_esc_on to 1 here in
> + * case we race with xive_esc_irq().
> + */
> + vcpu->arch.ceded = 0;
> + /*
> + * The escalation interrupts are special as we don't EOI them.
> + * There is no need to use the load-after-store ordering offset
> + * to set PQ to 10 as we won't use StoreEOI.
> + */
> + __raw_readq(esc_vaddr + XIVE_ESB_SET_PQ_10);
> + } else {
> + vcpu->arch.xive_esc_on = true;
> + mb();
> + __raw_readq(esc_vaddr + XIVE_ESB_SET_PQ_00);
> + }
> + mb();
> +}
> +EXPORT_SYMBOL_GPL(kvmppc_xive_cede_vcpu);
> +
> /*
> * This is a simple trigger for a generic XIVE IRQ. This must
> * only be called for interrupts that support a trigger page
> @@ -2106,6 +2140,42 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
> return 0;
> }
>
> +int kvmppc_xive_xics_hcall(struct kvm_vcpu *vcpu, u32 req)
> +{
> + struct kvmppc_vcore *vc = vcpu->arch.vcore;
> +
> + /*
> + * This test covers the case in which a vCPU does XICS hcalls without
> + * QEMU having connected the vCPU to a XICS ICP. The ICP is the KVM
> + * XICS device on P8 or XICS-on-XIVE on P9. It catches QEMU errors when
> + * the interrupt mode is negotiated, we don't want the OS to do XICS
> + * hcalls after having negotiated the XIVE interrupt mode.
> + */
> + if (!kvmppc_xics_enabled(vcpu))
> + return H_TOO_HARD;
> +
> + switch (req) {
> + case H_XIRR:
> + return xive_vm_h_xirr(vcpu);
> + case H_CPPR:
> + return xive_vm_h_cppr(vcpu, kvmppc_get_gpr(vcpu, 4));
> + case H_EOI:
> + return xive_vm_h_eoi(vcpu, kvmppc_get_gpr(vcpu, 4));
> + case H_IPI:
> + return xive_vm_h_ipi(vcpu, kvmppc_get_gpr(vcpu, 4),
> + kvmppc_get_gpr(vcpu, 5));
> + case H_IPOLL:
> + return xive_vm_h_ipoll(vcpu, kvmppc_get_gpr(vcpu, 4));
> + case H_XIRR_X:
> + xive_vm_h_xirr(vcpu);
> + kvmppc_set_gpr(vcpu, 5, get_tb() + vc->tb_offset);
> + return H_SUCCESS;
> + }
> +
> + return H_UNSUPPORTED;
> +}
> +EXPORT_SYMBOL_GPL(kvmppc_xive_xics_hcall);
> +
> int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
^ permalink raw reply
* [PATCH v2] powerpc/papr_scm: Implement support for H_SCM_FLUSH hcall
From: Shivaprasad G Bhat @ 2021-03-23 17:06 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc, linux-nvdimm, aneesh.kumar, ellerman
Cc: vaibhav, sbhat, linux-doc
Add support for ND_REGION_ASYNC capability if the device tree
indicates 'ibm,hcall-flush-required' property in the NVDIMM node.
Flush is done by issuing H_SCM_FLUSH hcall to the hypervisor.
If the flush request failed, the hypervisor is expected to
to reflect the problem in the subsequent dimm health request call.
This patch prevents mmap of namespaces with MAP_SYNC flag if the
nvdimm requires explicit flush[1].
References:
[1] https://github.com/avocado-framework-tests/avocado-misc-tests/blob/master/memory/ndctl.py.data/map_sync.c
Signed-off-by: Shivaprasad G Bhat <sbhat@linux.ibm.com>
---
v1 - https://www.spinics.net/lists/kvm-ppc/msg18272.html
Changes from v1:
- Hcall semantics finalized, all changes are to accomodate them.
Documentation/powerpc/papr_hcalls.rst | 14 ++++++++++
arch/powerpc/include/asm/hvcall.h | 3 +-
arch/powerpc/platforms/pseries/papr_scm.c | 39 +++++++++++++++++++++++++++++
3 files changed, 55 insertions(+), 1 deletion(-)
diff --git a/Documentation/powerpc/papr_hcalls.rst b/Documentation/powerpc/papr_hcalls.rst
index 48fcf1255a33..648f278eea8f 100644
--- a/Documentation/powerpc/papr_hcalls.rst
+++ b/Documentation/powerpc/papr_hcalls.rst
@@ -275,6 +275,20 @@ Health Bitmap Flags:
Given a DRC Index collect the performance statistics for NVDIMM and copy them
to the resultBuffer.
+**H_SCM_FLUSH**
+
+| Input: *drcIndex, continue-token*
+| Out: *continue-token*
+| Return Value: *H_SUCCESS, H_Parameter, H_P2, H_BUSY*
+
+Given a DRC Index Flush the data to backend NVDIMM device.
+
+The hcall returns H_BUSY when the flush takes longer time and the hcall needs
+to be issued multiple times in order to be completely serviced. The
+*continue-token* from the output to be passed in the argument list of
+subsequent hcalls to the hypervisor until the hcall is completely serviced
+at which point H_SUCCESS or other error is returned by the hypervisor.
+
References
==========
.. [1] "Power Architecture Platform Reference"
diff --git a/arch/powerpc/include/asm/hvcall.h b/arch/powerpc/include/asm/hvcall.h
index ed6086d57b22..9f7729a97ebd 100644
--- a/arch/powerpc/include/asm/hvcall.h
+++ b/arch/powerpc/include/asm/hvcall.h
@@ -315,7 +315,8 @@
#define H_SCM_HEALTH 0x400
#define H_SCM_PERFORMANCE_STATS 0x418
#define H_RPT_INVALIDATE 0x448
-#define MAX_HCALL_OPCODE H_RPT_INVALIDATE
+#define H_SCM_FLUSH 0x44C
+#define MAX_HCALL_OPCODE H_SCM_FLUSH
/* Scope args for H_SCM_UNBIND_ALL */
#define H_UNBIND_SCOPE_ALL (0x1)
diff --git a/arch/powerpc/platforms/pseries/papr_scm.c b/arch/powerpc/platforms/pseries/papr_scm.c
index 835163f54244..f0407e135410 100644
--- a/arch/powerpc/platforms/pseries/papr_scm.c
+++ b/arch/powerpc/platforms/pseries/papr_scm.c
@@ -93,6 +93,7 @@ struct papr_scm_priv {
uint64_t block_size;
int metadata_size;
bool is_volatile;
+ bool hcall_flush_required;
uint64_t bound_addr;
@@ -117,6 +118,38 @@ struct papr_scm_priv {
size_t stat_buffer_len;
};
+static int papr_scm_pmem_flush(struct nd_region *nd_region,
+ struct bio *bio __maybe_unused)
+{
+ struct papr_scm_priv *p = nd_region_provider_data(nd_region);
+ unsigned long ret_buf[PLPAR_HCALL_BUFSIZE];
+ uint64_t token = 0;
+ int64_t rc;
+
+ do {
+ rc = plpar_hcall(H_SCM_FLUSH, ret_buf, p->drc_index, token);
+ token = ret_buf[0];
+
+ /* Check if we are stalled for some time */
+ if (H_IS_LONG_BUSY(rc)) {
+ msleep(get_longbusy_msecs(rc));
+ rc = H_BUSY;
+ } else if (rc == H_BUSY) {
+ cond_resched();
+ }
+
+ } while (rc == H_BUSY);
+
+ if (rc) {
+ dev_err(&p->pdev->dev, "flush error: %lld", rc);
+ rc = -EIO;
+ } else {
+ dev_dbg(&p->pdev->dev, "flush drc 0x%x complete", p->drc_index);
+ }
+
+ return rc;
+}
+
static LIST_HEAD(papr_nd_regions);
static DEFINE_MUTEX(papr_ndr_lock);
@@ -943,6 +976,11 @@ static int papr_scm_nvdimm_init(struct papr_scm_priv *p)
ndr_desc.num_mappings = 1;
ndr_desc.nd_set = &p->nd_set;
+ if (p->hcall_flush_required) {
+ set_bit(ND_REGION_ASYNC, &ndr_desc.flags);
+ ndr_desc.flush = papr_scm_pmem_flush;
+ }
+
if (p->is_volatile)
p->region = nvdimm_volatile_region_create(p->bus, &ndr_desc);
else {
@@ -1088,6 +1126,7 @@ static int papr_scm_probe(struct platform_device *pdev)
p->block_size = block_size;
p->blocks = blocks;
p->is_volatile = !of_property_read_bool(dn, "ibm,cache-flush-required");
+ p->hcall_flush_required = of_property_read_bool(dn, "ibm,hcall-flush-required");
/* We just need to ensure that set cookies are unique across */
uuid_parse(uuid_str, (uuid_t *) uuid);
^ permalink raw reply related
* Re: [PATCH] xsysace: Remove SYSACE driver
From: Michal Simek @ 2021-03-23 16:28 UTC (permalink / raw)
To: Jens Axboe, Michal Simek, Davidlohr Bueso
Cc: devicetree, monstr, YueHaibing, linux-kernel, Rob Herring,
linux-block, Chris Packham, Paul Mackerras, git, linuxppc-dev
In-Reply-To: <66a774e3-f068-984e-e69f-b55667a494cf@kernel.dk>
On 3/23/21 5:28 PM, Jens Axboe wrote:
> On 3/23/21 10:25 AM, Michal Simek wrote:
>>
>>
>> On 3/23/21 5:23 PM, Jens Axboe wrote:
>>> On 3/22/21 6:04 PM, Davidlohr Bueso wrote:
>>>> Hi,
>>>>
>>>> On Mon, 09 Nov 2020, Michal Simek wrote:
>>>>
>>>>> Sysace IP is no longer used on Xilinx PowerPC 405/440 and Microblaze
>>>>> systems. The driver is not regularly tested and very likely not working for
>>>>> quite a long time that's why remove it.
>>>>
>>>> Is there a reason this patch was never merged? can the driver be
>>>> removed? I ran into this as a potential tasklet user that can be
>>>> replaced/removed.
>>>
>>> I'd be happy to merge it for 5.13.
>>>
>>
>> Can you just take this version? Or do you want me to send it again?
>
> Minor edits needed for fuzz, but I've applied this version.
Thanks,
Michal
^ permalink raw reply
* Re: [PATCH] xsysace: Remove SYSACE driver
From: Jens Axboe @ 2021-03-23 16:28 UTC (permalink / raw)
To: Michal Simek, Davidlohr Bueso
Cc: devicetree, monstr, YueHaibing, linux-kernel, Rob Herring,
linux-block, Chris Packham, Paul Mackerras, git, linuxppc-dev
In-Reply-To: <9c4911e6-92dc-0a0f-2f81-7d23e268144f@xilinx.com>
On 3/23/21 10:25 AM, Michal Simek wrote:
>
>
> On 3/23/21 5:23 PM, Jens Axboe wrote:
>> On 3/22/21 6:04 PM, Davidlohr Bueso wrote:
>>> Hi,
>>>
>>> On Mon, 09 Nov 2020, Michal Simek wrote:
>>>
>>>> Sysace IP is no longer used on Xilinx PowerPC 405/440 and Microblaze
>>>> systems. The driver is not regularly tested and very likely not working for
>>>> quite a long time that's why remove it.
>>>
>>> Is there a reason this patch was never merged? can the driver be
>>> removed? I ran into this as a potential tasklet user that can be
>>> replaced/removed.
>>
>> I'd be happy to merge it for 5.13.
>>
>
> Can you just take this version? Or do you want me to send it again?
Minor edits needed for fuzz, but I've applied this version.
--
Jens Axboe
^ permalink raw reply
* Re: [PATCH] xsysace: Remove SYSACE driver
From: Michal Simek @ 2021-03-23 16:25 UTC (permalink / raw)
To: Jens Axboe, Davidlohr Bueso, Michal Simek
Cc: devicetree, monstr, YueHaibing, linux-kernel, Rob Herring,
linux-block, Chris Packham, Paul Mackerras, git, linuxppc-dev
In-Reply-To: <6948510c-dc7e-d74a-62e3-e42be14cff16@kernel.dk>
On 3/23/21 5:23 PM, Jens Axboe wrote:
> On 3/22/21 6:04 PM, Davidlohr Bueso wrote:
>> Hi,
>>
>> On Mon, 09 Nov 2020, Michal Simek wrote:
>>
>>> Sysace IP is no longer used on Xilinx PowerPC 405/440 and Microblaze
>>> systems. The driver is not regularly tested and very likely not working for
>>> quite a long time that's why remove it.
>>
>> Is there a reason this patch was never merged? can the driver be
>> removed? I ran into this as a potential tasklet user that can be
>> replaced/removed.
>
> I'd be happy to merge it for 5.13.
>
Can you just take this version? Or do you want me to send it again?
Thanks,
Michal
^ permalink raw reply
* Re: [PATCH] xsysace: Remove SYSACE driver
From: Jens Axboe @ 2021-03-23 16:23 UTC (permalink / raw)
To: Davidlohr Bueso, Michal Simek
Cc: devicetree, monstr, YueHaibing, linux-kernel, Rob Herring,
linux-block, Chris Packham, Paul Mackerras, git, linuxppc-dev
In-Reply-To: <20210323000436.qm5rkiplwt5x5ttk@offworld>
On 3/22/21 6:04 PM, Davidlohr Bueso wrote:
> Hi,
>
> On Mon, 09 Nov 2020, Michal Simek wrote:
>
>> Sysace IP is no longer used on Xilinx PowerPC 405/440 and Microblaze
>> systems. The driver is not regularly tested and very likely not working for
>> quite a long time that's why remove it.
>
> Is there a reason this patch was never merged? can the driver be
> removed? I ran into this as a potential tasklet user that can be
> replaced/removed.
I'd be happy to merge it for 5.13.
--
Jens Axboe
^ permalink raw reply
* Re: [PATCH v4 44/46] KVM: PPC: Book3S HV P9: implement hash guest support
From: Fabiano Rosas @ 2021-03-23 15:53 UTC (permalink / raw)
To: Nicholas Piggin, kvm-ppc; +Cc: linuxppc-dev, Nicholas Piggin
In-Reply-To: <20210323010305.1045293-45-npiggin@gmail.com>
Nicholas Piggin <npiggin@gmail.com> writes:
> Guest entry/exit has to restore and save/clear the SLB, plus several
> other bits to accommodate hash guests in the P9 path.
>
> Radix host, hash guest support is removed from the P7/8 path.
>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---
<snip>
> diff --git a/arch/powerpc/kvm/book3s_hv_interrupt.c b/arch/powerpc/kvm/book3s_hv_interrupt.c
> index cd84d2c37632..03fbfef708a8 100644
> --- a/arch/powerpc/kvm/book3s_hv_interrupt.c
> +++ b/arch/powerpc/kvm/book3s_hv_interrupt.c
> @@ -55,6 +55,50 @@ static void __accumulate_time(struct kvm_vcpu *vcpu, struct kvmhv_tb_accumulator
> #define accumulate_time(vcpu, next) do {} while (0)
> #endif
>
> +static inline void mfslb(unsigned int idx, u64 *slbee, u64 *slbev)
> +{
> + asm volatile("slbmfev %0,%1" : "=r" (*slbev) : "r" (idx));
> + asm volatile("slbmfee %0,%1" : "=r" (*slbee) : "r" (idx));
> +}
> +
> +static inline void __mtslb(u64 slbee, u64 slbev)
> +{
> + asm volatile("slbmte %0,%1" :: "r" (slbev), "r" (slbee));
> +}
> +
> +static inline void mtslb(unsigned int idx, u64 slbee, u64 slbev)
> +{
> + BUG_ON((slbee & 0xfff) != idx);
> +
> + __mtslb(slbee, slbev);
> +}
> +
> +static inline void slb_invalidate(unsigned int ih)
> +{
> + asm volatile("slbia %0" :: "i"(ih));
> +}
Fyi, in my environment the assembler complains:
{standard input}: Assembler messages:
{standard input}:1293: Error: junk at end of line: `6'
{standard input}:2138: Error: junk at end of line: `6'
make[3]: *** [../scripts/Makefile.build:271:
arch/powerpc/kvm/book3s_hv_interrupt.o] Error 1
This works:
- asm volatile("slbia %0" :: "i"(ih));
+ asm volatile(PPC_SLBIA(%0) :: "i"(ih));
But I don't know what is going on.
^ permalink raw reply
* [PATCH] powerpc: Switch to relative jump labels
From: Christophe Leroy @ 2021-03-23 15:47 UTC (permalink / raw)
To: Benjamin Herrenschmidt, Paul Mackerras, Michael Ellerman
Cc: linuxppc-dev, linux-kernel
Convert powerpc to relative jump labels.
Before the patch, pseries_defconfig vmlinux.o has:
9074 __jump_table 0003f2a0 0000000000000000 0000000000000000 01321fa8 2**0
With the patch, the same config gets:
9074 __jump_table 0002a0e0 0000000000000000 0000000000000000 01321fb4 2**0
Size is 258720 without the patch, 172256 with the patch.
That's a 33% size reduction.
Largely copied from commit c296146c058c ("arm64/kernel: jump_label:
Switch to relative references")
Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
---
arch/powerpc/Kconfig | 1 +
arch/powerpc/include/asm/jump_label.h | 21 ++++++---------------
arch/powerpc/kernel/jump_label.c | 4 ++--
3 files changed, 9 insertions(+), 17 deletions(-)
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index d46db0bfb998..a52938c0f85b 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -182,6 +182,7 @@ config PPC
select HAVE_ARCH_AUDITSYSCALL
select HAVE_ARCH_HUGE_VMAP if PPC_BOOK3S_64 && PPC_RADIX_MMU
select HAVE_ARCH_JUMP_LABEL
+ select HAVE_ARCH_JUMP_LABEL_RELATIVE
select HAVE_ARCH_KASAN if PPC32 && PPC_PAGE_SHIFT <= 14
select HAVE_ARCH_KASAN_VMALLOC if PPC32 && PPC_PAGE_SHIFT <= 14
select HAVE_ARCH_KGDB
diff --git a/arch/powerpc/include/asm/jump_label.h b/arch/powerpc/include/asm/jump_label.h
index 09297ec9fa52..2d5c6bec2b4f 100644
--- a/arch/powerpc/include/asm/jump_label.h
+++ b/arch/powerpc/include/asm/jump_label.h
@@ -20,7 +20,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran
asm_volatile_goto("1:\n\t"
"nop # arch_static_branch\n\t"
".pushsection __jump_table, \"aw\"\n\t"
- JUMP_ENTRY_TYPE "1b, %l[l_yes], %c0\n\t"
+ ".long 1b - ., %l[l_yes] - .\n\t"
+ JUMP_ENTRY_TYPE "%c0 - .\n\t"
".popsection \n\t"
: : "i" (&((char *)key)[branch]) : : l_yes);
@@ -34,7 +35,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool
asm_volatile_goto("1:\n\t"
"b %l[l_yes] # arch_static_branch_jump\n\t"
".pushsection __jump_table, \"aw\"\n\t"
- JUMP_ENTRY_TYPE "1b, %l[l_yes], %c0\n\t"
+ ".long 1b - ., %l[l_yes] - .\n\t"
+ JUMP_ENTRY_TYPE "%c0 - .\n\t"
".popsection \n\t"
: : "i" (&((char *)key)[branch]) : : l_yes);
@@ -43,23 +45,12 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool
return true;
}
-#ifdef CONFIG_PPC64
-typedef u64 jump_label_t;
-#else
-typedef u32 jump_label_t;
-#endif
-
-struct jump_entry {
- jump_label_t code;
- jump_label_t target;
- jump_label_t key;
-};
-
#else
#define ARCH_STATIC_BRANCH(LABEL, KEY) \
1098: nop; \
.pushsection __jump_table, "aw"; \
- FTR_ENTRY_LONG 1098b, LABEL, KEY; \
+ .long 1098b - ., LABEL - .; \
+ FTR_ENTRY_LONG KEY; \
.popsection
#endif
diff --git a/arch/powerpc/kernel/jump_label.c b/arch/powerpc/kernel/jump_label.c
index 144858027fa3..ce87dc5ea23c 100644
--- a/arch/powerpc/kernel/jump_label.c
+++ b/arch/powerpc/kernel/jump_label.c
@@ -11,10 +11,10 @@
void arch_jump_label_transform(struct jump_entry *entry,
enum jump_label_type type)
{
- struct ppc_inst *addr = (struct ppc_inst *)(unsigned long)entry->code;
+ struct ppc_inst *addr = (struct ppc_inst *)jump_entry_code(entry);
if (type == JUMP_LABEL_JMP)
- patch_branch(addr, entry->target, 0);
+ patch_branch(addr, jump_entry_target(entry), 0);
else
patch_instruction(addr, ppc_inst(PPC_INST_NOP));
}
--
2.25.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox