[RFC PATCH 0/3] ppc/pnv: SMT support for powernv

[RFC PATCH 0/3] ppc/pnv: SMT support for powernv

[Nicholas Piggin]

These patches implement enough to boot a SMT powernv machine to Linux
and boot a SMP KVM guest inside that.

There are a few more SPRs that need to be done, and per-LPAR SPRs are
mostly not annotated yet so it can't run in 1LPAR mode. But it is
enough to run skiboot/Linux with SMT so I'll just post the minimal
patches for RFC because the concept isn't really different to add
more SPRs and things.

Thanks,
Nick

Nicholas Piggin (3):
  target/ppc: Add LPAR-per-core vs per-thread mode flag
  target/ppc: SMT support for the HID SPR
  ppc/pnv: SMT support for powernv

 hw/ppc/pnv.c             | 12 ++++++++++++
 hw/ppc/pnv_core.c        | 13 +++++--------
 hw/ppc/spapr_cpu_core.c  |  2 ++
 target/ppc/cpu.h         |  3 +++
 target/ppc/cpu_init.c    | 14 +++++++++++++-
 target/ppc/helper.h      |  1 +
 target/ppc/misc_helper.c | 21 +++++++++++++++++++++
 target/ppc/spr_common.h  |  1 +
 target/ppc/translate.c   | 32 +++++++++++++++++++++++++++++---
 9 files changed, 87 insertions(+), 12 deletions(-)

-- 
2.40.1

[RFC PATCH 1/3] target/ppc: Add LPAR-per-core vs per-thread mode flag

[Nicholas Piggin]

The Power ISA has the concept of sub-processors:

  Hardware is allowed to sub-divide a multi-threaded processor into
  "sub-processors" that appear to privileged programs as multi-threaded
  processors with fewer threads.

POWER9 and POWER10 have two modes, either every thread is a
sub-processor or all threads appear as one multi-threaded processor.
In the user manuals these are known as "LPAR-per-thread" and "LPAR
per core" (or "1LPAR"), respectively.

The practical difference is in LPAR-per-thread mode, non-hypervisor SPRs
are not shared between threads and msgsndp can not be used to message
siblings. In 1LPAR mode some SPRs are shared and msgsndp is usable. LPPT
allows multiple partitions to run concurrently on the same core,
and is a requirement for KVM to run on POWER9/10.

Traditionally, SMT in PAPR environments including PowerVM and the
pseries machine with KVM acceleration beahves as in 1LPAR mode. In
OPAL systems, LPAR-per-thread is used. When adding SMT to the powernv
machine, it is preferable to emulate OPAL LPAR-per-thread, so to
account for this difference a flag is added and SPRs may become either
per-thread, per-core shared, or per-LPAR shared. Per-LPAR registers
become either per-thread or per-core shared depending on the mode.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 hw/ppc/spapr_cpu_core.c |  2 ++
 target/ppc/cpu.h        |  3 +++
 target/ppc/cpu_init.c   | 12 ++++++++++++
 target/ppc/translate.c  | 16 +++++++++++++---
 4 files changed, 30 insertions(+), 3 deletions(-)

diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
index a4e3c2fadd..b482d9754a 100644
--- a/hw/ppc/spapr_cpu_core.c
+++ b/hw/ppc/spapr_cpu_core.c
@@ -270,6 +270,8 @@ static bool spapr_realize_vcpu(PowerPCCPU *cpu, SpaprMachineState *spapr,
     env->spr_cb[SPR_PIR].default_value = cs->cpu_index;
     env->spr_cb[SPR_TIR].default_value = thread_index;
 
+    cpu_ppc_set_1lpar(cpu);
+
     /* Set time-base frequency to 512 MHz. vhyp must be set first. */
     cpu_ppc_tb_init(env, SPAPR_TIMEBASE_FREQ);
 
diff --git a/target/ppc/cpu.h b/target/ppc/cpu.h
index 94497aa115..beddc5db5b 100644
--- a/target/ppc/cpu.h
+++ b/target/ppc/cpu.h
@@ -674,6 +674,8 @@ enum {
     POWERPC_FLAG_SCV      = 0x00200000,
     /* Has >1 thread per core                                                */
     POWERPC_FLAG_SMT      = 0x00400000,
+    /* Using "LPAR per core" mode  (as opposed to per-thread)                */
+    POWERPC_FLAG_1LPAR    = 0x00800000,
 };
 
 /*
@@ -1435,6 +1437,7 @@ void store_booke_tsr(CPUPPCState *env, target_ulong val);
 void ppc_tlb_invalidate_all(CPUPPCState *env);
 void ppc_tlb_invalidate_one(CPUPPCState *env, target_ulong addr);
 void cpu_ppc_set_vhyp(PowerPCCPU *cpu, PPCVirtualHypervisor *vhyp);
+void cpu_ppc_set_1lpar(PowerPCCPU *cpu);
 int ppcmas_tlb_check(CPUPPCState *env, ppcmas_tlb_t *tlb, hwaddr *raddrp,
                      target_ulong address, uint32_t pid);
 int ppcemb_tlb_search(CPUPPCState *env, target_ulong address, uint32_t pid);
diff --git a/target/ppc/cpu_init.c b/target/ppc/cpu_init.c
index aeff71d063..dc3a65a575 100644
--- a/target/ppc/cpu_init.c
+++ b/target/ppc/cpu_init.c
@@ -6601,6 +6601,18 @@ void cpu_ppc_set_vhyp(PowerPCCPU *cpu, PPCVirtualHypervisor *vhyp)
     env->msr_mask &= ~MSR_HVB;
 }
 
+void cpu_ppc_set_1lpar(PowerPCCPU *cpu)
+{
+    CPUPPCState *env = &cpu->env;
+
+    /*
+     * pseries SMT means "LPAR per core" mode, e.g., msgsndp is usable
+     * between threads.
+     */
+    if (env->flags & POWERPC_FLAG_SMT) {
+        env->flags |= POWERPC_FLAG_1LPAR;
+    }
+}
 #endif /* !defined(CONFIG_USER_ONLY) */
 
 #endif /* defined(TARGET_PPC64) */
diff --git a/target/ppc/translate.c b/target/ppc/translate.c
index 372ee600b2..ef186396b4 100644
--- a/target/ppc/translate.c
+++ b/target/ppc/translate.c
@@ -256,6 +256,16 @@ static inline bool gen_serialize_core(DisasContext *ctx)
 }
 #endif
 
+static inline bool gen_serialize_core_lpar(DisasContext *ctx)
+{
+    /* 1LPAR implies SMT */
+    if (ctx->flags & POWERPC_FLAG_1LPAR) {
+        return gen_serialize(ctx);
+    }
+
+    return true;
+}
+
 /* SPR load/store helpers */
 static inline void gen_load_spr(TCGv t, int reg)
 {
@@ -451,7 +461,7 @@ static void spr_write_CTRL_ST(DisasContext *ctx, int sprn, int gprn)
 
 void spr_write_CTRL(DisasContext *ctx, int sprn, int gprn)
 {
-    if (!(ctx->flags & POWERPC_FLAG_SMT)) {
+    if (!(ctx->flags & POWERPC_FLAG_1LPAR)) {
         spr_write_CTRL_ST(ctx, sprn, gprn);
         goto out;
     }
@@ -815,7 +825,7 @@ void spr_write_pcr(DisasContext *ctx, int sprn, int gprn)
 /* DPDES */
 void spr_read_dpdes(DisasContext *ctx, int gprn, int sprn)
 {
-    if (!gen_serialize_core(ctx)) {
+    if (!gen_serialize_core_lpar(ctx)) {
         return;
     }
 
@@ -824,7 +834,7 @@ void spr_read_dpdes(DisasContext *ctx, int gprn, int sprn)
 
 void spr_write_dpdes(DisasContext *ctx, int sprn, int gprn)
 {
-    if (!gen_serialize_core(ctx)) {
+    if (!gen_serialize_core_lpar(ctx)) {
         return;
     }
 
-- 
2.40.1

Re: [RFC PATCH 1/3] target/ppc: Add LPAR-per-core vs per-thread mode flag

[Joel Stanley]

On Thu, 29 Jun 2023 at 02:17, Nicholas Piggin <npiggin@gmail.com> wrote:
>
> The Power ISA has the concept of sub-processors:
>
>   Hardware is allowed to sub-divide a multi-threaded processor into
>   "sub-processors" that appear to privileged programs as multi-threaded
>   processors with fewer threads.
>
> POWER9 and POWER10 have two modes, either every thread is a
> sub-processor or all threads appear as one multi-threaded processor.
> In the user manuals these are known as "LPAR-per-thread" and "LPAR
> per core" (or "1LPAR"), respectively.
>
> The practical difference is in LPAR-per-thread mode, non-hypervisor SPRs
> are not shared between threads and msgsndp can not be used to message
> siblings. In 1LPAR mode some SPRs are shared and msgsndp is usable. LPPT
> allows multiple partitions to run concurrently on the same core,
> and is a requirement for KVM to run on POWER9/10.
>
> Traditionally, SMT in PAPR environments including PowerVM and the
> pseries machine with KVM acceleration beahves as in 1LPAR mode. In

behaves

> OPAL systems, LPAR-per-thread is used. When adding SMT to the powernv
> machine, it is preferable to emulate OPAL LPAR-per-thread, so to
> account for this difference a flag is added and SPRs may become either
> per-thread, per-core shared, or per-LPAR shared. Per-LPAR registers
> become either per-thread or per-core shared depending on the mode.
>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>

Nice description.

Reviewed-by: Joel Stanley <joel@jms.id.au>

As we make the emulation more accurate, we will want the 1LPAR state
to be reflected in the xscoms too.

> ---
>  hw/ppc/spapr_cpu_core.c |  2 ++
>  target/ppc/cpu.h        |  3 +++
>  target/ppc/cpu_init.c   | 12 ++++++++++++
>  target/ppc/translate.c  | 16 +++++++++++++---
>  4 files changed, 30 insertions(+), 3 deletions(-)
>
> diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> index a4e3c2fadd..b482d9754a 100644
> --- a/hw/ppc/spapr_cpu_core.c
> +++ b/hw/ppc/spapr_cpu_core.c
> @@ -270,6 +270,8 @@ static bool spapr_realize_vcpu(PowerPCCPU *cpu, SpaprMachineState *spapr,
>      env->spr_cb[SPR_PIR].default_value = cs->cpu_index;
>      env->spr_cb[SPR_TIR].default_value = thread_index;
>
> +    cpu_ppc_set_1lpar(cpu);
> +
>      /* Set time-base frequency to 512 MHz. vhyp must be set first. */
>      cpu_ppc_tb_init(env, SPAPR_TIMEBASE_FREQ);
>
> diff --git a/target/ppc/cpu.h b/target/ppc/cpu.h
> index 94497aa115..beddc5db5b 100644
> --- a/target/ppc/cpu.h
> +++ b/target/ppc/cpu.h
> @@ -674,6 +674,8 @@ enum {
>      POWERPC_FLAG_SCV      = 0x00200000,
>      /* Has >1 thread per core                                                */
>      POWERPC_FLAG_SMT      = 0x00400000,
> +    /* Using "LPAR per core" mode  (as opposed to per-thread)                */
> +    POWERPC_FLAG_1LPAR    = 0x00800000,
>  };
>
>  /*
> @@ -1435,6 +1437,7 @@ void store_booke_tsr(CPUPPCState *env, target_ulong val);
>  void ppc_tlb_invalidate_all(CPUPPCState *env);
>  void ppc_tlb_invalidate_one(CPUPPCState *env, target_ulong addr);
>  void cpu_ppc_set_vhyp(PowerPCCPU *cpu, PPCVirtualHypervisor *vhyp);
> +void cpu_ppc_set_1lpar(PowerPCCPU *cpu);
>  int ppcmas_tlb_check(CPUPPCState *env, ppcmas_tlb_t *tlb, hwaddr *raddrp,
>                       target_ulong address, uint32_t pid);
>  int ppcemb_tlb_search(CPUPPCState *env, target_ulong address, uint32_t pid);
> diff --git a/target/ppc/cpu_init.c b/target/ppc/cpu_init.c
> index aeff71d063..dc3a65a575 100644
> --- a/target/ppc/cpu_init.c
> +++ b/target/ppc/cpu_init.c
> @@ -6601,6 +6601,18 @@ void cpu_ppc_set_vhyp(PowerPCCPU *cpu, PPCVirtualHypervisor *vhyp)
>      env->msr_mask &= ~MSR_HVB;
>  }
>
> +void cpu_ppc_set_1lpar(PowerPCCPU *cpu)
> +{
> +    CPUPPCState *env = &cpu->env;
> +
> +    /*
> +     * pseries SMT means "LPAR per core" mode, e.g., msgsndp is usable
> +     * between threads.
> +     */
> +    if (env->flags & POWERPC_FLAG_SMT) {
> +        env->flags |= POWERPC_FLAG_1LPAR;
> +    }
> +}
>  #endif /* !defined(CONFIG_USER_ONLY) */
>
>  #endif /* defined(TARGET_PPC64) */
> diff --git a/target/ppc/translate.c b/target/ppc/translate.c
> index 372ee600b2..ef186396b4 100644
> --- a/target/ppc/translate.c
> +++ b/target/ppc/translate.c
> @@ -256,6 +256,16 @@ static inline bool gen_serialize_core(DisasContext *ctx)
>  }
>  #endif
>
> +static inline bool gen_serialize_core_lpar(DisasContext *ctx)
> +{
> +    /* 1LPAR implies SMT */
> +    if (ctx->flags & POWERPC_FLAG_1LPAR) {
> +        return gen_serialize(ctx);
> +    }
> +
> +    return true;
> +}
> +
>  /* SPR load/store helpers */
>  static inline void gen_load_spr(TCGv t, int reg)
>  {
> @@ -451,7 +461,7 @@ static void spr_write_CTRL_ST(DisasContext *ctx, int sprn, int gprn)
>
>  void spr_write_CTRL(DisasContext *ctx, int sprn, int gprn)
>  {
> -    if (!(ctx->flags & POWERPC_FLAG_SMT)) {
> +    if (!(ctx->flags & POWERPC_FLAG_1LPAR)) {
>          spr_write_CTRL_ST(ctx, sprn, gprn);
>          goto out;
>      }
> @@ -815,7 +825,7 @@ void spr_write_pcr(DisasContext *ctx, int sprn, int gprn)
>  /* DPDES */
>  void spr_read_dpdes(DisasContext *ctx, int gprn, int sprn)
>  {
> -    if (!gen_serialize_core(ctx)) {
> +    if (!gen_serialize_core_lpar(ctx)) {
>          return;
>      }
>
> @@ -824,7 +834,7 @@ void spr_read_dpdes(DisasContext *ctx, int gprn, int sprn)
>
>  void spr_write_dpdes(DisasContext *ctx, int sprn, int gprn)
>  {
> -    if (!gen_serialize_core(ctx)) {
> +    if (!gen_serialize_core_lpar(ctx)) {
>          return;
>      }
>
> --
> 2.40.1
>
>

[RFC PATCH 2/3] target/ppc: SMT support for the HID SPR

[Nicholas Piggin]

HID is a per-core shared register, skiboot sets this (e.g., setting
HILE) on one thread and that must affect all threads of the core.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 target/ppc/cpu_init.c    |  2 +-
 target/ppc/helper.h      |  1 +
 target/ppc/misc_helper.c | 21 +++++++++++++++++++++
 target/ppc/spr_common.h  |  1 +
 target/ppc/translate.c   | 16 ++++++++++++++++
 5 files changed, 40 insertions(+), 1 deletion(-)

diff --git a/target/ppc/cpu_init.c b/target/ppc/cpu_init.c
index dc3a65a575..a1b01dc479 100644
--- a/target/ppc/cpu_init.c
+++ b/target/ppc/cpu_init.c
@@ -5373,7 +5373,7 @@ static void register_book3s_ids_sprs(CPUPPCState *env)
     spr_register_hv(env, SPR_HID0, "HID0",
                  SPR_NOACCESS, SPR_NOACCESS,
                  SPR_NOACCESS, SPR_NOACCESS,
-                 &spr_read_generic, &spr_write_generic,
+                 &spr_read_generic, &spr_core_write_generic,
                  0x00000000);
     spr_register_hv(env, SPR_TSCR, "TSCR",
                  SPR_NOACCESS, SPR_NOACCESS,
diff --git a/target/ppc/helper.h b/target/ppc/helper.h
index fda40b8a60..719752b38a 100644
--- a/target/ppc/helper.h
+++ b/target/ppc/helper.h
@@ -704,6 +704,7 @@ DEF_HELPER_3(store_dcr, void, env, tl, tl)
 
 DEF_HELPER_2(load_dump_spr, void, env, i32)
 DEF_HELPER_2(store_dump_spr, void, env, i32)
+DEF_HELPER_3(spr_core_write_generic, void, env, i32, tl)
 DEF_HELPER_3(spr_write_CTRL, void, env, i32, tl)
 
 DEF_HELPER_4(fscr_facility_check, void, env, i32, i32, i32)
diff --git a/target/ppc/misc_helper.c b/target/ppc/misc_helper.c
index 1f1af21f33..0da335472e 100644
--- a/target/ppc/misc_helper.c
+++ b/target/ppc/misc_helper.c
@@ -43,6 +43,27 @@ void helper_store_dump_spr(CPUPPCState *env, uint32_t sprn)
              env->spr[sprn]);
 }
 
+void helper_spr_core_write_generic(CPUPPCState *env, uint32_t sprn,
+                                   target_ulong val)
+{
+    CPUState *cs = env_cpu(env);
+    CPUState *ccs;
+    uint32_t nr_threads = cs->nr_threads;
+    uint32_t core_id = env->spr[SPR_PIR] & ~(nr_threads - 1);
+
+    assert(core_id == env->spr[SPR_PIR] - env->spr[SPR_TIR]);
+
+    if (nr_threads == 1) {
+        env->spr[sprn] = val;
+        return;
+    }
+
+    THREAD_SIBLING_FOREACH(cs, ccs) {
+        CPUPPCState *cenv = &POWERPC_CPU(ccs)->env;
+        cenv->spr[sprn] = val;
+    }
+}
+
 void helper_spr_write_CTRL(CPUPPCState *env, uint32_t sprn,
                            target_ulong val)
 {
diff --git a/target/ppc/spr_common.h b/target/ppc/spr_common.h
index 4c0f2bed77..aae66ee966 100644
--- a/target/ppc/spr_common.h
+++ b/target/ppc/spr_common.h
@@ -82,6 +82,7 @@ void spr_noaccess(DisasContext *ctx, int gprn, int sprn);
 void spr_read_generic(DisasContext *ctx, int gprn, int sprn);
 void spr_write_generic(DisasContext *ctx, int sprn, int gprn);
 void spr_write_generic32(DisasContext *ctx, int sprn, int gprn);
+void spr_core_write_generic(DisasContext *ctx, int sprn, int gprn);
 void spr_write_MMCR0(DisasContext *ctx, int sprn, int gprn);
 void spr_write_MMCR1(DisasContext *ctx, int sprn, int gprn);
 void spr_write_PMC(DisasContext *ctx, int sprn, int gprn);
diff --git a/target/ppc/translate.c b/target/ppc/translate.c
index ef186396b4..f7a5ecc0c1 100644
--- a/target/ppc/translate.c
+++ b/target/ppc/translate.c
@@ -448,6 +448,22 @@ void spr_write_generic32(DisasContext *ctx, int sprn, int gprn)
 #endif
 }
 
+void spr_core_write_generic(DisasContext *ctx, int sprn, int gprn)
+{
+    if (!(ctx->flags & POWERPC_FLAG_SMT)) {
+        spr_write_generic(ctx, sprn, gprn);
+        return;
+    }
+
+    if (!gen_serialize(ctx)) {
+        return;
+    }
+
+    gen_helper_spr_core_write_generic(cpu_env, tcg_constant_i32(sprn),
+                                      cpu_gpr[gprn]);
+    spr_store_dump_spr(sprn);
+}
+
 static void spr_write_CTRL_ST(DisasContext *ctx, int sprn, int gprn)
 {
     /* This does not implement >1 thread */
-- 
2.40.1

[RFC PATCH 3/3] ppc/pnv: SMT support for powernv

[Nicholas Piggin]

Set the TIR default value with the SMT thread index, and place some
standard limits on SMT configurations. Now powernv is able to boot
skiboot and Linux with a SMT topology, including booting a KVM guest.

There are several other per-core SPRs, but they are not so important to
run OPAL/Linux. Some important per-LPAR ones to convert before powernv
could run in 1LPAR mode. Broadcast msgsnd is not yet implemented either,
but skiboot/Linux does not use that. KVM uses an implementation-specific
detail of POWER9/10 TLBs where TLBIEL invalidates translations of all
threads on a core, but that is not required here because KVM does not
cache translations across PID or LPID switch. Most of these I have or
aren't too hard to implement, but I start with a small bare bones for
comments.

Not-yet-Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 hw/ppc/pnv.c      | 12 ++++++++++++
 hw/ppc/pnv_core.c | 13 +++++--------
 2 files changed, 17 insertions(+), 8 deletions(-)

diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
index fc083173f3..f599ccad1d 100644
--- a/hw/ppc/pnv.c
+++ b/hw/ppc/pnv.c
@@ -887,6 +887,18 @@ static void pnv_init(MachineState *machine)
 
     pnv->num_chips =
         machine->smp.max_cpus / (machine->smp.cores * machine->smp.threads);
+
+    if (machine->smp.threads > 8) {
+        error_report("Cannot support more than 8 threads/core "
+                     "on a powernv machine");
+        exit(1);
+    }
+    if (!is_power_of_2(machine->smp.threads)) {
+        error_report("Cannot support %d threads/core on a powernv"
+                     "machine because it must be a power of 2",
+                     machine->smp.threads);
+        exit(1);
+    }
     /*
      * TODO: should we decide on how many chips we can create based
      * on #cores and Venice vs. Murano vs. Naples chip type etc...,
diff --git a/hw/ppc/pnv_core.c b/hw/ppc/pnv_core.c
index 0bc3ad41c8..acd83caee8 100644
--- a/hw/ppc/pnv_core.c
+++ b/hw/ppc/pnv_core.c
@@ -167,12 +167,13 @@ static const MemoryRegionOps pnv_core_power9_xscom_ops = {
     .endianness = DEVICE_BIG_ENDIAN,
 };
 
-static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp)
+static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp,
+                                 int thread_index)
 {
     CPUPPCState *env = &cpu->env;
     int core_pir;
-    int thread_index = 0; /* TODO: TCG supports only one thread */
     ppc_spr_t *pir = &env->spr_cb[SPR_PIR];
+    ppc_spr_t *tir = &env->spr_cb[SPR_TIR];
     Error *local_err = NULL;
     PnvChipClass *pcc = PNV_CHIP_GET_CLASS(pc->chip);
 
@@ -188,11 +189,7 @@ static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp)
 
     core_pir = object_property_get_uint(OBJECT(pc), "pir", &error_abort);
 
-    /*
-     * The PIR of a thread is the core PIR + the thread index. We will
-     * need to find a way to get the thread index when TCG supports
-     * more than 1. We could use the object name ?
-     */
+    tir->default_value = thread_index;
     pir->default_value = core_pir + thread_index;
 
     /* Set time-base frequency to 512 MHz */
@@ -241,7 +238,7 @@ static void pnv_core_realize(DeviceState *dev, Error **errp)
     }
 
     for (j = 0; j < cc->nr_threads; j++) {
-        pnv_core_cpu_realize(pc, pc->threads[j], &local_err);
+        pnv_core_cpu_realize(pc, pc->threads[j], &local_err, j);
         if (local_err) {
             goto err;
         }
-- 
2.40.1

Re: [RFC PATCH 3/3] ppc/pnv: SMT support for powernv

[Cédric Le Goater]

On 6/29/23 04:16, Nicholas Piggin wrote:
> Set the TIR default value with the SMT thread index, and place some
> standard limits on SMT configurations. Now powernv is able to boot
> skiboot and Linux with a SMT topology, including booting a KVM guest.
> 
> There are several other per-core SPRs, but they are not so important to
> run OPAL/Linux. Some important per-LPAR ones to convert before powernv
> could run in 1LPAR mode. Broadcast msgsnd is not yet implemented either,
> but skiboot/Linux does not use that. KVM uses an implementation-specific
> detail of POWER9/10 TLBs where TLBIEL invalidates translations of all
> threads on a core, but that is not required here because KVM does not
> cache translations across PID or LPID switch. Most of these I have or
> aren't too hard to implement, but I start with a small bare bones for
> comments.
> 
> Not-yet-Signed-off-by: Nicholas Piggin <npiggin@gmail.com>

I am glad this becoming possible. You can model the missing parts
later on.

Reviewed-by: Cédric Le Goater <clg@kaod.org>

Thanks,

C.

> ---
>   hw/ppc/pnv.c      | 12 ++++++++++++
>   hw/ppc/pnv_core.c | 13 +++++--------
>   2 files changed, 17 insertions(+), 8 deletions(-)
> 
> diff --git a/hw/ppc/pnv.c b/hw/ppc/pnv.c
> index fc083173f3..f599ccad1d 100644
> --- a/hw/ppc/pnv.c
> +++ b/hw/ppc/pnv.c
> @@ -887,6 +887,18 @@ static void pnv_init(MachineState *machine)
>   
>       pnv->num_chips =
>           machine->smp.max_cpus / (machine->smp.cores * machine->smp.threads);
> +
> +    if (machine->smp.threads > 8) {
> +        error_report("Cannot support more than 8 threads/core "
> +                     "on a powernv machine");
> +        exit(1);
> +    }
> +    if (!is_power_of_2(machine->smp.threads)) {
> +        error_report("Cannot support %d threads/core on a powernv"
> +                     "machine because it must be a power of 2",
> +                     machine->smp.threads);
> +        exit(1);
> +    }
>       /*
>        * TODO: should we decide on how many chips we can create based
>        * on #cores and Venice vs. Murano vs. Naples chip type etc...,
> diff --git a/hw/ppc/pnv_core.c b/hw/ppc/pnv_core.c
> index 0bc3ad41c8..acd83caee8 100644
> --- a/hw/ppc/pnv_core.c
> +++ b/hw/ppc/pnv_core.c
> @@ -167,12 +167,13 @@ static const MemoryRegionOps pnv_core_power9_xscom_ops = {
>       .endianness = DEVICE_BIG_ENDIAN,
>   };
>   
> -static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp)
> +static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp,
> +                                 int thread_index)
>   {
>       CPUPPCState *env = &cpu->env;
>       int core_pir;
> -    int thread_index = 0; /* TODO: TCG supports only one thread */
>       ppc_spr_t *pir = &env->spr_cb[SPR_PIR];
> +    ppc_spr_t *tir = &env->spr_cb[SPR_TIR];
>       Error *local_err = NULL;
>       PnvChipClass *pcc = PNV_CHIP_GET_CLASS(pc->chip);
>   
> @@ -188,11 +189,7 @@ static void pnv_core_cpu_realize(PnvCore *pc, PowerPCCPU *cpu, Error **errp)
>   
>       core_pir = object_property_get_uint(OBJECT(pc), "pir", &error_abort);
>   
> -    /*
> -     * The PIR of a thread is the core PIR + the thread index. We will
> -     * need to find a way to get the thread index when TCG supports
> -     * more than 1. We could use the object name ?
> -     */
> +    tir->default_value = thread_index;
>       pir->default_value = core_pir + thread_index;
>   
>       /* Set time-base frequency to 512 MHz */
> @@ -241,7 +238,7 @@ static void pnv_core_realize(DeviceState *dev, Error **errp)
>       }
>   
>       for (j = 0; j < cc->nr_threads; j++) {
> -        pnv_core_cpu_realize(pc, pc->threads[j], &local_err);
> +        pnv_core_cpu_realize(pc, pc->threads[j], &local_err, j);
>           if (local_err) {
>               goto err;
>           }

Re: [RFC PATCH 0/3] ppc/pnv: SMT support for powernv

[Cédric Le Goater]

On 6/29/23 04:16, Nicholas Piggin wrote:
> These patches implement enough to boot a SMT powernv machine to Linux
> and boot a SMP KVM guest inside that.
> 
> There are a few more SPRs that need to be done, and per-LPAR SPRs are
> mostly not annotated yet so it can't run in 1LPAR mode. But it is
> enough to run skiboot/Linux with SMT so I'll just post the minimal
> patches for RFC because the concept isn't really different to add
> more SPRs and things.

This is good material for 8.1. Please send a v1 and Cc: all pnv reviewers.

Thanks,

C.

Re: [RFC PATCH 0/3] ppc/pnv: SMT support for powernv

[Cédric Le Goater]

On 6/29/23 04:16, Nicholas Piggin wrote:
> These patches implement enough to boot a SMT powernv machine to Linux
> and boot a SMP KVM guest inside that.
> 
> There are a few more SPRs that need to be done, and per-LPAR SPRs are
> mostly not annotated yet so it can't run in 1LPAR mode. But it is
> enough to run skiboot/Linux with SMT so I'll just post the minimal
> patches for RFC because the concept isn't really different to add
> more SPRs and things.
> 
> Thanks,
> Nick

QEMU could boot a 2 sockets x 2 cores x 4 threads machine with MTTCG
in less than a minute. Very nice :)

Thanks,

C.

root@jammy:~# lscpu
Architecture:          ppc64le
   Byte Order:          Little Endian
CPU(s):                16
   On-line CPU(s) list: 0-15
Model name:            POWER9, altivec supported
   Model:               2.2 (pvr 004e 1202)
   Thread(s) per core:  4
   Core(s) per socket:  2
   Socket(s):           2
Caches (sum of all):
   L1d:                 128 KiB (4 instances)
   L1i:                 128 KiB (4 instances)
NUMA:
   NUMA node(s):        2
   NUMA node0 CPU(s):   0-7
   NUMA node1 CPU(s):   8-15




> Nicholas Piggin (3):
>    target/ppc: Add LPAR-per-core vs per-thread mode flag
>    target/ppc: SMT support for the HID SPR
>    ppc/pnv: SMT support for powernv
> 
>   hw/ppc/pnv.c             | 12 ++++++++++++
>   hw/ppc/pnv_core.c        | 13 +++++--------
>   hw/ppc/spapr_cpu_core.c  |  2 ++
>   target/ppc/cpu.h         |  3 +++
>   target/ppc/cpu_init.c    | 14 +++++++++++++-
>   target/ppc/helper.h      |  1 +
>   target/ppc/misc_helper.c | 21 +++++++++++++++++++++
>   target/ppc/spr_common.h  |  1 +
>   target/ppc/translate.c   | 32 +++++++++++++++++++++++++++++---
>   9 files changed, 87 insertions(+), 12 deletions(-)
>