Re: [PATCH 3/6] spapr_numa: translate regular NUMA distance to PAPR distance

[Greg Kurz <groug@kaod.org>]

On Wed, 23 Sep 2020 16:34:55 -0300
Daniel Henrique Barboza <danielhb413@gmail.com> wrote:

> QEMU allows the user to set NUMA distances in the command line.
> For ACPI architectures like x86, this means that user input is
> used to populate the SLIT table, and the guest perceives the
> distances as the user chooses to.
> 
> PPC64 does not work that way. In the PAPR concept of NUMA,
> associativity relations between the NUMA nodes are provided by
> the device tree, and the guest kernel is free to calculate the
> distances as it sees fit. Given how ACPI architectures works,
> this puts the pSeries machine in a strange spot - users expect
> to define NUMA distances like in the ACPI case, but QEMU does
> not have control over it. To give pSeries users a similar
> experience, we'll need to bring kernel specifics to QEMU
> to approximate the NUMA distances.
> 
> The pSeries kernel works with the NUMA distance range 10,
> 20, 40, 80 and 160. The code starts at 10 (local distance) and
> searches for a match in the first NUMA level between the
> resources. If there is no match, the distance is doubled and
> then it proceeds to try to match in the next NUMA level. Rinse
> and repeat for MAX_DISTANCE_REF_POINTS levels.
> 
> This patch introduces a spapr_numa_PAPRify_distances() helper

Funky naming but meaningful and funny, for me at least :)

> that translates the user distances to kernel distance, which
> we're going to use to determine the associativity domains for
> the NUMA nodes.
> 
> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com>
> ---
>  hw/ppc/spapr_numa.c | 44 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 44 insertions(+)
> 
> diff --git a/hw/ppc/spapr_numa.c b/hw/ppc/spapr_numa.c
> index 36aaa273ee..180800b2f3 100644
> --- a/hw/ppc/spapr_numa.c
> +++ b/hw/ppc/spapr_numa.c
> @@ -37,6 +37,49 @@ static bool spapr_numa_is_symmetrical(MachineState *ms)
>      return true;
>  }
>  
> +/*
> + * This function will translate the user distances into
> + * what the kernel understand as possible values: 10
> + * (local distance), 20, 40, 80 and 160. Current heuristic
> + * is:
> + *
> + *  - distances between 11 and 30 -> rounded to 20
> + *  - distances between 31 and 60 -> rounded to 40
> + *  - distances between 61 and 120 -> rounded to 80
> + *  - everything above 120 -> 160

It isn't clear what happens when the distances are exactly
30, 60 or 120...

> + *
> + * This step can also be done in the same time as the NUMA
> + * associativity domains calculation, at the cost of extra
> + * complexity. We chose to keep it simpler.
> + *
> + * Note: this will overwrite the distance values in
> + * ms->numa_state->nodes.
> + */
> +static void spapr_numa_PAPRify_distances(MachineState *ms)
> +{
> +    int src, dst;
> +    int nb_numa_nodes = ms->numa_state->num_nodes;
> +    NodeInfo *numa_info = ms->numa_state->nodes;
> +
> +    for (src = 0; src < nb_numa_nodes; src++) {
> +        for (dst = src; dst < nb_numa_nodes; dst++) {
> +            uint8_t distance = numa_info[src].distance[dst];
> +            uint8_t rounded_distance = 160;
> +
> +            if (distance > 11 && distance < 30) {
> +                rounded_distance = 20;
> +            } else if (distance > 31 && distance < 60) {
> +                rounded_distance = 40;
> +            } else if (distance > 61 && distance < 120) {
> +                rounded_distance = 80;
> +            }

... and this code doesn't convert them to PAPR-friendly values
actually. I guess < should be turned into <= .

> +
> +            numa_info[src].distance[dst] = rounded_distance;
> +            numa_info[dst].distance[src] = rounded_distance;
> +        }
> +    }
> +}
> +
>  void spapr_numa_associativity_init(SpaprMachineState *spapr,
>                                     MachineState *machine)
>  {
> @@ -95,6 +138,7 @@ void spapr_numa_associativity_init(SpaprMachineState *spapr,
>          exit(1);
>      }
>  
> +    spapr_numa_PAPRify_distances(machine);
>  }
>  
>  void spapr_numa_write_associativity_dt(SpaprMachineState *spapr, void *fdt,