Re: [PATCH] Xen sched: Fix multiple runqueues in credit2

[Juergen Gross <juergen.gross@ts.fujitsu.com>]

On 06.02.2014 09:58, Justin Weaver wrote:
> This patch attempts to address the issue of the Xen Credit 2
> Scheduler only creating one vCPU run queue on multiple physical
> processor systems. It should be creating one run queue per
> physical processor.
>
> CPU 0 does not get a starting callback, so it is hard coded to run
> queue 0. At the time this happens, socket information is not
> available for CPU 0.
>
> Socket information is available for each individual CPU when each
> gets the STARTING callback (I believe socket information is also
> available for CPU 0 by that time). This patch adds the following
> algorithm...
>
> IF cpu is on the same socket as CPU 0, add it to run queue 0

You should check whether cpu and CPU0 are in the same cpupool.

BTW: CPU0 is allowed to be moved to another cpupool, too.


Juergen

> ELSE, IF cpu is on socket 0, add it to a run queue based on the
>           socket CPU 0 is actually on
>        ELSE add it to a run queue based on the socket it is on
> ---
>   xen/common/sched_credit2.c |   37 +++++++++++++++++++++++++++----------
>   1 file changed, 27 insertions(+), 10 deletions(-)
>
> diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
> index 4e68375..c0ecb50 100644
> --- a/xen/common/sched_credit2.c
> +++ b/xen/common/sched_credit2.c
> @@ -85,8 +85,7 @@
>    * to a small value, and a fixed credit is added to everyone.
>    *
>    * The plan is for all cores that share an L2 will share the same
> - * runqueue.  At the moment, there is one global runqueue for all
> - * cores.
> + * runqueue.
>    */
>
>   /*
> @@ -1945,6 +1944,8 @@ static void deactivate_runqueue(struct csched_private *prv, int rqi)
>   static void init_pcpu(const struct scheduler *ops, int cpu)
>   {
>       int rqi;
> +    int cpu0_socket;
> +    int cpu_socket;
>       unsigned long flags;
>       struct csched_private *prv = CSCHED_PRIV(ops);
>       struct csched_runqueue_data *rqd;
> @@ -1962,12 +1963,28 @@ static void init_pcpu(const struct scheduler *ops, int cpu)
>       /* Figure out which runqueue to put it in */
>       rqi = 0;
>
> -    /* Figure out which runqueue to put it in */
>       /* NB: cpu 0 doesn't get a STARTING callback, so we hard-code it to runqueue 0. */
>       if ( cpu == 0 )
>           rqi = 0;
>       else
> -        rqi = cpu_to_socket(cpu);
> +    {
> +        cpu_socket = cpu_to_socket(cpu);
> +        cpu0_socket = cpu_to_socket(0);
> +
> +        /* If cpu is on the same socket as CPU 0, put it with CPU 0 on run queue 0 */
> +        if ( cpu_socket == cpu0_socket )
> +            rqi = 0;
> +        else
> +            /* If cpu is on socket 0, assign it to a run queue based on the
> +             * socket CPU 0 is actually on */
> +            if ( cpu_socket == 0 )
> +                rqi = cpu0_socket;
> +
> +            /* If cpu is NOT on socket 0, just assign it to a run queue based on
> +             * its own socket */
> +            else
> +                rqi = cpu_socket;
> +    }
>
>       if ( rqi < 0 )
>       {
> @@ -2010,13 +2027,11 @@ static void init_pcpu(const struct scheduler *ops, int cpu)
>   static void *
>   csched_alloc_pdata(const struct scheduler *ops, int cpu)
>   {
> -    /* Check to see if the cpu is online yet */
> -    /* Note: cpu 0 doesn't get a STARTING callback */
> -    if ( cpu == 0 || cpu_to_socket(cpu) >= 0 )
> +    /* This function is only for calling init_pcpu on CPU 0
> +     * because it does not get a STARTING callback */
> +
> +    if ( cpu == 0 )
>           init_pcpu(ops, cpu);
> -    else
> -        printk("%s: cpu %d not online yet, deferring initializatgion\n",
> -               __func__, cpu);
>
>       return (void *)1;
>   }
> @@ -2072,6 +2087,8 @@ csched_free_pdata(const struct scheduler *ops, void *pcpu, int cpu)
>   static int
>   csched_cpu_starting(int cpu)
>   {
> +    // This function is for calling init_pcpu on every CPU, except for CPU 0 */
> +
>       struct scheduler *ops;
>
>       /* Hope this is safe from cpupools switching things around. :-) */
>


-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 62060 2932
Fujitsu                                   e-mail: juergen.gross@ts.fujitsu.com
Mies-van-der-Rohe-Str. 8                Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html