schedule_credit not strict in cpu_affinity

schedule_credit not strict in cpu_affinity

[Alice Wan]

[-- Attachment #1.1: Type: text/plain, Size: 3608 bytes --]

hi all,

    recently, our group did some test with vcpu-pin, and found vcpu-list
result wasn't the same as we thought. our scheduler is default
scheduler(credit), our host has 16cores with ht.

netback-test1                        1     0    11   -b-       7.1 0-1,11-12
netback-test1                        1     1     0   -b-       0.3 0-1,11-12
netback-test1                        1     2    11   -b-       0.2 0-1,11-12
netback-test1                        1     3     0   -b-       0.4 0-1,11-12
netback-test2                        2     0     7   -b-       7.1 0-1,11-12
netback-test2                        2     1    11   -b-       0.2 0-1,11-12
netback-test2                        2     2     0   -b-       0.2 0-1,11-12
netback-test2                        2     3    11   -b-       0.3 0-1,11-12
netback-test3                        3     0    11   -b-       6.9 0-1,11-12
netback-test3                        3     1    13   -b-       0.2 0-1,11-12
netback-test3                        3     2    14   -b-       0.2 0-1,11-12
netback-test3                        3     3     0   -b-       0.3 0-1,11-12

    from the result, we found domain3's vcpu1, vcpu2 were running on the
cpus which weren't in the cpu_affinity

    then, i read some code about schedule_credit's picking up cpu and find
its implement like this

        nxt = cpumask_cycle(cpu, &cpus);

        if ( cpumask_test_cpu(cpu, per_cpu(cpu_core_mask, nxt)) )
        {
            /* We're on the same socket, so check the busy-ness of threads.
             * Migrate if # of idlers is less at all */
            ASSERT( cpumask_test_cpu(nxt, per_cpu(cpu_core_mask, cpu)) );
            migrate_factor = 1;
            cpumask_and(&cpu_idlers, &idlers, per_cpu(cpu_sibling_mask,
cpu));
            cpumask_and(&nxt_idlers, &idlers, per_cpu(cpu_sibling_mask,
nxt));
        }
        else
        {
            /* We're on different sockets, so check the busy-ness of cores.
             * Migrate only if the other core is twice as idle */
            ASSERT( !cpumask_test_cpu(nxt, per_cpu(cpu_core_mask, cpu)) );
            migrate_factor = 2;
            cpumask_and(&cpu_idlers, &idlers, per_cpu(cpu_core_mask, cpu));
            cpumask_and(&nxt_idlers, &idlers, per_cpu(cpu_core_mask, nxt));
        }

        weight_cpu = cpumask_weight(&cpu_idlers);
        weight_nxt = cpumask_weight(&nxt_idlers);
        /* smt_power_savings: consolidate work rather than spreading it */
        if ( sched_smt_power_savings ?
             weight_cpu > weight_nxt :
             weight_cpu * migrate_factor < weight_nxt )
        {
            cpumask_and(&nxt_idlers, &cpus, &nxt_idlers);
            spc = CSCHED_PCPU(nxt);
            cpu = cpumask_cycle(spc->idle_bias, &nxt_idlers);
            cpumask_andnot(&cpus, &cpus, per_cpu(cpu_sibling_mask, cpu));

    if the cpu_affinity which isn't in the same socket , the picked cpu
might be one of the intersection of nxt cpu_core_mask and idlers, which
isn't in the cpu_affinity and even is in the same socket, picked cpu might
be one of intersection of nxt cpu_sibling_mask and idlers.

    it really depends on the setting of cpu_affinity. if you set all the
siblings cpu of one socket to vcpu, it works well.

    in our test environment, we use xen4.0, code doesn't
have migrate_factor, so it's easier to trigger.

    i wanna know whether this is a bug or a design consideration, in our
opinion, this scheduling result must be the same as we set even though we
need better setting.

    can anyone give some ideas about my question, thanks in advance



regards,
wanjia

[-- Attachment #1.2: Type: text/html, Size: 4613 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: schedule_credit not strict in cpu_affinity

[Alice Wan]

[-- Attachment #1.1: Type: text/plain, Size: 4039 bytes --]

sorry, we already found patch 58a83b75d17c4346f0f8f20f9815e00d6f6ed5f1

_csched_cpu_pick(): don't return CPUs outside vCPU's affinity mask

actually, it should be a bug which is already fixed.


2013/12/17 Alice Wan <wanjia19870902@gmail.com>

> hi all,
>
>     recently, our group did some test with vcpu-pin, and found vcpu-list
> result wasn't the same as we thought. our scheduler is default
> scheduler(credit), our host has 16cores with ht.
>
> netback-test1                        1     0    11   -b-       7.1
> 0-1,11-12
> netback-test1                        1     1     0   -b-       0.3
> 0-1,11-12
> netback-test1                        1     2    11   -b-       0.2
> 0-1,11-12
> netback-test1                        1     3     0   -b-       0.4
> 0-1,11-12
> netback-test2                        2     0     7   -b-       7.1
> 0-1,11-12
> netback-test2                        2     1    11   -b-       0.2
> 0-1,11-12
> netback-test2                        2     2     0   -b-       0.2
> 0-1,11-12
> netback-test2                        2     3    11   -b-       0.3
> 0-1,11-12
> netback-test3                        3     0    11   -b-       6.9
> 0-1,11-12
> netback-test3                        3     1    13   -b-       0.2
> 0-1,11-12
> netback-test3                        3     2    14   -b-       0.2
> 0-1,11-12
> netback-test3                        3     3     0   -b-       0.3
> 0-1,11-12
>
>     from the result, we found domain3's vcpu1, vcpu2 were running on the
> cpus which weren't in the cpu_affinity
>
>     then, i read some code about schedule_credit's picking up cpu and find
> its implement like this
>
>         nxt = cpumask_cycle(cpu, &cpus);
>
>         if ( cpumask_test_cpu(cpu, per_cpu(cpu_core_mask, nxt)) )
>         {
>             /* We're on the same socket, so check the busy-ness of threads.
>              * Migrate if # of idlers is less at all */
>             ASSERT( cpumask_test_cpu(nxt, per_cpu(cpu_core_mask, cpu)) );
>             migrate_factor = 1;
>             cpumask_and(&cpu_idlers, &idlers, per_cpu(cpu_sibling_mask,
> cpu));
>             cpumask_and(&nxt_idlers, &idlers, per_cpu(cpu_sibling_mask,
> nxt));
>         }
>         else
>         {
>             /* We're on different sockets, so check the busy-ness of cores.
>              * Migrate only if the other core is twice as idle */
>             ASSERT( !cpumask_test_cpu(nxt, per_cpu(cpu_core_mask, cpu)) );
>             migrate_factor = 2;
>             cpumask_and(&cpu_idlers, &idlers, per_cpu(cpu_core_mask, cpu));
>             cpumask_and(&nxt_idlers, &idlers, per_cpu(cpu_core_mask, nxt));
>         }
>
>         weight_cpu = cpumask_weight(&cpu_idlers);
>         weight_nxt = cpumask_weight(&nxt_idlers);
>         /* smt_power_savings: consolidate work rather than spreading it */
>         if ( sched_smt_power_savings ?
>              weight_cpu > weight_nxt :
>              weight_cpu * migrate_factor < weight_nxt )
>         {
>             cpumask_and(&nxt_idlers, &cpus, &nxt_idlers);
>             spc = CSCHED_PCPU(nxt);
>             cpu = cpumask_cycle(spc->idle_bias, &nxt_idlers);
>             cpumask_andnot(&cpus, &cpus, per_cpu(cpu_sibling_mask, cpu));
>
>     if the cpu_affinity which isn't in the same socket , the picked cpu
> might be one of the intersection of nxt cpu_core_mask and idlers, which
> isn't in the cpu_affinity and even is in the same socket, picked cpu might
> be one of intersection of nxt cpu_sibling_mask and idlers.
>
>     it really depends on the setting of cpu_affinity. if you set all the
> siblings cpu of one socket to vcpu, it works well.
>
>     in our test environment, we use xen4.0, code doesn't
> have migrate_factor, so it's easier to trigger.
>
>     i wanna know whether this is a bug or a design consideration, in our
> opinion, this scheduling result must be the same as we set even though we
> need better setting.
>
>     can anyone give some ideas about my question, thanks in advance
>
>
>
> regards,
> wanjia
>
>
>
>
>
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 5251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: schedule_credit not strict in cpu_affinity

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 631 bytes --]

On mar, 2013-12-17 at 12:16 +0800, Alice Wan wrote:
> sorry, we already found patch 58a83b75d17c4346f0f8f20f9815e00d6f6ed5f1
> 
> _csched_cpu_pick(): don't return CPUs outside vCPU's affinity mask
> 
> 
> actually, it should be a bug which is already fixed.
> 
Right! :-)

BTW, what version of Xen are you using / are you interested in?

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: schedule_credit not strict in cpu_affinity

[Alice Wan]

[-- Attachment #1.1: Type: text/plain, Size: 1521 bytes --]

we're using a very old version of Xen4.0.x, due to  the upgrading problem
that we must stop machine, we haven't upgrade to higher version.  however,
we already prepare for upgrading.

of course, we're interested in the newest stable version which can have
better performance of disk io, network, scheduler, irq,etc and less bugs.

by the way, i also wanna know more details about the load balance
scalability of credit1 when host has many cores(64),  we said it didn't
scale well, is because it may spent much time on picking cpu when cores
become larger, and may migrate to other idle cpus easily, increasing the
schedule times ?

and what's the plan about credit2, when it become default scheduler and
finish a lot of todoes, especially ' "Discount" time run on a thread with
busy siblings'

thanks


regards,
wanjia


2013/12/17 Dario Faggioli <dario.faggioli@citrix.com>

> On mar, 2013-12-17 at 12:16 +0800, Alice Wan wrote:
> > sorry, we already found patch 58a83b75d17c4346f0f8f20f9815e00d6f6ed5f1
> >
> > _csched_cpu_pick(): don't return CPUs outside vCPU's affinity mask
> >
> >
> > actually, it should be a bug which is already fixed.
> >
> Right! :-)
>
> BTW, what version of Xen are you using / are you interested in?
>
> Regards,
> Dario
>
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>
>

[-- Attachment #1.2: Type: text/html, Size: 2269 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel