per_cpu problem

per_cpu problem

[Juergen Gross]

Hi,

I've been playing a little bit with cpu-bound tasklets. For this purpose I
defined a per_cpu list:

static DEFINE_PER_CPU(struct list_head, tasklet_list_pcpu);

The initialization is done in softirq_init:

void __init softirq_init(void)
{
    int i;

    for_each_possible_cpu ( i )
    {
        INIT_LIST_HEAD(&per_cpu(tasklet_list_pcpu, i));
    }
    tasklet_pcpu_inited = 1;
    open_softirq(TASKLET_SOFTIRQ, tasklet_action);
}

As soon as a cpu other than cpu 0 is accessing its tasklet_list_pcpu it is
seeing a non-empty list (I've put a printk in tasklet_action):

printk("tasklet_list_pcpu(%d) at %p not empty: %p\n", smp_processor_id(),
       &this_cpu(tasklet_list_pcpu), this_cpu(tasklet_list_pcpu).next);

Prints out:
(XEN) tasklet_list_pcpu(1) at ffff82c48026a100 not empty: ffff82c480268100

Somehow INIT_LIST_HEAD seems to put always the address of the list for cpu 0
in the list header. Could this be a compiler bug? Is this an artefact of the
per_cpu macro? Am I missing something?


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
TSP ES&S SWE OS6                       Telephone: +49 (0) 89 3222 2967
Fujitsu Technolgy Solutions               e-mail: juergen.gross@ts.fujitsu.com
Domagkstr. 28                           Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: per_cpu problem

[Juergen Gross]

Juergen Gross wrote:
> Hi,
> 
> I've been playing a little bit with cpu-bound tasklets. For this purpose I
> defined a per_cpu list:
> 
> static DEFINE_PER_CPU(struct list_head, tasklet_list_pcpu);
> 
> The initialization is done in softirq_init:
> 
> void __init softirq_init(void)
> {
>     int i;
> 
>     for_each_possible_cpu ( i )
>     {
>         INIT_LIST_HEAD(&per_cpu(tasklet_list_pcpu, i));
>     }
>     tasklet_pcpu_inited = 1;
>     open_softirq(TASKLET_SOFTIRQ, tasklet_action);
> }
> 
> As soon as a cpu other than cpu 0 is accessing its tasklet_list_pcpu it is
> seeing a non-empty list (I've put a printk in tasklet_action):
> 
> printk("tasklet_list_pcpu(%d) at %p not empty: %p\n", smp_processor_id(),
>        &this_cpu(tasklet_list_pcpu), this_cpu(tasklet_list_pcpu).next);
> 
> Prints out:
> (XEN) tasklet_list_pcpu(1) at ffff82c48026a100 not empty: ffff82c480268100
> 
> Somehow INIT_LIST_HEAD seems to put always the address of the list for cpu 0
> in the list header. Could this be a compiler bug? Is this an artefact of the
> per_cpu macro? Am I missing something?

Okay, I've narrowed things down. I've put a printk in the initialization loop
showing the list address and the contents of list->next:

(XEN) initialized tasklet_list_pcpu(1) at ffff82c48026a100 with ffff82c48026a100

Later the printk in tasklet_action says:

(XEN) tasklet_list_pcpu(1) at ffff82c48026a100 not empty: ffff82c480268100

Who is copying cpu0 data to cpu1???
If this isn't a bug but a feature, I suspect the following in schedule.c is
working only by luck:

void __init scheduler_init(void)
{
    int i;

    open_softirq(SCHEDULE_SOFTIRQ, schedule);

    for_each_possible_cpu ( i )
    {
        spin_lock_init(&per_cpu(schedule_data, i).schedule_lock);
        init_timer(&per_cpu(schedule_data, i).s_timer, s_timer_fn, NULL, i);
    }
...



Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
TSP ES&S SWE OS6                       Telephone: +49 (0) 89 3222 2967
Fujitsu Technolgy Solutions               e-mail: juergen.gross@ts.fujitsu.com
Domagkstr. 28                           Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: per_cpu problem

[Keir Fraser]

On 29/12/2009 13:22, "Juergen Gross" <juergen.gross@ts.fujitsu.com> wrote:

> Who is copying cpu0 data to cpu1???
> If this isn't a bug but a feature, I suspect the following in schedule.c is
> working only by luck:

It's a feature, kicking in too late. I'll fix this up for you.

 -- Keir

Re: per_cpu problem

[Keir Fraser]

On 29/12/2009 15:01, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

> On 29/12/2009 13:22, "Juergen Gross" <juergen.gross@ts.fujitsu.com> wrote:
> 
>> Who is copying cpu0 data to cpu1???
>> If this isn't a bug but a feature, I suspect the following in schedule.c is
>> working only by luck:
> 
> It's a feature, kicking in too late. I'll fix this up for you.

Fixed by changeset 20731.

 -- Keir