Re: [PATCH] sched_groups are expected to be circular linked list, make it so right after allocation

[Igor Mammedov <imammedo@redhat.com>]

On 05/09/2012 12:21 PM, Jiang Liu wrote:
> Hi Igor,
> 	Thanks for fixing this bug! We encountered the same issue with an
> IA64 systems too. That system could boot with 2.6.32, but can't boot with
> any 3.x.x kernels. We have just found the root cause today.
> 	--gerry

This patch fixes only build_sched_groups path, but there is another fail path
that results in below OOPS.
build_overlap_sched_groups() may exit without setting groups and later it will crash
init_sched_groups_power as well. But I just don't know how to fix it, so I've just
posted partial fix that reduces crash frequency. And I have to admit that
cpu_active_mask and siblings map are busted but we either should not exit from builder
funcs with NULL group or BUG there if it is impossible to come-up with sane group
for insane domain span.


>
> On 05/09/2012 06:38 PM, Igor Mammedov wrote:
>> if we have one cpu that failed to boot and boot cpu gave up on waiting for it
>> and then another cpu is being booted, kernel might crash with following OOPS:
>>
>> [  723.865765] BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
>> [  723.866616] IP: [<ffffffff812c3630>] __bitmap_weight+0x30/0x80
>> [  723.866616] PGD 7ba91067 PUD 7a205067 PMD 0
>> [  723.866616] Oops: 0000 [#1] SMP
>> [  723.898527] CPU 1
>> ...
>> [  723.898527] Pid: 1221, comm: offV2.sh Tainted: G        W    3.4.0-rc4+ #213 Red Hat KVM
>> [  723.898527] RIP: 0010:[<ffffffff812c3630>]  [<ffffffff812c3630>] __bitmap_weight+0x30/0x80
>> [  723.898527] RSP: 0018:ffff88007ab9dc18  EFLAGS: 00010246
>> [  723.898527] RAX: 0000000000000003 RBX: 0000000000000000 RCX: 0000000000000000
>> [  723.898527] RDX: 0000000000000018 RSI: 0000000000000100 RDI: 0000000000000018
>> [  723.898527] RBP: ffff88007ab9dc18 R08: 0000000000000000 R09: 0000000000000020
>> [  723.898527] R10: 0000000000000004 R11: 0000000000000000 R12: ffff88007c06ed60
>> [  723.898527] R13: ffff880037a94000 R14: 0000000000000003 R15: ffff88007c06ed60
>> [  723.898527] FS:  00007f1d6a7d8700(0000) GS:ffff88007fc80000(0000) knlGS:0000000000000000
>> [  723.898527] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [  723.898527] CR2: 0000000000000018 CR3: 000000007bb7f000 CR4: 00000000000007e0
>> [  723.898527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [  723.898527] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [  723.898527] Process offV2.sh (pid: 1221, threadinfo ffff88007ab9c000, task ffff88007b358000)
>> [  723.898527] Stack:
>> [  723.898527]  ffff88007ab9dcc8 ffffffff8108b9b6 ffff88007ab9dc58 ffff88007b4f2a00
>> [  723.898527]  ffff88007c06ed60 0000000000000003 000000037ab9dc58 0000000000010008
>> [  723.898527]  ffffffff81a308e8 0000000000000003 ffff88007b489cc0 ffff880037b6bd20
>> [  723.898527] Call Trace:
>> [  723.898527]  [<ffffffff8108b9b6>] build_sched_domains+0x7b6/0xa50
>> [  723.898527]  [<ffffffff8108bea9>] partition_sched_domains+0x259/0x3f0
>> [  723.898527]  [<ffffffff810c4485>] cpuset_update_active_cpus+0x85/0x90
>> [  723.898527]  [<ffffffff81084f65>] cpuset_cpu_active+0x25/0x30
>> [  723.898527]  [<ffffffff81545b45>] notifier_call_chain+0x55/0x80
>> [  723.898527]  [<ffffffff8107e59e>] __raw_notifier_call_chain+0xe/0x10
>> [  723.898527]  [<ffffffff81058be0>] __cpu_notify+0x20/0x40
>> [  723.898527]  [<ffffffff8153af08>] _cpu_up+0xc7/0x10e
>> [  723.898527]  [<ffffffff8153af9b>] cpu_up+0x4c/0x5c
>>
>> crash happens in  init_sched_groups_power() that expects sched_groups to be
>> circular linked list. However it is not always true, since sched_groups
>> preallocated in __sdt_alloc are initialized in build_sched_groups and it
>> may exit early
>>
>>          if (cpu != cpumask_first(sched_domain_span(sd)))
>>                  return 0;
>>
>> without initializing sd->groups->next field.
>>
>> Fix bug by initializing next field right after sched_group was allocated.
>>
>> Signed-off-by: Igor Mammedov<imammedo@redhat.com>
>> ---
>>   kernel/sched/core.c |    2 ++
>>   1 files changed, 2 insertions(+), 0 deletions(-)
>>
>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>> index 0533a68..e5212ae 100644
>> --- a/kernel/sched/core.c
>> +++ b/kernel/sched/core.c
>> @@ -6382,6 +6382,8 @@ static int __sdt_alloc(const struct cpumask *cpu_map)
>>   			if (!sg)
>>   				return -ENOMEM;
>>
>> +			sg->next = sg;
>> +
>>   			*per_cpu_ptr(sdd->sg, j) = sg;
>>
>>   			sgp = kzalloc_node(sizeof(struct sched_group_power),
>

-- 
-----
  Igor