Re: [3.4-rc1 crash]: NULL pointer deref in fs/sysfs/group.c:create_files -- sysctl related?

[David Ahern <dsahern@gmail.com>]

On 4/2/12 1:34 PM, Bruno Prémont wrote:
> [adding a few perf people to CC as might originate from perf]
>
> On Mon, 02 April 2012 Eric W. Biederman wrote:
>> Bruno Prémont writes:
>>> On Mon, 2 Apr 2012 16:27:16 Bruno Prémont wrote:
>>>> Trying to boot a freshly built 3.4-rc1 (x86_64) kernel I'm getting the following
>>>> trace (server is HP Proliant G4):
>>>>
>>>> [    0.986317] BUG: unable to handle kernel NULL pointer dereference at           (null)
>>>> [    0.990542] IP: [<ffffffff81152673>] internal_create_group+0x83/0x1a0
>>>> [    0.993693] PGD 0
>>>> [    0.994682] Oops: 0000 [#1] SMP
>>>> [    0.996198] CPU 0
>>>> [    0.996198] Modules linked in:
>>>> [    0.996198]
>>>> [    0.996198] Pid: 1, comm: swapper/0 Not tainted 3.4.0-rc1-x86_64 #3 HP ProLiant DL360 G4
>>>> [    0.996198] RIP: 0010:[<ffffffff81152673>]  [<ffffffff81152673>] internal_create_group+0x83/0x1a0
>>>> [    0.996198] RSP: 0018:ffff88019485fd70  EFLAGS: 00010202
>>>> [    0.996198] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000001
>>>> [    0.996198] RDX: ffff880192e99908 RSI: ffff880192e99630 RDI: ffffffff81a26c60
>>>> [    0.996198] RBP: ffff88019485fdc0 R08: 0000000000000000 R09: 0000000000000000
>>>> [    0.996198] R10: ffff880192e99908 R11: 0000000000000000 R12: ffffffff81a16a00
>>>> [    0.996198] R13: ffff880192e99908 R14: ffffffff81a16900 R15: 0000000000000000
>>>> [    0.996198] FS:  0000000000000000(0000) GS:ffff88019bc00000(0000) knlGS:0000000000000000
>>>> [    0.996198] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>>> [    0.996198] CR2: 0000000000000000 CR3: 0000000001a0c000 CR4: 00000000000007f0
>>>> [    0.996198] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>> [    0.996198] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>>> [    0.996198] Process swapper/0 (pid: 1, threadinfo ffff88019485e000, task ffff880194878000)
>>>> [    0.996198] Stack:
>>>> [    0.996198]  ffff88019485fdd0 ffff880192da9d60 0000000000000000 ffff880192e99908
>>>> [    0.996198]  ffff880192e995d8 0000000000000001 ffffffff81a16a00 ffff880192da9d60
>>>> [    0.996198]  0000000000000000 0000000000000000 ffff88019485fdd0 ffffffff811527be
>>>> [    0.996198] Call Trace:
>>>> [    0.996198]  [<ffffffff811527be>] sysfs_create_group+0xe/0x10
>>>> [    0.996198]  [<ffffffff81376ca6>] device_add_groups+0x46/0x80
>>>> [    0.996198]  [<ffffffff81377d3d>] device_add+0x46d/0x6a0
>>>> [    0.996198]  [<ffffffff81377891>] ? device_private_init+0x51/0x90
>>>> [    0.996198]  [<ffffffff81a98975>] ? utsname_sysctl_init+0x14/0x14
>>>> [    0.996198]  [<ffffffff810a7228>] pmu_dev_alloc+0x98/0xe0
>>>> [    0.996198]  [<ffffffff81a98975>] ? utsname_sysctl_init+0x14/0x14
>>>> [    0.996198]  [<ffffffff81a989c0>] perf_event_sysfs_init+0x4b/0x9a
>>>> [    0.996198]  [<ffffffff810002ad>] do_one_initcall+0x3d/0x170
>>>> [    0.996198]  [<ffffffff81a85cbd>] kernel_init+0x12d/0x1be
>>>> [    0.996198]  [<ffffffff81a85505>] ? rdinit_setup+0x28/0x28
>>>> [    0.996198]  [<ffffffff815f3714>] kernel_thread_helper+0x4/0x10
>>>> [    0.996198]  [<ffffffff81a85b90>] ? start_kernel+0x373/0x373
>>>> [    0.996198]  [<ffffffff815f3710>] ? gs_change+0xb/0xb
>>>> [    0.996198] Code: ff 85 c0 0f 85 bc 00 00 00 4c 8b 6d c8 4d 85 ed 74 15 41 8b 45 00 85 c0 0f 84 0b 01 00 00 f0 41 ff 45 00 4c 8b 6d c8 49 8b 5e 10<48>  8b 03 48 85 c0 74 71 45 31 e4 eb 44 49 8b 46 08 48 85 c0 74
>>>> [    0.996198] RIP  [<ffffffff81152673>] internal_create_group+0x83/0x1a0
>>>> [    0.996198]  RSP<ffff88019485fd70>
>>>> [    0.996198] CR2: 0000000000000000
>>>> [    1.131357] ---[ end trace 319c95c486d7d9cd ]---
>>>> [    1.133676] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009
>>>> [    1.133677]
>>>
>>> The patch below works around it and leaves exactly one trace for WARN_ON() matching
>>> above BUG.
>>> With it, system boots to userspace.
>>>
>>> Thanks,
>>> Bruno
>>>
>>> ---
>>> diff --git a/fs/sysfs/group.c b/fs/sysfs/group.c
>>> index dd1701c..0040ff2 100644
>>> --- a/fs/sysfs/group.c
>>> +++ b/fs/sysfs/group.c
>>> @@ -32,7 +32,8 @@ static int create_files(struct sysfs_dirent *dir_sd, struct kobject *kobj,
>>>   	struct attribute *const* attr;
>>>   	int error = 0, i;
>>>
>>> -	for (i = 0, attr = grp->attrs; *attr&&  !error; i++, attr++) {
>>> +	WARN_ON(!grp->attrs);
>>> +	for (i = 0, attr = grp->attrs; attr&&  *attr&&  !error; i++, attr++) {
>>>   		umode_t mode = 0;
>>>
>>>   		/* in update mode, we're changing the permissions or
>>
>> Sysfs has not changed in this area from 3.3.
>>
>> The sysctl in your backtrace looks like left over addresses on the stack.
>>
>> The backtrack indicates this is something perf related going wonky.
>>
>> I would suggest you try disabling your perf related options one by one
>> until the broken one shows up.  Or possibly just initially disable perf.
>
> Well, I didn't enable perf, all perf-related options that are enabled
> are selected by X86!
>
> Symbol: HAVE_PERF_EVENTS [=y]
>    Type  : boolean
>    Selected by: X86 [=y]
>
> Symbol: PERF_EVENTS [=y]
>    Type  : boolean
>    Prompt: Kernel performance events and counters
>            Defined at init/Kconfig:1157
>    Depends on: HAVE_PERF_EVENTS [=y]
>    Location:
>      ->  General setup
>        ->  Kernel Performance Events And Counters
>    Selects: ANON_INODES [=y]&&  IRQ_WORK [=y]
>    Selected by: X86 [=y] || KVM [=n]&&  VIRTUALIZATION [=n]&&  HAVE_KVM [=y]&&  PCI [=y]&&  NET [=y]
>
> Symbol: HAVE_PERF_EVENTS_NMI [=y]
>    Type  : boolean
>    Selected by: X86 [=y]
>
>> This looks like one of those crazy things where something registers
>> with the perf subsystem, and then perf later registers it with sysfs,
>> and whatever was registered did not have set the needed group attrs.
>
>
>  From quick-reading kernel/events/core.c which contains perf_event_sysfs_init()
> and pmu_dev_alloc() and commits from v3.3..v3.4 for that file
> commit 0c9d42ed4cee2aa1dfc3a260b741baae8615744f (perf, x86: Provide means
> for disabling userspace RDPMC) by Peter looks like a possible
> candidate or at least startpoint:
>
> diff --git a/kernel/events/core.c b/kernel/events/core.c
> index 05affc3..dcd4049 100644
> --- a/kernel/events/core.c
> +++ b/kernel/events/core.c
> @@ -5505,6 +5505,7 @@ static int pmu_dev_alloc(struct pmu *pmu)
>          if (!pmu->dev)
>                  goto out;
>
> +       pmu->dev->groups = pmu->attr_groups;
>          device_initialize(pmu->dev);
>          ret = dev_set_name(pmu->dev, "%s", pmu->name);
>          if (ret)
>
> Will try bisecting corresponding merge tomorrow when I have full access to affected
> system.

Perhaps:
641cc93 perf: Adding sysfs group format attribute for pmu device

David