Re: 2.6.26-git0: IDE oops during boot

[Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>]

Bartlomiej Zolnierkiewicz wrote:
> Hi,
> 
> On Monday 11 February 2008, Kamalesh Babulal wrote:
>> Nish Aravamudan wrote:
>>> On 2/7/08, Bartlomiej Zolnierkiewicz <bzolnier@gmail.com> wrote:
>>>> On Thursday 07 February 2008, Kamalesh Babulal wrote:
>>>>> Bartlomiej Zolnierkiewicz wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On Wednesday 06 February 2008, Pavel Machek wrote:
>>>>>>> On Wed 2008-02-06 11:53:34, Pavel Machek wrote:
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> Trying to boot 2.6.25-git0 (few days old), I get
>>>>>>>>
>>>>>>>> BUG: unable to handle kernel paging request at ffff..ffb0
>>>>>>>> IP at init_irq+0x42e
>>>>>> init_irq? hmm...
>>>>>>
>>>>>>>> Call trace:
>>>>>>>> ide_device_add_all
>>>>>> this comes from ide-generic
>>>>>> (Generic IDE host driver)
>>>>>>
>>>>>>>> ide_generic_init
>>>>>>>> kernel_init
>>>>>>>> child_rip
>>>>>>>> vgacon_cursor
>>>>>>>> kernel_init
>>>>>>>> child_rip
>>>>>>>>
>>>>>>>> Excerpt from config:
>>>>>>>>
>>>>>>>> CONFIG_IDE=y
>>>>>>>> CONFIG_BLK_DEV_IDE=y
>>>>>>> Disabling CONFIG_IDE made my machine boot, as it was using libata
>>>>>>> anyway.
>>>>>> Kamalesh/Pavel:
>>>>>>
>>>>>> Could you try latest git and see if the OOPS is still there?
>>>>>>
>>>>>> [ Yeah, I'm unable to reproduce it. :( ]
>>>>>>
>>>>>> Thanks,
>>>>>> Bart
>>>>> Hi Bart,
>>>>>
>>>>> The panic is reproducible with the 2.6.24-git16 kernel, the call trace is
>>>>> similar to the previous one
>>>> Thanks, I again reviewed ide-probe.c changes but nothing seems wrong...
>>>>
>>>> Could you please bisect it down to the guilty commit?
>>> Kamalesh, were you able to bisect this down? I just got hit by the
>>> same panic on a 4-way x86_64, with 2.6.24-git22.
>>>
>>> Thanks,
>>> Nish
>> Hi Nish,
>>
>> I tried bisecting and the guilty patch seems to be 
>>
>> 36501650ec45b1db308c3b51886044863be2d762 is first bad commit
>> commit 36501650ec45b1db308c3b51886044863be2d762
>> Author: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
>> Date:   Fri Feb 1 23:09:31 2008 +0100
>>
>>     ide: keep pointer to struct device instead of struct pci_dev in ide_hwif_t
>>
>>
>> the gdb output, also points to the changes made by the guilty patch
>>
>> (gdb) p ide_device_add_all
>> $1 = {int (u8 *, const struct ide_port_info *)} 0xffffffff804176ac <ide_device_add_all>
>> (gdb) p/x 0xffffffff804176ac+0xb60
>> $2 = 0xffffffff8041820c
>> (gdb) l *0xffffffff8041820c
>> 0xffffffff8041820c is in ide_device_add_all (drivers/ide/ide-probe.c:1249).
>> 1244                    goto out;
>> 1245            }
>> 1246
>> 1247            sg_init_table(hwif->sg_table, hwif->sg_max_nents);
>> 1248
>> 1249            if (init_irq(hwif) == 0)
>> 1250                    goto done;
>> 1251
>> 1252            old_irq = hwif->irq;
>> 1253            /*
>> (gdb) 
>>
>>
>> (gdb) p init_irq
>> $1 = {int (ide_hwif_t *)} 0xffffffff8041721f <init_irq>
>> (gdb) p/x 0xffffffff8041721f+0x1a4
>> $2 = 0xffffffff804173c3
>> (gdb) l *0xffffffff804173c3
>> 0xffffffff804173c3 is in init_irq (include/asm/pci.h:101).
>> 96      /* Returns the node based on pci bus */
>> 97      static inline int __pcibus_to_node(struct pci_bus *bus)
>> 98      {
>> 99              struct pci_sysdata *sd = bus->sysdata;
>> 100
>> 101             return sd->node;
>> 102     }
>> 103
>> 104     static inline cpumask_t __pcibus_to_cpumask(struct pci_bus *bus)
>> 105     {
>> (gdb) 
> 
> Thanks for the detailed analysis and sorry for the bug.
> 
> I think that this may has been just fixed by Andi's recent hwif_to_node()
> fix (patch below, it is in Linus' tree already), could please verify this?
> 
> commit 1f07e988290fc45932f5028c9e2a862c37a57336
> Author: Andi Kleen <andi@firstfloor.org>
> Date:   Mon Feb 11 01:35:20 2008 +0100
> 
>     Prevent IDE boot ops on NUMA system
>     
>     Without this patch a Opteron test system here oopses at boot with
>     current git.
>     
>     Calling to_pci_dev() on a NULL pointer gives a negative value so the
>     following NULL pointer check never triggers and then an illegal address
>     is referenced.  Check the unadjusted original device pointer for NULL
>     instead.
>     
>     Signed-off-by: Andi Kleen <ak@suse.de>
>     Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
> 
> diff --git a/include/linux/ide.h b/include/linux/ide.h
> index 23fad89..a3b69c1 100644
> --- a/include/linux/ide.h
> +++ b/include/linux/ide.h
> @@ -1295,7 +1295,7 @@ static inline void ide_dump_identify(u8 *id)
>  static inline int hwif_to_node(ide_hwif_t *hwif)
>  {
>  	struct pci_dev *dev = to_pci_dev(hwif->dev);
> -	return dev ? pcibus_to_node(dev->bus) : -1;
> +	return hwif->dev ? pcibus_to_node(dev->bus) : -1;
>  }
> 
>  static inline ide_drive_t *ide_get_paired_drive(ide_drive_t *drive)
Hi Bart,
Thanks !! the patch solves the kernel panic but when after applying the patch,kernel is not
able to mount the filesystem and panics, am i not sure what is likely causing the panic.

Creating root device.
Mounting root filesystem.
mount: could not  find filesystem
Kernel panic - not syncing: Attempted to kill init!


-- 
Thanks & Regards,
Kamalesh Babulal,
Linux Technology Center,
IBM, ISTL.