Re: Analyzed/Solved/Bisected: Booting 2.6.30-rc2-git7 very slow

Matthew Wilcox
2009-05-27T11:22:00Z

On Tue, May 26, 2009 at 11:31:02PM -0700, Andrew Morton wrote:
> On Wed, 20 May 2009 03:22:28 -0700 (PDT) Martin Knoblauch <knobi@knobisoft.de> wrote:
>
> >
> > ----- Original Message ----
> >
> > > From: Mike Galbraith <efault@gmx.de>
> > > To: Martin Knoblauch <knobi@knobisoft.de>
> > > Cc: Andrew Morton <akpm@linux-foundation.org>; viro@ZenIV.linux.org.uk; rjw@sisk.pl; linux-kernel@vger.kernel.org; tigran@aivazian.fsnet.co.uk
> > > Sent: Wednesday, May 6, 2009 10:37:45 AM
> > > Subject: Re: Analyzed/Solved: Booting 2.6.30-rc2-git7 very slow
> > >
> > > On Wed, 2009-05-06 at 00:55 -0700, Martin Knoblauch wrote:
> > >
> > > > just to bring this back to my problem :-)
> > >
> > > Good idea :-)
> > >
> > > > Last week I reported that the "new" sysfs entry in /proc/mounts already comes
> > > out of initrd. Does this ring a bell?
> > > >
> > > > http://lkml.indiana.edu/hypermail/linux/kernel/0904.3/03048.html
> > >
> > > Nope, no bells.
> > >
> > > The only thing I can suggest is that you try a bisection.
> > >
> > > -Mike
> >
> > OK, so I finally managed to bisect the issue down to the following commit. Not much that I can say about it. Someone else suggested that it might all be a question of timing. Might very well be. I will try it out on a system with a different SCSI/RAID controller. The failing system has an "Smart Array 6i" (cciss). "cciss", "ext3" and "jbd" are all modules coming from initrd.
> >
> > |commit 1120f8b8169fb2cb51219d326892d963e762edb6
> > |Author: Stephen Hemminger <shemminger@vyatta.com>
> > |Date: Thu Dec 18 09:17:16 2008 -0800
> > |
> > | PCI: handle long delays in VPD access
> > |
> > | Accessing the VPD area can take a long time. The existing
> > | VPD access code fails consistently on my hardware. There are comments
> > |
> > | Change the access routines to:
> > | * use a mutex rather than spinning with IRQ's disabled and lock held
> > | * have a much longer timeout
> > | * call cond_resched while spinning
> > |
> > | Signed-off-by: Stephen Hemminger <shemminger@vyatta.com>
> > | Reviewed-by: Matthew Wilcox <willy@linux.intel.com>
> > | Signed-off-by: Jesse Barnes <jbarnes@virtuousgeek.org>
> >
>
> <hello, any maintainers out there?>

This is the first I've seen of this report ...

> So afacit what's happening is that the above change caused one of your
> PCI devices to take a very long time to initialise, yes? Was it the
> CCISS driver?
>
> If you add "printk.time=y" to the kernel boot command line then you'll
> get timestamped boot messages which will make it easier to determine
> where the time was consumed. Adding `initcall_debug' to the boot line
> will help us delve further into the delay, assuming that the offending
> driver is build into vmlinux (which it might not be).

The two message logs posted show NTP starting up within a second of
each other. What was the problem again?

--
Matthew Wilcox Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours. We can't possibly take such
a retrograde step."