Re: [PATCH 0/14] init: deps: dependency based (parallelized) init

[Greg Kroah-Hartman <gregkh@linuxfoundation.org>]

On Sat, Oct 17, 2015 at 08:19:09PM +0200, Alexander Holler wrote:
> >>Another problem is that the link order can't be modified dynamically at
> >>boot time to include dependencies dictated by the hardware. To circumvent
> >>this, a brute-force trial-and-error mechanism called deferred probes has
> >>been introduced, but this approach, while beeing KISS, has its own
> >>problems.
> >
> >What problems does deferred probing have?  Why not just fix that if
> >there is issues with it, as it was supposed to solve this issue without
> >needing to annotate anything.
> 
> I've not looked why deferred probes are sometimes causing such a large
> delay. But giving that it's brutforce and non-deterministic, some drivers
> might be probed a dozen times, or important drivers might be probed very
> late, forcing all previous probed drivers to be probed again (late). Just
> think at the case the link order is a, b, c but drivers have to be called in
> the order c, b, a.

So how long does that really take to call all probe functions in all
possible order?  Real numbers please.  We have the tools to determine
where at boot time delays are happening, please use them to find the
problem drivers.

> >>To solve these problems I've written patches to use a topological sort at
> >>boot time which uses dependencies to calculate the order with which
> >>initcalls are called.
> >>
> >>Why? What are the benefits (assuming correct dependencies are available)?
> >>
> >>- It offers a clear in-source documentation for dependencies between
> >>   initcalls.
> >>- It is robust in regard to file or directory name changes and changes in
> >>   a Makefile.
> >>- If enabled, the order with which drivers for interfaces are called
> >>   (e.g. network interfaces, hard disks), can be defined independent of
> >>   the link order. These might result in more stable interface names or
> >>   numbers.
> >>- If enabled, it makes the the deferred probes obsolete, which might
> >>   result in faster boot times.
> >>- If enabled, it is possible to call initcalls in parallel. E.g. the
> >>   shipped kernel for Fedora 21 (4.1.7-100.fc21.x86_64) contains around
> >>   560 initcalls. These are all called in series. Also some of them use
> >>   asynchronous stuff by themself, most don't do.
> >
> >But that shipped kernel boots to X in less than 2 seconds, so there
> >isn't really a speed issue here, right?
> 
> It's noticeable if your phone, or any other thing you want to use (like your
> route planner, clock or whatever boots in one second instead of two when you
> turn it on.

My phone's kernel boots in less time than I can notice it, it's
userspace that takes forever to start up.  And there are other solutions
for that, look at what Sony has done for years with the cameras they
ship with Linux on them and boot insanely quick.

Again, real numbers please, show us where in the current scheme we are
taking too much time and we can work to resolve that.

> >>Drawbacks:
> >>
> >>- It requires a small amount of time to calculate the order a boot time.
> >>   But this time is most often smaller than the time saved by using
> >>   multiple cores to call initcalls or by not needing deferred probes.
> >
> >How much time is needed?
> 
> I've measured 3ms on a slow ARM box.

And how much time does deferred probe take in comparison?

> >>- Dependencies are required. For everything which can be build as a
> >>   module, looking at modules.dep might give some pointers. Looking at
> >>   the help from menuconfig also might give some pointers. But in the
> >>   end, the most preferable way would be if maintainers or other people
> >>   which have a deeper knowledge about the source and functionality
> >>   would add the dependencies.
> >
> >How will a "normal" driver author figure out what those dependancies are
> >in order to be able to write them down?  That's my biggest objection
> 
> Most drivers are done c&p from an existing driver. And if someone adds new
> code, he should know what these new code is for and on what it depends.

Trust me, as someone who reviews more new drivers than anyone else,
people don't know, they blindly cut-and-paste from other drivers and
don't stop to think what they are doing.

> >here, I have no idea how to add these, nor how to properly review such a
> >submission.  What about systems that have different ordering/dependancy
> >requirements for the same drivers due to different ways the hardware is
> >hooked up?  That is not going to work well here, unless I'm missing
> >something.
> 
> Hmm, how is that solved now?
> 
> If you have dependencies dictated by a special HW, this dependencies should
> come in by the HW description.

Yes, device tree shows this.

> The deferred probe mechanism exists just since 1 or 2 years. And once you
> had to search the source of non-displayed error in a set of several dozens
> possible sources (because many errors are now non-errors but -517) you might
> learn to hate deferred probes as much as I do. I'm sorry for the harsh words
> in regard to deferred probes.

I don't like it much either, but it seems to work.  If lots of error
messages are annoying, and are what is really taking a long time (serial
output can be a real delay), then let's turn off the log messages to
make things go faster.

> The way to use dependencies doesn't add new requirements, it just moves them
> away from the link order. Besides that these dependencies are making it
> possible to parallelize the stuff.

We can paralleize probing today, that's been around for a very long
time, nothing is preventing you from enabling that for the drivers you
know will work properly that way right now.  Moving to a "all probing in
parallel" model has been proven to not really speed things up at all,
and in fact, slows things down due to cache and multiple process issues.
See the lkml archives for the work done many years ago around this issue
for the details.

Again, link order is a crude way to handle dependancies for built-in
drivers, I know that, which is why I accepted the deferred probing which
ensures that it will always work eventually.  You are moving from having
things in link-order to be manually specified in another manner,
preventing systems that have different dependancy requirements to never
be able to work.  And this last issue is the big one, deferred probing
is the only solution that will always work for all systems.

Again, work to speed up the problem drivers that you see taking too long
at probe time, that's the real solution here and is what others have
spent a lot of time doing, resulting in my sub-second desktop boot
times.  If embedded systems want to also achieve that, then fix the
drivers, reordering the probe order is not going to solve the boot speed
issue.

thanks,

greg k-h