Atmel SoCs and the newly added CONFIG_DELAY_DEVICE_PROBES option

Atmel SoCs and the newly added CONFIG_DELAY_DEVICE_PROBES option

[Sylvain Rochet]

Hi,

FYI, to save you a git bisect, the recently added 
CONFIG_DELAY_DEVICE_PROBES (enabled by default) breaks Atmel SoCs.

For a few of quickly noticeable issues:
- PM is not working:   at91_pm_sram_init: sram pool unavailable!
- Watchdog is not even probed
- on -ek boards the wm8904 is not probed either.

Disabling CONFIG_DELAY_DEVICE_PROBES fixes all issues.

Tomeu, what should I provide to help find out what's happening ?

Sylvain

Re: Atmel SoCs and the newly added CONFIG_DELAY_DEVICE_PROBES option

[Tomeu Vizoso]

On 16 October 2015 at 20:11, Sylvain Rochet <sylvain.rochet@finsecur.com> wrote:
> Hi,
>
> FYI, to save you a git bisect, the recently added
> CONFIG_DELAY_DEVICE_PROBES (enabled by default) breaks Atmel SoCs.
>
> For a few of quickly noticeable issues:
> - PM is not working:   at91_pm_sram_init: sram pool unavailable!
> - Watchdog is not even probed
> - on -ek boards the wm8904 is not probed either.
>
> Disabling CONFIG_DELAY_DEVICE_PROBES fixes all issues.
>
> Tomeu, what should I provide to help find out what's happening ?

Hi Sylvain, I believe I can do some testing via kernelci on those
boards, latest on monday.

It will probably involve moving some initcalls, and probing more kinds
of devices on-demand. Sometimes the proper solution is to move code
from initcalls into proper drivers which can defer their probe if some
dependency isn't there at that point, but that's likely to be more
invasive than wanted at this point.

Regards,

Tomeu

> Sylvain
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Atmel SoCs and the newly added CONFIG_DELAY_DEVICE_PROBES option

[Alexandre Belloni]

On 16/10/2015 at 21:29:25 +0200, Tomeu Vizoso wrote :
> On 16 October 2015 at 20:11, Sylvain Rochet <sylvain.rochet@finsecur.com> wrote:
> > Hi,
> >
> > FYI, to save you a git bisect, the recently added
> > CONFIG_DELAY_DEVICE_PROBES (enabled by default) breaks Atmel SoCs.
> >
> > For a few of quickly noticeable issues:
> > - PM is not working:   at91_pm_sram_init: sram pool unavailable!
> > - Watchdog is not even probed
> > - on -ek boards the wm8904 is not probed either.
> >
> > Disabling CONFIG_DELAY_DEVICE_PROBES fixes all issues.
> >
> > Tomeu, what should I provide to help find out what's happening ?
> 
> Hi Sylvain, I believe I can do some testing via kernelci on those
> boards, latest on monday.
> 

This also probably broke PM on i.mx5 and i.mx6 (and I think socfpga)
boards as they use the sram driver the same way. That is why sram_init
is a postcore_initcall.

> It will probably involve moving some initcalls, and probing more kinds
> of devices on-demand. Sometimes the proper solution is to move code
> from initcalls into proper drivers which can defer their probe if some
> dependency isn't there at that point, but that's likely to be more
> invasive than wanted at this point.
> 

I guess that the PM code will be really difficult to move into drivers
but I'd be happy to see that done on AT91 as this is the only thing left
in mach-at91.

-- 
Alexandre Belloni, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[PATCH] driver core: Disable late probes by default

[Tomeu Vizoso]

To smooth the transition to late probes, make disabled the default for
DELAY_DEVICE_PROBES and let individual SoCs enable the option as they
get fixed.

Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Link: https://lkml.kernel.org/g/20151016181129.GA1764@gradator.net

---

Hi Rob,

I'm sending this in case you think it would be best to leave the
on-demand probe series in -next for now but have late probes disabled to
avoid hassle to some people.

Regards,

Tomeu
---
 drivers/base/Kconfig | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
index b449b17055c9..17e711a725e3 100644
--- a/drivers/base/Kconfig
+++ b/drivers/base/Kconfig
@@ -326,7 +326,7 @@ endif
 
 config DELAY_DEVICE_PROBES
 	bool "Allow delaying the probe of some devices" if EXPERT
-	default y
+	default n
 	help
 	  Devices can be matched to a driver and probed from the moment they
 	  are registered, but early during boot their probes are likely to be
@@ -340,6 +340,6 @@ config DELAY_DEVICE_PROBES
 	  In some platforms there may be implicit assumptions about when some
 	  devices are probed, so enabling this option could cause problems there.
 
-	  If unsure, say Y here.
+	  If unsure, say N here.
 
 endmenu
-- 
2.5.0

Re: [PATCH] driver core: Disable late probes by default

[Greg Kroah-Hartman]

On Mon, Oct 19, 2015 at 05:13:22PM +0200, Tomeu Vizoso wrote:
> To smooth the transition to late probes, make disabled the default for
> DELAY_DEVICE_PROBES and let individual SoCs enable the option as they
> get fixed.
> 
> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> Link: https://lkml.kernel.org/g/20151016181129.GA1764@gradator.net
> 
> ---
> 
> Hi Rob,
> 
> I'm sending this in case you think it would be best to leave the
> on-demand probe series in -next for now but have late probes disabled to
> avoid hassle to some people.

I would like Rob to just drop this series please, I don't agree with it
at all at the moment.

thanks,

greg k-h

Re: [PATCH] driver core: Disable late probes by default

[Tomeu Vizoso]

On 19 October 2015 at 17:19, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Mon, Oct 19, 2015 at 05:13:22PM +0200, Tomeu Vizoso wrote:
>> To smooth the transition to late probes, make disabled the default for
>> DELAY_DEVICE_PROBES and let individual SoCs enable the option as they
>> get fixed.
>>
>> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>> Link: https://lkml.kernel.org/g/20151016181129.GA1764@gradator.net
>>
>> ---
>>
>> Hi Rob,
>>
>> I'm sending this in case you think it would be best to leave the
>> on-demand probe series in -next for now but have late probes disabled to
>> avoid hassle to some people.
>
> I would like Rob to just drop this series please, I don't agree with it
> at all at the moment.

Hi Greg,

is it the case that you are satisfied with deferred probes as a way of
ordering device probing and that I should look at how to solve my
problem by improving it?

Thanks,

Tomeu

> thanks,
>
> greg k-h
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] driver core: Disable late probes by default

[Greg Kroah-Hartman]

On Tue, Oct 20, 2015 at 09:40:48AM +0200, Tomeu Vizoso wrote:
> On 19 October 2015 at 17:19, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> > On Mon, Oct 19, 2015 at 05:13:22PM +0200, Tomeu Vizoso wrote:
> >> To smooth the transition to late probes, make disabled the default for
> >> DELAY_DEVICE_PROBES and let individual SoCs enable the option as they
> >> get fixed.
> >>
> >> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> >> Link: https://lkml.kernel.org/g/20151016181129.GA1764@gradator.net
> >>
> >> ---
> >>
> >> Hi Rob,
> >>
> >> I'm sending this in case you think it would be best to leave the
> >> on-demand probe series in -next for now but have late probes disabled to
> >> avoid hassle to some people.
> >
> > I would like Rob to just drop this series please, I don't agree with it
> > at all at the moment.
> 
> Hi Greg,
> 
> is it the case that you are satisfied with deferred probes as a way of
> ordering device probing and that I should look at how to solve my
> problem by improving it?

Yes, especially given that you have said this does not speed up your
boot times, which I thought was your main goal here :(

If deferred probes doesn't work well, we can fix it, but for now it
seems our best option.

thanks,

greg k-h

Re: [PATCH] driver core: Disable late probes by default

[Tomeu Vizoso]

On 20 October 2015 at 16:05, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Tue, Oct 20, 2015 at 09:40:48AM +0200, Tomeu Vizoso wrote:
>> On 19 October 2015 at 17:19, Greg Kroah-Hartman
>> <gregkh@linuxfoundation.org> wrote:
>> > On Mon, Oct 19, 2015 at 05:13:22PM +0200, Tomeu Vizoso wrote:
>> >> To smooth the transition to late probes, make disabled the default for
>> >> DELAY_DEVICE_PROBES and let individual SoCs enable the option as they
>> >> get fixed.
>> >>
>> >> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>> >> Link: https://lkml.kernel.org/g/20151016181129.GA1764@gradator.net
>> >>
>> >> ---
>> >>
>> >> Hi Rob,
>> >>
>> >> I'm sending this in case you think it would be best to leave the
>> >> on-demand probe series in -next for now but have late probes disabled to
>> >> avoid hassle to some people.
>> >
>> > I would like Rob to just drop this series please, I don't agree with it
>> > at all at the moment.
>>
>> Hi Greg,
>>
>> is it the case that you are satisfied with deferred probes as a way of
>> ordering device probing and that I should look at how to solve my
>> problem by improving it?
>
> Yes, especially given that you have said this does not speed up your
> boot times, which I thought was your main goal here :(

Sorry if I'm repeating myself too often, but I have two goals: change
the probing order to not send deferred probes to the back of the queue
(getting the display up as fast as possible), and making easier to
understand the boot process on embedded platforms.

The concrete issue that would be fixed by achieving the first goal was
explained in this email from last year:

http://lists.freedesktop.org/archives/dri-devel/2014-August/066527.html

Because of that, ChromeOS had to use their own bindings for the panel
node so that the panel probe wouldn't be deferred, introducing a
sizable delta that is a barrier to rebasing on newer mainline releases
and for vendors to upstream their HW adaptation for chrome devices.

The goal of the project I'm working on is to help reduce the delta so
that ChromeOS (but will also help other downstreams) can rebase more
often and so that vendors have one excuse less to upstream support for
their SoCs in a timely way.

About simplifying the boot processes, I was really convinced that
people were as tired as me of debugging probing issues with deferred
probes in the middle and I'm very surprised that you and Russell don't
see any problem with it.

Regards,

Tomeu

> If deferred probes doesn't work well, we can fix it, but for now it
> seems our best option.
>
> thanks,
>
> greg k-h
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] driver core: Disable late probes by default

[Greg Kroah-Hartman]

On Tue, Oct 20, 2015 at 06:17:39PM +0200, Tomeu Vizoso wrote:
> On 20 October 2015 at 16:05, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> > On Tue, Oct 20, 2015 at 09:40:48AM +0200, Tomeu Vizoso wrote:
> >> On 19 October 2015 at 17:19, Greg Kroah-Hartman
> >> <gregkh@linuxfoundation.org> wrote:
> >> > On Mon, Oct 19, 2015 at 05:13:22PM +0200, Tomeu Vizoso wrote:
> >> >> To smooth the transition to late probes, make disabled the default for
> >> >> DELAY_DEVICE_PROBES and let individual SoCs enable the option as they
> >> >> get fixed.
> >> >>
> >> >> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> >> >> Link: https://lkml.kernel.org/g/20151016181129.GA1764@gradator.net
> >> >>
> >> >> ---
> >> >>
> >> >> Hi Rob,
> >> >>
> >> >> I'm sending this in case you think it would be best to leave the
> >> >> on-demand probe series in -next for now but have late probes disabled to
> >> >> avoid hassle to some people.
> >> >
> >> > I would like Rob to just drop this series please, I don't agree with it
> >> > at all at the moment.
> >>
> >> Hi Greg,
> >>
> >> is it the case that you are satisfied with deferred probes as a way of
> >> ordering device probing and that I should look at how to solve my
> >> problem by improving it?
> >
> > Yes, especially given that you have said this does not speed up your
> > boot times, which I thought was your main goal here :(
> 
> Sorry if I'm repeating myself too often, but I have two goals: change
> the probing order to not send deferred probes to the back of the queue
> (getting the display up as fast as possible), and making easier to
> understand the boot process on embedded platforms.
> 
> The concrete issue that would be fixed by achieving the first goal was
> explained in this email from last year:
> 
> http://lists.freedesktop.org/archives/dri-devel/2014-August/066527.html
> 
> Because of that, ChromeOS had to use their own bindings for the panel
> node so that the panel probe wouldn't be deferred, introducing a
> sizable delta that is a barrier to rebasing on newer mainline releases
> and for vendors to upstream their HW adaptation for chrome devices.

1.5 second delay is crazy (again, my laptop boots to X in less time than
that), if there are a zillion probe deferrs, then maybe you can blame
this type of system, but I would _strongly_ suggest fixing the broken
driver that is causing such a delay instead, as that's the real problem
here.

And again, you say you did this to save boot time, yet you didn't
actually test to see if you fixed this issue, sorry, but I need to see
real numbers.

> The goal of the project I'm working on is to help reduce the delta so
> that ChromeOS (but will also help other downstreams) can rebase more
> often and so that vendors have one excuse less to upstream support for
> their SoCs in a timely way.

That's great, and wonderful, and should be done, but again, find the
broken driver here and fix it.  We have the tools to determine what is
going wrong here, please use them.  Without that data, I'm going to
insist that it is not the deferred probe logic (hint, how many probe
functions can a processor make in 1.5 seconds?)

> About simplifying the boot processes, I was really convinced that
> people were as tired as me of debugging probing issues with deferred
> probes in the middle and I'm very surprised that you and Russell don't
> see any problem with it.

I don't see the problems on the hardware that I run, and if there are
problems, people don't tell me exactly what they are (hint, like what
I'm asking for here...)

thanks,

greg k-h

Re: [PATCH] driver core: Disable late probes by default

[Tomeu Vizoso]

On 21 October 2015 at 05:39, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Tue, Oct 20, 2015 at 06:17:39PM +0200, Tomeu Vizoso wrote:
>> On 20 October 2015 at 16:05, Greg Kroah-Hartman
>> <gregkh@linuxfoundation.org> wrote:
>> > On Tue, Oct 20, 2015 at 09:40:48AM +0200, Tomeu Vizoso wrote:
>> >> On 19 October 2015 at 17:19, Greg Kroah-Hartman
>> >> <gregkh@linuxfoundation.org> wrote:
>> >> > On Mon, Oct 19, 2015 at 05:13:22PM +0200, Tomeu Vizoso wrote:
>> >> >> To smooth the transition to late probes, make disabled the default for
>> >> >> DELAY_DEVICE_PROBES and let individual SoCs enable the option as they
>> >> >> get fixed.
>> >> >>
>> >> >> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>> >> >> Link: https://lkml.kernel.org/g/20151016181129.GA1764@gradator.net
>> >> >>
>> >> >> ---
>> >> >>
>> >> >> Hi Rob,
>> >> >>
>> >> >> I'm sending this in case you think it would be best to leave the
>> >> >> on-demand probe series in -next for now but have late probes disabled to
>> >> >> avoid hassle to some people.
>> >> >
>> >> > I would like Rob to just drop this series please, I don't agree with it
>> >> > at all at the moment.
>> >>
>> >> Hi Greg,
>> >>
>> >> is it the case that you are satisfied with deferred probes as a way of
>> >> ordering device probing and that I should look at how to solve my
>> >> problem by improving it?
>> >
>> > Yes, especially given that you have said this does not speed up your
>> > boot times, which I thought was your main goal here :(
>>
>> Sorry if I'm repeating myself too often, but I have two goals: change
>> the probing order to not send deferred probes to the back of the queue
>> (getting the display up as fast as possible), and making easier to
>> understand the boot process on embedded platforms.
>>
>> The concrete issue that would be fixed by achieving the first goal was
>> explained in this email from last year:
>>
>> http://lists.freedesktop.org/archives/dri-devel/2014-August/066527.html
>>
>> Because of that, ChromeOS had to use their own bindings for the panel
>> node so that the panel probe wouldn't be deferred, introducing a
>> sizable delta that is a barrier to rebasing on newer mainline releases
>> and for vendors to upstream their HW adaptation for chrome devices.
>
> 1.5 second delay is crazy (again, my laptop boots to X in less time than
> that),

1.5 seconds isn't crazy at all for the kernel to initialize all the
devices in an embedded board. That's the current state of affairs
today.

> if there are a zillion probe deferrs, then maybe you can blame
> this type of system, but I would _strongly_ suggest fixing the broken
> driver that is causing such a delay instead, as that's the real problem
> here.
>
> And again, you say you did this to save boot time, yet you didn't
> actually test to see if you fixed this issue, sorry, but I need to see
> real numbers.

I haven't said that I did this to save boot time, but I do have
repeated several times that the goal is to get the display up as early
as possible during the boot process.

Again, the problem is that a device that defers its probe is currently
sent to the back of the queue, so it will be given a second chance
only after most other devices have been probed.

On a tegra124-nyan-big with multi_v7_defconfig I get this difference:

[    0.310032] [drm] Initialized tegra 0.0.0 20120330 on minor 0
[    2.038431] [drm] Initialized tegra 0.0.0 20120330 on minor 0

In the first case all devices required for display were probed near
the start, but in the second case one or more were put in the deferred
queue and were processed in late_initcall.

This is very much visible to the user, as with my series the panel
gets to display stuff very shortly after the bootloader cedes control,
but without it there's a long period with a black screen. This is
something that the people making products care about.

I hope I have been more clear this time, but just in case: I'm *not*
trying to speed up the boot process, I'm proposing a change in the
probing *order*.

Regards,

Tomeu

>> The goal of the project I'm working on is to help reduce the delta so
>> that ChromeOS (but will also help other downstreams) can rebase more
>> often and so that vendors have one excuse less to upstream support for
>> their SoCs in a timely way.
>
> That's great, and wonderful, and should be done, but again, find the
> broken driver here and fix it.  We have the tools to determine what is
> going wrong here, please use them.  Without that data, I'm going to
> insist that it is not the deferred probe logic (hint, how many probe
> functions can a processor make in 1.5 seconds?)
>
>> About simplifying the boot processes, I was really convinced that
>> people were as tired as me of debugging probing issues with deferred
>> probes in the middle and I'm very surprised that you and Russell don't
>> see any problem with it.
>
> I don't see the problems on the hardware that I run, and if there are
> problems, people don't tell me exactly what they are (hint, like what
> I'm asking for here...)
>
> thanks,
>
> greg k-h
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] driver core: Disable late probes by default

[Greg Kroah-Hartman]

On Wed, Oct 21, 2015 at 04:35:58PM +0200, Tomeu Vizoso wrote:
> On 21 October 2015 at 05:39, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> > On Tue, Oct 20, 2015 at 06:17:39PM +0200, Tomeu Vizoso wrote:
> >> On 20 October 2015 at 16:05, Greg Kroah-Hartman
> >> <gregkh@linuxfoundation.org> wrote:
> >> > On Tue, Oct 20, 2015 at 09:40:48AM +0200, Tomeu Vizoso wrote:
> >> >> On 19 October 2015 at 17:19, Greg Kroah-Hartman
> >> >> <gregkh@linuxfoundation.org> wrote:
> >> >> > On Mon, Oct 19, 2015 at 05:13:22PM +0200, Tomeu Vizoso wrote:
> >> >> >> To smooth the transition to late probes, make disabled the default for
> >> >> >> DELAY_DEVICE_PROBES and let individual SoCs enable the option as they
> >> >> >> get fixed.
> >> >> >>
> >> >> >> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> >> >> >> Link: https://lkml.kernel.org/g/20151016181129.GA1764@gradator.net
> >> >> >>
> >> >> >> ---
> >> >> >>
> >> >> >> Hi Rob,
> >> >> >>
> >> >> >> I'm sending this in case you think it would be best to leave the
> >> >> >> on-demand probe series in -next for now but have late probes disabled to
> >> >> >> avoid hassle to some people.
> >> >> >
> >> >> > I would like Rob to just drop this series please, I don't agree with it
> >> >> > at all at the moment.
> >> >>
> >> >> Hi Greg,
> >> >>
> >> >> is it the case that you are satisfied with deferred probes as a way of
> >> >> ordering device probing and that I should look at how to solve my
> >> >> problem by improving it?
> >> >
> >> > Yes, especially given that you have said this does not speed up your
> >> > boot times, which I thought was your main goal here :(
> >>
> >> Sorry if I'm repeating myself too often, but I have two goals: change
> >> the probing order to not send deferred probes to the back of the queue
> >> (getting the display up as fast as possible), and making easier to
> >> understand the boot process on embedded platforms.
> >>
> >> The concrete issue that would be fixed by achieving the first goal was
> >> explained in this email from last year:
> >>
> >> http://lists.freedesktop.org/archives/dri-devel/2014-August/066527.html
> >>
> >> Because of that, ChromeOS had to use their own bindings for the panel
> >> node so that the panel probe wouldn't be deferred, introducing a
> >> sizable delta that is a barrier to rebasing on newer mainline releases
> >> and for vendors to upstream their HW adaptation for chrome devices.
> >
> > 1.5 second delay is crazy (again, my laptop boots to X in less time than
> > that),
> 
> 1.5 seconds isn't crazy at all for the kernel to initialize all the
> devices in an embedded board. That's the current state of affairs
> today.

Then someone needs to fix that, that really is crazy.  What takes so
long here?  Why aren't you using async probing to do things in parallel
when you need to sleep in device probe (I'm hoping you are sleeping in
device probe, otherwise that's really broken)?

Have you used the tools we have to find where the time is being spent?

> > if there are a zillion probe deferrs, then maybe you can blame
> > this type of system, but I would _strongly_ suggest fixing the broken
> > driver that is causing such a delay instead, as that's the real problem
> > here.
> >
> > And again, you say you did this to save boot time, yet you didn't
> > actually test to see if you fixed this issue, sorry, but I need to see
> > real numbers.
> 
> I haven't said that I did this to save boot time, but I do have
> repeated several times that the goal is to get the display up as early
> as possible during the boot process.

Ok, sorry, I implied that this was speeding it up, you are just
reordering things, makes sense.  But you shouldn't have to touch all
different subsystems just to reorder things, why not do that to the
device tree file itself before you parse it?

> Again, the problem is that a device that defers its probe is currently
> sent to the back of the queue, so it will be given a second chance
> only after most other devices have been probed.
> 
> On a tegra124-nyan-big with multi_v7_defconfig I get this difference:
> 
> [    0.310032] [drm] Initialized tegra 0.0.0 20120330 on minor 0
> [    2.038431] [drm] Initialized tegra 0.0.0 20120330 on minor 0

Again, broken drivers, go fix them, we have the tools to do so, that's
a crazy and unacceptable delay.

greg k-h

Re: [PATCH] driver core: Disable late probes by default

[Tomeu Vizoso]

On 21 October 2015 at 17:14, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Wed, Oct 21, 2015 at 04:35:58PM +0200, Tomeu Vizoso wrote:
>> On 21 October 2015 at 05:39, Greg Kroah-Hartman
>> <gregkh@linuxfoundation.org> wrote:
>> > On Tue, Oct 20, 2015 at 06:17:39PM +0200, Tomeu Vizoso wrote:
>> >> On 20 October 2015 at 16:05, Greg Kroah-Hartman
>> >> <gregkh@linuxfoundation.org> wrote:
>> >> > On Tue, Oct 20, 2015 at 09:40:48AM +0200, Tomeu Vizoso wrote:
>> >> >> On 19 October 2015 at 17:19, Greg Kroah-Hartman
>> >> >> <gregkh@linuxfoundation.org> wrote:
>> >> >> > On Mon, Oct 19, 2015 at 05:13:22PM +0200, Tomeu Vizoso wrote:
>> >> >> >> To smooth the transition to late probes, make disabled the default for
>> >> >> >> DELAY_DEVICE_PROBES and let individual SoCs enable the option as they
>> >> >> >> get fixed.
>> >> >> >>
>> >> >> >> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>> >> >> >> Link: https://lkml.kernel.org/g/20151016181129.GA1764@gradator.net
>> >> >> >>
>> >> >> >> ---
>> >> >> >>
>> >> >> >> Hi Rob,
>> >> >> >>
>> >> >> >> I'm sending this in case you think it would be best to leave the
>> >> >> >> on-demand probe series in -next for now but have late probes disabled to
>> >> >> >> avoid hassle to some people.
>> >> >> >
>> >> >> > I would like Rob to just drop this series please, I don't agree with it
>> >> >> > at all at the moment.
>> >> >>
>> >> >> Hi Greg,
>> >> >>
>> >> >> is it the case that you are satisfied with deferred probes as a way of
>> >> >> ordering device probing and that I should look at how to solve my
>> >> >> problem by improving it?
>> >> >
>> >> > Yes, especially given that you have said this does not speed up your
>> >> > boot times, which I thought was your main goal here :(
>> >>
>> >> Sorry if I'm repeating myself too often, but I have two goals: change
>> >> the probing order to not send deferred probes to the back of the queue
>> >> (getting the display up as fast as possible), and making easier to
>> >> understand the boot process on embedded platforms.
>> >>
>> >> The concrete issue that would be fixed by achieving the first goal was
>> >> explained in this email from last year:
>> >>
>> >> http://lists.freedesktop.org/archives/dri-devel/2014-August/066527.html
>> >>
>> >> Because of that, ChromeOS had to use their own bindings for the panel
>> >> node so that the panel probe wouldn't be deferred, introducing a
>> >> sizable delta that is a barrier to rebasing on newer mainline releases
>> >> and for vendors to upstream their HW adaptation for chrome devices.
>> >
>> > 1.5 second delay is crazy (again, my laptop boots to X in less time than
>> > that),
>>
>> 1.5 seconds isn't crazy at all for the kernel to initialize all the
>> devices in an embedded board. That's the current state of affairs
>> today.
>
> Then someone needs to fix that, that really is crazy.  What takes so
> long here?  Why aren't you using async probing to do things in parallel
> when you need to sleep in device probe (I'm hoping you are sleeping in
> device probe, otherwise that's really broken)?

I'm a bit surprised now. During all the time that I have been pushing
this forward I have been regularly testing on more than a dozen boards
with different socs and 1.5 seconds to probe all the devices isn't
that much. This is basically due to having to wait for the hardware a
bit here and there, and to the sheer number of devices involved.

Of course people have been looking at speeding up boot on ARM devices
for years now and this is what we have come with up to now.

> Have you used the tools we have to find where the time is being spent?

Have to recognize that my starting point has been that probe order was
the cause of the problem and haven't profiled the whole boot process,
but I don't see how probe ordering would become irrelevant unless we
got total probing time down to 200ms. And that would give us a
fabulously fast boot, which I don't think is as realistic as you seem
to believe.

>> > if there are a zillion probe deferrs, then maybe you can blame
>> > this type of system, but I would _strongly_ suggest fixing the broken
>> > driver that is causing such a delay instead, as that's the real problem
>> > here.
>> >
>> > And again, you say you did this to save boot time, yet you didn't
>> > actually test to see if you fixed this issue, sorry, but I need to see
>> > real numbers.
>>
>> I haven't said that I did this to save boot time, but I do have
>> repeated several times that the goal is to get the display up as early
>> as possible during the boot process.
>
> Ok, sorry, I implied that this was speeding it up, you are just
> reordering things, makes sense.  But you shouldn't have to touch all
> different subsystems just to reorder things, why not do that to the
> device tree file itself before you parse it?

Because the code that parses the DT according to bindings is in each
subsystem. If I was to reorder the DT file at compilation time as
Alexander proposes I would still have to either modify all subsystems
or just duplicate that code somewhere else.

Note that in the other thread people are discussing using the
dependency information for PM and unbind, so we would be back here
again if we just patched deferred probe this time.

Regards,

Tomeu

>> Again, the problem is that a device that defers its probe is currently
>> sent to the back of the queue, so it will be given a second chance
>> only after most other devices have been probed.
>>
>> On a tegra124-nyan-big with multi_v7_defconfig I get this difference:
>>
>> [    0.310032] [drm] Initialized tegra 0.0.0 20120330 on minor 0
>> [    2.038431] [drm] Initialized tegra 0.0.0 20120330 on minor 0
>
> Again, broken drivers, go fix them, we have the tools to do so, that's
> a crazy and unacceptable delay.
>
>
> greg k-h
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] driver core: Disable late probes by default

[Greg Kroah-Hartman]

On Wed, Oct 21, 2015 at 05:53:13PM +0200, Tomeu Vizoso wrote:
> On 21 October 2015 at 17:14, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> > On Wed, Oct 21, 2015 at 04:35:58PM +0200, Tomeu Vizoso wrote:
> >> On 21 October 2015 at 05:39, Greg Kroah-Hartman
> >> <gregkh@linuxfoundation.org> wrote:
> >> > On Tue, Oct 20, 2015 at 06:17:39PM +0200, Tomeu Vizoso wrote:
> >> >> On 20 October 2015 at 16:05, Greg Kroah-Hartman
> >> >> <gregkh@linuxfoundation.org> wrote:
> >> >> > On Tue, Oct 20, 2015 at 09:40:48AM +0200, Tomeu Vizoso wrote:
> >> >> >> On 19 October 2015 at 17:19, Greg Kroah-Hartman
> >> >> >> <gregkh@linuxfoundation.org> wrote:
> >> >> >> > On Mon, Oct 19, 2015 at 05:13:22PM +0200, Tomeu Vizoso wrote:
> >> >> >> >> To smooth the transition to late probes, make disabled the default for
> >> >> >> >> DELAY_DEVICE_PROBES and let individual SoCs enable the option as they
> >> >> >> >> get fixed.
> >> >> >> >>
> >> >> >> >> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> >> >> >> >> Link: https://lkml.kernel.org/g/20151016181129.GA1764@gradator.net
> >> >> >> >>
> >> >> >> >> ---
> >> >> >> >>
> >> >> >> >> Hi Rob,
> >> >> >> >>
> >> >> >> >> I'm sending this in case you think it would be best to leave the
> >> >> >> >> on-demand probe series in -next for now but have late probes disabled to
> >> >> >> >> avoid hassle to some people.
> >> >> >> >
> >> >> >> > I would like Rob to just drop this series please, I don't agree with it
> >> >> >> > at all at the moment.
> >> >> >>
> >> >> >> Hi Greg,
> >> >> >>
> >> >> >> is it the case that you are satisfied with deferred probes as a way of
> >> >> >> ordering device probing and that I should look at how to solve my
> >> >> >> problem by improving it?
> >> >> >
> >> >> > Yes, especially given that you have said this does not speed up your
> >> >> > boot times, which I thought was your main goal here :(
> >> >>
> >> >> Sorry if I'm repeating myself too often, but I have two goals: change
> >> >> the probing order to not send deferred probes to the back of the queue
> >> >> (getting the display up as fast as possible), and making easier to
> >> >> understand the boot process on embedded platforms.
> >> >>
> >> >> The concrete issue that would be fixed by achieving the first goal was
> >> >> explained in this email from last year:
> >> >>
> >> >> http://lists.freedesktop.org/archives/dri-devel/2014-August/066527.html
> >> >>
> >> >> Because of that, ChromeOS had to use their own bindings for the panel
> >> >> node so that the panel probe wouldn't be deferred, introducing a
> >> >> sizable delta that is a barrier to rebasing on newer mainline releases
> >> >> and for vendors to upstream their HW adaptation for chrome devices.
> >> >
> >> > 1.5 second delay is crazy (again, my laptop boots to X in less time than
> >> > that),
> >>
> >> 1.5 seconds isn't crazy at all for the kernel to initialize all the
> >> devices in an embedded board. That's the current state of affairs
> >> today.
> >
> > Then someone needs to fix that, that really is crazy.  What takes so
> > long here?  Why aren't you using async probing to do things in parallel
> > when you need to sleep in device probe (I'm hoping you are sleeping in
> > device probe, otherwise that's really broken)?
> 
> I'm a bit surprised now. During all the time that I have been pushing
> this forward I have been regularly testing on more than a dozen boards
> with different socs and 1.5 seconds to probe all the devices isn't
> that much. This is basically due to having to wait for the hardware a
> bit here and there, and to the sheer number of devices involved.
> 
> Of course people have been looking at speeding up boot on ARM devices
> for years now and this is what we have come with up to now.
> 
> > Have you used the tools we have to find where the time is being spent?
> 
> Have to recognize that my starting point has been that probe order was
> the cause of the problem and haven't profiled the whole boot process,
> but I don't see how probe ordering would become irrelevant unless we
> got total probing time down to 200ms. And that would give us a
> fabulously fast boot, which I don't think is as realistic as you seem
> to believe.

So you aren't using the tools that we have today that were created years
ago, to help to reduce boot time problems like this and instead work on
changing the driver core to try to guess at what the real issue is here?

Come on, until you really know where you are taking so long, how can you
know what you need to fix?  I strongly recommend doing that here first,
that's why those tools were written in the first place.

{sigh}

greg k-h

Re: [PATCH] driver core: Disable late probes by default

[Rob Herring]

On Wed, Oct 21, 2015 at 11:06 AM, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Wed, Oct 21, 2015 at 05:53:13PM +0200, Tomeu Vizoso wrote:
>> On 21 October 2015 at 17:14, Greg Kroah-Hartman
>> <gregkh@linuxfoundation.org> wrote:
>> > On Wed, Oct 21, 2015 at 04:35:58PM +0200, Tomeu Vizoso wrote:
>> >> On 21 October 2015 at 05:39, Greg Kroah-Hartman
>> >> <gregkh@linuxfoundation.org> wrote:
>> >> > On Tue, Oct 20, 2015 at 06:17:39PM +0200, Tomeu Vizoso wrote:
>> >> >> On 20 October 2015 at 16:05, Greg Kroah-Hartman
>> >> >> <gregkh@linuxfoundation.org> wrote:
>> >> >> > On Tue, Oct 20, 2015 at 09:40:48AM +0200, Tomeu Vizoso wrote:
>> >> >> >> On 19 October 2015 at 17:19, Greg Kroah-Hartman
>> >> >> >> <gregkh@linuxfoundation.org> wrote:
>> >> >> >> > On Mon, Oct 19, 2015 at 05:13:22PM +0200, Tomeu Vizoso wrote:
>> >> >> >> >> To smooth the transition to late probes, make disabled the default for
>> >> >> >> >> DELAY_DEVICE_PROBES and let individual SoCs enable the option as they
>> >> >> >> >> get fixed.
>> >> >> >> >>
>> >> >> >> >> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
>> >> >> >> >> Link: https://lkml.kernel.org/g/20151016181129.GA1764@gradator.net
>> >> >> >> >>
>> >> >> >> >> ---
>> >> >> >> >>
>> >> >> >> >> Hi Rob,
>> >> >> >> >>
>> >> >> >> >> I'm sending this in case you think it would be best to leave the
>> >> >> >> >> on-demand probe series in -next for now but have late probes disabled to
>> >> >> >> >> avoid hassle to some people.
>> >> >> >> >
>> >> >> >> > I would like Rob to just drop this series please, I don't agree with it
>> >> >> >> > at all at the moment.
>> >> >> >>
>> >> >> >> Hi Greg,
>> >> >> >>
>> >> >> >> is it the case that you are satisfied with deferred probes as a way of
>> >> >> >> ordering device probing and that I should look at how to solve my
>> >> >> >> problem by improving it?
>> >> >> >
>> >> >> > Yes, especially given that you have said this does not speed up your
>> >> >> > boot times, which I thought was your main goal here :(
>> >> >>
>> >> >> Sorry if I'm repeating myself too often, but I have two goals: change
>> >> >> the probing order to not send deferred probes to the back of the queue
>> >> >> (getting the display up as fast as possible), and making easier to
>> >> >> understand the boot process on embedded platforms.
>> >> >>
>> >> >> The concrete issue that would be fixed by achieving the first goal was
>> >> >> explained in this email from last year:
>> >> >>
>> >> >> http://lists.freedesktop.org/archives/dri-devel/2014-August/066527.html
>> >> >>
>> >> >> Because of that, ChromeOS had to use their own bindings for the panel
>> >> >> node so that the panel probe wouldn't be deferred, introducing a
>> >> >> sizable delta that is a barrier to rebasing on newer mainline releases
>> >> >> and for vendors to upstream their HW adaptation for chrome devices.
>> >> >
>> >> > 1.5 second delay is crazy (again, my laptop boots to X in less time than
>> >> > that),
>> >>
>> >> 1.5 seconds isn't crazy at all for the kernel to initialize all the
>> >> devices in an embedded board. That's the current state of affairs
>> >> today.
>> >
>> > Then someone needs to fix that, that really is crazy.  What takes so
>> > long here?  Why aren't you using async probing to do things in parallel
>> > when you need to sleep in device probe (I'm hoping you are sleeping in
>> > device probe, otherwise that's really broken)?
>>
>> I'm a bit surprised now. During all the time that I have been pushing
>> this forward I have been regularly testing on more than a dozen boards
>> with different socs and 1.5 seconds to probe all the devices isn't
>> that much. This is basically due to having to wait for the hardware a
>> bit here and there, and to the sheer number of devices involved.
>>
>> Of course people have been looking at speeding up boot on ARM devices
>> for years now and this is what we have come with up to now.
>>
>> > Have you used the tools we have to find where the time is being spent?
>>
>> Have to recognize that my starting point has been that probe order was
>> the cause of the problem and haven't profiled the whole boot process,
>> but I don't see how probe ordering would become irrelevant unless we
>> got total probing time down to 200ms. And that would give us a
>> fabulously fast boot, which I don't think is as realistic as you seem
>> to believe.
>
> So you aren't using the tools that we have today that were created years
> ago, to help to reduce boot time problems like this and instead work on
> changing the driver core to try to guess at what the real issue is here?
>
> Come on, until you really know where you are taking so long, how can you
> know what you need to fix?  I strongly recommend doing that here first,
> that's why those tools were written in the first place.

For something everyone is or should be doing for years, there is
surprisingly zero information I can find. It is perf timechart you are
talking about, right? Everything I find on it is all after userspace
starts. I know perf has command line options, but I never could get it
to do what I wanted (which was dumping events up until a boot hang).

Still, the 1.5 sec doesn't surprise me either.

Rob

Re: [PATCH] driver core: Disable late probes by default

[Greg Kroah-Hartman]

On Wed, Oct 21, 2015 at 01:09:55PM -0500, Rob Herring wrote:
> On Wed, Oct 21, 2015 at 11:06 AM, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> > On Wed, Oct 21, 2015 at 05:53:13PM +0200, Tomeu Vizoso wrote:
> >> On 21 October 2015 at 17:14, Greg Kroah-Hartman
> >> <gregkh@linuxfoundation.org> wrote:
> >> > On Wed, Oct 21, 2015 at 04:35:58PM +0200, Tomeu Vizoso wrote:
> >> >> On 21 October 2015 at 05:39, Greg Kroah-Hartman
> >> >> <gregkh@linuxfoundation.org> wrote:
> >> >> > On Tue, Oct 20, 2015 at 06:17:39PM +0200, Tomeu Vizoso wrote:
> >> >> >> On 20 October 2015 at 16:05, Greg Kroah-Hartman
> >> >> >> <gregkh@linuxfoundation.org> wrote:
> >> >> >> > On Tue, Oct 20, 2015 at 09:40:48AM +0200, Tomeu Vizoso wrote:
> >> >> >> >> On 19 October 2015 at 17:19, Greg Kroah-Hartman
> >> >> >> >> <gregkh@linuxfoundation.org> wrote:
> >> >> >> >> > On Mon, Oct 19, 2015 at 05:13:22PM +0200, Tomeu Vizoso wrote:
> >> >> >> >> >> To smooth the transition to late probes, make disabled the default for
> >> >> >> >> >> DELAY_DEVICE_PROBES and let individual SoCs enable the option as they
> >> >> >> >> >> get fixed.
> >> >> >> >> >>
> >> >> >> >> >> Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> >> >> >> >> >> Link: https://lkml.kernel.org/g/20151016181129.GA1764@gradator.net
> >> >> >> >> >>
> >> >> >> >> >> ---
> >> >> >> >> >>
> >> >> >> >> >> Hi Rob,
> >> >> >> >> >>
> >> >> >> >> >> I'm sending this in case you think it would be best to leave the
> >> >> >> >> >> on-demand probe series in -next for now but have late probes disabled to
> >> >> >> >> >> avoid hassle to some people.
> >> >> >> >> >
> >> >> >> >> > I would like Rob to just drop this series please, I don't agree with it
> >> >> >> >> > at all at the moment.
> >> >> >> >>
> >> >> >> >> Hi Greg,
> >> >> >> >>
> >> >> >> >> is it the case that you are satisfied with deferred probes as a way of
> >> >> >> >> ordering device probing and that I should look at how to solve my
> >> >> >> >> problem by improving it?
> >> >> >> >
> >> >> >> > Yes, especially given that you have said this does not speed up your
> >> >> >> > boot times, which I thought was your main goal here :(
> >> >> >>
> >> >> >> Sorry if I'm repeating myself too often, but I have two goals: change
> >> >> >> the probing order to not send deferred probes to the back of the queue
> >> >> >> (getting the display up as fast as possible), and making easier to
> >> >> >> understand the boot process on embedded platforms.
> >> >> >>
> >> >> >> The concrete issue that would be fixed by achieving the first goal was
> >> >> >> explained in this email from last year:
> >> >> >>
> >> >> >> http://lists.freedesktop.org/archives/dri-devel/2014-August/066527.html
> >> >> >>
> >> >> >> Because of that, ChromeOS had to use their own bindings for the panel
> >> >> >> node so that the panel probe wouldn't be deferred, introducing a
> >> >> >> sizable delta that is a barrier to rebasing on newer mainline releases
> >> >> >> and for vendors to upstream their HW adaptation for chrome devices.
> >> >> >
> >> >> > 1.5 second delay is crazy (again, my laptop boots to X in less time than
> >> >> > that),
> >> >>
> >> >> 1.5 seconds isn't crazy at all for the kernel to initialize all the
> >> >> devices in an embedded board. That's the current state of affairs
> >> >> today.
> >> >
> >> > Then someone needs to fix that, that really is crazy.  What takes so
> >> > long here?  Why aren't you using async probing to do things in parallel
> >> > when you need to sleep in device probe (I'm hoping you are sleeping in
> >> > device probe, otherwise that's really broken)?
> >>
> >> I'm a bit surprised now. During all the time that I have been pushing
> >> this forward I have been regularly testing on more than a dozen boards
> >> with different socs and 1.5 seconds to probe all the devices isn't
> >> that much. This is basically due to having to wait for the hardware a
> >> bit here and there, and to the sheer number of devices involved.
> >>
> >> Of course people have been looking at speeding up boot on ARM devices
> >> for years now and this is what we have come with up to now.
> >>
> >> > Have you used the tools we have to find where the time is being spent?
> >>
> >> Have to recognize that my starting point has been that probe order was
> >> the cause of the problem and haven't profiled the whole boot process,
> >> but I don't see how probe ordering would become irrelevant unless we
> >> got total probing time down to 200ms. And that would give us a
> >> fabulously fast boot, which I don't think is as realistic as you seem
> >> to believe.
> >
> > So you aren't using the tools that we have today that were created years
> > ago, to help to reduce boot time problems like this and instead work on
> > changing the driver core to try to guess at what the real issue is here?
> >
> > Come on, until you really know where you are taking so long, how can you
> > know what you need to fix?  I strongly recommend doing that here first,
> > that's why those tools were written in the first place.
> 
> For something everyone is or should be doing for years, there is
> surprisingly zero information I can find. It is perf timechart you are
> talking about, right? Everything I find on it is all after userspace
> starts. I know perf has command line options, but I never could get it
> to do what I wanted (which was dumping events up until a boot hang).

scripts/bootgraph.pl combined with 'initcall_debug' on the kernel
command line.  Landed in Linus's tree back in 2008, perf is not needed.

thanks,

greg k-h

Re: [PATCH] driver core: Disable late probes by default

[Rob Herring]

On Wed, Oct 21, 2015 at 1:45 PM, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Wed, Oct 21, 2015 at 01:09:55PM -0500, Rob Herring wrote:
>> On Wed, Oct 21, 2015 at 11:06 AM, Greg Kroah-Hartman
>> <gregkh@linuxfoundation.org> wrote:
>> > On Wed, Oct 21, 2015 at 05:53:13PM +0200, Tomeu Vizoso wrote:
>> >> On 21 October 2015 at 17:14, Greg Kroah-Hartman
>> >> <gregkh@linuxfoundation.org> wrote:
>> >> > On Wed, Oct 21, 2015 at 04:35:58PM +0200, Tomeu Vizoso wrote:
>> >> >> On 21 October 2015 at 05:39, Greg Kroah-Hartman
>> >> >> <gregkh@linuxfoundation.org> wrote:
>> >> >> > On Tue, Oct 20, 2015 at 06:17:39PM +0200, Tomeu Vizoso wrote:
>> >> >> >> On 20 October 2015 at 16:05, Greg Kroah-Hartman
>> >> >> >> <gregkh@linuxfoundation.org> wrote:

[...]

>> >> >> >> Because of that, ChromeOS had to use their own bindings for the panel
>> >> >> >> node so that the panel probe wouldn't be deferred, introducing a
>> >> >> >> sizable delta that is a barrier to rebasing on newer mainline releases
>> >> >> >> and for vendors to upstream their HW adaptation for chrome devices.
>> >> >> >
>> >> >> > 1.5 second delay is crazy (again, my laptop boots to X in less time than
>> >> >> > that),
>> >> >>
>> >> >> 1.5 seconds isn't crazy at all for the kernel to initialize all the
>> >> >> devices in an embedded board. That's the current state of affairs
>> >> >> today.
>> >> >
>> >> > Then someone needs to fix that, that really is crazy.  What takes so
>> >> > long here?  Why aren't you using async probing to do things in parallel
>> >> > when you need to sleep in device probe (I'm hoping you are sleeping in
>> >> > device probe, otherwise that's really broken)?
>> >>
>> >> I'm a bit surprised now. During all the time that I have been pushing
>> >> this forward I have been regularly testing on more than a dozen boards
>> >> with different socs and 1.5 seconds to probe all the devices isn't
>> >> that much. This is basically due to having to wait for the hardware a
>> >> bit here and there, and to the sheer number of devices involved.
>> >>
>> >> Of course people have been looking at speeding up boot on ARM devices
>> >> for years now and this is what we have come with up to now.
>> >>
>> >> > Have you used the tools we have to find where the time is being spent?
>> >>
>> >> Have to recognize that my starting point has been that probe order was
>> >> the cause of the problem and haven't profiled the whole boot process,
>> >> but I don't see how probe ordering would become irrelevant unless we
>> >> got total probing time down to 200ms. And that would give us a
>> >> fabulously fast boot, which I don't think is as realistic as you seem
>> >> to believe.
>> >
>> > So you aren't using the tools that we have today that were created years
>> > ago, to help to reduce boot time problems like this and instead work on
>> > changing the driver core to try to guess at what the real issue is here?
>> >
>> > Come on, until you really know where you are taking so long, how can you
>> > know what you need to fix?  I strongly recommend doing that here first,
>> > that's why those tools were written in the first place.
>>
>> For something everyone is or should be doing for years, there is
>> surprisingly zero information I can find. It is perf timechart you are
>> talking about, right? Everything I find on it is all after userspace
>> starts. I know perf has command line options, but I never could get it
>> to do what I wanted (which was dumping events up until a boot hang).
>
> scripts/bootgraph.pl combined with 'initcall_debug' on the kernel
> command line.  Landed in Linus's tree back in 2008, perf is not needed.

Okay, well that I have used. I was thinking something more granular
than that. It will get you driver probe times, but the problem could
be some underlying dependency causing the actual problem. For example,
a driver enabling its regulator which happens to be connected via a
bit-banged I2C bus. Obviously we can dig down from there, but it is
not as simple as enabling the tool, running it, and instantly
identifying the problem.

Rob

Re: [PATCH] driver core: Disable late probes by default

[Greg Kroah-Hartman]

On Wed, Oct 21, 2015 at 03:53:31PM -0500, Rob Herring wrote:
> On Wed, Oct 21, 2015 at 1:45 PM, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> > On Wed, Oct 21, 2015 at 01:09:55PM -0500, Rob Herring wrote:
> >> On Wed, Oct 21, 2015 at 11:06 AM, Greg Kroah-Hartman
> >> <gregkh@linuxfoundation.org> wrote:
> >> > On Wed, Oct 21, 2015 at 05:53:13PM +0200, Tomeu Vizoso wrote:
> >> >> On 21 October 2015 at 17:14, Greg Kroah-Hartman
> >> >> <gregkh@linuxfoundation.org> wrote:
> >> >> > On Wed, Oct 21, 2015 at 04:35:58PM +0200, Tomeu Vizoso wrote:
> >> >> >> On 21 October 2015 at 05:39, Greg Kroah-Hartman
> >> >> >> <gregkh@linuxfoundation.org> wrote:
> >> >> >> > On Tue, Oct 20, 2015 at 06:17:39PM +0200, Tomeu Vizoso wrote:
> >> >> >> >> On 20 October 2015 at 16:05, Greg Kroah-Hartman
> >> >> >> >> <gregkh@linuxfoundation.org> wrote:
> 
> [...]
> 
> >> >> >> >> Because of that, ChromeOS had to use their own bindings for the panel
> >> >> >> >> node so that the panel probe wouldn't be deferred, introducing a
> >> >> >> >> sizable delta that is a barrier to rebasing on newer mainline releases
> >> >> >> >> and for vendors to upstream their HW adaptation for chrome devices.
> >> >> >> >
> >> >> >> > 1.5 second delay is crazy (again, my laptop boots to X in less time than
> >> >> >> > that),
> >> >> >>
> >> >> >> 1.5 seconds isn't crazy at all for the kernel to initialize all the
> >> >> >> devices in an embedded board. That's the current state of affairs
> >> >> >> today.
> >> >> >
> >> >> > Then someone needs to fix that, that really is crazy.  What takes so
> >> >> > long here?  Why aren't you using async probing to do things in parallel
> >> >> > when you need to sleep in device probe (I'm hoping you are sleeping in
> >> >> > device probe, otherwise that's really broken)?
> >> >>
> >> >> I'm a bit surprised now. During all the time that I have been pushing
> >> >> this forward I have been regularly testing on more than a dozen boards
> >> >> with different socs and 1.5 seconds to probe all the devices isn't
> >> >> that much. This is basically due to having to wait for the hardware a
> >> >> bit here and there, and to the sheer number of devices involved.
> >> >>
> >> >> Of course people have been looking at speeding up boot on ARM devices
> >> >> for years now and this is what we have come with up to now.
> >> >>
> >> >> > Have you used the tools we have to find where the time is being spent?
> >> >>
> >> >> Have to recognize that my starting point has been that probe order was
> >> >> the cause of the problem and haven't profiled the whole boot process,
> >> >> but I don't see how probe ordering would become irrelevant unless we
> >> >> got total probing time down to 200ms. And that would give us a
> >> >> fabulously fast boot, which I don't think is as realistic as you seem
> >> >> to believe.
> >> >
> >> > So you aren't using the tools that we have today that were created years
> >> > ago, to help to reduce boot time problems like this and instead work on
> >> > changing the driver core to try to guess at what the real issue is here?
> >> >
> >> > Come on, until you really know where you are taking so long, how can you
> >> > know what you need to fix?  I strongly recommend doing that here first,
> >> > that's why those tools were written in the first place.
> >>
> >> For something everyone is or should be doing for years, there is
> >> surprisingly zero information I can find. It is perf timechart you are
> >> talking about, right? Everything I find on it is all after userspace
> >> starts. I know perf has command line options, but I never could get it
> >> to do what I wanted (which was dumping events up until a boot hang).
> >
> > scripts/bootgraph.pl combined with 'initcall_debug' on the kernel
> > command line.  Landed in Linus's tree back in 2008, perf is not needed.
> 
> Okay, well that I have used. I was thinking something more granular
> than that. It will get you driver probe times, but the problem could
> be some underlying dependency causing the actual problem. For example,
> a driver enabling its regulator which happens to be connected via a
> bit-banged I2C bus. Obviously we can dig down from there, but it is
> not as simple as enabling the tool, running it, and instantly
> identifying the problem.

But it will show you the time taken in the driver probe calls, which is
what you need to know.  And it will show what order things ended up
happening in, which is also what you need to see (i.e. the regulator
that came after the i2c controller.)

Yeah, it's not a "here's where the bug is, go fix it" type tool, but it
kind of is given that you can see where your time is spent, which is the
most important thing to learn, so you can find what to fix.

thanks,

greg k-h